linux.git/kernel/smp.c, branch v7.3-rc2

Merge tag 'smp-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2026-08-18T22:29:53+00:00

Pull SMP core updates from Thomas Gleixner:

 - Reduce the preemption disabled sections in smp_call_function*().

   The various smp call functions keep preemption disabled accross the
   full operation which includes the wait for completion. Especially the
   latter can take some time when one of the target CPUs is not
   immediately responding to the IPI, which can result in large latency
   spikes.

   To improve this provide a per task CPU mask to track the CPUs to wait
   for. That makes the information required for the wait task local and
   therefore allows to reenable preemption before the wait.

   While this comes with moderate extra memory cost this reduces SMP
   function call induced latency measured in a fleet for high priority
   tasks from ~17ms to ~1.5ms (~90%).

 - Reduce the overhead of the CSD debug code by replacing the heavy
   memory barriers with smp_store_release()/acquire()

 - Remove obsolute unused hotplug states

* tag 'smp-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  scftorture: Remove preempt_disable() in scftorture_invoke_one()
  smp: Remove preempt_disable() from on_each_cpu_cond_mask()
  smp: Remove preempt_disable() from smp_call_function()
  smp: Enable preemption early in smp_call_function_many_cond()
  smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once
  smp: Use task-local IPI cpumask in smp_call_function_many_cond()
  smp: Refactor remote CPU selection in smp_call_function_any()
  smp: Enable preemption early in smp_call_function_single()
  smp: Disable preemption explicitly in __csd_lock_wait()
  cpu/hotplug: Remove CPUHP_AP_ARM_CORESIGHT_CTI_STARTING
  smp: Use release stores for csd_lock_record() state

smp: Make CSD lock acquisition atomic for debug mode

2026-07-22T18:57:08+00:00

Commit b0473dcd4b1d ("smp: Improve smp_call_function_single()
CSD-lock diagnostics") changed smp_call_function_single() so that,
when CSD lock debugging is enabled, async !wait calls use the
destination CPU csd_data. That improves diagnostics, but it also removes
the single-writer property that made the old csd_lock() safe: multiple
CPUs can now prepare the same destination CPU CSD concurrently.

csd_lock() currently waits for CSD_FLAG_LOCK to clear and then sets the
bit with a non-atomic read-modify-write. Two senders can both see an
unlocked CSD, set the bit, overwrite the callback fields, and enqueue
the same llist node. Re-adding a node that is already the queue head can
make node->next point to itself, leaving the target CPU stuck walking
call_single_queue. Later synchronous work, such as a TLB shootdown, can
then remain queued and trigger soft-lockup warnings or panics.

Keep the single csd_lock() implementation, but when CSD lock debugging is
enabled, acquire CSD_FLAG_LOCK with try_cmpxchg_acquire(). This makes the
destination CPU CSD a real atomic lock in the only configuration where it
can be shared by multiple remote senders, while preserving the existing
non-debug fast path.

Fixes: b0473dcd4b1d ("smp: Improve smp_call_function_single() CSD-lock diagnostics")
Signed-off-by: Chuyi Zhou 
Signed-off-by: Paul E. McKenney 
Signed-off-by: Thomas Gleixner 
Link: https://patch.msgid.link/20260716004539.13983-2-paulmck@kernel.org

smp: Avoid invalid per-CPU CSD lookup with CSD lock debug

2026-07-22T18:57:07+00:00

Commit b0473dcd4b1d ("smp: Improve smp_call_function_single()
CSD-lock diagnostics") made smp_call_function_single() use the destination
CPU's csd_data when CSD lock debugging is enabled. That lets the debug code
associate a stuck CSD lock with the target CPU, but it also means the CPU
argument is used in per_cpu_ptr() before generic_exec_single() has a chance
to validate it.

This becomes unsafe when smp_call_function_any() cannot find an online CPU
in the supplied mask. In that case the selected CPU can be nr_cpu_ids, and
the !wait path calls get_single_csd_data(cpu) before generic_exec_single()
returns -ENXIO. With csdlock_debug_enabled set, that indexes the per-CPU
offset array with an invalid CPU number.

Use the destination CPU's csd_data only when the CPU number is within
nr_cpu_ids. For invalid CPU numbers, fall back to the local CPU's csd_data
and let generic_exec_single() perform the existing validation and return
-ENXIO.

Fixes: b0473dcd4b1d ("smp: Improve smp_call_function_single() CSD-lock diagnostics")
Signed-off-by: Chuyi Zhou 
Signed-off-by: Paul E. McKenney 
Signed-off-by: Thomas Gleixner 
Reviewed-by: Paul E. McKenney 
Acked-by: Muchun Song 
Link: https://patch.msgid.link/20260716004539.13983-1-paulmck@kernel.org

smp: Remove preempt_disable() from on_each_cpu_cond_mask()

2026-07-16T07:24:55+00:00

smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. on_each_cpu_cond_mask() only builds the call
flags and forwards the request to that helper.

Remove the outer preempt_disable() and preempt_enable() pair from
on_each_cpu_cond_mask().

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Muchun Song 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-9-zhouchuyi@bytedance.com

smp: Remove preempt_disable() from smp_call_function()

2026-07-16T07:24:55+00:00

smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. smp_call_function() only forwards the request to
that helper for cpu_online_mask and does not access CPU-local state on
its own.

Remove the outer preempt_disable() and preempt_enable() pair from
smp_call_function().

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Muchun Song 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-8-zhouchuyi@bytedance.com

smp: Enable preemption early in smp_call_function_many_cond()

2026-07-16T07:24:55+00:00

smp_call_function_many_cond() still has to keep the caller pinned to the
current CPU while the remote IPI request is built and dispatched. This
protects the queueing state and CPU-hotplug boundary that are required
before the synchronous wait starts:

 - It protects the current CPU's per-CPU scratch cpumask,
   cfd->cpumask_ipi. Another task running on the same CPU could otherwise
   enter smp_call_function_many_cond() and reuse that scratch cpumask
   before the current caller has finished building and sending the IPI
   request.

 - It provides the CPU-hotplug exclusion required by the CSD queueing
   side. New CSDs must not be queued after smpcfd_dying_cpu() has flushed
   the outgoing CPU's callback queue. Keeping preemption disabled until
   all required CSDs have been queued and the corresponding IPIs have
   been sent prevents CPU offline from crossing that boundary in the
   middle of the queueing operation.

The CSD acquisition side also relies on that caller-side CPU pinning.
csd_lock() waits for CSD_FLAG_LOCK to clear and then marks the CSD busy
with a regular store, so another task on the same CPU must not be
allowed to acquire and reinitialize the same per-CPU CSD concurrently.

After the callbacks have been queued and the IPIs have been sent, the
caller only performs the final csd_lock_wait() completion wait. If it is
preempted there, another task running on the original CPU may enter
smp_call_function_many_cond(), but any attempt to reuse the same per-CPU
CSD will block in csd_lock() until the previous callback clears
CSD_FLAG_LOCK. The final csd_lock_wait() does not acquire or reinitialize
the CSD, so it does not need the same caller-side preemption-disabled
protection.

The wait mask is task-local, so it cannot be overwritten by another task
on the original CPU. The per-CPU CSD storage also remains allocated
across CPU offline, so csd_lock_wait() can safely dereference it even if
the target CPU is offlined after the caller is unpinned.

With those requirements satisfied, enable preemption before the
synchronous csd_lock_wait() loop. This makes the potentially long wait
preemptible and migratable while keeping the CPU-pinned section around
the remote CPU selection and IPI dispatch.

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-7-zhouchuyi@bytedance.com

smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once

2026-07-16T07:24:55+00:00

smp_call_function_many_cond() uses per-CPU CSD objects when queueing
callbacks to remote CPUs, and the wait path later dereferences those CSDs
from csd_lock_wait().

Making the wait path preemptible allows the initiating task to be
preempted or migrated before it waits for completion. A target CPU can be
offlined in that window. If smpcfd_dead_cpu() frees the target CPU's
per-CPU CSD storage, csd_lock_wait() can later dereference freed memory.

One way to protect the CSD storage is to free it via RCU or after a
synchronization step in the CPU offline path, but that would add
unnecessary complexity and can delay CPU shutdown.

Allocate the per-CPU CSD storage the first time a CPU comes up and keep
it allocated when the CPU is offlined. This allows csd_lock_wait() to
access the CSD even when the target CPU is offlined after preemption is
re-enabled and before the wait is invoked.

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Sebastian Andrzej Siewior 
Acked-by: Muchun Song 
Link: https://patch.msgid.link/20260709122933.4021501-6-zhouchuyi@bytedance.com

smp: Use task-local IPI cpumask in smp_call_function_many_cond()

2026-07-16T07:24:55+00:00

smp_call_function_many_cond() uses the per-CPU cfd->cpumask as the list
of remote CPUs to wait for. That is safe while the caller remains pinned
to the current CPU for the whole operation, because another task cannot
run on the same CPU and reuse the per-CPU mask.

The synchronous wait is the long-latency part of the operation. To make
that wait preemptible, the mask iterated by csd_lock_wait() must remain
stable even if the task is preempted or migrates. If the wait used the
per-CPU cfd->cpumask after dropping CPU pinning, another task scheduled
on the original CPU could enter smp_call_function_many_cond() and
overwrite the mask while the first task is still iterating it.

Give each task private IPI cpumask storage and use it as the wait mask in
smp_call_function_many_cond(). Other cpumask storage choices do not fit
this use case:

 - Per-CPU storage is the state that becomes unsafe once the wait is
   made preemptible. After the caller drops CPU pinning, another task
   scheduled on the original CPU can enter smp_call_function_many_cond()
   and reuse the same per-CPU mask.

 - Stack storage is not suitable for large NR_CPUS or
   CONFIG_CPUMASK_OFFSTACK=y configurations. The wait mask needs to
   scale with cpumask_size(), and putting that storage on the stack is
   not acceptable on large systems.

 - Allocating the mask inside smp_call_function_many_cond() would put an
   allocation and a failure path in the generic IPI path. A sleeping
   allocation is not suitable because callers have historically only
   provided a preempt-disabled context, not a sleepable one. GFP_ATOMIC
   would avoid sleeping, but a failure fallback would make the latency
   improvement opportunistic instead of guaranteed.

The users are not limited to a small, pre-identifiable class of tasks. On
x86, ordinary tasks can reach this path through TLB flushes during exit,
unmap and reclaim, so allocating the mask only for a known subset of
tasks is not straightforward.

The memory cost is explicit: one word is added to task_struct. When
cpumask_size() fits in that word, the mask is stored inline and no
separate allocation is needed. Larger systems allocate cpumask_size() per
task; on x86-64 NR_CPUS=8192 this is 1 KiB per task. For context,
x86 already carries several KiB of per-task architecture and FPU state,
depending on the enabled features and configuration. That does not make
the extra cpumask free, but it puts the large-NR_CPUS case in
perspective.

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-5-zhouchuyi@bytedance.com

smp: Refactor remote CPU selection in smp_call_function_any()

2026-07-16T07:24:55+00:00

smp_call_function_any() disables preemption across the entire operation:
selecting a target CPU, enqueueing the IPI, and synchronously waiting for
the remote CPU. smp_call_function_single() already re-enables preemption
before the synchronous csd_lock_wait(), so callers of
smp_call_function_any() should benefit from the same shorter
preemption-disabled section.

Simply removing get_cpu() and put_cpu() from smp_call_function_any()
would leave the preemption disablement entirely to
smp_call_function_single(). That opens a preemption window between
selecting the remote CPU, for example via sched_numa_find_nth_cpu(), and
dispatching the IPI in smp_call_function_single(). If the selected CPU is
fully offlined in that window, smp_call_function_single() fails its
cpu_online() check and returns -ENXIO to the caller, violating the
guarantee that smp_call_function_any() executes on any online CPU in the
mask.

Move the remote CPU selection into a common
__smp_call_function_single() helper. Keep the target CPU selection and
IPI dispatch within the same preemption-disabled region, while still
allowing the wait path to use the shorter preemption-disabled section
provided by smp_call_function_single().

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-4-zhouchuyi@bytedance.com

smp: Enable preemption early in smp_call_function_single()

2026-07-16T07:24:55+00:00

smp_call_function_single() disables preemption while it validates the
target CPU, prepares the call single data, queues the callback and sends
the IPI.

For the !wait case, preemption protects the per-CPU csd_data from
concurrent modification by another task on the same CPU. For the wait
case, the CSD is stack allocated and no other task can reuse it. CPU
pinning is still required until the callback has been queued and the IPI
has been sent, to ensure that the target CPU cannot be offlined after the
online check but before dispatch.

After generic_exec_single() has queued the callback, the synchronous
csd_lock_wait() invocation at the end of the execution does not require
the caller to remain pinned to the current CPU.

Enable preemption before csd_lock_wait() to shorten the
preemption-disabled section.

Signed-off-by: Chuyi Zhou 
Signed-off-by: Thomas Gleixner 
Tested-by: Paul E. McKenney 
Reviewed-by: Muchun Song 
Reviewed-by: Steven Rostedt (Google) 
Reviewed-by: Sebastian Andrzej Siewior 
Link: https://patch.msgid.link/20260709122933.4021501-3-zhouchuyi@bytedance.com