Pull RCU updates from Paul McKenney:
 "Make expedited grace periods expedite normal RCU callbacks

  Miscellaneous fixes:
   - Improve diagnostic output with character task states
   - Mark accesses to inform KCSAN of concurrency design
   - Move from kmalloc() to kmalloc_obj()
   - Documentation updates
   - Improve handling of RCU deferred quiescent states
   - Clean up unused function arguments and structure fields
   - Reduce show_rcu_gp_kthreads() stack space

  Tasks RCU updates:
   - Clean up after SRCU re-implementation of Tasks Trace RCU
   - Mark accesses to inform KCSAN of concurrency design
   - Add ->lazy_timer status to diagnostic output
   - Remove an unnecessary memory barrier
   - Fix a data race, courtesy of KCSAN
   - Documentation updates
   - Convert cond_resched_tasks_rcu_qs() from macro to static inline
     function

  SRCU updates:
   - Add Rust helpers for SRCU
   - Avoid losing queued work at cleanup_srcu_struct() time

  Torture-test updates:
   - Preparation work for immediate RCU priority deboosting
   - Test RCU readers from real interrupt handlers (as opposed to
     softirq)
   - Simplify code through use of cpumask_next_wrap()
   - Improve diagnostic output with character task states
   - Add rcutorture.nwriters parameter to allow lightweight stall
     testing, and rcutorture.stall_only to make doing so easier
   - Test an RCU Tasks Trace grace period implying an RCU grace period
   - Make RCU Tasks Trace torturing track reader batches
   - Fix a data race, courtesy of KCSAN
   - Plug a shuffle_tmp_mask memory leak on kthread spawn failure"

* tag 'rcu.2026.08.18a' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (59 commits)
  rcu: Add closing parenthesis in comment in rcu_read_unlock_strict()
  rcutorture: Make {,s}rcu_read_delay() better handle forward-progress testing
  rcutorture: Announce declining to forward-progress test
  torture: Don't leak shuffle_tmp_mask when shuffler kthread fails to start
  rcutorture: Use this_cpu_inc() for rcu_torture_count[] and rcu_torture_batch[]
  rcutorture: Make RCU Tasks Trace track Reader Batches
  rcutorture: Test RCU Tasks Trace GP implying RCU GP
  rcutorture: Add a stall_only module parameter
  rcutorture: Add nwriters module parameter
  rcutorture: Use task_state_to_char() for task-state reporting
  rcutorture: Use cpumask_next_wrap() in rcu_torture_preempt()
  rcutorture: Test RCU readers from hardware interrupt handlers
  rcutorture: Check for immediate deboosting at reader end
  srcu: Queue sdp->work when the delay timer is successfully deleted
  rcu-tasks: Convert cond_resched_tasks_rcu_qs() to static inline
  rcu-tasks: Fix some comments for call_rcu_tasks() and call_rcu_tasks_rude()
  rcu-tasks: Rename tasks_rcu_exit_srcu_stall_timer to tasks_rcu_exit_stall_timer
  rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s
  rcu: Reduce stack usage in show_rcu_gp_kthreads()
  rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs
  ...

Changes:

Make expedited grace periods expedite normal RCU callbacks

Miscellaneous fixes:
 * Improve diagnostic output with character task states.
 * Mark accesses to inform KCSAN of concurrency design.
 * Move from kmalloc() to kmalloc_obj().
 * Documentation updates.
 * Improve handling of RCU deferred quiescent states.
 * Clean up unused function arguments and structure fields.
 * Reduce show_rcu_gp_kthreads() stack space.

Tasks RCU updates:
 * Clean up after SRCU re-implementation of Tasks Trace RCU.
 * Mark accesses to inform KCSAN of concurrency design.
 * Add ->lazy_timer status to diagnostic output.
 * Remove an unnecessary memory barrier.
 * Fix a data race, courtesy of KCSAN.
 * Documentation updates.
 * Convert cond_resched_tasks_rcu_qs() from macro to static inline
   function.

SRCU updates:
 * Add Rust helpers for SRCU.
 * Avoid losing queued work at cleanup_srcu_struct() time.

Torture-test updates:
 * Preparation work for immediate RCU priority deboosting.
 * Test RCU readers from real interrupt handlers (as opposed to softirq).
 * Simplify code through use of cpumask_next_wrap().
 * Improve diagnostic output with character task states.
 * Add rcutorture.nwriters parameter to allow lightweight stall testing,
   and rcutorture.stall_only to make doing so easier.
 * Test an RCU Tasks Trace grace period implying an RCU grace period.
 * Make RCU Tasks Trace torturing track reader batches.
 * Fix a data race, courtesy of KCSAN.
 * Plug a shuffle_tmp_mask memory leak on kthread spawn failure.

The rcu_check_gp_start_stall() works entirely on rnp parameter,
and never uses it's rdp parameter. this commit therefore drop it,
updating both callers and the declaration.

No functional change.

Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

The rdp->cpu_no_qs.b.norm field is accessed only by the current CPU,
but can be accessed both at task level and from interrupt handlers.
All accesses from interrupts-enabled code must therefore be marked.
This commit therefore converts from __this_cpu_read() to this_cpu_read(),
from __this_cpu_write() to this_cpu_write(), and plain C-language accesses
to READ_ONCE() and WRITE_ONCE(), but only in interrupts-enabled code.

KCSAN located this issue.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Although the rdp->rcu_cpu_has_work field is accessed only by the
corresponding CPU, it can be accessed by both interrupt handlers via
invoke_rcu_core_kthread() and at task level via rcu_cpu_kthread().
This means that we need this_cpu_read() rather than __this_cpu_read(),
this_cpu_write() rather than __this_cpu_write(), and READ_ONCE()
rather than plain C-language loads.  The exception is the boot-time
rcu_spawn_core_kthreads(), which cannot race with kthreads that have
not yet been spawned.

This commit therefore makes it so.

KCSAN located this issue.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Ever since the kvfree_rcu() tracing moved out of the callback-enqueue
path, rcutree_enqueue() no longer looks at the callback function
pointer:  By the time it is invoked, __call_rcu_common() has already
stored the function into rhp->func, and the enqueue path only adds
the rcu_head to the segmented callback list and emits tracepoints
that do not take the function pointer.

Nevertheless, the function pointer is still threaded through
call_rcu_core(), call_rcu_nocb(), and rcutree_enqueue(), forcing
each level to carry a dead argument.

Remove the parameter from all three functions, from the no-CBs stub,
and from the corresponding declarations.  Anything needing the
callback function can still get it from rhp->func.

No functional change.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Currently raw_cpu_write() is used to clear the ->rcu_need_heavy_qs
field of the per-CPU rcu_data structure.  However, on x86 this is a
normal assignment, which does not play well with concurrent accesses.
This commit therefore upgrades the uses of raw_cpu_write() to its
concurrency-safe counterpart WRITE_ONCE() of an rdp pointer obtained
from this_cpu_ptr(&rcu_data).

KCSAN located this issue.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Even when rcu_pending() triggers rcu_core(), the normal callback
advancement path through note_gp_changes() -> __note_gp_changes() bails
out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
called and callbacks remain stuck in RCU_WAIT_TAIL.

Add a direct callback advancement block in rcu_core() that checks for GP
completion via rcu_segcblist_nextgp() combined with
poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
GP kthread if rcu_advance_cbs() requests a new grace period.

Uses trylock to avoid adding contention on rnp->lock. If the lock is
contended, callbacks will be advanced on the next tick.

Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcu_pending() decides whether rcu_core() should run on the current CPU's
timer tick.  It does not account for expedited grace periods: after an
expedited GP completes, a non-offloaded CPU's callbacks remain in
RCU_WAIT_TAIL (not yet advanced to RCU_DONE_TAIL) and rcu_core() is
never invoked to advance them.

Detect that case via rcu_segcblist_nextgp() combined with a new
memory-ordering-free poll variant,
poll_state_synchronize_rcu_full_unordered().  This keeps rcu_pending()
cheap: it runs on every tick that has pending callbacks, so it must
not pay for the two memory barriers in
poll_state_synchronize_rcu_full().  The check is only a hint to run
rcu_core(); the ordered re-check and the actual callback advancement
happen there.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

When an expedited grace period completes, rcu_exp_wait_wake() wakes
waiters on rnp->exp_wq[] but does not notify the NOCB rcuog kthreads.  An
rcuog kthread that is waiting for a grace period sleeps on the leaf
rcu_node's ->nocb_gp_wq[] with a wait condition based on the grace-period
state, so without a wakeup, callbacks on offloaded CPUs that could
benefit from the expedited GP wait until the rcuog kthread wakes for some
other reason (e.g. the next normal GP or a timer).

Make the rcuog grace-period wait honour expedited GPs and wake it when
one completes:

 - nocb_gp_wait() now records the grace period to wait for as a struct
   rcu_gp_seq (both normal and expedited), tracks the earliest pending
   normal and expedited sequence across the group, and releases the wait
   via poll_state_synchronize_rcu_full() so it wakes for whichever
   completes first.  ->nocb_gp_seq is widened to struct rcu_gp_seq
   accordingly.

 - rcu_exp_wait_wake() calls the new rcu_nocb_exp_cleanup() on leaf
   nodes, which wakes both ->nocb_gp_wq[0] and ->nocb_gp_wq[1] (the
   expedited sequence does not share parity with the normal ->gp_seq the
   waiter indexed with).  Both this path and rcu_nocb_gp_cleanup() use
   the shared rcu_nocb_cleanup_wake() helper, which checks swait_active()
   first; the smp_mb() in rcu_gp_cleanup()/rcu_exp_wait_wake() orders the
   grace-period state update before that check.

A stub rcu_nocb_exp_cleanup() is provided for CONFIG_RCU_NOCB_CPU=n.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>