linux-stable.git/kernel/rcu/tree_exp.h, branch linux-rolling-lts

rcu/exp: Remove needless CPU up quiescent state report

2025-07-08T17:51:13+00:00

A CPU coming online checks for an ongoing grace period and reports
a quiescent state accordingly if needed. This special treatment that
shortcuts the expedited IPI finds its origin as an optimization purpose
on the following commit:

	338b0f760e84 (rcu: Better hotplug handling for synchronize_sched_expedited()

The point is to avoid an IPI while waiting for a CPU to become online
or failing to become offline.

However this is pointless and even error prone for several reasons:

* If the CPU has been seen offline in the first round scanning offline
  and idle CPUs, no IPI is even tried and the quiescent state is
  reported on behalf of the CPU.

* This means that if the IPI fails, the CPU just became offline. So
  it's unlikely to become online right away, unless the cpu hotplug
  operation failed and rolled back, which is a rare event that can
  wait a jiffy for a new IPI to be issued.

* But then the "optimization" applying on failing CPU hotplug down only
  applies to !PREEMPT_RCU.

* This force reports a quiescent state even if ->cpu_no_qs.b.exp is not
  set. As a result it can race with remote QS reports on the same rdp.
  Fortunately it happens to be OK but an accident is waiting to happen.

For all those reasons, remove this optimization that doesn't look worthy
to keep around.

Reported-by: Paul E. McKenney 
Signed-off-by: Frederic Weisbecker 
Reviewed-by: Paul E. McKenney 
Signed-off-by: Joel Fernandes 
Signed-off-by: Neeraj Upadhyay (AMD)

rcu/exp: Protect against early QS report

2025-06-25T03:06:18+00:00

When a grace period is started, the ->expmask of each node is set up
from sync_exp_reset_tree(). Then later on each leaf node also initialize
its ->exp_tasks pointer.

This means that the initialization of the quiescent state of a node and
the initialization of its blocking tasks happen with an unlocked node
gap in-between.

It happens to be fine because nothing is expected to report an exp
quiescent state within this gap, since no IPI have been issued yet and
every rdp's ->cpu_no_qs.b.exp should be false.

However if it were to happen by accident, the quiescent state could be
reported and propagated while ignoring tasks that blocked _before_ the
start of the grace period.

Prevent such trouble to happen in the future and initialize both the
quiescent states mask to report and the blocked tasks head from the same
node locked block.

If a task blocks within an RCU read side critical section before
sync_exp_reset_tree() is called and is then unblocked between
sync_exp_reset_tree() and __sync_rcu_exp_select_node_cpus(), the QS
won't be reported because no RCU exp IPI had been issued to request it
through the setting of srdp->cpu_no_qs.b.exp.

Reviewed-by: Paul E. McKenney 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Joel Fernandes 
Signed-off-by: Neeraj Upadhyay (AMD)

rcu: Remove swake_up_one_online() bandaid

2025-05-16T13:00:54+00:00

It's now ok to perform a wake-up from an offline CPU because the
resulting armed scheduler bandwidth hrtimers are now correctly targeted
by hrtimer infrastructure.

Remove the obsolete hackerry.

Link: https://lore.kernel.org/all/20241231170712.149394-3-frederic@kernel.org/
Reviewed-by: Usama Arif 
Reviewed-by: Joel Fernandes 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Joel Fernandes

rcu: Split rcu_report_exp_cpu_mult() mask parameter and use for tracing

2025-02-05T05:52:38+00:00

This commit renames the rcu_report_exp_cpu_mult() function from "mask"
to "mask_in" and introduced a "mask" local variable to better support
upcoming event-tracing additions.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Boqun Feng

rcu: Add lockdep_assert_irqs_disabled() to rcu_exp_need_qs()

2024-12-14T16:10:38+00:00

Callers to rcu_exp_need_qs() are supposed to disable interrupts, so this
commit enlists lockdep's aid in checking this.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Add KCSAN exclusive-writer assertions for rdp->cpu_no_qs.b.exp

2024-12-14T16:10:29+00:00

The value of rdp->cpu_no_qs.b.exp may be changed only by the corresponding
CPU, and that CPU is not even allowed to race with itself, for example,
via interrupt handlers.  This commit therefore adds KCSAN exclusive-writer
assertions to check this constraint.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Make preemptible rcu_exp_handler() check idempotency

2024-12-14T16:10:20+00:00

Although the non-preemptible implementation of rcu_exp_handler()
contains checks to enforce idempotency, the preemptible version does not.
The reason for this omission is that in preemptible kernels, there is
no reporting of quiescent states from CPU hotplug notifiers, and thus
no need for idempotency.

In theory, anyway.

In practice, accidents happen.  This commit therefore adds checks under
WARN_ON_ONCE() to catch any such accidents.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Replace open-coded rcu_exp_need_qs() from rcu_exp_handler() with call

2024-12-14T16:10:14+00:00

Currently, the preemptible implementation of rcu_exp_handler()
almost open-codes rcu_exp_need_qs().  A call to that function would be
shorter and would improve expediting in cases where rcu_exp_handler()
interrupted a preemption-disabled or bh-disabled region of code.
This commit therefore moves rcu_exp_need_qs() out of the non-preemptible
leg of the enclosing #ifdef and replaces the open coding in preemptible
rcu_exp_handler() with a call to rcu_exp_need_qs().

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Move rcu_report_exp_rdp() setting of ->cpu_no_qs.b.exp under lock

2024-12-14T16:10:07+00:00

This commit reduces the state space of rcu_report_exp_rdp() by moving
the setting of ->cpu_no_qs.b.exp under the rcu_node structure's ->lock.
The lock isn't really all that important here, given that this per-CPU
field is supposed to be written only by its CPU, but the disabling of
interrupts excludes things like rcu_exp_handler(), which also can write
to this same field.  Avoiding this sort of interleaved access reduces
the state space.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Make rcu_report_exp_cpu_mult() caller acquire lock

2024-12-14T16:09:59+00:00

There is a hard-to-trigger bug in the expedited grace-period computation
whose fix requires that the __sync_rcu_exp_select_node_cpus() function
to check that the grace-period sequence number has not changed before
invoking rcu_report_exp_cpu_mult().  However, this check must be done
while holding the leaf rcu_node structure's ->lock.

This commit therefore prepares for that fix by moving this lock's
acquisition from rcu_report_exp_cpu_mult() to its callers (all two
of them).

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)