linux.git/kernel/rcu/tree_plugin.h, branch v3.18-rc2

rcu: Eliminate deadlock between CPU hotplug and expedited grace periods

2014-09-18T23:22:27+00:00

Currently, the expedited grace-period primitives do get_online_cpus().
This greatly simplifies their implementation, but means that calls
to them holding locks that are acquired by CPU-hotplug notifiers (to
say nothing of calls to these primitives from CPU-hotplug notifiers)
can deadlock.  But this is starting to become inconvenient, as can be
seen here: https://lkml.org/lkml/2014/8/5/754.  The problem in this
case is that some developers need to acquire a mutex from a CPU-hotplug
notifier, but also need to hold it across a synchronize_rcu_expedited().
As noted above, this currently results in deadlock.

This commit avoids the deadlock and retains the simplicity by creating
a try_get_online_cpus(), which returns false if the get_online_cpus()
reference count could not immediately be incremented.  If a call to
try_get_online_cpus() returns true, the expedited primitives operate as
before.  If a call returns false, the expedited primitives fall back to
normal grace-period operations.  This falling back of course results in
increased grace-period latency, but only during times when CPU hotplug
operations are actually in flight.  The effect should therefore be
negligible during normal operation.

Signed-off-by: Paul E. McKenney 
Cc: Josh Triplett 
Cc: "Rafael J. Wysocki" 
Tested-by: Lan Tianyu

Merge branch 'rcu-tasks.2014.09.10a' into HEAD

2014-09-16T17:10:44+00:00

rcu-tasks.2014.09.10a: Add RCU-tasks flavor of RCU.

Merge branches 'doc.2014.09.07a', 'fixes.2014.09.10a', 'nocb-nohz.2014.09.16b' and 'torture.2014.09.07a' into HEAD

2014-09-16T17:08:34+00:00

doc.2014.09.07a: Documentation updates.
fixes.2014.09.10a: Miscellaneous fixes.
nocb-nohz.2014.09.16b: No-CBs CPUs and NO_HZ_FULL updates.
torture.2014.09.07a: Torture-test updates.

rcu: Avoid misordering in nocb_leader_wait()

2014-09-16T17:08:03+00:00

The NOCB follower wakeup ordering depends on the store to the tail
pointer happening before the wakeup.  However, because atomic_long_add()
does not return a value, it does not provide ordering guarantees, and
the locking in wake_up() only guarantees that the store will happen
before the unlock, which might be too late.  Even though this is only a
theoretical issue, this commit adds a smp_mb__after_atomic() after the
final atomic_long_add() to provide the needed ordering guarantee.

Reported-by: Amit Shah 
Signed-off-by: Paul E. McKenney 
Tested-by: Paul Gortmaker

rcu: Handle NOCB callbacks from irq-disabled idle code

2014-09-16T17:08:03+00:00

If an RCU callback is queued on a no-CBs CPU from idle code with irqs
disabled, and if that CPU stays idle forever after, the callback will
never be invoked.  This commit therefore adds a check for this situation
in ____call_rcu_nocb(), invoking the RCU core solely for the purpose
of the ensuing return-to-idle transition.  (If the CPU doesn't return
to idle, the next scheduling-clock interrupt will fix things up.)

Reported-by: Amit Shah 
Signed-off-by: Paul E. McKenney 
Tested-by: Paul Gortmaker

rcu: Avoid misordering in __call_rcu_nocb_enqueue()

2014-09-16T17:08:03+00:00

The NOCB leader wakeup ordering depends on the store to the header
happening before the check for the leader already being awake.  However,
because atomic_long_add() does not return a value, it does not provide
ordering guarantees, the incorrect comment in wake_nocb_leader()
notwithstanding.  This commit therefore adds a smp_mb__after_atomic()
after the final atomic_long_add() to provide the needed ordering
guarantee.

Reported-by: Amit Shah 
Signed-off-by: Paul E. McKenney 
Tested-by: Paul Gortmaker

rcu: Don't track sysidle state if no nohz_full= CPUs

2014-09-16T17:08:02+00:00

If there are no nohz_full= CPUs, then there is currently no reason to
track sysidle state.  This commit therefore short-circuits this state
tracking if !tick_nohz_full_enabled().

Note that these checks will need to be revisited if nohz_full= state
can ever be changed at runtime.

Signed-off-by: Paul E. McKenney 
Acked-by: Frederic Weisbecker 
Tested-by: Paul Gortmaker

rcu: Eliminate redundant rcu_sysidle_state variable

2014-09-16T17:08:02+00:00

Now that we have rcu_state_p, which references rcu_preempt_state for
TREE_PREEMPT_RCU and rcu_sched_state for TREE_RCU, we don't need a
separate rcu_sysidle_state variable.  This commit therefore eliminates
rcu_preempt_state in favor of rcu_state_p.

Signed-off-by: Paul E. McKenney 
Reviewed-by: Pranith Kumar 
Acked-by: Frederic Weisbecker 
Tested-by: Paul Gortmaker

rcu: Check for have_rcu_nocb_mask instead of rcu_nocb_mask

2014-09-16T17:08:02+00:00

If we configure a kernel with CONFIG_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_NONE=y and
CONFIG_CPUMASK_OFFSTACK=n and do not pass in a rcu_nocb= boot parameter, the
cpumask rcu_nocb_mask can be garbage instead of NULL.

Hence this commit replaces checks for rcu_nocb_mask == NULL with a check for
have_rcu_nocb_mask.

Signed-off-by: Pranith Kumar 
Signed-off-by: Paul E. McKenney 
Tested-by: Paul Gortmaker

rcu: Create rcuo kthreads only for onlined CPUs

2014-09-16T17:08:02+00:00

RCU currently uses for_each_possible_cpu() to spawn rcuo kthreads,
which can result in more rcuo kthreads than one would expect, for
example, derRichard reported 64 CPUs worth of rcuo kthreads on an
8-CPU image.  This commit therefore creates rcuo kthreads only for
those CPUs that actually come online.

This was reported by derRichard on the OFTC IRC network.

Reported-by: Richard Weinberger 
Signed-off-by: Paul E. McKenney 
Reviewed-by: Josh Triplett 
Tested-by: Paul Gortmaker