linux-stable.git/kernel/rcu/srcutree.c, branch v4.20.2

srcu: Lock srcu_data structure in srcu_gp_start()

2019-01-13T08:24:05+00:00

commit eb4c2382272ae7ae5d81fdfa5b7a6c86146eaaa4 upstream.

The srcu_gp_start() function is called with the srcu_struct structure's
->lock held, but not with the srcu_data structure's ->lock.  This is
problematic because this function accesses and updates the srcu_data
structure's ->srcu_cblist, which is protected by that lock.  Failing to
hold this lock can result in corruption of the SRCU callback lists,
which in turn can result in arbitrarily bad results.

This commit therefore makes srcu_gp_start() acquire the srcu_data
structure's ->lock across the calls to rcu_segcblist_advance() and
rcu_segcblist_accelerate(), thus preventing this corruption.

Reported-by: Bart Van Assche 
Reported-by: Christoph Hellwig 
Reported-by: Sebastian Kuzminsky 
Signed-off-by: Dennis Krein 
Signed-off-by: Paul E. McKenney 
Tested-by: Dennis Krein 
Cc:  # 4.16.x
Signed-off-by: Greg Kroah-Hartman

Merge branches 'doc.2018.08.30a', 'dynticks.2018.08.30b', 'srcu.2018.08.30b' and 'torture.2018.08.29a' into HEAD

2018-08-30T23:12:53+00:00

doc.2018.08.30a: Documentation updates
dynticks.2018.08.30b: RCU flavor consolidation updates and cleanups
srcu.2018.08.30b: SRCU updates
torture.2018.08.29a: Torture-test updates

srcu: Make early-boot call_srcu() reuse workqueue lists

2018-08-30T23:10:49+00:00

Allocating a list_head structure that is almost never used, and, when
used, is used only during early boot (rcu_init() and earlier), is a bit
wasteful.  This commit therefore eliminates that list_head in favor of
the one in the work_struct structure.  This is safe because the work_struct
structure cannot be used until after rcu_init() returns.

Reported-by: Steven Rostedt 
Signed-off-by: Paul E. McKenney 
Cc: Tejun Heo 
Cc: Lai Jiangshan 
Tested-by: Steven Rostedt (VMware)

srcu: Make call_srcu() available during very early boot

2018-08-30T23:10:19+00:00

Event tracing is moving to SRCU in order to take advantage of the fact
that SRCU may be safely used from idle and even offline CPUs.  However,
event tracing can invoke call_srcu() very early in the boot process,
even before workqueue_init_early() is invoked (let alone rcu_init()).
Therefore, call_srcu()'s attempts to queue work fail miserably.

This commit therefore detects this situation, and refrains from attempting
to queue work before rcu_init() time, but does everything else that it
would have done, and in addition, adds the srcu_struct to a global list.
The rcu_init() function now invokes a new srcu_init() function, which
is empty if CONFIG_SRCU=n.  Otherwise, srcu_init() queues work for
each srcu_struct on the list.  This all happens early enough in boot
that there is but a single CPU with interrupts disabled, which allows
synchronization to be dispensed with.

Of course, the queued work won't actually be invoked until after
workqueue_init() is invoked, which happens shortly after the scheduler
is up and running.  This means that although call_srcu() may be invoked
any time after per-CPU variables have been set up, there is still a very
narrow window when synchronize_srcu() won't work, and this window
extends from the time that the scheduler starts until the time that
workqueue_init() returns.  This can be fixed in a manner similar to
the fix for synchronize_rcu_expedited() and friends, but until someone
actually needs to use synchronize_srcu() during this window, this fix
is added churn for no benefit.

Finally, note that Tree SRCU's new srcu_init() function invokes
queue_work() rather than the queue_delayed_work() function that is
invoked post-boot.  The reason is that queue_delayed_work() will (as you
would expect) post a timer, and timers have not yet been initialized.
So use of queue_work() avoids the complaints about use of uninitialized
spinlocks that would otherwise result.  Besides, some delay is already
provide by the aforementioned fact that the queued work won't actually
be invoked until after the scheduler is up and running.

Requested-by: Steven Rostedt 
Signed-off-by: Paul E. McKenney 
Tested-by: Steven Rostedt (VMware)

rcu: Clean up flavor-related definitions and comments in srcutree.h

2018-08-30T23:03:34+00:00

Signed-off-by: Paul E. McKenney

rcu: Remove rsp parameter from rcu_node tree accessor macros

2018-08-30T23:03:16+00:00

There now is only one rcu_state structure in a given build of the Linux
kernel, so there is no need to pass it as a parameter to RCU's rcu_node
tree's accessor macros.  This commit therefore removes the rsp parameter
from those macros in kernel/rcu/rcu.h, and removes some now-unused rsp
local variables while in the area.

Signed-off-by: Paul E. McKenney

srcu: Add grace-period number to rcutorture statistics printout

2018-07-12T22:39:25+00:00

This commit adds the SRCU grace-period number to the rcutorture statistics
printout, which allows it to be compared to the rcutorture "Writer stall
state" message.

Signed-off-by: Paul E. McKenney

rcu: Use pr_fmt to prefix "rcu: " to logging output

2018-07-12T22:39:13+00:00

This commit also adjusts some whitespace while in the area.

Signed-off-by: Joe Perches 
Signed-off-by: Paul E. McKenney 
[ paulmck: Revert string-breaking %s as requested by Andy Shevchenko. ]

rcutorture: Convert rcutorture_get_gp_data() to ->gp_seq

2018-07-12T21:27:57+00:00

SRCU has long used ->srcu_gp_seq, and now RCU uses ->gp_seq.  This
commit therefore moves the rcutorture_get_gp_data() function from
a ->gpnum / ->completed pair to ->gp_seq.

Signed-off-by: Paul E. McKenney

srcu: Add address of first callback to rcutorture output

2018-06-25T18:26:24+00:00

This commit adds the address of the first callback to the per-CPU rcutorture
output in order to allow lost wakeups to be more efficiently tracked down.

Signed-off-by: Paul E. McKenney