linux.git/kernel/rcu/tree_nocb.h, branch v6.7

rcu: dynamically allocate the rcu-lazy shrinker

2023-10-04T17:32:24+00:00

Use new APIs to dynamically allocate the rcu-lazy shrinker.

Link: https://lkml.kernel.org/r/20230911094444.68966-16-zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Reviewed-by: Joel Fernandes (Google) 
Acked-by: Muchun Song 
Cc: Abhinav Kumar 
Cc: Alasdair Kergon 
Cc: Alexander Viro 
Cc: Alyssa Rosenzweig 
Cc: Andreas Dilger 
Cc: Andreas Gruenbacher 
Cc: Anna Schumaker 
Cc: Arnd Bergmann 
Cc: Bob Peterson 
Cc: Borislav Petkov 
Cc: Carlos Llamas 
Cc: Chandan Babu R 
Cc: Chao Yu 
Cc: Chris Mason 
Cc: Christian Brauner 
Cc: Christian Koenig 
Cc: Chuck Lever 
Cc: Coly Li 
Cc: Dai Ngo 
Cc: Daniel Vetter 
Cc: Daniel Vetter 
Cc: "Darrick J. Wong" 
Cc: Dave Chinner 
Cc: Dave Hansen 
Cc: David Airlie 
Cc: David Hildenbrand 
Cc: David Sterba 
Cc: Dmitry Baryshkov 
Cc: Gao Xiang 
Cc: Greg Kroah-Hartman 
Cc: Huang Rui 
Cc: Ingo Molnar 
Cc: Jaegeuk Kim 
Cc: Jani Nikula 
Cc: Jan Kara 
Cc: Jason Wang 
Cc: Jeff Layton 
Cc: Jeffle Xu 
Cc: Joonas Lahtinen 
Cc: Josef Bacik 
Cc: Juergen Gross 
Cc: Kent Overstreet 
Cc: Kirill Tkhai 
Cc: Marijn Suijten 
Cc: "Michael S. Tsirkin" 
Cc: Mike Snitzer 
Cc: Minchan Kim 
Cc: Nadav Amit 
Cc: Neil Brown 
Cc: Oleksandr Tyshchenko 
Cc: Olga Kornievskaia 
Cc: Paul E. McKenney 
Cc: Richard Weinberger 
Cc: Rob Clark 
Cc: Rob Herring 
Cc: Rodrigo Vivi 
Cc: Roman Gushchin 
Cc: Sean Paul 
Cc: Sergey Senozhatsky 
Cc: Song Liu 
Cc: Stefano Stabellini 
Cc: Steven Price 
Cc: "Theodore Ts'o" 
Cc: Thomas Gleixner 
Cc: Tomeu Vizoso 
Cc: Tom Talpey 
Cc: Trond Myklebust 
Cc: Tvrtko Ursulin 
Cc: Vlastimil Babka 
Cc: Xuan Zhuo 
Cc: Yue Hu 
Signed-off-by: Andrew Morton

rcu: Make the rcu_nocb_poll boot parameter usable via boot config

2023-08-16T21:27:41+00:00

The rcu_nocb_poll kernel boot parameter is defined via early_param(),
whose parsing functions are invoked from parse_early_param() which
is in turn invoked by setup_arch(), which is very early indeed.  It
is invoked so early that the console output timestamps read 0.000000,
in other words, before time begins.

This use of early_param() means that the rcu_nocb_poll kernel boot
parameter cannot usefully be embedded into the kernel image.  Yes, you
can embed it, but setup_boot_config() is invoked from start_kernel()
too late for it to be parsed.

But it makes no sense to parse this parameter so early.  After all,
it cannot do anything until the rcuog kthreads are created, which is
long after rcu_init() time, let alone setup_boot_config() time.

This commit therefore switches the rcu_nocb_poll kernel boot parameter
from early_param() to __setup(), which allows boot-config parsing of
this parameter, in turn allowing it to be embedded into the kernel image.

Signed-off-by: Paul E. McKenney 
Reviewed-by: Joel Fernandes (Google)

rcu/nocb: Make shrinker iterate only over NOCB CPUs

2023-05-11T20:44:50+00:00

Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating
over the NOCB mask is enough for both counting and scanning. Just lock
the mostly uncontended barrier mutex on counting as well in order to
keep rcu_nocb_mask stable.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker

2023-05-10T00:26:59+00:00

The ->lazy_len is only checked locklessly. Recheck again under the
->nocb_lock to avoid spending more time on flushing/waking if not
necessary. The ->lazy_len can still increment concurrently (from 1 to
infinity) but under the ->nocb_lock we at least know for sure if there
are lazy callbacks at all (->lazy_len > 0).

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Fix shrinker race against callback enqueuer

2023-05-10T00:26:58+00:00

The shrinker resets the lazy callbacks counter in order to trigger the
pending lazy queue flush though the rcuog kthread. The counter reset is
protected by the ->nocb_lock against concurrent accesses...except
for one of them. Here is a list of existing synchronized readers/writer:

1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under
   ->nocb_lock and ->nocb_bypass_lock.

2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so
   under ->nocb_bypass_lock _only_.

3) The lazy flush checks and resets to 0 under ->nocb_lock and
	->nocb_bypass_lock.

The shrinker protects its ->lazy_len reset against cases 1) and 3) but
not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock
may be cancelled right away by an overwrite from an enqueuer, leading
rcuog to ignore the flush.

To avoid that, use the proper bypass flush API which takes care of all
those details.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Protect lazy shrinker against concurrent (de-)offloading

2023-05-10T00:26:58+00:00

The shrinker may run concurrently with callbacks (de-)offloading. As
such, calling rcu_nocb_lock() is very dangerous because it does a
conditional locking. The worst outcome is that rcu_nocb_lock() doesn't
lock but rcu_nocb_unlock() eventually unlocks, or the reverse, creating
an imbalance.

Fix this with protecting against (de-)offloading using the barrier mutex.
Although if the barrier mutex is contended, which should be rare, then
step aside so as not to trigger a mutex VS allocation
dependency chain.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu: Register rcu-lazy shrinker only for CONFIG_RCU_LAZY=y kernels

2023-04-05T13:47:43+00:00

The lazy_rcu_shrink_count() shrinker function is registered even in
kernels built with CONFIG_RCU_LAZY=n, in which case this function
uselessly consumes cycles learning that no CPU has any lazy callbacks
queued.

This commit therefore registers this shrinker function only in the kernels
built with CONFIG_RCU_LAZY=y, where it might actually do something useful.

Signed-off-by: Zqiang 
Reviewed-by: Frederic Weisbecker 
Reviewed-by: Joel Fernandes (Google) 
Signed-off-by: Paul E. McKenney 
Signed-off-by: Joel Fernandes (Google)

rcu: Shrinker for lazy rcu

2022-11-29T22:02:52+00:00

The shrinker is used to speed up the free'ing of memory potentially held
by RCU lazy callbacks. RCU kernel module test cases show this to be
effective. Test is introduced in a later patch.

Signed-off-by: Vineeth Pillai 
Signed-off-by: Joel Fernandes (Google) 
Signed-off-by: Paul E. McKenney

rcu: Refactor code a bit in rcu_nocb_do_flush_bypass()

2022-11-29T22:02:52+00:00

This consolidates the code a bit and makes it cleaner. Functionally it
is the same.

Reported-by: Paul E. McKenney 
Signed-off-by: Joel Fernandes (Google) 
Reviewed-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu: Make call_rcu() lazy to save power

2022-11-29T22:02:23+00:00

Implement timer-based RCU callback batching (also known as lazy
callbacks). With this we save about 5-10% of power consumed due
to RCU requests that happen when system is lightly loaded or idle.

By default, all async callbacks (queued via call_rcu) are marked
lazy. An alternate API call_rcu_hurry() is provided for the few users,
for example synchronize_rcu(), that need the old behavior.

The batch is flushed whenever a certain amount of time has passed, or
the batch on a particular CPU grows too big. Also memory pressure will
flush it in a future patch.

To handle several corner cases automagically (such as rcu_barrier() and
hotplug), we re-use bypass lists which were originally introduced to
address lock contention, to handle lazy CBs as well. The bypass list
length has the lazy CB length included in it. A separate lazy CB length
counter is also introduced to keep track of the number of lazy CBs.

[ paulmck: Fix formatting of inline call_rcu_lazy() definition. ]
[ paulmck: Apply Zqiang feedback. ]
[ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ]

Suggested-by: Paul McKenney 
Acked-by: Frederic Weisbecker 
Signed-off-by: Joel Fernandes (Google) 
Signed-off-by: Paul E. McKenney