linux-stable.git/kernel/rcu/tasks.h, branch v6.0

rcu-tasks: Use delayed_work to delay rcu_tasks_verify_self_tests()

2022-06-21T22:49:38+00:00

Commit 2585014188d5 ("rcu-tasks: Be more patient for RCU Tasks
boot-time testing") fixes false positive rcu_tasks verification check
failure by repeating the test once every second until timeout using
schedule_timeout_uninterruptible().

Since rcu_tasks_verify_selft_tests() is called from do_initcalls()
as a late_initcall, this has the undesirable side effect of delaying
other late_initcall's queued after it by a second or more.  Fix this by
instead using delayed_work to repeat the verification check.

Fixes: 2585014188d5 ("rcu-tasks: Be more patient for RCU Tasks boot-time testing")
Signed-off-by: Waiman Long 
Signed-off-by: Paul E. McKenney

rcu-tasks: Be more patient for RCU Tasks boot-time testing

2022-06-21T22:49:38+00:00

The RCU-Tasks family of grace-period primitives can take some time to
complete, and the amount of time can depend on the exact hardware and
software configuration.  Some configurations boot up fast enough that the
RCU-Tasks verification process gets false-positive failures.  This commit
therefore allows up to 30 seconds for the grace periods to complete, with
this value adjustable downwards using the rcupdate.rcu_task_stall_timeout
kernel boot parameter.

Reported-by: Matthew Wilcox 
Reported-by: Zhouyi Zhou 
Signed-off-by: Paul E. McKenney 
Tested-by: Zhouyi Zhou 
Tested-by: Mark Rutland

rcu-tasks: Update comments

2022-06-21T22:49:38+00:00

This commit updates comments to reflect the changes in the series
of commits that eliminated the full task-list scan.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Disable and enable CPU hotplug in same function

2022-06-21T22:49:38+00:00

The rcu_tasks_trace_pregp_step() function invokes cpus_read_lock() to
disable CPU hotplug, and a later call to the rcu_tasks_trace_postscan()
function invokes cpus_read_unlock() to re-enable it.  This was absolutely
necessary in the past in order to protect the intervening scan of the full
tasks list, but there is no longer such a scan.  This commit therefore
improves readability by moving the cpus_read_unlock() call to the end
of the rcu_tasks_trace_pregp_step() function.  This commit is a pure
code-motion commit without any (intended) change in functionality.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Eliminate RCU Tasks Trace IPIs to online CPUs

2022-06-21T22:49:38+00:00

Currently, the RCU Tasks Trace grace-period kthread IPIs each online CPU
using smp_call_function_single() in order to track any tasks currently in
RCU Tasks Trace read-side critical sections during which the corresponding
task has neither blocked nor been preempted.  These IPIs are annoying
and are also not strictly necessary because any task that blocks or is
preempted within its current RCU Tasks Trace read-side critical section
will be tracked on one of the per-CPU rcu_tasks_percpu structure's
->rtp_blkd_tasks list.  So the only time that this is a problem is if
one of the CPUs runs through a long-duration RCU Tasks Trace read-side
critical section without a context switch.

Note that the task_call_func() function cannot help here because there is
no safe way to identify the target task.  Of course, the task_call_func()
function will be very useful later, when processing the list of tasks,
but it needs to know the task.

This commit therefore creates a cpu_curr_snapshot() function that returns
a pointer the task_struct structure of some task that happened to be
running on the specified CPU more or less during the time that the
cpu_curr_snapshot() function was executing.  If there was no context
switch during this time, this function will return a pointer to the
task_struct structure of the task that was running throughout.  If there
was a context switch, then the outgoing task will be taken care of by
RCU's context-switch hook, and the incoming task was either already taken
care during some previous context switch, or it is not currently within an
RCU Tasks Trace read-side critical section.  And in this latter case, the
grace period already started, so there is no need to wait on this task.

This new cpu_curr_snapshot() function is invoked on each CPU early in
the RCU Tasks Trace grace-period processing, and the resulting tasks
are queued for later quiescent-state inspection.

Signed-off-by: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Maintain a count of tasks blocking RCU Tasks Trace grace period

2022-06-21T22:49:38+00:00

This commit maintains a new n_trc_holdouts counter that tracks the number
of tasks blocking the RCU Tasks grace period.  This counter is useful
for debugging, and its value has been added to a diagostic message.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Stop RCU Tasks Trace from scanning full tasks list

2022-06-21T22:37:28+00:00

This commit takes off the training wheels and relies only on scanning
currently running tasks and tasks that have blocked or been preempted
within their current RCU Tasks Trace read-side critical section.

Before this commit, the time complexity of an RCU Tasks Trace grace
period is O(T), where T is the number of tasks.  After this commit,
this time complexity is O(C+B), where C is the number of CPUs and B
is the number of tasks that have blocked (or been preempted) at least
once during their current RCU Tasks Trace read-side critical sections.
Of course, if all tasks have blocked (or been preempted) at least once
during their current RCU Tasks Trace read-side critical sections, this is
still O(T), but current expectations are that RCU Tasks Trace read-side
critical section will be short and that there will normally not be large
numbers of tasks blocked within such a critical section.

Dave Marchevsky kindly measured the effects of this commit on the RCU
Tasks Trace grace-period latency and the rcu_tasks_trace_kthread task's
CPU consumption per RCU Tasks Trace grace period over the course of a
fixed test, all in milliseconds:

		Before			After

GP latency	22.3 ms stddev > 0.1	17.0 ms stddev < 0.1

GP CPU		 2.3 ms stddev 0.3	 1.1 ms stddev 0.2

This was on a system with 15,000 tasks, so it is reasonable to expect
much larger savings on the systems on which this issue was first noted,
given that they sport well in excess of 100,000 tasks.  CPU consumption
was measured using profiling techniques.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh 
Tested-by: Dave Marchevsky

rcu-tasks: Stop RCU Tasks Trace from scanning idle tasks

2022-06-20T16:22:29+00:00

Now that RCU scans both running tasks and tasks that have blocked within
their current RCU Tasks Trace read-side critical section, there is no
need for it to scan the idle tasks.  After all, an idle loop should not
be remain within an RCU Tasks Trace read-side critical section across
exit from idle, and from a BPF viewpoint, functions invoked from the
idle loop should not sleep.  So only running idle tasks can be within
RCU Tasks Trace read-side critical sections.

This commit therefore removes the scan of the idle tasks from the
rcu_tasks_trace_postscan() function.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Pull in tasks blocked within RCU Tasks Trace readers

2022-06-20T16:22:29+00:00

This commit scans each CPU's ->rtp_blkd_tasks list, adding them to
the list of holdout tasks.  This will cause the current RCU Tasks Trace
grace period to wait until these tasks exit their RCU Tasks Trace
read-side critical sections.  This commit will enable later work
omitting the scan of the full task list.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh

rcu-tasks: Scan running tasks for RCU Tasks Trace readers

2022-06-20T16:22:29+00:00

A running task might be within an RCU Tasks Trace read-side critical
section for any length of time, but will not be placed on any of the
per-CPU rcu_tasks_percpu structure's ->rtp_blkd_tasks lists.  Therefore
any RCU Tasks Trace grace-period processing that does not scan the full
task list must interact with the running tasks.

This commit therefore causes the rcu_tasks_trace_pregp_step() function
to IPI each CPU in order to place the corresponding task on the holdouts
list and to record whether or not it was in an RCU Tasks Trace read-side
critical section.  Yes, it is possible to avoid adding it to that list
if it is not a reader, but that would prevent the system from remembering
that this task was in a quiescent state.  Which is why the running tasks
are unconditionally added to the holdout list.

Signed-off-by: Paul E. McKenney 
Cc: Neeraj Upadhyay 
Cc: Eric Dumazet 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Martin KaFai Lau 
Cc: KP Singh