linux.git/kernel/sched/psi.c, branch v6.16

treewide, timers: Rename from_timer() to timer_container_of()

2025-06-08T07:07:37+00:00

Move this API to the canonical timer_*() namespace.

[ tglx: Redone against pre rc1 ]

Signed-off-by: Ingo Molnar 
Signed-off-by: Thomas Gleixner 
Link: https://lore.kernel.org/all/aB2X0jCKQO56WdMt@gmail.com

treewide: Switch/rename to timer_delete[_sync]()

2025-04-05T08:30:12+00:00

timer_delete[_sync]() replaces del_timer[_sync](). Convert the whole tree
over and remove the historical wrapper inlines.

Conversion was done with coccinelle plus manual fixups where necessary.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar

sched, psi: Don't account irq time if sched_clock_irqtime is disabled

2025-01-13T13:10:26+00:00

sched_clock_irqtime may be disabled due to the clock source. When disabled,
irq_time_read() won't change over time, so there is nothing to account. We
can save iterating the whole hierarchy on every tick and context switch.

Signed-off-by: Yafang Shao 
Signed-off-by: Yafang Shao 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Michal Koutný 
Acked-by: Johannes Weiner 
Link: https://lore.kernel.org/r/20250103022409.2544-4-laoar.shao@gmail.com

sched: psi: fix bogus pressure spikes from aggregation race

2024-10-03T23:03:16+00:00

Brandon reports sporadic, non-sensical spikes in cumulative pressure
time (total=) when reading cpu.pressure at a high rate. This is due to
a race condition between reader aggregation and tasks changing states.

While it affects all states and all resources captured by PSI, in
practice it most likely triggers with CPU pressure, since scheduling
events are so frequent compared to other resource events.

The race context is the live snooping of ongoing stalls during a
pressure read. The read aggregates per-cpu records for stalls that
have concluded, but will also incorporate ad-hoc the duration of any
active state that hasn't been recorded yet. This is important to get
timely measurements of ongoing stalls. Those ad-hoc samples are
calculated on-the-fly up to the current time on that CPU; since the
stall hasn't concluded, it's expected that this is the minimum amount
of stall time that will enter the per-cpu records once it does.

The problem is that the path that concludes the state uses a CPU clock
read that is not synchronized against aggregators; the clock is read
outside of the seqlock protection. This allows aggregators to race and
snoop a stall with a longer duration than will actually be recorded.

With the recorded stall time being less than the last snapshot
remembered by the aggregator, a subsequent sample will underflow and
observe a bogus delta value, resulting in an erratic jump in pressure.

Fix this by moving the clock read of the state change into the seqlock
protection. This ensures no aggregation can snoop live stalls past the
time that's recorded when the state concludes.

Reported-by: Brandon Duffany 
Link: https://bugzilla.kernel.org/show_bug.cgi?id=219194
Link: https://lore.kernel.org/lkml/20240827121851.GB438928@cmpxchg.org/
Fixes: df77430639c9 ("psi: Reduce calls to sched_clock() in psi")
Cc: stable@vger.kernel.org
Signed-off-by: Johannes Weiner 
Reviewed-by: Chengming Zhou 
Signed-off-by: Linus Torvalds

Merge branch 'sched/urgent' into sched/core, to pick up fixes and refresh the branch

2024-07-11T08:42:33+00:00

Signed-off-by: Ingo Molnar

sched/psi: Optimise psi_group_change a bit

2024-07-04T13:59:52+00:00

The current code loops over the psi_states only to call a helper which
then resolves back to the action needed for each state using a switch
statement. That is effectively creating a double indirection of a kind
which, given how all the states need to be explicitly listed and handled
anyway, we can simply remove. Both the for loop and the switch statement
that is.

The benefit is both in the code size and CPU time spent in this function.
YMMV but on my Steam Deck, while in a game, the patch makes the CPU usage
go from ~2.4% down to ~1.2%. Text size at the same time went from 0x323 to
0x2c1.

Signed-off-by: Tvrtko Ursulin 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Chengming Zhou 
Acked-by: Johannes Weiner 
Link: https://lkml.kernel.org/r/20240625135000.38652-1-tursulin@igalia.com

sched: Move psi_account_irqtime() out of update_rq_clock_task() hotpath

2024-07-01T11:01:44+00:00

It was reported that in moving to 6.1, a larger then 10%
regression was seen in the performance of
clock_gettime(CLOCK_THREAD_CPUTIME_ID,...).

Using a simple reproducer, I found:
5.10:
100000000 calls in 24345994193 ns => 243.460 ns per call
100000000 calls in 24288172050 ns => 242.882 ns per call
100000000 calls in 24289135225 ns => 242.891 ns per call

6.1:
100000000 calls in 28248646742 ns => 282.486 ns per call
100000000 calls in 28227055067 ns => 282.271 ns per call
100000000 calls in 28177471287 ns => 281.775 ns per call

The cause of this was finally narrowed down to the addition of
psi_account_irqtime() in update_rq_clock_task(), in commit
52b1364ba0b1 ("sched/psi: Add PSI_IRQ to track IRQ/SOFTIRQ
pressure").

In my initial attempt to resolve this, I leaned towards moving
all accounting work out of the clock_gettime() call path, but it
wasn't very pretty, so it will have to wait for a later deeper
rework. Instead, Peter shared this approach:

Rework psi_account_irqtime() to use its own psi_irq_time base
for accounting, and move it out of the hotpath, calling it
instead from sched_tick() and __schedule().

In testing this, we found the importance of ensuring
psi_account_irqtime() is run under the rq_lock, which Johannes
Weiner helpfully explained, so also add some lockdep annotations
to make that requirement clear.

With this change the performance is back in-line with 5.10:
6.1+fix:
100000000 calls in 24297324597 ns => 242.973 ns per call
100000000 calls in 24318869234 ns => 243.189 ns per call
100000000 calls in 24291564588 ns => 242.916 ns per call

Reported-by: Jimmy Shiu 
Originally-by: Peter Zijlstra 
Signed-off-by: John Stultz 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Chengming Zhou 
Reviewed-by: Qais Yousef 
Link: https://lore.kernel.org/r/20240618215909.4099720-1-jstultz@google.com

sched: Fix spelling in comments

2024-05-27T15:00:21+00:00

Do a spell-checking pass.

Signed-off-by: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar

sched/psi: Update poll => rtpoll in relevant comments

2023-10-16T11:42:49+00:00

The PSI trigger code is now making a distinction between privileged and
unprivileged triggers, after the following commit:

 65457b74aa94 ("sched/psi: Rename existing poll members in preparation")

But some comments have not been modified along with the code, so they
need to be updated.

This will help readers better understand the code.

Signed-off-by: Fan Yu 
Signed-off-by: Ingo Molnar 
Cc: Johannes Weiner 
Cc: Suren Baghdasaryan 
Cc: Peter Ziljstra 
Link: https://lore.kernel.org/r/202310161920399921184@zte.com.cn

sched/psi: Bail out early from irq time accounting

2023-10-13T07:56:29+00:00

We could bail out early when psi was disabled.

Signed-off-by: Haifeng Xu 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Chengming Zhou 
Link: https://lore.kernel.org/r/20230926115722.467833-1-haifeng.xu@shopee.com