linux-stable.git/kernel/sched/rt.c, branch v7.3-rc2 Linux kernel stable tree sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate 2026-09-02T07:17:49+00:00 Seiji Nishikawa snishika@redhat.com 2026-08-30T07:37:46+00:00 dae5c0292080dd7b9c7d784268dcf443f1f3d15e A migrate_disable()'d RT task cannot be moved to another CPU, but the scheduler still keeps such a task on that CPU's pushable list (rq->rt.pushable_tasks) and still marks the runqueue RT-overloaded (rq->rt.overloaded = 1). So the RT balancer keeps treating this CPU as having a task to move away, and keeps trying to move the task, but the push can never succeed. When the head is pinned, push_rt_task() does not give up either. It falls back to pushing rq->curr instead, using the per-CPU stopper, as added by commit a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing"). The CPU spends tens of milliseconds in this retry loop. The core is isolated for real-time work, but during the loop nearly half of its time is consumed by pushes that cannot succeed. An ftrace capture of the affected CPU, with sched_switch enabled and commit 94894c9c477e ("sched/rt: Skip currently executing CPU in rto_next_cpu()") applied, shows where the CPU time went. Two SCHED_FIFO tasks at equal priority shared the CPU, taskA migrate_disable()'d and queued, taskB as rq->curr. In one 89 ms window, taskB got only 52 ms of CPU. The other 37 ms went to the stopper thread. The scheduler kept trying to push taskA, the pinned head of the pushable list, fell back to pushing taskB instead, and woke the stopper 5204 times. Every one of those pushes failed and no task was moved. taskA stayed runnable and queued the whole time, and never ran. Pushing taskB fails on a re-check. find_lock_lowest_rq() drops the rq lock to take the target rq lock, then checks again with "task != pick_next_pushable_task(rq)". The task being pushed is taskB, but the pick returns taskA, the head of the pushable list. taskB is rq->curr, and set_next_task_rt() removes the running task from that list, so taskB can never be the head. The check expects a candidate taken from the pushable list, but the fallback pushes rq->curr, which is never on that list. So the check fails every time. .--> push-IPI arrives | | | v | pushable head = taskA -> pinned, cannot be pushed | | | v | so push taskB instead -> wake migration/N, a stop-class | | thread, so it preempts taskB | v | re-check compares taskB against the pushable head, | which is still taskA -> give up | | | v | nothing moved, taskA still queued, rq still overloaded | | '----------' repeats every ~17 us, 5204 times, for 89 ms The loop cannot stop itself. Every round leaves the runqueue exactly as it was, so the next push-IPI does the same thing. In the capture it ended only when taskB went to sleep on its own. taskA was then picked locally and left the pushable list. CPU time per task in the window, from sched_switch: taskB 51.95 ms real work migration/N 37.18 ms nothing moved taskA 0.00 ms queued the whole time, never picked idle 0.01 ms Counts over the same window: 7667 push-IPIs handled on this CPU 17481 pick_next_pushable_task() returned taskA, still pinned 5204 find_lock_lowest_rq() gave up on the re-check 1 push that actually completed 0 migrations of taskA The CPU times and the window length come from the standard sched_switch tracepoint. The counts needed tracepoints added inside the RT balancer for this investigation. The self-IPI path is closed by the rto_next_cpu() fix above, and that part works. But the runqueue is still marked overloaded, because the pinned task is still advertised as pushable. Other CPUs now send the push-IPIs during their own RT balancing, and the same loop runs again. Closing the self-IPI path did not stop a pinned task from triggering push balancing. A pinned task should never have been returned as a push candidate in the first place. A migrate_disable()'d task cannot be migrated, so it belongs in the same skip that was added for on_cpu tasks by commit e0ca8991b2de ("sched: Make class_schedulers avoid pushing current, and get rid of proxy_tag_curr()"). Add is_migration_disabled() to the skip condition in pick_next_pushable_task() and pick_next_pushable_dl_task(). With the skip in place, if the pinned task is the only extra runnable task the helpers return NULL, push_rt_task() and push_dl_task() give up early, and no stopper is woken. The pinned task then runs locally once curr yields. If a task that really can be migrated is queued behind the pinned head, it is now picked and pushed for real. This makes the fallback that pushes rq->curr unreachable when the pushable head is migrate-disabled. Nothing is lost, because that path was always stopped by the re-check described above. In the capture it ran 5204 times and moved nothing. Fixes: a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing") Signed-off-by: Seiji Nishikawa <snishika@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260830073746.2189355-1-snishika@redhat.com 
A migrate_disable()'d RT task cannot be moved to another CPU, but the
scheduler still keeps such a task on that CPU's pushable list
(rq->rt.pushable_tasks) and still marks the runqueue RT-overloaded
(rq->rt.overloaded = 1). So the RT balancer keeps treating this CPU as
having a task to move away, and keeps trying to move the task, but the
push can never succeed. When the head is pinned, push_rt_task() does not
give up either. It falls back to pushing rq->curr instead, using the
per-CPU stopper, as added by commit a7c81556ec4d ("sched: Fix
migrate_disable() vs rt/dl balancing").

The CPU spends tens of milliseconds in this retry loop. The core is
isolated for real-time work, but during the loop nearly half of its time
is consumed by pushes that cannot succeed.

An ftrace capture of the affected CPU, with sched_switch enabled and
commit 94894c9c477e ("sched/rt: Skip currently executing CPU in
rto_next_cpu()") applied, shows where the CPU time went. Two SCHED_FIFO
tasks at equal priority shared the CPU, taskA migrate_disable()'d and
queued, taskB as rq->curr. In one 89 ms window, taskB got only 52 ms of
CPU. The other 37 ms went to the stopper thread.

The scheduler kept trying to push taskA, the pinned head of the pushable
list, fell back to pushing taskB instead, and woke the stopper 5204
times. Every one of those pushes failed and no task was moved. taskA
stayed runnable and queued the whole time, and never ran.

Pushing taskB fails on a re-check. find_lock_lowest_rq() drops the rq
lock to take the target rq lock, then checks again with
"task != pick_next_pushable_task(rq)".

The task being pushed is taskB, but the pick returns taskA, the head of
the pushable list. taskB is rq->curr, and set_next_task_rt() removes the
running task from that list, so taskB can never be the head. The check
expects a candidate taken from the pushable list, but the fallback
pushes rq->curr, which is never on that list. So the check fails every
time.

   .--> push-IPI arrives
   |          |
   |          v
   |     pushable head = taskA   ->  pinned, cannot be pushed
   |          |
   |          v
   |     so push taskB instead   ->  wake migration/N, a stop-class
   |          |                      thread, so it preempts taskB
   |          v
   |     re-check compares taskB against the pushable head,
   |     which is still taskA                        ->  give up
   |          |
   |          v
   |     nothing moved, taskA still queued, rq still overloaded
   |          |
   '----------'
         repeats every ~17 us, 5204 times, for 89 ms

   The loop cannot stop itself. Every round leaves the runqueue
   exactly as it was, so the next push-IPI does the same thing. In
   the capture it ended only when taskB went to sleep on its own.
   taskA was then picked locally and left the pushable list.

   CPU time per task in the window, from sched_switch:

     taskB         51.95 ms   real work
     migration/N   37.18 ms   nothing moved
     taskA          0.00 ms   queued the whole time, never picked
     idle           0.01 ms

   Counts over the same window:

      7667  push-IPIs handled on this CPU
     17481  pick_next_pushable_task() returned taskA, still pinned
      5204  find_lock_lowest_rq() gave up on the re-check
         1  push that actually completed
         0  migrations of taskA

   The CPU times and the window length come from the standard
   sched_switch tracepoint. The counts needed tracepoints added inside
   the RT balancer for this investigation.

The self-IPI path is closed by the rto_next_cpu() fix above, and that
part works. But the runqueue is still marked overloaded, because the
pinned task is still advertised as pushable. Other CPUs now send the
push-IPIs during their own RT balancing, and the same loop runs again.
Closing the self-IPI path did not stop a pinned task from triggering
push balancing.

A pinned task should never have been returned as a push candidate in the
first place. A migrate_disable()'d task cannot be migrated, so it
belongs in the same skip that was added for on_cpu tasks by
commit e0ca8991b2de ("sched: Make class_schedulers avoid pushing
current, and get rid of proxy_tag_curr()"). Add is_migration_disabled()
to the skip condition in pick_next_pushable_task() and
pick_next_pushable_dl_task().

With the skip in place, if the pinned task is the only extra runnable
task the helpers return NULL, push_rt_task() and push_dl_task() give up
early, and no stopper is woken. The pinned task then runs locally once
curr yields. If a task that really can be migrated is queued behind the
pinned head, it is now picked and pushed for real.

This makes the fallback that pushes rq->curr unreachable when the
pushable head is migrate-disabled. Nothing is lost, because that path
was always stopped by the re-check described above. In the capture it
ran 5204 times and moved nothing.

Fixes: a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing")
Signed-off-by: Seiji Nishikawa <snishika@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260830073746.2189355-1-snishika@redhat.com
sched/core: Fix inter-class wakeup_preempt() 2026-06-30T08:56:51+00:00 Peter Zijlstra peterz@infradead.org 2026-06-26T07:36:52+00:00 fa02b2868420d9f33d64ddcb15ef0f96880b2a6d The way wakeup_preempt() works since commit 704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()") is that it will call rq->next_class->wakeup_preempt(rq, p) when p is of an equal or higher class, and raise ->next_class when higher. This means that: running idle task wakeup fair-A (next_class == idle) if (sched_class_above(fair, idle)) { wakeup_preempt_idle(fair-A); resched_curr(rq); next_class = fair; } wakeup fair-B (next_class == fair) if (fair == fair) wakeup_preempt_fair(fair-B); (but current is idle) All wakeup_preempt_$class() methods, except for wakeup_preempt_scx() (for whoem this was build) ignore cross-class wakeups by testing if @p is of the right class, but per the above case, it also should check current. This is mostly harmless in the current form, but will lead to trouble with later patches. Fixes: 704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()") Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260626074605.GB2568396%40noisy.programming.kicks-ass.net 
The way wakeup_preempt() works since commit 704069649b5b ("sched/core: Rework
sched_class::wakeup_preempt() and rq_modified_*()") is that it will call
rq->next_class->wakeup_preempt(rq, p) when p is of an equal or higher class,
and raise ->next_class when higher.

This means that:

	running idle task

	wakeup fair-A
	  (next_class == idle)
	  if (sched_class_above(fair, idle)) {
	     wakeup_preempt_idle(fair-A);
	     resched_curr(rq);
	     next_class = fair;
	  }

	wakeup fair-B
	  (next_class == fair)
	  if (fair == fair)
	    wakeup_preempt_fair(fair-B);
	      (but current is idle)

All wakeup_preempt_$class() methods, except for wakeup_preempt_scx() (for whoem
this was build) ignore cross-class wakeups by testing if @p is of the right
class, but per the above case, it also should check current.

This is mostly harmless in the current form, but will lead to trouble with
later patches.

Fixes: 704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260626074605.GB2568396%40noisy.programming.kicks-ass.net
sched: Rework prev_balance() to avoid stale prev references 2026-06-02T10:26:06+00:00 John Stultz jstultz@google.com 2026-05-12T02:56:11+00:00 7a3a6bfbd62a2ba3e0ef1e92d6b71abb66890825 Historically, the prev value from __schedule() was the rq->curr. This prev value is passed down through numerous functions, and used in the class scheduler implementations. The fact that prev was on_cpu until the end of __schedule(), meant it was stable across the rq lock drops that the class->balance() implementations often do. However, with proxy-exec, the prev passed to functions called by __schedule() is rq->donor, which may not be the same as rq->curr and may not be on_cpu, this makes the prev value potentially unstable across rq lock drops. A recently found issue with proxy-exec, is when we begin doing return migration from try_to_wake_up(), its possible we may be waking up the rq->donor. When we do this, we proxy_resched_idle() to put_prev_set_next() setting the rq->donor to rq->idle, allowing the rq->donor to be return migrated and allowed to run. This however runs into trouble, as on another cpu we might be in the middle of calling __schedule(). Conceptually the rq lock is held for the majority of the time, but in calling prev_balance() its possible the class->balance() handler call may briefly drop the rq lock. This opens a window for try_to_wake_up() to wake and return migrate the rq->donor before the class logic reacquires the rq lock. Unfortunately prev_balance() pass in a prev argument, to which we pass rq->donor. However this prev value can now become stale and incorrect across a rq lock drop. So, to correct this, rework the prev_balance() call so that it does not take a "prev" argument. Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260512025635.2840817-2-jstultz@google.com 
Historically, the prev value from __schedule() was the rq->curr.
This prev value is passed down through numerous functions, and
used in the class scheduler implementations. The fact that
prev was on_cpu until the end of __schedule(), meant it was
stable across the rq lock drops that the class->balance()
implementations often do.

However, with proxy-exec, the prev passed to functions called
by __schedule() is rq->donor, which may not be the same as
rq->curr and may not be on_cpu, this makes the prev value
potentially unstable across rq lock drops.

A recently found issue with proxy-exec, is when we begin doing
return migration from try_to_wake_up(), its possible we may be
waking up the rq->donor.  When we do this, we proxy_resched_idle()
to put_prev_set_next() setting the rq->donor to rq->idle, allowing
the rq->donor to be return migrated and allowed to run.

This however runs into trouble, as on another cpu we might be in
the middle of calling __schedule(). Conceptually the rq lock is
held for the majority of the time, but in calling prev_balance()
its possible the class->balance() handler call may briefly drop the rq lock.
This opens a window for try_to_wake_up() to wake and return migrate the
rq->donor before the class logic reacquires the rq lock.

Unfortunately prev_balance() pass in a prev argument, to which we pass
rq->donor. However this prev value can now become stale and incorrect across a
rq lock drop.

So, to correct this, rework the prev_balance() call so that it does not take a
"prev" argument.

Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260512025635.2840817-2-jstultz@google.com
sched/rt: Update default bandwidth for real-time tasks to ONE 2026-05-19T10:17:35+00:00 Yuri Andriaccio yurand2000@gmail.com 2026-04-30T21:38:25+00:00 c2e390197ad1360db6686a8c89abaafaf83adf72 Set the default total bandwidth for SCHED_DEADLINE tasks and servers to ONE. FIFO/RR tasks are already throttled by fair-servers and ext-servers, and the sysctl_sched_rt_runtime parameter now only defines the total bw that is allowed to deadline entities. Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260430213835.62217-22-yurand2000@gmail.com 
Set the default total bandwidth for SCHED_DEADLINE tasks and servers
to ONE.  FIFO/RR tasks are already throttled by fair-servers and
ext-servers, and the sysctl_sched_rt_runtime parameter now only
defines the total bw that is allowed to deadline entities.

Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260430213835.62217-22-yurand2000@gmail.com
sched/rt: Cleanup global RT bandwidth functions 2026-04-08T11:11:44+00:00 Michal Koutný mkoutny@suse.com 2026-03-23T12:39:39+00:00 985215804dcbf02ab675977e770708e3f084e9fc The commit 5f6bd380c7bdb ("sched/rt: Remove default bandwidth control") and followup changes made a few of the functions unnecessary, drop them for simplicity. Signed-off-by: Michal Koutný <mkoutny@suse.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-3-1e7d5ed6b249@suse.com 
The commit 5f6bd380c7bdb ("sched/rt: Remove default bandwidth control")
and followup changes made a few of the functions unnecessary, drop them
for simplicity.

Signed-off-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-3-1e7d5ed6b249@suse.com
sched/rt: Move group schedulability check to sched_rt_global_validate() 2026-04-08T11:11:44+00:00 Michal Koutný mkoutny@suse.com 2026-03-23T12:39:38+00:00 4f70a0456d090303d5a6c915dd7d9db9da56cb16 The sched_rt_global_constraints() function is a remnant that used to set up global RT throttling but that is no more since commit 5f6bd380c7bdb ("sched/rt: Remove default bandwidth control") and the function ended up only doing schedulability check. Move the check into the validation function where it fits better. (The order of validations sched_dl_global_validate() and sched_rt_global_validate() shouldn't matter.) Signed-off-by: Michal Koutný <mkoutny@suse.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-2-1e7d5ed6b249@suse.com 
The sched_rt_global_constraints() function is a remnant that used to set
up global RT throttling but that is no more since commit 5f6bd380c7bdb
("sched/rt: Remove default bandwidth control") and the function ended up
only doing schedulability check.
Move the check into the validation function where it fits better.
(The order of validations sched_dl_global_validate() and
sched_rt_global_validate() shouldn't matter.)

Signed-off-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-2-1e7d5ed6b249@suse.com
sched/rt: Skip group schedulable check with rt_group_sched=0 2026-04-08T11:11:44+00:00 Michal Koutný mkoutny@suse.com 2026-03-23T12:39:37+00:00 8b016dcec9365675be81d26be88f2c09cf983bd4 The warning from the commit 87f1fb77d87a6 ("sched: Add RT_GROUP WARN checks for non-root task_groups") is wrong -- it assumes that only task_groups with rt_rq are traversed, however, the schedulability check would iterate all task_groups even when rt_group_sched=0 is disabled at boot time but some non-root task_groups exist. The schedulability check is supposed to validate: a) that children don't overcommit its parent, b) no RT task group overcommits global RT limit. but with rt_group_sched=0 there is no (non-trivial) hierarchy of RT groups, therefore skip the validation altogether. Otherwise, writes to the global sched_rt_runtime_us knob will be rejected with incorrect validation error. This fix is immaterial with CONFIG_RT_GROUP_SCHED=n. Fixes: 87f1fb77d87a6 ("sched: Add RT_GROUP WARN checks for non-root task_groups") Signed-off-by: Michal Koutný <mkoutny@suse.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-1-1e7d5ed6b249@suse.com 
The warning from the commit 87f1fb77d87a6 ("sched: Add RT_GROUP WARN
checks for non-root task_groups") is wrong -- it assumes that only
task_groups with rt_rq are traversed, however, the schedulability check
would iterate all task_groups even when rt_group_sched=0 is disabled at
boot time but some non-root task_groups exist.

The schedulability check is supposed to validate:
  a) that children don't overcommit its parent,
  b) no RT task group overcommits global RT limit.
but with rt_group_sched=0 there is no (non-trivial) hierarchy of RT groups,
therefore skip the validation altogether. Otherwise, writes to the
global sched_rt_runtime_us knob will be rejected with incorrect
validation error.

This fix is immaterial with CONFIG_RT_GROUP_SCHED=n.

Fixes: 87f1fb77d87a6 ("sched: Add RT_GROUP WARN checks for non-root task_groups")
Signed-off-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260323-sched-rert_groups-v3-1-1e7d5ed6b249@suse.com
sched: Use u64 for bandwidth ratio calculations 2026-04-07T07:23:52+00:00 Joseph Salisbury joseph.salisbury@oracle.com 2026-04-03T21:00:14+00:00 c6e80201e057dfb7253385e60bf541121bf5dc33 to_ratio() computes BW_SHIFT-scaled bandwidth ratios from u64 period and runtime values, but it returns unsigned long. tg_rt_schedulable() also stores the current group limit and the accumulated child sum in unsigned long. On 32-bit builds, large bandwidth ratios can be truncated and the RT group sum can wrap when enough siblings are present. That can let an overcommitted RT hierarchy pass the schedulability check, and it also narrows the helper result for other callers. Return u64 from to_ratio() and use u64 for the RT group totals so bandwidth ratios are preserved and compared at full width on both 32-bit and 64-bit builds. Fixes: b40b2e8eb521 ("sched: rt: multi level group constraints") Assisted-by: Codex:GPT-5 Signed-off-by: Joseph Salisbury <joseph.salisbury@oracle.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260403210014.2713404-1-joseph.salisbury@oracle.com 
to_ratio() computes BW_SHIFT-scaled bandwidth ratios from u64 period and
runtime values, but it returns unsigned long.  tg_rt_schedulable() also
stores the current group limit and the accumulated child sum in unsigned
long.

On 32-bit builds, large bandwidth ratios can be truncated and the RT
group sum can wrap when enough siblings are present.  That can let an
overcommitted RT hierarchy pass the schedulability check, and it also
narrows the helper result for other callers.

Return u64 from to_ratio() and use u64 for the RT group totals so
bandwidth ratios are preserved and compared at full width on both 32-bit
and 64-bit builds.

Fixes: b40b2e8eb521 ("sched: rt: multi level group constraints")
Assisted-by: Codex:GPT-5
Signed-off-by: Joseph Salisbury <joseph.salisbury@oracle.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260403210014.2713404-1-joseph.salisbury@oracle.com
sched: Make class_schedulers avoid pushing current, and get rid of proxy_tag_curr() 2026-04-03T12:23:38+00:00 John Stultz jstultz@google.com 2026-03-24T19:13:16+00:00 e0ca8991b2de6c9dfe6fcd8a0364951b2bd56797 With proxy-execution, the scheduler selects the donor, but for blocked donors, we end up running the lock owner. This caused some complexity, because the class schedulers make sure to remove the task they pick from their pushable task lists, which prevents the donor from being migrated, but there wasn't then anything to prevent rq->curr from being migrated if rq->curr != rq->donor. This was sort of hacked around by calling proxy_tag_curr() on the rq->curr task if we were running something other then the donor. proxy_tag_curr() did a dequeue/enqueue pair on the rq->curr task, allowing the class schedulers to remove it from their pushable list. The dequeue/enqueue pair was wasteful, and additonally K Prateek highlighted that we didn't properly undo things when we stopped proxying, leaving the lock owner off the pushable list. After some alternative approaches were considered, Peter suggested just having the RT/DL classes just avoid migrating when task_on_cpu(). So rework pick_next_pushable_dl_task() and the rt pick_next_pushable_task() functions so that they skip over the first pushable task if it is on_cpu. Then just drop all of the proxy_tag_curr() logic. Fixes: be39617e38e0 ("sched: Fix proxy/current (push,pull)ability") Closes: https://lore.kernel.org/lkml/e735cae0-2cc9-4bae-b761-fcb082ed3e94@amd.com/ Reported-by: K Prateek Nayak <kprateek.nayak@amd.com> Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260324191337.1841376-2-jstultz@google.com 
With proxy-execution, the scheduler selects the donor, but for
blocked donors, we end up running the lock owner.

This caused some complexity, because the class schedulers make
sure to remove the task they pick from their pushable task
lists, which prevents the donor from being migrated, but there
wasn't then anything to prevent rq->curr from being migrated
if rq->curr != rq->donor.

This was sort of hacked around by calling proxy_tag_curr() on
the rq->curr task if we were running something other then the
donor. proxy_tag_curr() did a dequeue/enqueue pair on the
rq->curr task, allowing the class schedulers to remove it from
their pushable list.

The dequeue/enqueue pair was wasteful, and additonally K Prateek
highlighted that we didn't properly undo things when we stopped
proxying, leaving the lock owner off the pushable list.

After some alternative approaches were considered, Peter
suggested just having the RT/DL classes just avoid migrating
when task_on_cpu().

So rework pick_next_pushable_dl_task() and the rt
pick_next_pushable_task() functions so that they skip over the
first pushable task if it is on_cpu.

Then just drop all of the proxy_tag_curr() logic.

Fixes: be39617e38e0 ("sched: Fix proxy/current (push,pull)ability")
Closes: https://lore.kernel.org/lkml/e735cae0-2cc9-4bae-b761-fcb082ed3e94@amd.com/
Reported-by: K Prateek Nayak <kprateek.nayak@amd.com>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260324191337.1841376-2-jstultz@google.com
sched: Fix incorrect schedstats for rt and dl thread 2026-02-23T17:04:11+00:00 Dengjun Su dengjun.su@mediatek.com 2026-02-04T11:59:29+00:00 c0e1832ba6dad7057acf3f485a87e0adccc23141 For RT and DL thread, only 'set_next_task_(rt/dl)' will call 'update_stats_wait_end_(rt/dl)' to update schedstats information. However, during the migration process, 'update_stats_wait_start_(rt/dl)' will be called twice, which will cause the values of wait_max and wait_sum to be incorrect. The specific output as follows: $ cat /proc/6046/task/6046/sched | grep wait wait_start : 0.000000 wait_max : 496717.080029 wait_sum : 7921540.776553 A complete schedstats information update flow of migrate should be __update_stats_wait_start() [enter queue A, stage 1] -> __update_stats_wait_end() [leave queue A, stage 2] -> __update_stats_wait_start() [enter queue B, stage 3] -> __update_stats_wait_end() [start running on queue B, stage 4] Stage 1: prev_wait_start is 0, and in the end, wait_start records the time of entering the queue. Stage 2: task_on_rq_migrating(p) is true, and wait_start is updated to the waiting time on queue A. Stage 3: prev_wait_start is the waiting time on queue A, wait_start is the time of entering queue B, and wait_start is expected to be greater than prev_wait_start. Under this condition, wait_start is updated to (the moment of entering queue B) - (the waiting time on queue A). Stage 4: the final wait time = (time when starting to run on queue B) - (time of entering queue B) + (waiting time on queue A) = waiting time on queue B + waiting time on queue A. The current problem is that stage 2 does not call __update_stats_wait_end to update wait_start, which causes the final computed wait time = waiting time on queue B + the moment of entering queue A, leading to incorrect wait_max and wait_sum. Add 'update_stats_wait_end_(rt/dl)' in 'update_stats_dequeue_(rt/dl)' to update schedstats information when dequeue_task. Signed-off-by: Dengjun Su <dengjun.su@mediatek.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260204115959.3183567-1-dengjun.su@mediatek.com 
For RT and DL thread, only 'set_next_task_(rt/dl)' will call
'update_stats_wait_end_(rt/dl)' to update schedstats information.
However, during the migration process,
'update_stats_wait_start_(rt/dl)' will be called twice, which
will cause the values of wait_max and wait_sum to be incorrect.
The specific output as follows:
$ cat /proc/6046/task/6046/sched | grep wait
wait_start                                   :             0.000000
wait_max                                     :        496717.080029
wait_sum                                     :       7921540.776553

A complete schedstats information update flow of migrate should be
__update_stats_wait_start() [enter queue A, stage 1] ->
__update_stats_wait_end()   [leave queue A, stage 2] ->
__update_stats_wait_start() [enter queue B, stage 3] ->
__update_stats_wait_end()   [start running on queue B, stage 4]

    Stage 1: prev_wait_start is 0, and in the end, wait_start records the
    time of entering the queue.
    Stage 2: task_on_rq_migrating(p) is true, and wait_start is updated to
    the waiting time on queue A.
    Stage 3: prev_wait_start is the waiting time on queue A, wait_start is
    the time of entering queue B, and wait_start is expected to be greater
    than prev_wait_start. Under this condition, wait_start is updated to
    (the moment of entering queue B) - (the waiting time on queue A).
    Stage 4: the final wait time = (time when starting to run on queue B)
    - (time of entering queue B) + (waiting time on queue A) = waiting
    time on queue B + waiting time on queue A.

The current problem is that stage 2 does not call __update_stats_wait_end
to update wait_start, which causes the final computed wait time = waiting
time on queue B + the moment of entering queue A, leading to incorrect
wait_max and wait_sum.

Add 'update_stats_wait_end_(rt/dl)' in 'update_stats_dequeue_(rt/dl)' to
update schedstats information when dequeue_task.

Signed-off-by: Dengjun Su <dengjun.su@mediatek.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260204115959.3183567-1-dengjun.su@mediatek.com