linux-stable.git/include/linux/sched/ext.h, branch v7.3-rc2

sched_ext: Use runnable_at for the default core-sched task ordering

2026-08-16T02:07:49+00:00

The default core-sched ordering runs the longest waiting task first by
comparing p->scx.core_sched_at stamps. The stamp is maintained under two
rules. touch_core_sched() stamps when a task starts waiting for a CPU and
when its slice runs out. If the scheduler implements
ops.core_sched_before(), touch_core_sched_dispatch() re-stamps on every
dispatch.

A comparison can see one stamp taken under each rule, which isn't a
meaningful ordering. The dispatch rule also buys little - it only aligns
bypass-mode comparisons with the local DSQ order. Multiple schedulers make
the mixed comparisons more common.

Wait time is what p->scx.runnable_at already tracks for the stall watchdog.
Delete core_sched_at with both touch functions and compare runnable_at in
the scx_prio_less() fallback.

runnable_at is refreshed only on enqueue and goes stale while a task keeps
occupying its CPU. Instead of re-stamping, order a running task after every
waiting task as it is the most recently serviced.

Signed-off-by: Tejun Heo

sched_ext: Eject the top rescue consumer on overload

2026-08-03T21:01:36+00:00

When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The
watchdog blames the waiting task's owner, but the misbehaving party is
whoever floods the queue, not whoever happens to time out.

Track each sched's recent rescue consumption per cpu as a decaying average.
Once the oldest waiter on a cpu's rescue DSQ has been queued past a
threshold derived from the rescue knobs (4s at the defaults), the rescue
timer ejects the sub with the highest recent consumption on that cpu with
SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and
nothing is ejected - the generic stall watchdog eventually blames the
waiter's owner instead. Ejections on a cpu are spaced one threshold apart so
the freed bandwidth can drain the backlog before another sub is judged.

The overload check only wins the race against the stall watchdog when the
watchdog timeout clears the threshold, and a single in-budget wait must not
cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit
and on knobs whose funding period exceeds half the threshold.

v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
      wraps 2^31 ticks after the last ejection and suppresses ejections.
      (sashiko AI)

    - Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay
      delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage
      average in the victim pick.

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Add bandwidth-limited rescue execution for stranded tasks

2026-08-03T21:01:29+00:00

A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
bounced back through ops.enqueue() so the scheduler can re-decide. That
recovery assumes the scheduler has somewhere legal to send the task. When it
doesn't, e.g. when the task's affinity is restricted to cids delegated away,
the task starves until the stall watchdog ejects the scheduler. An exiting
task is worse - it skips ops.enqueue() and the rejection becomes a
self-requeuing cycle that burns the CPU until the watchdog fires.

Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the
insert would be rejected for missing caps, the kernel takes over and runs
the task on the target CPU without consulting the owning scheduler. The
kernel sets the flag itself when enqueueing an exiting task.

Rescue is a last-resort forward-progress backstop with a persistent
disadvantage, not a way around cap enforcement. A per-CPU token bucket
accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at
a time in arrival order. Each is granted a slice of the rescue_quantum_us
(default 5ms) quantum divided across the waiters, waits at the tail of the
local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival
once the slice is served.

The schedulers keep their normal control over an admitted rescuee and may
preempt or reslice it. Service is measured on CPU time actually received, so
neither shortens the rescue. Prolonged denial escalates - the remaining
slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee
preempts the current task. Escalation is paced by the same bucket, and
delivered service converges on the configured bandwidth no matter how
aggressively the schedulers dispatch.

Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making
SCX_ENQ_RESCUE inserts reject as usual.

v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form
      schedulers setting them still load on older kernels. (Andrea)

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Add SCX_TASK_PROTECTED

2026-08-03T21:01:20+00:00

A BPF scheduler can displace any of its tasks at will - cut a running one's
slice with an SCX_ENQ_PREEMPT dispatch, an SCX_KICK_PREEMPT kick or a direct
shortening, and jump a queued one with HEAD insertions. Sometimes the kernel
needs a slice and a DSQ position to stick regardless.

Add SCX_TASK_PROTECTED, guarding both:

- The slice becomes immutable. Every scheduler-reachable write is refused
  and counted as SCX_EV_SLICE_DENIED. Higher scheduling classes are
  unaffected. PREEMPT|IMMED can't preempt a running protected task and gets
  reenqueued.

- A protected task that reached the head of its DSQ keeps it - HEAD
  insertions land behind the leading run of protected tasks and reenqueue
  sweeps skip them. Only rq-owned DSQs can hold protected tasks, so the walk
  runs only for them.

The bit lives in p->scx.flags so that both the refusal and the head walk
read it under the rq lock that protects it.

Protection ends when the slice is consumed, when the task leaves the rq
except for a save/restore on the running task, on a yield, when the
scheduler enters bypass, and when the task leaves scx. The flag is
kernel-internal and not used yet.

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Synchronize slice and dsq_vtime writes

2026-08-03T21:01:14+00:00

p->scx.slice and p->scx.dsq_vtime writes have no synchronization rules. The
dsq insert kfuncs write both fields synchronously from whatever context
they're called in - a direct dispatch from ops.select_cpu() writes with only
pi_lock held - and, as the kfuncs are safe to call spuriously with the
invalid dispatch discarded later, a scheduler can modify any task's slice by
spuriously calling them. The latter stands in the way of an upcoming patch
which adds kernel-granted slices that the schedulers must not be able to
modify.

Give both fields explicit rules. While the task is running, sleeping or
queued on an rq-owned DSQ, the rq lock protects them - these are the states
where the kernel consumes the slice. While queued on a user DSQ or on the
BPF side, the kernel neither consumes nor decides on the fields and every
writer acts for the BPF scheduler - synchronizing the writers is the
scheduler's responsibility and whichever write lands last wins.

To conform, an insert kfunc no longer writes the fields when called. The
values travel with the dispatch and take effect when the task is inserted. A
discarded dispatch has no side effects. The rq lock rule is asserted at the
slice store.

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Bound per-task reenqueues and eject the owning scheduler

2026-07-26T21:12:19+00:00

Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.

Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.

Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.

The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.

v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
    reenqueue, not on every reenqueue.

v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
      carry over to the next owner across sched class switches, scheduler
      replacement or sub-scheduler rehoming (Andrea Righi).

    - Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
      (Andrea Righi).

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Hand over cgroups at sub-scheduler enable/disable

2026-07-20T07:10:57+00:00

Sub-schedulers don't get cgroups yet: every task_group is inited on the root
sched and the routing added by the previous patches always resolves to it.
Add the handover: an enabling sub-scheduler takes over the cgroups in its
subtree and a disabling one returns them to its parent.

scx_cgroup_claim_subtree() runs while the sub enables, after the subtree's
cgrp->scx_sched's are set and before any task is claimed. It inits each
subtree task_group on the sub, exits it from the parent and updates
tg->scx.sched. A failed ops.cgroup_init() unwinds the sub-side inits and
aborts the enable with the parent untouched.

Disabling reverses it with scx_cgroup_return_subtree(): exit each cgroup
from the sub, then re-init it on the parent with the current tg->scx.*
values, resyncing weight and bandwidth changes made while the sub had it.
When a re-init fails, the parent is failed and the remaining task_groups
still transfer uninited and get no cgroup ops - the same punting done for
tasks. The dying parent's own disable moves them onward.

The handover walks include dying but not yet offlined task_groups, the same
as root's bulk walks: a removed cgroup keeps hosting scheduling events until
its dying tasks finish their final context switches, and its
ops.cgroup_exit() must follow the last of them. tg on/offlining is excluded
through cgroup_lock(), so either ordering against an rmdir of a subtree
cgroup delivers balanced init/exit pairs.

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

sched_ext: Deliver cgroup ops to each task_group's sched

2026-07-20T07:10:56+00:00

With sub-schedulers claiming cgroup subtrees, cgroup ops must be delivered
to each task_group's sched rather than always to root. Add tg->scx.sched to
track which sched initialized the task_group. It is set and cleared together
with SCX_TG_INITED.

Deliver the ops accordingly:

- ops.cgroup_exit() goes to the sched whose ops.cgroup_init() it pairs with.

- ops.cgroup_prep_move/move/cancel_move() go to the task's sched, and only
  for moves that don't re-home the task. A re-homing move is reported
  through the ops.exit_task/init_task() pair instead. The cgroups passed to
  the move ops can be outside the sched's inited set as the cpu controller
  can be coarser than the sub-scheduler topology.

- Knobs of a cgroup belong to the parent, so ops.set_weight/idle/bandwidth()
  go to the parent task_group's sched.

All task_groups currently resolve to the root sched, so no behavior changes
until sub-schedulers start claiming cgroups.

While at it, scx_cgroup_init() is restructured so both paths share the
recording.

Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi

Merge branch 'for-7.2-fixes' into for-7.3

2026-07-18T07:50:21+00:00

Pull to receive:

 477869bfafea ("sched_ext: Reject setting disallow from init_task outside the enable path")
 5f8b69642d18 ("sched_ext: Take cgroup_lock() first in scx_cgroup_lock()")
 8c13364db9c9 ("sched_ext: Skip sub-disable teardown for never-linked sub-schedulers")
 5cdc92859809 ("sched_ext: Don't enable non-ext tasks in the sub-sched task loops")

as dependencies for the upcoming cgroup migration patchset and to
resolve the conflicts with the ext.c/sub.c split on for-7.3.

5f8b69642d18 comments scx_cgroup_lock() which for-7.3 exported for
sub.c. Resolved by keeping the exported version with the comment.

8c13364db9c9 and 5cdc92859809 patch the pre-split sub-sched enable and
disable paths in ext.c which for-7.3 moved to sub.c. Resolved by
applying the never-linked teardown skip and the class gates to sub.c.

Signed-off-by: Tejun Heo

sched_ext: Reject setting disallow from init_task outside the enable path

2026-07-18T07:28:57+00:00

The p->scx.disallow revert assumes the root enable path, where the switching
loop reads the reverted policy right afterwards and leaves the task off SCX.
The sub-scheduler disable path also reaches it when re-initializing the
returned tasks on a root parent. Nothing reads the policy there: the task is
enabled on root anyway and keeps running on the ext class with a silently
rewritten policy.

Kill the sched instead, matching the fork and non-root branches, and update
the disallow documentation, which equated !fork with the load path and
pointed at a stale debugfs path for nr_rejected.

Fixes: 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
Signed-off-by: Tejun Heo 
Reviewed-by: Andrea Righi