wq_worker_tick() accounts CPU time and detects CPU-intensive work for
the kworker that is actually running. With proxy execution, rq->donor
is the scheduling context while rq->curr is the execution context.

Calling the hook with rq->donor can skip workqueue accounting when a
kworker is executing on behalf of a donor task. It can also account a
blocked kworker when the donor is a worker but rq->curr is the task
actually executing. The former can delay WORKER_CPU_INTENSIVE handling
and pool concurrency management, which can delay pending kernel work
and userspace operations depending on it.

Use rq->curr for the workqueue tick hook while retaining rq->donor for
scheduler accounting.

Fixes: af0c8b2bf67b ("sched: Split scheduler and execution contexts")
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260902150208.1209922-2-sh_def@163.com

Proxy execution separates the scheduling context from the execution
context. Commit aa4f74dfd42b ("sched: Fix runtime accounting w/ split
exec & sched contexts") made per-task and thread-group runtime
accounting follow the task that actually executes, while cgroup CPU
usage is charged to the donor.

When the donor and execution task belong to different cgroups, this
makes a task's execution time count against a different cgroup from the
one the task belongs to.

Cgroup CPU usage should follow the execution context, matching the
per-task, thread-group, and cgroup user/system accounting. Keep
scheduling state associated with the donor, but charge cgroup CPU
usage to rq->curr.

A reproducer with the donor and execution task in separate cgroups
showed the execution task accumulating runtime while cgroup CPU usage
was charged to the donor's cgroup. With this change, the execution
task's cgroup accumulates the CPU usage instead. The same behavior was
verified with an RT donor and with legacy cpuacct accounting.

Fixes: aa4f74dfd42b ("sched: Fix runtime accounting w/ split exec & sched contexts")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260904034707.268416-1-sh_def@163.com

The eevdf rb tree maintains 3 augmented fields but only one is currently
copied when balancing the tree.

Add a more generic define that can be used when there are several augmented
fields. In this case, we provide a function that takes care of copying all
fields.

Fixes: aef6987d8954 ("sched/eevdf: Propagate min_slice up the cgroup hierarchy")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260909150522.858312-1-vincent.guittot@linaro.org

Similarly to se->min_slice, init se->max_slice with se->slice before
enqueueing the entity so the augmented callback computes it correctly
at parent level.

Fixes: 6e3c0a4e1ad1 ("sched/fair: Fix lag clamp")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260907123855.1297976-1-vincent.guittot@linaro.org

Pull scheduler fixes from Ingo Molnar:

 - Fix a timestamping bug in pick_task_fair() and yield_task_fair()
   (Zhan Xusheng)

 - Skip migrate-disabled tasks when picking a push candidate in the
   RT and DL schedulers (Seiji Nishikawa)

 - Skip rq->avg_idle update without a valid idle_stamp (Shubhang
   Kaushik)

 - Fix throttling bug in throttle_cfs_rq(), caused by the recent
   single-runqueue conversion (Wanwu Li)

 - Fix bandwidth calculation bug in distribute_cfs_runtime(),
   caused by the single-runqueue conversion (Wanwu Li)

 - Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello)

 - Avoid creating misfits during cache-aware load-balancing on hybrid
   systems (Tim Chen)

* tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  sched/fair: Avoid creating misfits during cache-aware balancing
  x86/itmt: Don't make ITMT enablement depend on debugfs
  sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime()
  sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq()
  sched/core: Skip rq->avg_idle update without a valid idle_stamp
  sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate
  sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers

Pull locking fixes from Ingo Molnar:

 - Fix a softirq processing delay bug in local_interrupt_disable(),
   which should mostly only affect the Rust runtime (Boqun Feng)

 - Remove the hardirq_disable_count() function which caused the
   previous bug and is now unused & unnecessary (Boqun Feng)

 - lockdep: Invalidate stale class_cache entries for zapped classes
   (Eric Dumazet)

 - Fix rt_mutex specific futex scheduling helpers
   (Sebastian Andrzej Siewior)

 - Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)

* tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  futex: Prevent rcuwait use-after-free during requeue PI
  futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
  locking/lockdep: Invalidate stale class_cache entries for zapped classes
  preempt: Remove hardirq_disable_count()
  interrupt: Disable interrupt before modifying hardirq_disable counter

This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci

This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.

Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.

Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>

There is rt_mutex_{pre|post}_schedule() around
rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/
sched_update_worker() is invoked before we schedule out and block on
rt_mutex while waiting for it become available.

The reason is that blocking on rt_mutex assigns a pi_waiter for the PI
chain and sched_submit_work() will also assign a pi_waiter if it blocks
on lock but a this point we already have a waiter assigned.
We can't skip sched_submit_work() entirely because I/O relies on the
fact that I/O queue is flushed while it blocks on a sleeping lock.
Therefore sched_submit_work() is moved before we block on the lock.

Sleeping lock in this context means mutex or rw_semaphore not spinlock_t
on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same
abstraction as the futex proxy lock, the futex code ended up using
rt_mutex_{pre|post}_schedule(), too.
Using it is/ was just to keep the task_struct::sched_rt_mutex assertion
happy. Futex proxy lock is used only in the syscall context of a task.
At this point it never got any I/O that needs to be flushed and it can't
be a workqueue that needs to notify that it will be scheduled out.
Therefore sched_submit_work() does nothing here.

By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not
get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed
because in this callchain the lock is (usually) not contended and so
rt_mutex_slowlock_block() does not schedule, triggering the assert.

Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ)
because at this point there is a pi_waiter recorded and invoking
sched_submit_work() with a possible lock contention would be wrong.

Add rt_mutex_futex_{pre|post}_schedule() which toggles the
sched_rt_mutex assert and does not involve sched_submit_work(). Add
asserts here to ensure that sched_submit_work() would do nothing. Use it
only in futex proxy lock case which is rt_mutex_wait_proxy_lock().
Remove it from futex_lock_pi().

Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers")
Reported-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de
Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com

Cache-aware load balancing biases tasks toward their preferred LLC. On
asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may
contain CPUs that are too small to run the task. Pulling the task there
turns it into a misfit, trading a cache-locality gain for a capacity loss
that's more detrimental to performance.

Guard both cache-aware migration entry points against this:

 - can_migrate_llc_task(): forbid the LLC migration when the task fits its
   source CPU but would not fit the destination CPU.
 - alb_break_llc(): veto the active balance under the same condition so the
   runnable task is not pushed onto a CPU that cannot accommodate it.

Both checks are gated with checks for hybrid processors, so symmetric
systems are unaffected. Tasks that already do not fit their source CPU
are left to the existing LLC policy, since the move cannot make their
fitness worse (this also preserves misfit up-migration to bigger CPUs).

Additionally, if there are misfit tasks found in the load balancing
classification phase, prioritize misfit task migrations
over LLC load aggregation on asymmetric systems. A better fitting
CPU will boost performance more than better cache locality.

Reviewed-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Tested-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com

distribute_cfs_runtime() refreshes the rq clock and accounts elapsed
runtime with update_curr() before redistributing bandwidth, but gates
this on cfs_rq->curr. Since commit 85570f10a4c6 ("sched/eevdf: Move to
a single runqueue") cfs_rq->curr is only maintained on the root
cfs_rq, so for the cgroup cfs_rqs it walks, the check never fires and
the refresh is dead code.

Use cfs_rq->h_curr, the per-level current entity, restoring the
intended behaviour: only refresh when something is actually running at
the throttled level, i.e. within the deferred throttle window.
Without this, runtime consumed by a still-running task of the
throttled hierarchy is not docked before redistribution;
unthrottle_cfs_rq() catches up unconditionally since
commit 28ad5427682b ("sched/fair: Call update_curr() before
unthrottling the hierarchy"), so this is not a correctness hole today,
but the refresh the check was written for is gone.

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Aaron Lu <ziqianlu@bytedance.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260831101141.391382-3-liwanwu@kylinos.cn