Cache-aware load balancing biases tasks toward their preferred LLC. On
asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may
contain CPUs that are too small to run the task. Pulling the task there
turns it into a misfit, trading a cache-locality gain for a capacity loss
that's more detrimental to performance.

Guard both cache-aware migration entry points against this:

 - can_migrate_llc_task(): forbid the LLC migration when the task fits its
   source CPU but would not fit the destination CPU.
 - alb_break_llc(): veto the active balance under the same condition so the
   runnable task is not pushed onto a CPU that cannot accommodate it.

Both checks are gated with checks for hybrid processors, so symmetric
systems are unaffected. Tasks that already do not fit their source CPU
are left to the existing LLC policy, since the move cannot make their
fitness worse (this also preserves misfit up-migration to bigger CPUs).

Additionally, if there are misfit tasks found in the load balancing
classification phase, prioritize misfit task migrations
over LLC load aggregation on asymmetric systems. A better fitting
CPU will boost performance more than better cache locality.

Reviewed-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Tested-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com

distribute_cfs_runtime() refreshes the rq clock and accounts elapsed
runtime with update_curr() before redistributing bandwidth, but gates
this on cfs_rq->curr. Since commit 85570f10a4c6 ("sched/eevdf: Move to
a single runqueue") cfs_rq->curr is only maintained on the root
cfs_rq, so for the cgroup cfs_rqs it walks, the check never fires and
the refresh is dead code.

Use cfs_rq->h_curr, the per-level current entity, restoring the
intended behaviour: only refresh when something is actually running at
the throttled level, i.e. within the deferred throttle window.
Without this, runtime consumed by a still-running task of the
throttled hierarchy is not docked before redistribution;
unthrottle_cfs_rq() catches up unconditionally since
commit 28ad5427682b ("sched/fair: Call update_curr() before
unthrottling the hierarchy"), so this is not a correctness hole today,
but the refresh the check was written for is gone.

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Aaron Lu <ziqianlu@bytedance.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260831101141.391382-3-liwanwu@kylinos.cn

After commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue"),
cfs_rq->curr is only maintained on the root cfs_rq (set/cleared from
set_next_task_fair()/put_prev_task_fair()), while cfs_rq->h_curr is
the per-level current entity, set by set_next_entity() at every level
of the hierarchy. For an intermediate cfs_rq (a cgroup), cfs_rq->curr
is always NULL, but cfs_rq->h_curr is the group entity at that level.

throttle_cfs_rq() reads cfs_rq->curr to decide whether there is a
running entity at the throttled level, in which case it should request
a full sched_cfs_bandwidth_slice() of runtime and arm the deferred
throttle task_work via task_throttle_setup_work(). For intermediate
cfs_rqs the check is always false, so bandwidth-controlled cgroups
always get just 1ns of runtime and never arm the deferred throttle
work; the running task then escapes throttling until the next pick
arms the work instead, even though there is an on-rq entity at this
level.

Switch the read to cfs_rq->h_curr so intermediate bandwidth-controlled
cgroups behave consistently with the root cfs_rq, matching the
existing usage of cfs_rq->h_curr in update_curr() and
check_enqueue_throttle().

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Aaron Lu <ziqianlu@bytedance.com>
Tested-by: Aaron Lu <ziqianlu@bytedance.com>
Link: https://patch.msgid.link/20260831101141.391382-2-liwanwu@kylinos.cn

pick_task_fair() and yield_task_fair() call update_curr(&rq->cfs) to bring
curr up to date before they look at the eevdf state.  With cgroups that
does not happen: update_curr() reads ->h_curr, which on the root cfs_rq is
the top level group entity, and returns at the !entity_is_task() check
before touching vruntime.  Both then read ->curr, so the guard and the
update disagree about which entity they mean.

Counting how often ->h_curr and ->curr differ at pick_task_fair(), on one
CPU for 10s with three busy tasks and one 200us-periodic task:

  all tasks in the root cgroup        43321 calls,     0 no-ops
  busy tasks in G0, periodic in G1    45211 calls, 45193 no-ops

Whether that matters depends on what precedes the pick.  Since
commit 68e37487810a ("sched/fair: Fix flat hierarchy") the tick and
enqueue/dequeue all update curr correctly, so on the normal reschedule
path only the microseconds between those and the pick are missing, and I
could not measure a latency difference there.  Three paths have nothing
before them on that rq though:

  - pick_task() on the sibling rqs of a core under core scheduling
    (kernel/sched/core.c), which updates that rq's clock first for
    exactly this reason
  - fair_server_pick_task()
  - yield_task_fair(), where the stale value feeds the entity_eligible()
    test that guards forfeiting the remaining vruntime

There curr can be a full tick behind, as it was before that commit.

No new behaviour for the entity being updated: without cgroups ->h_curr
is already the task, so these two call sites already run the full
update_curr() including update_deadline(), dl_server_update() and the
resched_curr_lazy() at the end.  This makes the cgroup case do the same.

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260822105930.2352761-1-zhanxusheng1024@gmail.com

Revert commit 1b0866874833 ("locking: Switch to _irq_{disable,enable}()
variants in cleanup guards").

While the guards are properly nested, not all wrapped code is nice, as already
highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which
does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq).
Combined with this patch, that goes sideways most spectacular.

Undo this until we've developed stronger tools / debug for such issues.

Fixes: 1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824105523.GA4121620%40noisy.programming.kicks-ass.net

tg_cpus() returns cpuset_num_cpus() unfloored, while its sibling
tg_tasks() already floors its result at 1. calc_concur_shares() feeds

	nr = min(tg_tasks(tg), tg_cpus(tg))

into __calc_smp_shares() as shares_max, so an nr of 0 makes shares_max 0.
__calc_smp_shares() ends with

	return clamp_t(long, shares, MIN_SHARES, shares_max);

and clamp() yields hi when hi < lo, so a zero shares_max silently defeats
the MIN_SHARES floor and returns 0 -- the exact case the comment above
that line says must return MIN_SHARES instead of 0.

That leaves a group sched_entity with load.weight == 0, and
__calc_prop_weight() then divides by cfs_rq->load.weight:

	weight *= se->load.weight;
	if (parent_entity(se))
		weight /= cfs_rq->load.weight;

which takes a #DE inside enqueue_task_fair():

  Oops: divide error: 0000 [#1] SMP NOPTI
  RIP: 0010:enqueue_task_fair+0x422/0x950
  Call Trace:
   <TASK>
   enqueue_task+0x8e/0x250
   wake_up_new_task+0x148/0x2e0
   kernel_clone+0x1c6/0x390
   __x64_sys_clone+0xcc/0x100
   do_syscall_64+0x147/0x3c0
   </TASK>

This is not survivable in practice: with panic_on_oops=0 the kernel took
the first #DE and continued for 476 ms, then faulted at the same RIP with
identical register state and an identical RSP, because the oops recovery
path (kill task -> schedule()) re-enters the same enqueue while the rq
lock is held mid-enqueue. The second fault escalates to a panic.

Flooring tg_cpus() at 1 makes it symmetric with tg_tasks() and keeps
shares_max >= tg_shares, so the MIN_SHARES floor in __calc_smp_shares()
can no longer be bypassed.

Note this only removes the division hazard. Whether cpuset_num_cpus() can
legitimately return 0 -- via the cpu hotplug/suspend path where a v2
cpuset may transiently become empty, or via an RCU race -- is a separate
question still open on the report thread.

Fixes: 90ac22ffef48 ("sched/fair: Add cgroup_mode: max")
Signed-off-by: Jake Steinman <j@metarealtyinc.ca>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/all/20260818231333.1441757-1-j@metarealtyinc.ca/
Link: https://patch.msgid.link/20260819132104.2148918-1-j@metarealtyinc.ca

Pull timer and timekeeping core updates from Thomas Gleixner:

 - Fix a subtly inconsistency in the timekeeping code, which fails to
   account for the monotonicity adjustment in ntp_error.

   For small changes of the clocksource multiplicator (+/-1) which are
   typically used by the NTP PLL this is hard to observe. But for larger
   adjustments, e.g. caused by a direct frequency setting through
   adjtimex() the one-time uncompensated offset is significant.

   Cure this by adjusting ntp_error with the resulting offset so that
   the discrepancy is smoothed away over time

 - Make tick length calculations correct in NTP.

   The timekeeping core takes the quantisation of the clocksource into
   account when calculating the tick length to compensate for the
   deviation of the nominal NTP_INTERVAL_LENGTH.

   While timekeeping gets this right, NTP is not aware of that, which
   means it operates on the nominal value and not on the actual value
   which is determined by the clock source frequency. The rounding of a
   coarse clocksource like the ACPI PM timer results in a +127 PPM
   deviation.

   Cure this by exposing the deviation to the NTP code so that it can
   operate on the same data as the timekeeping core. This is purely
   kernel internal. User space still sees the nominal tick lenght via
   adjtimex().

 - The accuracy of the NTP adjustments is fairly approximate as the code
   assumes that the invocations are precisely in NTP interval frequency
   ticks and the final adjustment can over and under-run.

   Cure this by adjusting ntp_error by the intended skew on each tick to
   achieve the desired rate.

 - Handle the two competing skews of time offset and time adjustment
   correctly by calculating the conflict portion between the skews and
   adjusting both accordingly.

 - A set of updates and improvements for the selftests

 - The usual small fixes and improvements all over the place

* tag 'timers-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (58 commits)
  selftests: timers: nsleep-lat: Check all calls to clock_nanosleep() and clock_gettime()
  selftests: timers: nsleep-lat: Reuse kselftest error numbers
  selftests: timers: nsleep-lat: Explicitly list the tested clocks
  selftests: timers: nsleep-lat: Use NSEC_PER_MSEC define for unreasonable latency
  selftests: timers: nanosleep: Report each test separately
  selftests: timers: nanosleep: Explicitly handle timer_delete() failure
  selftests: timers: nanosleep: Move all single clock tests out of the loop in main()
  selftests: timers: nanosleep: Reuse kselftest error numbers
  selftests: timers: nanosleep: Explicitly list the tested clocks
  selftests: timers: nanosleep: Drop output alignment
  selftests: timers: Use clock_name() and constants from clock-helpers.h
  selftests: Add clock-helpers.h
  timer_list: Use ktime_t over nanoseconds
  timer_list: Use standard 'long long' format placeholders
  hrtimer: Add a lockdep assertion to hrtimer_update_base()
  timekeeping: Use u32 for clock_was_set_seq
  timekeeping: Rename clockid_aux_valid() to clockid_is_aux_clock()
  hrtimer: Account nr_retries on recovered interrupt retries
  timers/itimer: Zero-init old itimerval before copy to userspace
  nohz: Replace dead select with choice default
  ...

Pull scheduler updates from Ingo Molnar:
 "Load-balancing updates:

   - 'flatten the pick': improve cgroup scheduling, which has always
     been problematic and painful, which has caused various scheduling
     misbehavior such as the mishandling of reniced tasks et al.

     Add various cgroup weight distribution methods via cgroup_mode:
     'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
     which is the most precise yet also most expensive version.

     Finally, change cgroup scheduling to a single runqueue (Peter
     Zijlstra)

   - Series to improve the scheduling latency of short slice tasks
     (Vincent Guittot)

   - Series to fix cluster scheduling in the presence of asymmetric
     capacity (Ricardo Neri)

   - Prefer fully idle cores for NOHZ balancing (Andrea Righi)

   - Don't trigger active load-balancing if src_rq->curr is not on_rq
     (Xin Zhao)

  PSI updates:

   - Skip irqtime accounting when no new irq time has elapsed (Usama
     Arif)

  Scheduler debugging updates:

   - Remove unused schedstats (Shrikanth Hegde)

   - Defer freeing of cpumask memblock memory to initcall (Waiman Long)

  Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
  Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng"

* tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (33 commits)
  sched/fair: Fix flat hierarchy
  sched/isolation: Defer freeing of cpumask memblock memory to initcall
  sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity
  sched/fair: Allow load balancing between CPUs of identical capacity
  sched/fair: Skip misfit load accounting when the destination CPU cannot help
  sched/fair: Check CPU capacity before comparing group types during load balance
  sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY
  sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings
  sched/fair: Prefer fully idle cores for NOHZ balancing
  stop_machine: Make stop_one_cpu_nowait() return void
  sched/eevdf: Delayed dequeue task can't preempt
  sched/fair: Fix stale comments referring to removed CFS concepts
  sched/debug: Remove unused schedstats
  sched/psi: skip irqtime accounting when no new irq time has elapsed
  sched/fair: Reflow sched_balance_rq()
  sched/fair: Simplify balance_interval reset logic in sched_balance_rq()
  sched/fair: Don't trigger active lb if src_rq->curr is not on_rq
  sched/eevdf: Speedup short slice task scheduling
  sched/eevdf: Always update slice protection
  sched/eevdf: Cancel slice protection if short slice task is eligible
  ...

Pull locking updates from Ingo Molnar:
 "Futexes:

   - Use runtime constants for futex_hash computation (K Prateek Nayak,
     Peter Zijlstra)

   - Optimise the size check get_futex_key() (Sebastian Andrzej Siewior)

   - Avoid private hash use-after-free on final put (Felix Hoffmann)

   - Tell kmemleak we're not leaking __futex_queues (Peter Zijlstra)

  Rust integration updates:

   - Implement refcounted interrupt disable and SpinLockIrq for Rust
     (Boqun Feng, Heiko Carstens, Joel Fernandes, Lyude Paul)

   - Rust sync: add helpers for mb, dma_mb and friends; add generic
     memory barriers and use LKMM atomics instead of Rust atomics in the
     revocable code (Gary Guo)

   - Add abstraction and integrate synchronize_rcu() (Philipp Stanner)

  Lock debugging:

   - Add qspinlock contended_release tracepoint (Dmitry Ilvokhin, Peter
     Zijlstra)

   - Enable the printing of held locks of remote running tasks and print
     task CPU (Ingo Molnar)

   - percpu-rwsem: Annotate intentional data race in readers_active_check()
     (Sun Shaojie)

  Misc fixes and updates by Boqun Feng, Peter Zijlstra, Fangrui Song,
  Naveen Kumar Chaudhary and Thomas Huth"

* tag 'locking-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (44 commits)
  rust: sync: Introduce SpinLockIrq::lock_with() and friends
  rust: sync: Add SpinLockIrq
  rust: sync: Use super::* in spinlock.rs
  rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
  rust: Introduce interrupt module
  s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
  arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
  preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
  sched: Avoid signed comparison of preempt_count() in __cant_migrate()
  sched: Remove the unused preempt_offset parameter of __cant_sleep()
  locking: Switch to _irq_{disable,enable}() variants in cleanup guards
  irq: Add KUnit test for refcounted interrupt enable/disable
  irq,spin_lock: Add counted interrupt disabling/enabling
  openrisc: Include <linux/cpumask.h> in smp.h
  preempt: Introduce __preempt_count_{sub,add}_return()
  preempt: Introduce HARDIRQ_DISABLE_BITS
  preempt: Track NMI nesting to separate per-CPU counter
  futex: Tell kmemleak we're not leaking __futex_queues
  x86/paravirt: Trace contended_release on unlock
  tracing/lock: Use TRACE_EVENT_FN() for contended_release
  ...

When a fair task is enqueued, we must update curr and more precisely
its vruntime before placing the enqueued task so avg vruntime will take
into account the last exec phase.

Example:
TA is an always running task in cgroup G0.
TB is a short running task (cyclictest) in cgroup G1.
The lag of TB always increases up the clamp limit because TB is placed
before TA(curr) is updated (since the last tick). When curr(TA) is
finally updated, its last exec phase provide positive lag to TB

Because TA and TB don't belong to the same group, enqueue_hierarchy() will not
update TA's entity when updating curr but only G0's entity at root level.

The same applies when dequeuing.

This is because update_curr() uses ->h_curr, rather than ->curr, and therefore,
while it is invoked on the root cfs_rq, which contains all the eevdf bits, it
does not do the right thing.

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260812125039.1717249-1-vincent.guittot@linaro.org