commit 62f3d531e41a7e6e0dc8d37ea377c50488decb56 upstream.

scx_bpf_dsq_reenq() is registered with KF_IMPLICIT_ARGS and its kernel BTF
prototype omits the trailing bpf_prog_aux argument. The ___compat extern
declares the argument explicitly, so libbpf never matches the prototype and
the weak extern silently stays unresolved on every kernel. The wrapper
always takes the old fallback path, which disables generic reenq users like
scx_qmap's lowpri mechanism and fails non-local reenq with "kernel too old"
even on kernels that have the kfunc.

Drop the explicit aux argument. Also correct the stale v6.20 reference, the
kfunc was added in v7.1.

Fixes: 9c34c5074d1b ("sched_ext: Introduce scx_bpf_dsq_reenq() for remote local DSQ reenqueue")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 56bbc91219c0711f1580a8ba7cd60471b6e1684e upstream.

libbpf strips the last ___flavor suffix when resolving kfunc externs, so
the bare ___v2 declaration resolves to scx_bpf_dsq_move_to_local, whose
BTF proto lacks @enq_flags. The extern never matches, bpf_ksym_exists()
returns false on every kernel that has the ___v2 kfunc, and the macro
falls back to ___v1, silently dropping @enq_flags.

Add the trailing ___compat suffix used by the other versioned externs in
this file (scx_bpf_dsq_insert___v2, scx_bpf_reenqueue_local___v2).

Any caller passing non-zero @enq_flags through the compat macro silently
loses them.

Fixes: 860683763ebf ("sched_ext: Add enq_flags to scx_bpf_dsq_move_to_local()")
Cc: stable@vger.kernel.org # v7.1+
Assisted-by: Z.ai:glm-5.2
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit ffaab58d217581cb75353168f8812a16e10463fc upstream.

SCX_RQ_BAL_KEEP tells the pick to keep running the previous task, a leftover
from when balancing and picking were separate operations. An rq-level flag
only works while dispatches and picks pair up one to one, which core
scheduling breaks: selections interleave through dispatch's lock drops and a
pick can consume a stale flag, keeping a task that has since been dequeued.
Fixing core scheduling support requires the decision to travel with the
dispatch that made it. Make scx_dispatch_sched() and balance_one() return an
explicit verdict instead and drop the flag's plumbing from the tools autogen
enum headers.

Also factor the pick-side invocation, its follow-up queueing and the
post-dispatch checks out of do_pick_task_scx() into dispatch_pick(). No
functional changes intended.

v2: Drop the SCX_RQ_BAL_KEEP plumbing from the tools autogen enum headers
    as well (Andrea).

Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

cmask_equal(), cmask_weight() and cmask_subset() bounded their word walks
with CMASK_NR_WORDS(nr_cids), which pads by one word and can't tell the last
word in use without @base. The walks could thus cover a slack word past the
active range, which cmask_reframe() leaves non-zero: a stale bit there gave
cmask_equal() a spurious mismatch, cmask_weight() an inflated count, and
cmask_subset() a spurious violation. cmask_subset() could also read
@b->bits[] one word past its allocation (within the arena's fault-recovered
range, so harmless), and deviated from the kernel scx_cmask_subset() by
failing any @a range that doesn't nest inside @b's even when the overhanging
bits are all clear.

Bound the cmask_equal() and cmask_weight() walks by the words the range
actually spans, with early returns for empty ranges. Rewrite cmask_subset()
to match the kernel semantics: scan @a's overhangs for set bits with
cmask_next_set() and walk the words of the range intersection.
cmask_subset() moves below cmask_next_set(), which it now uses. Padding bits
don't need masking as every cmask helper keeps them clear.

Fixes: a58e6b79b432 ("sched_ext: Add cmask, a base-windowed bitmap over cid space")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

The sched_ext sources had grown to ten ext* files directly under
kernel/sched/. Move them into a new kernel/sched/ext/ subdirectory and drop
the now-redundant ext_ prefix. ext.c/h keep their names.

  kernel/sched/ext.{c,h}       -> kernel/sched/ext/ext.{c,h}
  kernel/sched/ext_internal.h  -> kernel/sched/ext/internal.h
  kernel/sched/ext_types.h     -> kernel/sched/ext/types.h
  kernel/sched/ext_idle.{c,h}  -> kernel/sched/ext/idle.{c,h}
  kernel/sched/ext_cid.{c,h}   -> kernel/sched/ext/cid.{c,h}
  kernel/sched/ext_arena.{c,h} -> kernel/sched/ext/arena.{c,h}

The include paths in build_policy.c and sched.h, the MAINTAINERS glob, and a
few documentation and comment references are updated to match. No code or
symbol changes.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

Switch scx_bpf_kick_cid() from void to s32 so future cap enforcement can
surface failures. cid interface is introduced in this cycle and has no
external users, so the ABI change is safe. Subsequent patches will add
-EPERM returns when the calling sub-sched lacks the required cap on the
target cid.

v2: Return scx_cid_to_cpu()'s errno instead of -EINVAL. (Andrea)

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

The BPF arena single-cid cmask helpers take the cmask first and the cid
second. Reorder them to (cid, mask) to match the kernel-side helpers and
the test_bit(nr, addr), cpumask_test_cpu(cpu, mask) convention. Range and
iteration helpers keep (mask, start).

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

ops_cid.set_cmask() expects a cmask. The kernel couldn't write into the
arena, so it translated cpumask -> cmask in kernel memory and passed the
result as a trusted pointer. The BPF cmask helpers all operate on arena
cmasks though, so the BPF side had to word-by-word probe-read the kernel
cmask into an arena cmask via cmask_copy_from_kernel() before any helper
could touch it. It works, but is clumsy.

With direct kernel-side arena access now in place, build the cmask in the
arena. The kernel writes to it through the kern_va side of the dual mapping.
BPF directly dereferences it via an __arena pointer like any other arena
struct.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>

scx_cmask carries @base and @nr_cids but not the bits[] allocation size, so
helpers reshaping the active range have no way to check it fits and later
kfuncs taking caller-provided storage can't validate it.

Add @alloc_words (u64 word count) annotated with __counted_by, and split the
bit-range API into three helpers:

- SCX_CMASK_DEFINE() / __SCX_CMASK_DEFINE() define an on-stack cmask, the
  latter taking an explicit capacity for oversized storage.
  SCX_CMASK_DEFINE_SHARD() is a thin wrapper that always reserves
  SCX_CID_SHARD_MAX_CPUS bits of storage.

- scx_cmask_init() / __scx_cmask_init() initialize a cmask, with the same
  tight-vs-explicit split.

- scx_cmask_reframe() reshapes the active range without resizing storage.

The BPF mirror (cmask_init / __cmask_init / cmask_reframe) gets the same
shape.

Add scx_cmask_clear() and scx_cmask_fill() to zero and set the
active-range bits respectively. scx_cpumask_to_cmask() uses
scx_cmask_clear(); scx_cmask_init() would otherwise re-write @alloc_words
on every call.

A later patch uses @alloc_words in scx_cmask_ref_shard() to refuse output
storage that can't hold the requested shard.

v2: Init per-CPU scx_set_cmask_scratch (was zero-init, emitted empty
    cmasks). Add nr_cids/alloc_cids check in BPF __cmask_init().
    (sashiko AI)
    Widen SCX_CMASK_NR_WORDS()/CMASK_NR_WORDS() to compute in u64 so that
    @nr_cids near U32_MAX no longer wraps to a small value and bypasses
    the bounds check in cmask_reframe(). (Andrea)

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

struct scx_cmask is a base-windowed bitmap over cid space. Each bit
represents one cid, so the count of active bits is the count of cids. The
sibling struct scx_cid_shard already uses nr_cids. Rename as a prep so the
following patches that grow the cmask API can use the consistent name.

v2: Also rename src->nr_bits / dst->nr_bits in
    cmask_copy_from_kernel(). (sashiko AI)

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>