| Age | Commit message (Collapse) | Author |
|---|
|
This commit adds a check for failure to have fully deboosted a
multi-segmented RCU reader at the end of the full read-side critical
section. This check only happens for fully task-level readers, because
a a handler might have interrupted an already-boosted task-level RCU
reader, and a reader in that handler could then cause false positives.
The first failed check (due to an RCU reader that was not immediately
deboosted) causes a splat, but only when the disabled-by-default
deboost_timeliness_check module parameter is enabled. Regardless of the
value of this parameter, it produces a list of the segments making up that
RCU reader following a "Slow-deboost rcutorture reader segments" heading.
Subsequent failures fail silently, all in the name of keeping console
output down to a dull roar.
Although most uses of RCU priority boosting serve as debugging aids,
this might change, and in fact might already have changed. And allowing
(for example) RCU priority boosting to persist until the next scheduler
tick could cause an aggressively real-time system to miss sub-millisecond
deadlines. So we do need to find this sort of problem during testing,
and preferably not in the field.
The name and type of the newly added rcu_torture_ops function pointer
(named "->is_task_rcu_boosted()") may need to change should other
end-of-reader checks be needed. But let's start simple.
Oh, and Claude figured out that rcu_is_task_rcu_boosted() could be
lockless. Perhaps there is hope for AI yet! ;-)
[ paulmck: Apply Akira Yokosawa feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
scx_fork() initializes tasks when scx_init_task_enabled is set, but
scx_cancel_fork() only exits them when scx_enabled() is true. A fork
that fails in the enable window (between releasing scx_fork_rwsem and
setting __scx_enabled) runs ops.init_task() but never ops.exit_task().
Gate scx_cancel_fork() on scx_init_task_enabled.
Fixes: 4269c603cc26 ("sched_ext: Enable scx_ops_init_task() separately")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
- inlines.h: scx_bpf_dispatch() doesn't exist; the comment means
scx_bpf_sub_dispatch()
- internal.h: name %SCX_DEQ_SCHED_CHANGE instead of the never-defined
%SCX_DEQ_SAVE
- internal.h: @name shows up in the ops file in the scheduler's sysfs
directory, not a "kernel.sched_ext_ops" sysctl
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_process_sync_ecaps() sets up the dispatch context for
ops.sub_ecaps_updated() in the target cpu's pcpu context recovered from the
llist node. However, the context is per executing cpu: the dispatch kfuncs
resolve it with this_cpu_ptr() and the dispatch buffer lives in it. What the
dispatches target is determined by the rq recorded in the context, not by
which cpu's context it is. Under core scheduling the pick runs balance_one()
for sibling rqs, so a sync processed for a sibling invokes the op with the
executing cpu's context not set up and its dispatch kfuncs misoperate on a
NULL or stale rq.
Set up the executing cpu's dsp_ctx instead, matching scx_dispatch_sched().
The recorded rq keeps the dispatches targeting the synced cpu.
Fixes: b81a6c018cde ("sched_ext: Add sub_ecaps_updated() effective-cap change notifier")
Reported-by: David Carlier <devnexen@gmail.com>
Link: https://lore.kernel.org/all/20260813045931.8691-1-devnexen@gmail.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
ring_buffer_per_cpu::mapped tracks if a ring-buffer is either mapped by
user-space or if it is a persistent buffer. We already have user_mapped
for the former and ring_meta for the latter. Get rid of mapped and
instead create rb_is_static(). A static ring-buffer cannot be resized,
swapped or have its pages extracted.
Link: https://patch.msgid.link/20260813131152.3589632-10-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The 'cpus' field in struct trace_buffer became useless in commit
8e7b58c27b3c ("ring-buffer: Just update the subbuffers when changing their
allocation order"). Remove it
Link: https://patch.msgid.link/20260813131152.3589632-9-vdonnefort@google.com
Fixes: 8e7b58c27b3c ("ring-buffer: Just update the subbuffers when changing their allocation order")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The ring buffer order can be dynamically modified and temporarily
disables writing to do so. It is therefore safe to use the updated value
to calculate the maximum event size which can be written onto the ring
buffer.
However, notice it is hardly making any difference for trace_marker
because of the TRACE_MARKER_MAX_SIZE limit. For an 8KiB subbuf size,
trace_marker can take 4096 characters while it can 'only' take 4054
bytes for smaller subbufs.
Link: https://patch.msgid.link/20260813131152.3589632-8-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
ring_buffer_alloc_read_page() is racy with ring_buffer_subbuf_order_set,
it can allocate a reader page with an outdated order. This isn't a big
issue, the user can still re-allocate a new reader page and try again.
However, what is more problematic is if the value of subbuf_order
changes in the middle of ring_buffer_alloc_read_page(). In that case,
bpage->order might not match the actual allocated memory.
Use bpage->order for the allocation to prevent this race.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-6-vdonnefort@google.com
Fixes: bce761d75745 ("ring-buffer: Read and write to ring buffers with custom sub buffer size")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
trace_buffer subbuf_size is read lockless in ring_buffer_read_page() and
ring_buffer_read_start(), while it can simultaneously be resized with
ring_buffer_subbuf_order_set().
Instead of trace_buffer::subbuf_size, use bpage::order in
ring_buffer_read_start() and ring_buffer_read_page().
In ring_buffer_read_start(), even with resize_disabled, there is still a
possibility of a race with a buffer modification. Hold the trace_buffer
mutex to synchronise with any pending ring buffer order modification.
trace_buffer::subbuf_size is now actually useless, remove it. Also,
create accessors rb_subbuf_capacity() and rb_page_capacity() which
return the actual size available for storing events, while
rb_subbuf_size() returns the actual subbuf page-size.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-5-vdonnefort@google.com
Fixes: f9b94daa542a ("ring-buffer: Set new size of the ring buffer sub page")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260805153225.2096152-1-vdonnefort%40google.com # patch 1
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Discarding a cached reader page after a concurrent ring buffer resize
uses the new global subbuf_order for the free_pages() call. This
mismatched order may crashes the kernel or leaks memory because the cached
page was allocated under the old size.
Save the actual free_page order alongside the page address to ensure we
always refer to the correct value and do not rely on the potentially
stalled cpu_buffer->subbuf_order value. The simplest is to make
free_page a buffer_data_read_page which already covers exactly what we
need: a page address and a page order.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-4-vdonnefort@google.com
Fixes: 8e7b58c27b3c ("ring-buffer: Just update the subbuffers when changing their allocation order")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Because, ring_buffer_subbuf_order_set() can clear cpu_buffer->free_page,
hold cpu_buffer->lock to prevent races with
ring_buffer_alloc_read_page() and ring_buffer_free_read_page().
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-3-vdonnefort@google.com
Fixes: 8e7b58c27b3c ("ring-buffer: Just update the subbuffers when changing their allocation order")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260810125633.3344684-1-vdonnefort%40google.com # patch 3
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When sub-buffers use an order greater than 0, cpu_buffer->free_page is
allocated with subbuf_order. Use the correct order for
cpu_buffer->free_page.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260813131152.3589632-2-vdonnefort@google.com
Fixes: f9b94daa542a ("ring-buffer: Set new size of the ring buffer sub page")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260806211306.3704194-1-vdonnefort%40google.com # patch 3
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
arm64 passes zero as the stack pointer while walking BPF frames, so
bpf_throw() leaves stale KASAN stack poison after jumping to the
exception callback.
Use the frame pointer as the fallback stack watermark.
Fixes: e74cb1b42213 ("arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT")
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260812-hello_world-v1-1-c3c2ddcb362d@meta.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
- Don't warn when a mount is completed from another user namespace.
fsopen() records the caller's user namespace in fc->user_ns and
hands back an ordinary file descriptor. The task that calls
fsconfig(FSCONFIG_CMD_CREATE) doesn't have to be the one that
created the context, and mount_capable() lets it through as long
as the caller has CAP_SYS_ADMIN over fc->user_ns, which anyone in
an ancestor namespace does. So fc->user_ns != current_user_ns()
is something an unprivileged user can arrange.
Both overlayfs and binfmt_misc WARN_ON() that. Overlayfs already
has the same check as a plain error return in ovl_parse_param().
Drop the WARN_ON() and just refuse. Add selftests for both cases.
- Reject pid allocations through dead ancestor pid namespaces.
Require PIDNS_ADDING in every namespace that will receive the pid
before publishing any of them. That preserves the invariant that
free_pid() never decrements pid_allocated in a namespace whose
child_reaper is no longer live. The existing ENOMEM behavior is
unchanged.
* tag 'vfs-7.2-rc8.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
pid: reject allocations through dead ancestor pid namespaces
selftests/filesystems: test completing a context from another user namespace
binfmt_misc: don't warn when the mount is completed from another user namespace
ovl: don't warn when the mount is completed from another user namespace
|
|
When a fair task is enqueued, we must update curr and more precisely
its vruntime before placing the enqueued task so avg vruntime will take
into account the last exec phase.
Example:
TA is an always running task in cgroup G0.
TB is a short running task (cyclictest) in cgroup G1.
The lag of TB always increases up the clamp limit because TB is placed
before TA(curr) is updated (since the last tick). When curr(TA) is
finally updated, its last exec phase provide positive lag to TB
Because TA and TB don't belong to the same group, enqueue_hierarchy() will not
update TA's entity when updating curr but only G0's entity at root level.
The same applies when dequeuing.
This is because update_curr() uses ->h_curr, rather than ->curr, and therefore,
while it is invoked on the root cfs_rq, which contains all the eevdf bits, it
does not do the right thing.
Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260812125039.1717249-1-vincent.guittot@linaro.org
|
|
Pull in dependents, the flat hierarchy fix depends on this.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
|
|
In order to compute the right lag, it is required to update time to 'now'.
Without this, the delayed entity might appear younger than it really is and
receive less compensation for having waited.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
|
|
The function trace_event_update_all() does a scan of events looking to
replace enums with their values in the strings that get exported to the
event format files. It's run at boot up on all events and again when a
module loads.
The issue is that when a module loads, it still runs on *all* events.
There's no reason to process every event when a module loads as the
previous events have already been processed. Only execute on the events
that are loaded with the module.
Link: https://patch.msgid.link/20260813204226.29563591@gandalf.local.home
Fixes: 3673b8e4ce723 ("tracing: Allow for modules to convert their enums to values")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The check that prevents userspace from sending siginfo with si_code values
reserved to the kernel is duplicated across do_rt_sigqueueinfo(),
do_rt_tgsigqueueinfo() and do_pidfd_send_signal(). Move the check into a
helper so the rule lives in one place.
Link: https://lore.kernel.org/20260806133013.4341-1-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
cmd_attr_register_cpumask() and cmd_attr_deregister_cpumask() differed
only in which attribute they parsed and which action they passed on, so
take both as arguments. __free(free_cpumask_var) then removes the goto.
No functional change.
Link: https://lore.kernel.org/20260728202104.17839-3-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Cc: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "taskstats: tidy up the cpumask command path".
Two small cleanups from reading kernel/taskstats.c. No functional change in
either one.
This patch (of 2):
taskstats_user_cmd() only calls the cpumask handlers after checking the
same info->attrs[] entry, so parse() never sees a NULL attribute. Drop
the check and its odd "return 1", which no caller tested for anyway.
No functional change.
Link: https://lore.kernel.org/20260728202104.17839-1-include@grrlz.net
Link: https://lore.kernel.org/20260728202104.17839-2-include@grrlz.net
Signed-off-by: Bradley Morgan <include@grrlz.net>
Cc: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The verifier keeps per-subprogram metadata in three parallel arrays:
subprog_info, func_info, and func_info_aux. Dead code elimination can
remove whole subprograms, and adjust_subprog_starts_after_remove()
shifts subprog_info and func_info to close the gap, but leaves
func_info_aux in place. From that point on, func_info_aux[i] no longer
describes subprogram i.
Shift func_info_aux together with func_info so the three arrays stay
aligned after subprogram removal.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com
|
|
The following sequence may leads race between event_define_fields()
and update_event_fields():
CPU0 (loads module A) CPU1 (loads module B)
=============================== ===============================
load_module(A) load_module(B)
notifier_call_chain notifier_call_chain
trace_module_notify trace_module_notify
mutex_lock(&event_mutex) trace_event_update_all()
trace_module_add_events(A) down_write(&trace_event_sem)
__register_event(call_A)
__add_event_to_tracers(call_A)
event_define_fields(call_A)
for each f: list_for_each_entry(field,
list_add(&f->link, &class->fields, link)
&class->fields) field = class->fields->next;
Where access to the class->fields is not protected by the event_mutex in
trace_event_update_all().
This produces the following panic:
Unable to handle kernel access ... at virtual address 0000000000000018
pc : update_event_fields+0xf8/0x368
Call trace:
update_event_fields+0xf8/0x368
trace_event_update_all+0x7c/0x2b4
trace_module_notify+0x4c/0x1dc
notifier_call_chain+0x84/0x168
blocking_notifier_call_chain_robust+0x64/0xd4
load_module+0x10c8/0x123c
__arm64_sys_finit_module+0x230/0x31c
Fix by taking event_mutex in trace_event_update_all() before
trace_event_sem.
Cc: stable@vger.kernel.org
Fixes: b3bc8547d3be ("tracing: Have TRACE_DEFINE_ENUM affect trace event types as well")
Link: https://patch.msgid.link/2e5730d2-c631-da41-3a3a-ae35bb4895f3@allwinnertech.com
Signed-off-by: Michael Wu <michael@allwinnertech.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A module-only event filter such as ":mod:foo" is cached with a NULL
event_mod->match when foo has not been loaded. If a later write tries to
remove a specific match from the same module, remove_cache_mod() passes
the NULL cached match to strcmp(), causing a NULL pointer dereference.
The issue can be reproduced from userspace:
echo ':mod:trace_events_kunit_missing' > /sys/kernel/tracing/set_event
echo '!foo_bar:mod:trace_events_kunit_missing' >> /sys/kernel/tracing/set_event
The second write must be a concatenation (">>") to not include O_TRUNC as
that would cause ftrace_clear_events() to clear the cached modules lines.
The crash was reproduced on x86_64 QEMU while KUnit workers contended on
the event tracing path:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
RIP: 0010:strcmp+0x10/0x30
Call Trace:
__ftrace_set_clr_event_nolock+0x373/0x4a0
ftrace_set_clr_event+0xf0/0x180
ftrace_event_write+0xdf/0x110
vfs_write+0xf6/0x440
ksys_write+0x68/0xe0
do_syscall_64+0xf9/0x540
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Check event_mod->match before comparing it, consistent with the existing
NULL checks for the cached system and event fields. The mismatched removal
continues to return -EINVAL; a broad cached module filter is removed with
"!:mod:<module>".
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260811173902.1927376-2-sh_def@163.com
Fixes: b355247df104 ("tracing: Cache \":mod:\" events for modules not loaded yet")
Reported-by: syzbot+4d3143c8e28f6266c636@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/lkml/6a7a6b7f.9c11d2ce.289b96.00f8.GAE@google.com/
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Cross-merge networking fixes after downstream PR (net-7.2-rc8).
No conflicts.
Adjacent changes:
drivers/net/ethernet/wangxun/ngbe/ngbe_main.c
5f3a13e0bb5e ("net: ngbe: fix NULL pointer dereference in non-MSI-X interrupt enabling")
d661abdc30c2 ("net: ngbe: correct misleading interrupt comment")
drivers/net/ipvlan/ipvlan_main.c
e16e960d55a4 ("ipvlan: inherit needed_headroom and needed_tailroom from phy_dev")
00a40d809207 ("ipvlan: Support per-netns netdev unregistration.")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Introduce global percpu data, inspired by the commit
6316f78306c1 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
|
|
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
|
|
There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.
Drop them for cleanup.
No functional changes intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev
|
|
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can
update it concurrently. KCSAN reports:
BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8
race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by
task 1817002 on cpu 40:
kick_pool_pick+0xf8/0x2d8
process_scheduled_works+0x2bc/0x888
worker_thread+0x394/0x548
kthread+0x1b8/0x1f0
ret_from_fork+0x10/0x20
value changed: 0x0000002b -> 0x0000002f
The race is harmless. wake_cpu is a best-effort placement hint: every
writer stores a valid CPU id and the wakeup path validates it through
select_task_rq(), so a stale value only affects which CPU the worker
wakes up on.
Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that
they are intentionally racy and to stop the compiler from reloading or
tearing them.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When testing a linux-next kernel with commit 59bd1d914bb5 ("memblock:
warn when freeing reserved memory before memory map is initialized"),
the following warning was hit when there was a "nohz_full" kernel boot
parameter.
Cannot free reserved memory because of deferred initialization of the memory map
WARNING: mm/memblock.c:904 at __free_reserved_area+0xde/0xf0, CPU#0: swapper/0/0
:
Call Trace:
<TASK>
memblock_phys_free+0xcb/0x100
housekeeping_init+0x14c/0x170
start_kernel+0x207/0x450
x86_64_start_reservations+0x24/0x30
x86_64_start_kernel+0xda/0xe0
common_startup_64+0x13e/0x141
</TASK>
IOW, we shouldn't free memblock allocated memory so early
in the boot process when memory map isn't fully initialized in
deferred_init_memmap().
Fix it by saving the housekeeping cpumask memblock memory to be
freed into a llist free list in housekeeping_init() and add a new
housekeeping_late_init() helper to defer the actual freeing of memblock
memory to when initcall's are being processed. The cpumask memblock
memory is treated as a llist_node with the size of a "long" type which
is also smallest cpumask size that can be allocated.
The non-atomic version of the llist APIs are used as there is no
contention.
This commit depends on the presence of commit 7c2eee9c1367 ("memblock:
don't touch memblock arrays when memblock_free() is called late")
to prevent a KASAN UAF bug report [1].
[1] https://lore.kernel.org/lkml/20260505051821.1107133-1-longman@redhat.com/
Fixes: 27c3a5967f05 ("sched/isolation: Convert housekeeping cpumasks to rcu pointers")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <pauld@redhat.com>
Link: https://patch.msgid.link/20260701195810.477326-1-longman@redhat.com
|
|
Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
performs an atomic_long_add_return() plus a CAS loop on the global
used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
All CPUs contend on the same cacheline, causing measurable throughput
degradation at scale.
Historically these counters were only compiled in under CONFIG_DEBUG_FS,
which means production kernels with debugfs paid the atomic overhead
unconditionally. Make the tracking boot-time opt-in instead so that it
is disabled by default with near-zero overhead via static_call, and can
be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.
Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled,
the "io_tlb_used" metric reports an approximate value rather than an
instantaneously exact one.
Suggested-by: Fan Du <fan.du@intel.com>
Signed-off-by: Jun Miao <jun.miao@intel.com>
Co-developed-by: Fan Du <fan.du@intel.com>
Signed-off-by: Fan Du <fan.du@intel.com>
Tested-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Signed-off-by: chenhuguanshen <chenhgs@chinatelecom.cn>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Link: https://lore.kernel.org/r/20260812070459.637077-1-frankchen158@126.com
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
The commit 7619a0ee9340 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
bpf_trampoline_multi_detach() always returns 0 and the sole caller
ignores the return value. Change it to return void and drop the
WARN_ON_ONCE at the call site.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
When bpf_trampoline_update() fails before modify_fentry_multi()/
unregister_fentry_multi() is called, cur_image is unchanged
(cur_image == old_image) and ftrace still calls into it. Freeing
old_image in that case causes a UAF.
Only free old_image when it differs from cur_image.
Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
audit_del_rule() is used for both netlink deletion templates and internal
fsnotify autoremove. The former passes a parsed template which owns a
temporary tree reference; the latter passes the installed entry itself.
The unconditional audit_put_tree() at the end of audit_del_rule() assumes
the template case. For mixed AUDIT_DIR plus AUDIT_EXE rules, an fsnotify
autoremove event therefore drops the installed rule's live tree reference.
Repeating this across rules sharing the same tree can free the tree while
another rule still references it, and a later autoremove dereferences the
freed pathname while comparing rules.
Move the temporary-tree put to audit_rule_change(), the caller that owns
deletion templates. Keep it in the AUDIT_DEL_RULE cleanup so both
successful deletion and -ENOENT still release the parser-owned tree.
Cc: stable@kernel.org
Fixes: 34d99af52ad4 ("audit: implement audit by executable")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Ricardo Robaina <rrobaina@redhat.com>
Tested-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: dropped unnecessary comment for line length reasons]
Signed-off-by: Paul Moore <paul@paul-moore.com>
|
|
scx_bpf_cid_override() predates the cid-form arena transition and takes its
arrays as verifier-checked mem+size buffers, forcing scx_qmap to keep the
cpu_to_cid and shard_start arrays in writable bss while the rest of its
state lives in the arena. Unify on arena arguments before cid-form
schedulers start seeing real use.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Take the arrays as __arena arguments, with the counts passed in
entries. The counts now size the snapshot copies and are bounds-checked
before them.
scx_qmap moves the arrays into struct qmap_arena. As the arena is mmapped at
load, the loader populates them between load and attach instead of before
load.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling this kfunc load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The sub-cap kfuncs take their cmask arguments as __ign pointers. The values
cross the kfunc boundary as unchecked scalars and scx_cmask_ref_init()
rebases them into the arena by hand.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the cmask arguments __arena so the kfuncs receive kernel
addresses and scx_cmask_ref_init() loses the hand-rolled conversion. The
optional denied_out keeps its NULL not-provided signal via
__arena__nullable. The mandatory masks use plain __arena.
scx_qmap's call sites drop the (void *)(long) casts since the BPF-side
declarations type the cmask arguments __arena and take arena pointers
directly.
The arena argument address translation is currently implemented only on
x86-64. Schedulers calling these kfuncs load only there for now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.
BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.
The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
|
|
scx_bpf_cid_to_cpu(), scx_bpf_cpu_to_cid() and scx_bpf_cid_topo() live in
the scx_kfunc_ids_cid set, but scx_kfunc_context_filter() doesn't check
that set. The filter's first test treats any kfunc outside its known sets
as non-SCX and allows it, so these three kfuncs can be called from any
struct_ops program - e.g. a TCP congestion control program.
Add scx_kfunc_ids_cid to the filter's known sets, matching how in_any and
in_idle are handled.
Fixes: e9b55af47edf ("sched_ext: Add topological CPU IDs (cids)")
Assisted-by: Z.ai:glm-5.2
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.
Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but
every caller dereferences the result right away, so the WARN_ON_ONCE()
only moves the oops one frame up, as raised by Tejun.
Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE();
Fixes: b72fdc651056 ("workqueue: account nr_active by the backing pool")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation
path fills it in with plain loads and stores, which sparse flags:
kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@
Allocate the array as __rcu pointers and publish each pwq with
rcu_assign_pointer() once it is initialized and linked, the order
install_unbound_pwq() uses.
The warnings are not new: commit 79f23600bc7b ("workqueue: factor out
get_percpu_pool()") only turned the flagged assignment into an
initializer.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain
load, so sparse complains about the dropped __rcu:
kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@
A pwq is released with kfree_rcu() and the read is protected by the
surrounding preempt_disable(), which is what
commit fd5081f4ef33 ("workqueue: Remove redundant rcu_read_lock/unlock() in
workqueue_congested()") relied on when it dropped the rcu_read_lock()
here.
Use the rcu_dereference_sched() helper to make that explicit.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive:
c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d2175 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e44a ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da9587118d ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205c1 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d2175 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d2175 and 3dd52416e44a update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da9587118d instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205c1 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
It was an effort to enhance fscache as a kernel cache for lazy
pulling (at least according to previous Incremental FS discussion [1])
and EROFS over fscache was the in-tree user of this mode.
fscache has since evolved to be netfslib-oriented, serving network
filesystem inodes via the netfs library, but EROFS never acts as a
network filesystem and we need to cache golden filesystem images rather
than individual EROFS inodes.
Since EROFS over fscache is now removed, clean up netfs/fscache/
cachefiles upstream too.
[1] https://lore.kernel.org/r/CAOQ4uxi4dzxArY24YO=+kBCK2gGoq3Ptb8WkzCqSogPgU_R3dQ@mail.gmail.com
[dh] Fixed up comments on:
https://sashiko.dev/#/patchset/20260716103030.3065561-1-dhowells%40redhat.com
https://sashiko.dev/#/patchset/20260722130218.78958-1-dhowells%40redhat.com
Signed-off-by: Gao Xiang <xiang@kernel.org>
Signed-off-by: David Howells <dhowells@redhat.com>
Link: https://patch.msgid.link/1046393.1786544127@warthog.procyon.org.uk
cc: Paulo Alcantara <pc@manguebit.org>
cc: netfs@lists.linux.dev
cc: linux-erofs@lists.ozlabs.org
cc: bpf@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
