| Age | Commit message (Collapse) | Author |
|---|
|
wq_worker_tick() bumps pwq->stats[PWQ_STAT_CPU_TIME] on every scheduler
tick before pool->lock is taken. For unbound workqueues the
pool_workqueue is shared by all workers of the pool across CPUs, so
concurrent ticks on different CPUs perform an unsynchronized 64-bit
read-modify-write on the same counter. KCSAN reports this as a
data-race:
BUG: KCSAN: data-race in wq_worker_tick / wq_worker_tick
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 29:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
update_process_times+0x200/0x260
tick_nohz_handler+0x230/0x2f8
__hrtimer_run_queues+0x1ec/0x6c8
hrtimer_interrupt+0x174/0x4b8
...
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 24:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
...
value changed: 0x000000000010a1d0 -> 0x000000000010a9a0
The counter is purely advisory, so an occasional lost update is
harmless, and every other stats[] update already runs under pool->lock.
Annotate the update with data_race().
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
For debugging probe events, it is helpful to verify the compiled
fetch instructions for each probe argument. This introduces a new
kernel config CONFIG_PROBE_EVENTS_DUMP_FETCHARG to decode the
instruction sequence of each argument and display it under a
commented line starting with '#' immediately following the dynamic
event definition (such as in dynamic_events, kprobe_events,
uprobe_events, etc.).
For example:
/sys/kernel/tracing # cat dynamic_events
p:kprobes/p_vfs_read_0 vfs_read arg1=+0(file):ustring arg2=%ax:x16
# arg1: ARG(0) -> ST_USTRING(offset=0,size=4) -> END
# arg2: REG(80) -> ST_RAW(size=2) -> END
Link: https://lore.kernel.org/all/178271359508.1176915.11895926894608440183.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
The commit 69efd863a785 ("tracing/eprobes: Allow use of BTF names
to dereference pointers") allows eprobe to use event field without
"$" prefix when it is used with typecast, it is natual to allow it
without typecast.
Link: https://lore.kernel.org/all/178271358117.1176915.13520279547897961254.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
We can pass NULL as id pointer to btf_type_skip_modifiers if we do not
need the id of the returned btf_type.
Link: https://lore.kernel.org/all/20260623132937.3494895-1-martin@kaiser.cx/
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Since FETCH_OP_DATA is used solely to store immediate string
values, rename it to the more specific FETCH_OP_IMMSTR.
No behavior change, just rename it.
Link: https://lore.kernel.org/all/178217906914.643090.2958799486090467704.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
The shared probe argument parser rejects file offsets for kernel probes.
This path is used outside the kprobe event parser too, but the diagnostic
currently says "with kprobe" even when emitted from another probe path.
Make the diagnostic probe-agnostic.
Link: https://lore.kernel.org/all/20260622160032.99834-1-pyudistira519@gmail.com/
Signed-off-by: Yudistira Putra <pyudistira519@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
The function name should be __register_tracepoint_fprobe.
Link: https://lore.kernel.org/all/20260507081041.885781-5-martin@kaiser.cx/
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
perf_event_remove_on_exec() removes events by calling
perf_event_exit_event(). For top-level events, this removes the event from
the context with DETACH_EXIT only.
This can leave inconsistent group state when a removed event is a group
leader and the group contains siblings without remove_on_exec. If the group
was active, the surviving siblings can remain active and attached to the
removed leader's sibling list, but are no longer represented by a valid
group leader on the PMU context active lists.
A later close of the removed leader uses DETACH_GROUP and can promote the
still-active siblings from this stale group state. The next schedule-in can
then add an already-linked active_list entry again, corrupting the PMU
context active list.
With DEBUG_LIST enabled, this is caught as a list_add double-add in
merge_sched_in().
Fix this by detaching group relationships when remove_on_exec removes an
event. This preserves the existing task-exit and revoke behavior, while
ensuring surviving siblings are ungrouped before the removed event leaves
the context.
Fixes: 2e498d0a74e5 ("perf: Add support for event removal on exec")
Signed-off-by: Taeyang Lee <0wn@theori.io>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/ai65GgZcC0LAlWLG@Taeyangs-MacBook-Pro.local
|
|
Load balancing can migrate delayed dequeue tasks to even the load between
CPUs.
sched_balance_rq()
-> detach_task()
-> deactivate_task(DEQUEUE_NOCLOCK)
-> set_task_cpu(dst_cpu)
-> attach_task()
-> activate_task(ENQUEUE_NOCLOCK)
-> wakeup_preempt()
A delayed task with shorter slice can be dequeued during pick_next_entity()
but then jump to preempt because eligible.
Always jump to update for delayed dequeue task even with shorter slice.
Fixes: ba0d3bf5f97b ("sched/eevdf: Cancel slice protection if short slice task is eligible")
Reported-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260701164920.1571352-1-vincent.guittot@linaro.org
|
|
The sub-scheduler implementation has grown and will continue to expand. Move
the sub-scheduler functions from ext.c into a new kernel/sched/ext/sub.c.
sub.h holds the prototypes and the !CONFIG_EXT_SUB_SCHED no-op stubs.
scx_dispatch_sched() is shared: balance_one() in ext.c and the
scx_bpf_sub_dispatch() kfunc in sub.c both call it, and the latter re-enters
it as sub-scheduler dispatch nests. It moves into sub.h as a static
__always_inline so both callers keep it inlined and per-level stack stays
bounded across the recursion. The event macros it uses move to internal.h.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The following trivial helpers in ext.c are called from both ext.c and the
sub-scheduler code. Define them as static inline in internal.h.
- scx_bypass_dsq()
- scx_bypass_dsp_enabled()
- scx_ops_sanitize_err()
- scx_schedule_reenq_local()
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The sub-scheduler implementation is about to move into its own sub.c, from
where it calls a set of ext.c helpers and shares a few ext.c globals. Make
those reachable across the new file boundary ahead of the move.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A later change moves the sub-scheduler implementation out of ext.c into its
own file, from where it calls a number of file-local ext.c helpers. Give
those helpers the scx_ prefix that cross-file sched_ext symbols carry, ahead
of the move so the mechanical rename stays out of the code-motion patch. No
functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
for-7.3 will reorganize sub-sched code. Pull in for-7.2-fixes in
preparation.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
__bpf_prog_put_rcu() is the call_rcu() callback for non-sleepable programs.
security_bpf_prog_free() called from there fires bpf_prog_free in softirq;
if a sleepable LSM prog is attached to that hook, might_fault() BUGs:
BUG: sleeping function called from invalid context
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 5038
preempt_count: 101, expected: 0
Call Trace:
<IRQ>
__bpf_prog_enter_sleepable+0x1cd/0x320 kernel/bpf/trampoline.c:1255
bpf_trampoline_6442549705+0x53/0xd7
security_bpf_prog_free+0xde/0x130 security/security.c:5465
__bpf_prog_put_rcu+0xab/0xd0 kernel/bpf/syscall.c:2365
rcu_do_batch kernel/rcu/tree.c:2617 [inline]
handle_softirqs+0x236/0x800 kernel/softirq.c:622
</IRQ>
The call_rcu/call_rcu_tasks_trace split reflects the freed program's
sleepability, not that of any attached observer.
security_bpf_prog_free() also frees prog->aux->security, which has to stay
after the grace period, so drop bpf_prog_free from sleepable_lsm_hooks
rather than move the call. Non-sleepable observers still run there.
Fixes: 1b67772e4e3f ("bpf,lsm: Refactor bpf_prog_alloc/bpf_prog_free LSM hooks")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260701080757.1394144-1-rhkrqnwk98@gmail.com
|
|
In bpf_do_misc_fixups(), the conversion from a function address to a
BPF immediate value is handled using the BPF_CALL_IMM macro inside the
'patch_map_ops_generic' label block. However, immediately following it
in the 'patch_call_imm' label block, the immediate value is calculated
manually by subtracting __bpf_call_base from fn->func.
Inspired by KaFai Wan's review comments on fixing helper call offsets,
use the BPF_CALL_IMM macro in 'patch_call_imm' as well to clean this up.
This removes the redundant manual pointer arithmetic and ensures coding
style consistency across adjacent label blocks within the same function.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260701035107.8069-1-yangtiezhu@loongson.cn
|
|
Adding BPF_OBJ_GET_INFO_BY_FD support for tracing_multi links.
We expose following tracing_multi link data:
- attach_type of the program
- number of ids
- array of BTF ids
- array of its related kernel addresses
- array of cookies
The change follows the kprobe_multi and uprobe_multi link-info convention
of optional output arrays with an in/out count,
On top of standard tracing link data we also expose addresses, because they
are useful info for user (especially when the attachment was done via pattern).
This data is hidden when kallsyms does not allow exposing kernel pointer values.
Assisted-by: Codex:GPT-5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260629212208.895962-2-jolsa@kernel.org
|
|
To identify the task most likely responsible for a stall, add
last_woken_worker (L: pool->lock) to worker_pool and record it in
kick_pool() just before wake_up_process(). This captures the idle
worker that was kicked to take over when the last running worker went to
sleep; if the pool is now stuck with no running worker, that task is the
prime suspect and its backtrace is dumped by show_pool_no_running_worker().
Using struct worker * rather than struct task_struct * avoids any
lifetime concern: workers are only destroyed via set_worker_dying()
which requires pool->lock, and set_worker_dying() clears
last_woken_worker when the dying worker matches.
show_cpu_pool_busy_workers() holds pool->lock while calling
sched_show_task(), so last_woken_worker is either NULL or points to a
live worker with a valid task. More precisely, set_worker_dying() clears
last_woken_worker before setting WORKER_DIE, so a non-NULL
last_woken_worker means the kthread has not yet exited and worker->task
is still alive.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is
intact.
2. if we're worried about relocating the kernel into a region used by
some DMA that wasn't properly cancelled.
If KHO is enabled then relocations will happen to KHO scratch, which
is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to
happen at all.
Therefore, we can safely disable checksum verification in both of those
cases.
Instead of adding a new variable to purgatory, just skip adding regions
and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the
live-update project.
Signed-off-by: Michal Clapinski <mclapinski@google.com>
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Acked-by: Baoquan He <baoquan.he@linux.dev>
Link: https://patch.msgid.link/20260602123311.1841746-1-mclapinski@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
The pack allocator only flushes predictors when reusing a dirty pack for
cBPF, eBPF allocations never trigger a flush. Currently, eBPF picks the
first free pack, which could be a clean pack. As an optimization, leaving
a clean pack for cBPF can avoid flushes.
Prefer dirty packs for eBPF and keep clean packs free for cBPF. This
mirrors the existing cBPF preference for clean packs: each program kind
prefers the pack that avoids an extra flush, and falls back to the other
kind only when no preferred pack has room. eBPF reuse of a dirty pack is
harmless since eBPF being privileged does not flush.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
Currently BPF pack allocator picks the chunks from the first available
pack. While this is okay, it naturally leads to more frequent flushes
when there are multiple packs in the system that weren't used since the
last flush.
As an optimization prefer allocating the new programs from packs that
are unused since last flush. When all packs are dirty, allocation forces
a flush and marks all packs clean.
Below are some future optimizations ideas:
1. Currently, the "dirty" tracking is only done at the pack-level.
Flush frequency can further be reduced with chunk-level tracking.
This requires a new bitmap per-pack to track the dirty state.
2. IBPB flush is done on all CPUs, even if only a single CPU ran the
BPF program. On a system with hundreds of CPUs this could be a
major bottleneck forcing hundreds of IPIs to deliver the flush.
The solution is to track the CPUs where a BPF program ran, and
issue IBPB only on those CPUs.
3. Avoid IBPB when flush is already done at other sources (e.g.
context switch).
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
bpf_prog_pack_alloc() issues IBPB on all CPUs on every cBPF allocation,
even when reusing chunks from an existing pack where no new memory was
touched since the last IBPB.
Since IBPB on all CPUs is heavy, Dave Hansen suggested to track allocation
since last IBPB, and only issue IBPB at reuse for the chunks that have not
seen an IBPB since they were last freed.
Track per-pack whether an IBPB is needed via arch_flush_needed. Set it when
allocating a chunk, reset on IBPB flush. On reuse, conditionally issue the
flush. Since IBPB invalidates all BTB entries, clear the flag on all packs
after flushing.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
Currently predictor flush on memory reuse is done for all BPF JIT
allocations, but only cBPF programs can be loaded by an unprivileged user.
eBPF is privileged by default, and flushing predictors for all CPUs on
every eBPF reuse penalizes the common case for no security benefit.
eBPF allocations can be frequent on busy systems, only flush predictors
for cBPF programs. Trampoline and dispatcher allocations also skip the
flush as they are eBPF-only.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
The BPF JIT allocator packs many small programs into larger executable
allocations and reuses space within those allocations as programs are
loaded and freed. When fresh code is written into space that a previous
program occupied, an indirect jump into the new program can reuse a branch
prediction left behind by the old one.
Flush the indirect branch predictors before reusing JIT memory so that
indirect jumps into a newly written program don't reuse predictions from an
old program that occupied the same space.
Introduce bpf_arch_pred_flush_enabled static key and bpf_arch_pred_flush
static call for flushing the branch predictors on JIT memory reuse.
Architectures that need a flush, can update it to a predictor flush
function. By default, its a NOP and does not emit any CALL.
Allocations larger than a pack are not covered by this flush. That is safe
because cBPF programs (the unprivileged attack surface) are bounded well
below a pack size. Issue a warning if this assumption is ever violated
while the flush is active.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fixes from Masami Hiramatsu:
"fprobe fixes and spelling typos:
- Fix NULL pointer dereference in fprobe_fgraph_entry(). Prevent
general protection faults by checking shadow-stack reservation
bounds. Skip mid-flight registered fprobes that were not counted
during sizing.
eprobe: fix string pointer extraction
- Correct the casting of string pointers read from the ringbuffer to
prevent truncation of base event pointer variables when
dereferencing FILTER_PTR_STRING fields.
tracing/probes: clean up argument parsing and BTF helper logic
- Make the $ prefix mandatory for comm access: Require the $ prefix
for special fetcharg variables like $comm and $COMM, preventing
naming conflicts with regular BTF-based event fields.
- Fix double addition of offset for @+FOFFSET: Clear the temporary
offset variable after setting the FETCH_OP_FOFFS instruction to
avoid applying the offset multiple times.
- Remove WARN_ON_ONCE from parse_btf_arg: Prevent triggering a kernel
warning via user-space input when creating a kprobe event on a raw
address.
- Fix typo in a log message: Correct a spelling error ("$-valiable")
in trace probe log messages.
samples/trace_events: improve error checking
- Validate the thread pointer returned from kthread_run() in the
trace events sample code to properly handle thread creation
failures"
* tag 'probes-fixes-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing/probes: Make the $ prefix mandatory for comm access
tracing/fprobe: Fix NULL pointer dereference in fprobe_fgraph_entry()
tracing/probes: Fix double addition of offset for @+FOFFSET
tracing: eprobe: read the complete FILTER_PTR_STRING pointer
tracing/events: Fix to check the simple_tsk_fn creation
tracing/probes: Remove WARN_ON_ONCE from parse_btf_arg
tracing: probes: fix typo in a log message
|
|
Export irq_can_set_affinity() for loadable drivers that need a runtime
check for IRQ affinity capability.
In hierarchical IRQ setups where the effective irqchip path lacks
.irq_set_affinity(), drivers may need to switch to a fallback policy.
Without this export, module drivers cannot use the core helper and have
to open-code equivalent checks.
Signed-off-by: Hangtian Zhu <hangtian.zhu@oss.qualcomm.com>
Acked-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260519011627.713068-2-hangtian.zhu@oss.qualcomm.com
Signed-off-by: Jeff Johnson <jeff.johnson@oss.qualcomm.com>
|
|
Multiple readers access audit_queue.qlen via skb_queue_len() without
holding the queue lock or using READ_ONCE(), while kauditd writes to
this field via the skb_dequeue() → __skb_unlink() path with WRITE_ONCE()
protected by a spinlock. This constitutes data races.
All affected skb_queue_len(&audit_queue) call sites:
- kauditd_thread() wait_event_freezable() condition
- audit_receive_msg() AUDIT_GET handler (s.backlog assignment)
- audit_receive() backlog check
- audit_log_start() backlog check and pr_warn()
KCSAN reports the following conflicting access pattern (one example):
==================================================================
BUG: KCSAN: data-race in audit_log_start / skb_dequeue
write (marked) to 0xffffffff8512ee20 of 4 bytes by task 661 on cpu 57:
skb_dequeue+0x70/0xf0
kauditd_send_queue+0x71/0x220
kauditd_thread+0x1cb/0x430
kthread+0x1c2/0x210
ret_from_fork+0x162/0x1a0
ret_from_fork_asm+0x1a/0x30
read to 0xffffffff8512ee20 of 4 bytes by task 36586 on cpu 1:
audit_log_start+0x2a0/0x6b0
audit_core_dumps+0x64/0xa0
do_coredump+0x14b/0x1260
get_signal+0xeb2/0xf70
arch_do_signal_or_restart+0x41/0x170
exit_to_user_mode_loop+0xa2/0x1c0
do_syscall_64+0x1a3/0x1c0
entry_SYSCALL_64_after_hwframe+0x76/0xe0
value changed: 0x00000001 -> 0x00000000
==================================================================
Resolve the race by switching to lockless helper skb_queue_len_lockless(),
which internally uses READ_ONCE() and properly pairs with the WRITE_ONCE()
write accesses already present on the writer side.
Cc: stable@vger.kernel.org
Fixes: 3197542482df ("audit: rework audit_log_start()")
Signed-off-by: Chi Wang <wangchi@kylinos.cn>
Reviewed-by: Ricardo Robaina <rrobaina@redhat.com>
[PM: line length tweak]
Signed-off-by: Paul Moore <paul@paul-moore.com>
|
|
Fix checkpatch "Macros with complex values should be enclosed in
parentheses" error.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
clock_base_next_timer_safe() was added by commit a64ad57e41c7 ("hrtimer:
Simplify run_hrtimer_queues()") but has never had a caller; the queue
iteration in __hrtimer_run_queues() uses clock_base_next_timer() instead.
The two are functionally equivalent: struct hrtimer embeds the timerqueue
node at offset 0, so container_of() of a NULL node yields NULL. Thus
clock_base_next_timer() already returns NULL on an empty queue and the
explicit NULL check in the _safe variant is redundant.
Being a static __always_inline function it does not trigger
-Wunused-function, so the dead code has gone unnoticed. Remove it.
No functional change.
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260625140901.929554-1-zhanxusheng@xiaomi.com
|
|
When a CPU pool is stalled with no running worker, the task occupying the
CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace
for the stalled CPU to capture what it is currently executing.
The CPU is snapshotted under pool->lock and the backtrace is triggered
after releasing the lock to avoid any potential issues with NMI delivery.
Skip the backtrace when the CPU is offline. A pool disassociated by CPU
hotplug keeps its pool->cpu, and an NMI to an offline CPU is never acked,
so nmi_trigger_cpumask_backtrace() would busy-wait for its full timeout
in the watchdog's timer context.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
show_cpu_pool_busy_workers() dumps every in-flight worker in the pool's
busy_hash, including workers that are not currently running on the CPU.
Restore the task_is_running() filter so only running workers are dumped.
When no running worker is found the pool may be stuck, unable to wake an
idle worker to process pending work, and the watchdog would otherwise
give no feedback. Add show_pool_no_running_worker() to report the pool
id, CPU, idle state, and worker counts in that case.
The pool info message is printed inside pool->lock using
printk_deferred_enter/exit, the same pattern used by the existing
busy-worker loop, to avoid deadlocks with console drivers that queue
work while holding locks also taken in their write paths.
This has been running on the Meta fleet for a while and caught some real
issues, for instance EFI stalls stalling the workqueue [1].
Link: https://lore.kernel.org/all/20260616-efi_timeout-v3-0-76dd1d26657b@debian.org/ [1]
Suggested-by: Petr Mladek <pmladek@suse.com>
Fixes: 8823eaef45da7 ("workqueue: Show all busy workers in stall diagnostics")
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The crash memory exclude of crashk_res and crashk_cma memory on powerpc
are almost identical to the generic crash_exclude_core_ranges().
By introducing the architecture-specific arch_crash_exclude_mem_range()
function with a default implementation of crash_exclude_mem_range(),
and using crash_exclude_mem_range_guarded as powerpc's separate
implementation, the generic crash_exclude_core_ranges() helper function
can be reused.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Hari Bathini <hbathini@linux.ibm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shivang Upadhyay <shivangu@linux.ibm.com>
Acked-by: Breno leitao <leitao@debian.org>
Acked-by: Baoquan He <bhe@redhat.com>
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260629094746.191843-9-ruanjinjie@huawei.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
The crash memory alloc, and the exclude of crashk_res, crashk_low_res
and crashk_cma memory are almost identical across different architectures,
handling them in the crash core would eliminate a lot of duplication, so
add crash_prepare_headers() helper to handle them in the common code.
To achieve the above goal, three architecture-specific functions are
introduced:
- arch_get_system_nr_ranges(). Pre-counts the max number of memory ranges.
- arch_crash_populate_cmem(). Collects the memory ranges and fills them
into cmem.
- arch_crash_exclude_ranges(). Architecture's additional crash memory
ranges exclusion, defaulting to empty.
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260629094746.191843-4-ruanjinjie@huawei.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
Since $comm or $COMM are not event field but special fetcharg
variables to access current->comm, It should not be accessed
without '$' prefix even with typecast.
Link: https://lore.kernel.org/all/178231209724.732967.12049805699091810641.stgit@devnote2/
Fixes: 69efd863a785 ("tracing/eprobes: Allow use of BTF names to dereference pointers")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
fprobe_fgraph_entry() sizes a shadow-stack reservation in one walk of
the per-ip fprobe list and fills it in a second walk, both under
rcu_read_lock() only. A fprobe registered on an already-live ip can
become visible between the two walks, so the fill walk processes an
exit_handler the sizing walk did not count and used runs past
reserved_words. If the sizing walk counted nothing, fgraph_data is NULL
and the first write_fprobe_header() faults:
Oops: general protection fault, probably for non-canonical address ...
KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007]
RIP: 0010:fprobe_fgraph_entry+0xa38/0xf10 kernel/trace/fprobe.c:167
Call Trace:
<TASK>
function_graph_enter_regs+0x44c/0xa10 kernel/trace/fgraph.c:677
ftrace_graph_func+0xc5/0x140 arch/x86/kernel/ftrace.c:671
__kernel_text_address+0x9/0x40 kernel/extable.c:78
arch_stack_walk+0x117/0x170 arch/x86/kernel/stacktrace.c:26
kmem_cache_free+0x188/0x580 mm/slub.c:6378
tcp_data_queue+0x18d/0x6550 net/ipv4/tcp_input.c:5590
[...]
</TASK>
The list cannot be frozen across the two walks, so skip a node that does
not fit the reservation and count it as missed.
Link: https://lore.kernel.org/all/20260619184425.3824774-1-rhkrqnwk98@gmail.com/
Fixes: 4346ba160409 ("fprobe: Rewrite fprobe on function-graph tracer")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Since commit 533059281ee5 ("tracing: probeevent: Introduce new argument
fetching code") wrongly use @offset local variable during the parsing,
the offset value is added twice when dereferencing.
Reset the @offset after setting it in FETCH_OP_FOFFS.
Link: https://lore.kernel.org/all/178217905962.643090.1978577464942171332.stgit@devnote2/
Fixes: 533059281ee5 ("tracing: probeevent: Introduce new argument fetching code")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Cc: stable@vger.kernel.org
|
|
For a char * element in an event, the FILTER_PTR_STRING filter type is
used. When the event occurs, a pointer is stored in the ringbuffer.
If an eprobe references such a char * element of a "base event", the
stored pointer is truncated when it's read from the ringbuffer.
$ cd /sys/kernel/tracing
$ echo 'e rcu.rcu_utilization $s:x64 $s:string' > dynamic_events
$ echo 1 > tracing_on
$ echo 1 > events/eprobes/enable
$ sleep 1
$ echo 0 > events/eprobes/enable
$ cat trace
<idle>-0 ...: (rcu.rcu_utilization) arg1=0x4f arg2=(fault)
<idle>-0 ...: (rcu.rcu_utilization) arg1=0x2 arg2=(fault)
The problem is in get_event_field
val = (unsigned long)(*(char *)addr);
addr points to the position in the ringbuffer where the pointer was
stored. The assignment reads only the lowest byte of the pointer.
Fix the cast to read the whole pointer. The output of the test above
is now
<idle>-0 ... arg1=0xffffffff81c7d3f3 arg2="Start scheduler-tick"
<idle>-0 ... arg1=0xffffffff81c57340 arg2="End scheduler-tick"
Link: https://lore.kernel.org/all/20260620145339.3234726-1-martin@kaiser.cx/
Fixes: f04dec93466a ("tracing/eprobes: Fix reading of string fields")
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Sashiko found that user can cause this WARN_ON_ONCE() easily
with adding a kprobe event based on a raw address with BTF
parameter.
Since this is not an unexpected condition, remove the
WARN_ON_ONCE().
Link: https://lore.kernel.org/all/178177265367.2059927.13789953014706792126.stgit@mhiramat.tok.corp.google.com/
Link: https://sashiko.dev/#/patchset/178165816303.269421.7302603996990753309.stgit%40devnote2
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: b576e09701c7 ("tracing/probes: Support function parameters if BTF is available")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
When CONFIG_BPF_LSM=y is set, BPF inode storage maps
(BPF_MAP_TYPE_INODE_STORAGE) are compiled into the kernel. However,
if the BPF LSM is not explicitly enabled at boot time (e.g. omitted
from the "lsm=" boot parameter), lsm_prepare() is never executed for
the BPF LSM.
Consequently, the BPF inode security blob offset
(bpf_lsm_blob_sizes.lbs_inode) is never initialized and remains at
its default compiled size of 8 bytes instead of being updated to a
valid offset past the reserved struct rcu_head (typically 16 bytes
or more).
When a privileged user creates and updates a BPF_MAP_TYPE_INODE_STORAGE
map, bpf_inode() evaluates inode->i_security + 8. This erroneously
aliases the struct rcu_head.func callback pointer at the beginning
of the inode->i_security blob. During subsequent map element cleanup
or inode destruction, writing NULL to owner_storage clears the queued
RCU callback pointer. When rcu_do_batch() later executes the queued
callback, it attempts an instruction fetch at address 0x0, triggering
an immediate kernel panic.
Fix this by introducing a global bpf_lsm_initialized boolean flag
marked with __ro_after_init. Set this flag to true inside bpf_lsm_init()
when the LSM framework successfully registers the BPF LSM. Gate map
allocation in inode_storage_map_alloc() on this flag, returning
-EOPNOTSUPP if the BPF LSM is in turn uninitialized.
This fail-fast approach prevents userspace from allocating inode
storage maps when the supporting BPF LSM infrastructure is absent,
avoiding zombie map states.
Fixes: 8ea636848aca ("bpf: Implement bpf_local_storage for inodes")
Reported-by: oxsignal <awo@kakao.com>
Signed-off-by: Matt Bobrowski <mattbobrowski@google.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260628201103.3624525-1-mattbobrowski@google.com
|
|
Three sites enumerate every scx_event_stats field by name:
scx_read_events(), scx_attr_events_show(), and scx_dump_state().
Adding a new SCX_EV_* event requires three separate manual edits.
Missing any one silently drops counts or omits the field from
diagnostic output with no compile-time indication.
SCX_EVENTS_LIST(SCX_EVENT) in internal.h becomes the single
authoritative list of event names, so all three sites stay in sync.
No functional changes.
Signed-off-by: Cheng-Yang Chou <yphbchou0911@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
put_prev_task_scx() warns when a runnable task drops to a lower sched_class
without SCX_OPS_ENQ_LAST, on the assumption that balance_one() would have
kept it running. Core scheduling breaks that: a forced-idle SMT sibling
reschedules through the core_pick fast path in pick_next_task(), which skips
pick_task_scx() and thus balance_one(), so a runnable task can drop to idle
with ENQ_LAST unset.
Gate the warning on sched_cpu_cookie_match(): a cookie mismatch means core
scheduling forced the idle, while a match (or core scheduling off) still
catches a genuine missing-ENQ_LAST drop.
Fixes: 7c65ae81ea86 ("sched_ext: Don't call put_prev_task_scx() before picking the next task")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A few comments still describe the pre-EEVDF CFS world:
- sysctl_sched_base_slice is documented as "Minimal preemption
granularity for CPU-bound tasks". That was the wording of the old
sysctl_sched_min_granularity, renamed in commit e4ec3318a17f
("sched/debug: Rename sysctl_sched_min_granularity to
sysctl_sched_base_slice"). Under EEVDF it is the default base time
slice / request size (r_i) used to compute the virtual deadline, as
documented in update_deadline().
- Two comments still mention sched_slice(), which was removed when the
fair class committed to EEVDF in commit 5e963f2bd465 ("sched/fair:
Commit to EEVDF"). The dequeue-path comment should simply refer to
the task's slice (se->slice); the forced-idle comment describes the
slice accounting now performed by __entity_slice_used(), which is the
function actually used right below it.
No functional changes.
[ mingo: Ported to a more recent scheduler base ]
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/20260629030200.3165589-1-zhanxusheng@xiaomi.com
|
|
perf_event_open() currently gates perf_allow_kernel() only on
!attr.exclude_kernel.
However, users can still request kernel callchain collection with
attr.exclude_callchain_kernel == 0 even when attr.exclude_kernel == 1.
That still requires kernel profiling privilege, but the existing check
does not enforce it.
Update the permission check to call perf_allow_kernel() when either
kernel sampling is requested or kernel callchains are requested.
This keeps permission checks aligned with requested data and prevents
unprivileged use of kernel callchain capture.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260616044654.3468742-9-dapeng1.mi@linux.intel.com
|
|
An unprivileged hardware perf event using exclude_kernel=1 can leak kernel
register data to user space via PERF_SAMPLE_REGS_INTR or PERF_SAMPLE_IP.
Due to hardware skid, a PMI may trigger after the CPU has already entered
kernel space (Ring 0), bypassing the perf_allow_kernel() privilege
barrier.
This security vulnerability is severely exacerbated by upcoming support
for SIMD register sampling via XSAVES, which could expose sensitive kernel
FPU states (such as active cryptographic keys).
Fix this by ensuring that sampled register data is dropped if the event's
exclude_kernel attribute is set but the PMI catches the CPU in kernel mode.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/all/20260529085613.CCAFB1F00893@smtp.kernel.org/
Link: https://patch.msgid.link/20260616044654.3468742-8-dapeng1.mi@linux.intel.com
|
|
nr_migrations_cold, nr_wakeups_passive and nr_wakeups_idle are not
being updated anywhere. So remove them.
These are per process stats. So updating sched stats version isn't
necessary.
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260625124648.802832-2-sshegde@linux.ibm.com
|
|
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative. A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clock
So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window. Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change. The existing behaviour for
delta > 0 is untouched.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Link: https://patch.msgid.link/20260617175219.2494857-2-usama.arif@linux.dev
|
|
Reflow to reduce indenting.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260618105627.GP49951@noisy.programming.kicks-ass.net
|
|
Because active_balance is initialized to 0, and need_active_balance()
is a pre-condition for setting it to 1, the condition '!active_balance
|| need_active_balance()' is a truism and can be removed.
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260617072151.1173416-3-jackzxcui1989@163.com
|
|
Active load balancing relies on migration threads, which temporarily preempt
tasks on the source runqueue (src_rq). This preemption can negatively impact
overall system performance. The active balancing logic includes a check to
verify whether the current task (curr) on src_rq can actually run on the
destination runqueue (dst_rq). We have observed that when curr is a CFS task
and its on_rq flag is 0, the active balancing failure rate is exceptionally
high. The following table summarizes test data collected over 300 seconds on an
18-CPU platform under a specific fillback task scenario:
fair: busiest->curr->sched_class == &fair_sched_class
on_rq: busiest->curr->on_rq
total: active balance count triggered of correspondent type
fail: fail to migrate one task in active_load_balance_cpu_stop()
fair && !on_rq !fair && !on_rq
domain total fail total fail
cpu0 0x00003 0 0 0 0
cpu0 0x3ffff 33 33 1 1
cpu1 0x00003 0 0 0 0
cpu1 0x3ffff 42 42 0 0
cpu2 0x0003c 4 4 0 0
cpu2 0x3ffff 12 12 0 0
cpu3 0x0003c 3 3 0 0
cpu3 0x3ffff 8 7 0 0
cpu4 0x0003c 2 2 0 0
cpu4 0x3ffff 5 4 0 0
cpu5 0x0003c 4 4 0 0
cpu5 0x3ffff 8 8 0 0
cpu6 0x003c0 60 60 0 0
cpu6 0x3ffff 28 27 0 0
cpu7 0x003c0 194 184 0 0
cpu7 0x3ffff 35 35 1 1
cpu8 0x003c0 240 228 0 0
cpu8 0x3ffff 28 28 0 0
cpu9 0x003c0 0 0 0 0
cpu9 0x3ffff 10 10 0 0
cpu10 0x03c00 52 50 0 0
cpu10 0x3ffff 0 0 0 0
cpu11 0x03c00 70 68 0 0
cpu11 0x3ffff 1 1 0 0
cpu12 0x03c00 73 72 0 0
cpu12 0x3ffff 0 0 0 0
cpu13 0x03c00 79 76 0 0
cpu13 0x3ffff 0 0 0 0
cpu14 0x3c000 0 0 0 0
cpu14 0x3ffff 57 55 1 0
cpu15 0x3c000 53 52 1 0
cpu15 0x3ffff 30 29 0 0
cpu16 0x3c000 344 341 10 6
cpu16 0x3ffff 103 100 2 1
cpu17 0x3c000 183 179 2 2
cpu17 0x3ffff 78 77 0 0
sum 1839 1791 18 11
In __schedule(), before curr is updated to next, pick_next_task() invokes
sched_balance_rq(). This function temporarily unlocks and relocks the runqueue,
creating a window where other CPUs may observe rq->curr->on_rq as 0.
We can safely skip active balancing when src_rq->curr->on_rq == 0, as other
eligible tasks have likely already been evaluated.
We retain the affinity check on dst_rq to trigger active balancing, since such
tasks are often woken by (or wake up) tasks on src_rq that share similar
affinity constraints. Furthermore, detach_tasks() releases the runqueue lock;
any tasks awakened during this window may preempt the previous CFS task. My
testing (data not shown) indicates that active balancing succeeds in 98.4% of
cases where !fair && on_rq.
This scenario does not require a stop-work callback, but would necessitate an
additional detach/attach path. As Valentin and Vincent have already discussed,
this addition does not appear justified at this time (see [1]).
Since can_migrate_task() already checks on_cpu during the cfs_tasks traversal,
adding an on_rq check will have negligible performance overhead due to cache
locality.
There are two reasons for not combining the on_rq check with the
cpumask_test_cpu() check:
- Avoiding new scenarios that would skip the logic for resetting
balance_interval to min_interval.
- The existing check for whether the busiest CPU recently triggered active
load balancing already filters more cases than the on_rq check.
[1]: https://lore.kernel.org/lkml/20190815145107.5318-5-valentin.schneider@arm.com/
Signed-off-by: Xin Zhao <jackzxcui1989@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Link: https://patch.msgid.link/20260617072151.1173416-2-jackzxcui1989@163.com
|
|
When a task with a shorter slice is enqueued, we protect the running
task which has a longer slice until it becomes ineligible instead of a
full slice in order to speedup the switch to other tasks until the task
with the shortest slice is scheduled. This helps to the task to not wait
too many full slices before running.
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260624151229.1710703-7-vincent.guittot@linaro.org
|
