| Age | Commit message (Collapse) | Author |
|---|
|
MAX_COMMON_HEAD_LEN (63) was used to allocate a temporary buffer for
formatting command heads in trace_kprobe_match_command_head() and
trace_uprobe_match_command_head(). However, the buffer size is too
short for some longer symbols. Especially, with rust code, the symbol
can be mangled and become very long.
Refactor trace_kprobe_match_command_head() to perform direct string
comparisons using strcmp() and strncmp(), eliminating the need for a
temporary buffer and removing the MAX_COMMON_HEAD_LEN string length
restriction on probe symbol names.
For trace_uprobe_match_command_head(), since tu->filename is already
matched via strncmp(), use a fixed 64-byte stack buffer solely for
formatting offset and ref_ctr_offset (which requires at most 39 bytes).
With all users converted, remove the MAX_COMMON_HEAD_LEN definition from
trace_probe.h.
Link: https://lore.kernel.org/all/178521361102.34226.9650586522488974115.stgit@devnote2/
Reported-by: Zhan Xusheng <zhanxusheng1024@gmail.com>
Link: https://lore.kernel.org/all/20260724023317.624074-1-zhanxusheng@xiaomi.com/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
In preparation for removing the strlcat API[1],
replace the string concatenation logic with a struct seq_buf,
which tracks the current position and the remaining space internally.
Use seq_buf_str() to NUL-terminate before passing to early_enable_events().
Link: https://github.com/KSPP/linux/issues/370 [1]
Link: https://patch.msgid.link/20260713045249.69942-1-woradorn.laon@gmail.com
Signed-off-by: Woradorn Laodhanadhaworn <woradorn.laon@gmail.com>
[ Moved placement of #include <linux/seq_buf.h> ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Replace strcpy() with strscpy() in __trace_find_cmdline() for
consistency with the existing strscpy() call in the same function,
and to avoid potential buffer overflow as flagged by the Kernel
Self Protection Project.
Link: https://patch.msgid.link/20260705173648.5418-1-posheng.lin.tw@gmail.com
Signed-off-by: Po-Sheng Lin <posheng.lin.tw@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Currently on boot up and when modules are loaded, the trace event
infrastructure will examine the TP_printk's of every event looking to see
if it dereferences pointers on the ring buffer via printk formats like
"%pB" and such. What it doesn't do is check if the arguments themselves
do a dereference from a pointer.
This was brought with a fix[1] to the fsl_edma event that had in the
arguments of the TP_printk(): "__entry->edma->membase"
The __entry->edma is a pointer saved in the ring buffer. The dereference
from TP_printk() happens when the user reads the "trace" file which can be
seconds, minutes, hours, days, weeks, or even months later! There is no
guarantee that the __entry->edma pointer will still be pointing to what it
was when it was recorded, and could crash the kernel when a user reads the
event.
Add logic to the test_event_printk() that also checks for this case and
warn if the event dereferences a pointer from the ring buffer.
[1] https://lore.kernel.org/all/20260630200022.1826420-1-martin@kaiser.cx/
Link: https://patch.msgid.link/20260630184836.74d477b6@gandalf.local.home
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Reviewed-by: Martin Kaiser <martin@kaiser.cx>
Reviewed-by: Vinod Koul <vkoul@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
preemptirq_delay_test accepts cpu_affinity as a module parameter and,
when it is non-negative, writes that CPU directly into a temporary
cpumask from the worker thread. Values outside nr_cpu_ids can set a bit
outside the allocated cpumask before the test reports a normal affinity
error.
Validate the requested CPU in preemptirq_delay_run() before setting it
in the temporary cpumask. Invalid affinity requests are reported by
the test thread and skipped before cpumask_set_cpu() can touch an
out-of-range bit.
Link: https://patch.msgid.link/20260628131021.2208632.6a5c6c959813.preemptirq-delay-test-invalid-cpu-affinity@trailofbits.com
Assisted-by: Codex:gpt-5.5-cyber-preview
Signed-off-by: Samuel Moelius <sam.moelius@trailofbits.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A single closing curly bracket should be put into a trace sequence buffer.
Thus use the corresponding function “trace_seq_putc”.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/d215fa89-9a62-4067-86ec-833290f35c80@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
user_seq_show()
A single string should be put into a sequence within a loop.
Thus use the corresponding function “seq_puts” for one selected call.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/1cf327f0-49a6-477f-a06f-2b22a167db24@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Single characters should be put into a sequence. Thus use the corresponding
function “seq_putc” for selected calls.
The source code was transformed by using the Coccinelle software.
Link: https://patch.msgid.link/6bcaa4da-05c6-4097-90f5-3969f8a1dfbc@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
expr_str() allocates a fixed MAX_FILTER_STR_VAL buffer and then builds
expression names with a series of raw strcat() appends. Nested operands,
constants, field flags, and generated field names can push the rendered
string past that fixed limit before the name is attached to the hist
field.
Build expression strings with seq_buf and return -E2BIG when the
rendered name would exceed MAX_FILTER_STR_VAL. This keeps the existing
tracing-side limit while replacing the raw append logic with bounded
construction.
Link: https://patch.msgid.link/20260611055945.22348-4-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
expr_str() currently reports all failure cases as NULL, so callers cannot
distinguish invalid recursion depth from allocation failure or later
string construction errors.
Return ERR_PTR()-encoded errors from expr_str() and make parse_unary()
and parse_expr() propagate them. Clear expr->name before destroying the
hist field so the error pointer is not freed as a string.
Link: https://patch.msgid.link/20260611055945.22348-3-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
expr_str() allocates a temporary expression buffer and manually frees it
on some error paths.
Convert the buffer to __free(kfree) and return it with return_ptr() on
success. This keeps ownership handling separate from the later ERR_PTR()
conversion and string-bound change.
Link: https://patch.msgid.link/20260611055945.22348-2-pengpeng@iscas.ac.cn
Signed-off-by: Pengpeng Hou <pengpeng@iscas.ac.cn>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Link: https://patch.msgid.link/20260606202633.5018-34-david.laight.linux@gmail.com
Signed-off-by: David Laight <david.laight.linux@gmail.com>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The HIST_FIELD_FL_CONST path uses the fixed "u64" type string.
Point hist_field->type directly to the string literal, matching the
HIST_FIELD_FL_HITCOUNT path. The release path already uses kfree_const(),
so no duplication is needed.
Link: https://patch.msgid.link/20260527023450.2137639-1-pengyu@kylinos.cn
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Add events/<sys>/<event>/btf_ids, a per-template file that exposes
the BTF ids resolve_btfids fills in for each tracepoint:
btf_obj_id BTF object owning the ids below
raw_btf_id FUNC_PROTO of __bpf_trace_<call> (named args), consumed
by raw_tp / tp_btf BPF programs
tp_btf_id trace_event_raw_<call> ring-buffer record, consumed by
classic BPF_PROG_TYPE_TRACEPOINT programs
DECLARE_EVENT_CLASS now emits a 2-entry BTF_ID_LIST (FUNC __bpf_trace_*
and STRUCT trace_event_raw_*) and stores the pointer in
trace_event_class.
Per-syscall events under syscalls/ share the handcrafted classes
event_class_syscall_{enter,exit} instead of going through
DECLARE_EVENT_CLASS. Wire those classes to the BTF id lists
generated for sys_enter / sys_exit so all ~700 per-syscall
events expose the shared dispatcher prototype and record.
The per-syscall events do not own their own tracepoint
(they share sys_enter/sys_exit), so raw_btf_id is reported as 0
on those events; the meaningful raw_btf_id is exposed on
raw_syscalls/sys_{enter,exit}/btf_ids where raw_tp / tp_btf
programs can actually attach.
Link: https://patch.msgid.link/20260518-generic_tracepoint-v2-2-b755a5cf67bb@meta.com
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Drop the static qualifier and add prototypes to <linux/btf.h> so the
tracing core can look up module BTF and translate ids stored by
resolve_btfids (which are local to a module's split BTF) into the
runtime ids used by the kernel.
Used by the upcoming events/<sys>/<event>/btf_ids tracefs interface.
Link: https://patch.msgid.link/20260518-generic_tracepoint-v2-1-b755a5cf67bb@meta.com
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next
Johannes Berg says:
====================
wireless-next-2026-07-26
Mostly driver changes this time:
- new driver mm81x for an S1G device
- new driver nxpwifi for NXP devices
(mostly forked off from mwifiex)
- ath12k: much kernel infrastructure integration work
- brcmfmac: DPP support, some Cypress part update
- nl80211: per-link statistics support
====================
Link: https://patch.msgid.link/20260726105205.942922-60-johannes@sipsolutions.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
There is no particular reason to keep bpf_res_spin_lock() disabled in
tracing programs, since it is safe against reentrancy and deadlocks.
Remove the restriction for tracing programs covered by the predicate
is_tracing_prog_type().
This is a prerequisite before the definition of is_tracing_prog_type()
is updated to include raw_tp, fentry, fexit, and fmod_ret. Existing
tracing programs will be updated to use bpf_res_spin_lock() instead when
it is available.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260719113551.1294284-2-memxor@gmail.com
|
|
scx_link_sched() carries each failure out of the locked section through
err_msg and ret because scx_error() used to take scx_sched_lock and couldn't
be called under it. That restriction is gone, so report each failure at the
site it's detected and return directly. The scx_error() here claims the exit
on the sched being linked, which has no descendants yet, and the locked
propagation walk is deferred, so nothing reacquires scx_sched_lock inline.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_hardlockup() defers the abort to an irq_work because exit claiming used
to take scx_sched_lock and couldn't run from NMI. The deferral is now
unnecessary - claiming is NMI-safe and asserting ->aborting is exactly what
breaks the live-locks that hard-lock CPUs. Call handle_lockup() directly and
drop the irq_work. This also makes the self-detected case recoverable: the
perf watchdog fires on the hard-locked CPU itself, where a queued irq_work
never runs with IRQs off.
Also fix the return value: %true used to be returned whenever sched_ext was
loaded, suppressing the kernel's hardlockup report even when the abort was
refused. Return %true only when this call initiated the abort.
Fixes: bd2d76455b65 ("sched_ext: Defer scx_hardlockup() out of NMI")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The per-cpu kick lists are protected by IRQ masking which doesn't stop NMIs,
so scx_bpf_kick_cpu() from NMI silently drops the kick after a one-time
warning. A dropped kick can leave a CPU idle when the scheduler believes it
was woken, which is a correctness problem for the scheduler even if the
kernel is fine. Now that scx_error() works from NMI, abort the scheduler
instead so that the bug is surfaced deterministically. The warned_nmi_kick
tracking is no longer needed.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The bstr exit kfuncs format the message into a shared static buffer under a
raw spinlock before initiating the exit. The lock can't be taken from NMI
and needlessly serializes all bstr exits system-wide.
Now that exit claiming is lock-free, reverse the order: claim the exit first
and format directly into the exit_info message buffer which the claim winner
owns exclusively. The new scx_exit_bstr() implements the sequence, replacing
scx_bstr_format(), and the shared buffer and lock are deleted; the formatter
itself is what bpf_trace_printk() already runs from NMI. scx_prog_sched()
callers were relying on the lock for RCU protection, which is now provided
explicitly.
A malformed format no longer changes or fails the requested operation:
scx_bpf_exit_bstr() keeps its graceful exit kind and scx_bpf_sub_kill_bstr()
still kills the child, with a fallback message carrying the formatting
errno, while the sched that supplied the bad format is aborted for its bug.
Before this and the previous patch, an "any" category kfunc called from NMI
context could trigger scx_error() and deadlock - e.g. a tracing prog
attached to a function running in NMI calling scx_bpf_dsq_peek() on a
non-existent DSQ would try to grab scx_sched_lock, which may be held by the
interrupted CPU. This and the previous patch fix the deadlock: scx_error()
and the bstr exit kfuncs, and thus scx_bpf_error() and scx_bpf_exit(), are
now safe to call from any context including NMI.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_claim_exit() claims descendants' exits by walking the subtree under
scx_sched_lock, making exit claiming, and thus scx_error(), unusable from
NMI and from under scx_sched_lock. However, kfuncs raising errors can run
from NMI-attached BPF progs, the hardlockup handler runs in NMI, and
scx_link_sched() wants to report failures under the lock.
The walk does two things with different urgencies: ->aborting must be
asserted synchronously to break IRQs-off dispatch-path live-locks, while the
descendants' exit_kind claims can happen later. Split them: sweep ->aborting
locklessly under RCU to unwedge the system and defer the locked
SCX_EXIT_PARENT walk to a new irq_work, both of which are NMI-safe.
The sweep stores each node's ->aborting and then reads its children list
while scx_link_sched() inserts and then checks the parent's ->aborting, the
two sides paired by full barriers - one side always sees the other. A link
that sees ->aborting undoes its insert and fails. As the undo's
list_del_rcu() leaves ->sibling non-empty, list_empty() can no longer
identify a never-linked sched during teardown - add sch->linked instead.
trace_sched_ext_exit can now fire from NMI and is called after the
->aborting stores so that its callbacks don't hold up live-lock recovery.
The exit backtrace is skipped for NMI exits as stack_trace_save()'s
NMI-safety is arch-dependent and undocumented.
v2: Move trace_sched_ext_exit() after the ->aborting stores (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
New HW no longer implements a separate class for latency-sensitive QPs
and advertises this by a new cap bit.
Signed-off-by: Leon Romanovsky <leon@kernel.org>
|
|
For when a device is not DMA capable, the max mapping size would be 0, so
make dma_max_mapping_size() reflect that.
Signed-off-by: John Garry <john.g.garry@oracle.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260629085310.2298552-2-john.g.garry@oracle.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.
Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.
Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.
The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.
v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
reenqueue, not on every reenqueue.
v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
carry over to the next owner across sched class switches, scheduler
replacement or sub-scheduler rehoming (Andrea Righi).
- Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
(Andrea Righi).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Move rb_desc->nr_page_va before updating dynamic array
The rb_descr->page_va is a dynamic array counted by nr_page_va. But
the updating of the page_va[] is done before the nr_page_va is
incremented causing a build with CONFIG_UBSAN_BOUNDS to flag it as an
overflow.
Move the increment of the counted by value before the array element
is updated.
- Propagate errors from remote event bulk updates
The return value of trace_remote_enable_event() was not being checked
by remote_events_dir_enable_write() where it would silently fail.
Have it check the return value and propagate that back up to user
space.
- Fix resource leak on mmiotrace trace_pipe close
The mmiotrace tracer was created in 2008 before the trace_pipe had a
close callback to allow tracers to do clean up from trace_pipe open.
The trace_pipe close cleanup callback was added in 2009 but the
mmiotrace tracer was not updated. It had a hack to do the cleanup in
the read call, where it may leak if user space did not read the
entire buffer.
Add a callback to mmiotrace trace_pipe close do to the cleanup
properly.
- Fix a possible NULL pointer dereference in the mmiotrace tracer
If the mmio_pipe_open() fails to find a PCI device, it will set the
hiter->dev pointer to NULL. The read function will blindly
dereference that pointer. Fix the read call to check to see if that
pointer is populated before dereferencing it.
- Fix union collision of module and refcnt for dynamic events
In 'struct trace_event_call', the 'module' pointer and the 'refcnt'
atomic variable share the same memory space in a union. The filter on
module logic only checked if the 'module' was set to determine if the
event belonged to the module. As dynamic events are always builtin,
it doesn't need the 'module' field of the structure and used a
refcount. But the module filtering logic would then mistaken these
dynamic events as a module and call module_name(event->module) on it.
Add a check to see if the event is a dynamic event and if so, do not
check it for being part of the given module.
- Reset the top level buffer in selftests before running instances
The ftracetest selftest initializes each instance before executing
the tests. But it does not reset the top level buffer. Dynamic events
are only added and removed by the top level so any left over dynamic
events will not be removed by the reset in the instances.
Left over dynamic events can cause the tests to incorrectly fail.
Reset the top level buffer before running the instances.
- Make the context_switch counter 64 bit
The code to read user space for a system call trace event or for a
trace_marker will disable migration, enable preemption, read user
space into a per CPU buffer, disable preemption and enable migration
again. It checks if the per CPU context switch counter to see if it
changed, and if it did not, it would know that the per CPU buffer was
not touched by another task.
But the save counter was 32 bit and it would compare it to the 64 bit
context_switch variable. A long running system could have the
context_switch variable greater that 1<<32 in which case the compare
will always fail. The compare will promote the 32 bit int saved value
to 64 bit and compare it to the full 64 bit counter. Since the top 32
bits of the saved value was zero, it would never match.
- Fix a use-after-free of the event_enable trigger
The event_enable trigger allows for enabling one event when another
event is triggered. When the trigger is removed, it must go through a
synchronization phase to make sure it is not triggered again. The
trigger itself is delayed by the "bulk delay" logic that was recently
added. But the code that frees the event_enable data used to rely on
the trigger code to do the synchronization. Now that the code uses
the call RCU functions (and a workqueue), that delay no longer is
there.
Add a callback private_data_free() function that allows triggers to
clean up data after the synchronization phase has completed.
- Move the module_ref counter into the delay callback
Since an event of the event_enable trigger can enable an event for a
module, it ups the module ref count for that event's module. This
prevents the event from trying to enable an event that no longer
exists and cause a use-after-free bug.
The ref counter was set back down when the trigger was removed but
not after thy synchronization phase. This could lead to the module
data being accessed after module was unloaded.
Move the module ref decrement into the private_data_free() callback
of the event_enable trigger.
- Add mutex to protect parser in ftrace filtering
The set_ftrace_filter file uses a parsing descriptor that is
allocated at open and modified by writes. If multiple threads were to
write to the descriptor at the same time, it can corrupt the parser.
Add a mutex around the modifications of the parser descriptor.
- Fix possible corruption in perf syscall tracing
The perf system call trace events can now read user space. To do so,
the reads of user space enable preemption and disables it again.
During this time that preemption is enabled, the task can migrate.
The perf event list head is assigned via a per CPU pointer. It is
done before the user space part is called. If the user space reading
migrates the task to another CPU, then the head pointer is no longer
valid.
Re-assign the head pointer after the reading of user space to keep it
using the correct data.
* tag 'trace-v7.2-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing: perf: Fix stale head for perf syscall tracing
ftrace: Add global mutex to serialize trace_parser access
tracing: Delay module ref count for "enable_event" trigger
tracing: Fix use-after-free freeing trigger private data
tracing: Fix context switch counter truncation
selftests/ftrace: Reset triggers at top level before instance loop
tracing: Fix union collision of module and refcnt for dynamic events
tracing: Fix mmiotrace possible NULL dereferencing of hiter->dev
tracing: Fix resource leak on mmiotrace trace_pipe close
tracing: Propagate errors from remote event bulk updates
tracing/remotes: Fix page_va[] access before counter update in trace_remote_alloc_buffer()
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull SMP debug fixes from Ingo Molnar:
- SMP-call fixes when CSD lock debugging is enabled (Chuyi Zhou)
* tag 'smp-urgent-2026-07-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
smp: Make CSD lock acquisition atomic for debug mode
smp: Avoid invalid per-CPU CSD lookup with CSD lock debug
|
|
RDMA device names are unique only within a network namespace, but an
RDMA cgroup can account resources for devices from multiple namespaces.
Duplicate names therefore make cgroup output ambiguous and can cause
limit writes to select the wrong device.
Use the system-wide RDMA device index to distinguish duplicate names
while preserving the existing UAPI for unique names. Reject ambiguous
name-only writes with -ENOTUNIQ and expose a complete device view to
administrators.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://patch.msgid.link/20260716132316.1495242-9-jiri@resnulli.us
Signed-off-by: Leon Romanovsky <leon@kernel.org>
|
|
dsq->first_task is __rcu for the lockless scx_bpf_dsq_peek(). The task
removal path compares it against the departing task with a plain load, which
sparse flags. The comparison runs under the dsq lock and only tests
identity, so rcu_access_pointer() is the fit.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_root is __rcu and naked accesses were left as transitional markers for
the multi-scheduler transition, to be converted to accesses through the
associated scheduler instances. Most accesses have since been converted to
resolve the sched from the program or task at hand. The remaining naked
sites divide into ones that semantically always want the root sched, which
this patch resolves, and one that is left to a later patch.
The resolved sites:
- The SCX_OPS_TID_TO_TASK validation and the ecaps sync kick already hold a
sched whose ancestors[] pins the root as entry 0 with plain pointers
stable for the sched's lifetime. Reach the root through the sched at hand.
- The dispatch entry, class switch, idle notification and fork init paths
only execute while the scheduler is live and scx_root never changes inside
the live window, so no update can race them. Add scx_root_protected_live()
which documents that invariant and resolves with a plain load.
- The hotplug path, including the ecaps reseeds, runs with the hotplug lock
held, which excludes the scx_root writers. Add scx_root_protected(), which
accepts either the hotplug lock or scx_enable_mutex.
- Is-root tests use a zero level instead of comparing against the global.
touch_core_sched_dispatch() stays naked, to be resolved by a later patch.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
cgrp->scx_sched is __rcu and published with rcu_assign_pointer() but every
reader loads it with a plain access, so sparse flags all of them. The reads
are lock-protected: enable/disable paths rewrite the field under all of
scx_enable_mutex, scx_fork_rwsem and cgroup_mutex, and cgroup creation
inherits the parent's sched under cgroup_mutex before the new cgroup is
reachable, so holding any one of the three locks makes the read stable.
Add scx_cgroup_sched() which states the protection with
rcu_dereference_check() and convert the readers. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
While booting with KHO, the following crash was observed:
BUG: unable to handle page fault for address: ff19164fffff8328
RIP: 0010:__free_one_page+0x1a1/0x6b0
Call Trace:
<TASK>
[<ffffffff913208bf>] free_one_page+0xaf/0x240
[<ffffffff93973288>] deferred_free_pages+0xa8/0xd0
[<ffffffff93971b4f>] deferred_init_memmap_chunk+0x10f/0x1b0
[<ffffffff9396e265>] padata_mt_helper+0x65/0xa0
[<ffffffff90fac402>] process_scheduled_works+0x202/0x410
[<ffffffff90fae739>] worker_thread+0x1f9/0x2d0
[<ffffffff90fb62fd>] kthread+0x27d/0x2f0
[<ffffffff90fae540>] ? __pfx_worker_thread+0x10/0x10
[<ffffffff90fb6080>] ? __pfx_kthread+0x10/0x10
[<ffffffff90efdc55>] ret_from_fork+0x145/0x280
[<ffffffff90fb6080>] ? __pfx_kthread+0x10/0x10
[<ffffffff90e2e46a>] ret_from_fork_asm+0x1a/0x30
</TASK>
deferred_init_memmap_chunk() interleaves initialization of struct pages
with freeing them. This works fine without KHO because free regions
will never be buddy neighbors. However, with KHO, free memory will be split
into (free && scratch) and (free && !scratch), that can be buddy neighbors.
KHO scratch is aligned to CMA_MIN_ALIGNMENT_PAGES pages but buddy looks
at the neighborhood of MAX_ORDER_NR_PAGES pages. These values are
configurable but CMA_MIN_ALIGNMENT_PAGES is always less or equal to
MAX_ORDER_NR_PAGES. In the crashing configuration they were set as
follows:
CMA_MIN_ALIGNMENT_PAGES = 1 << 9
MAX_ORDER_NR_PAGES = 1 << 10
So while freeing one chunk, buddy accessed uninitialized struct pages
from another chunk, tried to merge the blocks and crashed.
To fix this, let's just align KHO scratch to MAX_ORDER_NR_PAGES pages.
Fixes: c6073743d0c7 ("kho: make preserved pages compatible with deferred struct page init")
Signed-off-by: Michal Clapinski <mclapinski@google.com>
Link: https://patch.msgid.link/20260717134028.2880508-1-mclapinski@google.com
[rppt: massaged the changelog]
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
LUO keeps track of successful retrieve attempts on an FLB. It does so
to avoid multiple retrievals of the same FLB. Multiple retrievals cause
problems because once the FLB is retrieved, the serialized data
structures are likely freed and the FLB is likely in a very different
state from what the code expects.
All this works well when retrieve succeeds. When it fails,
luo_flb_retrieve_one() returns the error immediately, without ever
storing anywhere that a retrieve was attempted or what its error code
was. If the user attempts to retrieve another file registered with the
same FLB, LUO will attempt to call the FLB's retrieve() callback again.
The retry is problematic for much of the same reasons listed above. The
FLB is likely in a very different state than what the retrieve logic
normally expects (e.g. some KHO pages may have already been restored and
freed).
There is no sane way of attempting the retrieve again. Remember the
error retrieve returned and directly return it on a retry.
This is done by changing the retrieved bool to a retrieve_status
integer. A value of 0 means retrieve was never attempted, a positive
value means it succeeded, and a negative value means it failed and the
error code is the value.
This is similar to commit f85b1c6af5bc ("liveupdate: luo_file: remember
retrieve() status") which did the same for LUO files.
Fixes: cab056f2aae7 ("liveupdate: luo_flb: introduce File-Lifecycle-Bound global state")
Assisted-by: Gemini:gemini-3-pro-preview
Signed-off-by: David Matlack <dmatlack@google.com>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260528174140.1921129-3-dmatlack@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
Increment the outgoing FLB refcount in liveupdate_flb_get_outgoing() so
that the FLB structure cannot be freed while the caller is actively
using it. Add an additional liveupdate_flb_put_outgoing() function so
the caller can explicitly indicate when it is done using the outgoing
FLB.
During a Live Update, the kernel may need to fetch the outgoing FLB
outside of the scope of a file handler's preserve() and unpreserve()
callbacks. In that situation there is no way for the caller to protect
itself against the outgoing FLB from being freed while it is using it.
Incrementing the reference count in liveupdate_flb_get_outgoing()
ensures it cannot be freed.
This change also aligns the outgoing FLB lifecycle management with the
incoming FLB, since the latter uses the same get/put semantics.
Fixes: cab056f2aae7 ("liveupdate: luo_flb: introduce File-Lifecycle-Bound global state")
Assisted-by: Gemini:gemini-3-pro-preview
Signed-off-by: David Matlack <dmatlack@google.com>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Link: https://patch.msgid.link/20260528174140.1921129-2-dmatlack@google.com
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
The UAPI documents liveupdate_session_finish::reserved as requiring zero,
but luo_session_finish() currently ignores it and finishes the session.
Accepting nonzero values prevents the field from being safely repurposed
by a future extension.
Reject nonzero reserved values before changing session state, matching
LIVEUPDATE_SESSION_GET_NAME.
Fixes: 16cec0d26521 ("liveupdate: luo_session: add ioctls for file preservation")
Assisted-by: Codex:gpt-5.6-sol
Reviewed-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Link: https://patch.msgid.link/20260716012607.22020-2-liu.yun@linux.dev
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
Cross-merge BPF and other fixes after downstream PR.
Conflicts:
net/core/filter.c
Changes [2] in bpf-next conflict with a recent fix [1]
from the 'net' tree. Resolved by using [1] as a base and
applying same flags handling logic as in [2] in the
bpf_redirect_peer() helper.
[1] https://lore.kernel.org/all/20260706185609.330006-2-daniel@iogearbox.net/
[2] https://lore.kernel.org/all/20260618182035.43811-2-jordan@jrife.io/
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
The code that can read the user space parameters of a system call may
enable preemption and migrate. The head of the per CPU perf events list
may be pointing to the wrong CPU event if the code migrates the task.
Reassign the head pointer if the system call event called the code that
may have caused a migration.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260724193210.03fae1d6@gandalf.local.home
Reported-by: Sashiko <>
Link: https://sashiko.dev/#/patchset/20260717173252.3431565-1-usama.arif%40linux.dev
Fixes: edca33a56297d ("tracing: Fix failure to read user space from system call trace events")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
In ftrace, the trace_parser structure is allocated and initialized when
a trace file is opened, and is subsequently used across write and release
handlers to parse user input.
The affected handler paths and their specific functions are:
- Open paths: ftrace_regex_open(), ftrace_graph_open()
- Write paths: ftrace_regex_write(), ftrace_graph_write()
- Release paths: ftrace_regex_release(), ftrace_graph_release()
If userspace opens a trace file descriptor and shares it across multiple
threads, concurrent write calls will race on the parser's internal state,
specifically the 'idx', 'cont', and 'buffer' fields, leading to corrupted
input or undefined behavior.
Fix this by adding a global mutex, parser_lock, to serialize all access
to trace_parser across write and release paths, preventing concurrent
corruption of parser state.
Fixes: e704eff3ff51 ("ftrace: Have set_graph_function handle multiple functions in one write")
Fixes: 689fd8b65d66 ("tracing: trace parser support for function and graph")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260725024721.1983675-1-wutengda@huaweicloud.com
Signed-off-by: Tengda Wu <wutengda@huaweicloud.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Pull bpf fixes from Eduard Zingerman:
- Fix tcp_bpf_sendmsg() error path mistaking a concurrently-freed
sk_psock->cork for the local temporary message and freeing it again
(Chengfeng Ye)
- Reject passing scalar NULL to nonnull arg of a global subprog.
Previously the verifier did not account for the cases directly
passing scalars to a global subprog, e.g.: 'global_func(0);' would
pass even if 'global_func' argument was marked nonnull (Amery Hung)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
bpf, sockmap: Fix cork use-after-free in tcp_bpf_sendmsg()
selftests/bpf: Test passing scalar NULL to nonnull global subprog
bpf: Reject passing scalar NULL to nonnull arg of a global subprog
|
|
Add 'P' flag to the diagnostic line to indicate whether any per-cpu
rtpcp's lazy_timer is pending. this helps diagnose stalls where
rcu-task callbacks are queued but the kthread stay sleep because
the lazy_timer has not yet fired and no grace period has started.
The output is as follows:
[ 31.319540][ T77] call_rcu_tasks() has failed boot-time tests.
[ 31.320205][ T77] rcu_tasks: RTGS_WAIT_CBS(11) since 7518 g:4 i:0 kCuUP l:150000
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit updates the comment "We can't create the thread unless
interrupts are enabled." from 'commit 4929c913bda5 ("rcu: Make
call_rcu_tasks() tolerate first call with irqs disabled")' to be more
clear and also to cover deferred wakeup and to take into account the fact
that kthread creation has been move to core_initcall() time by 'commit
c63eb17ff06d ("rcu: Create call_rcu_tasks() kthread at boot time")'.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Although the rdp->rcu_cpu_has_work field is accessed only by the
corresponding CPU, it can be accessed by both interrupt handlers via
invoke_rcu_core_kthread() and at task level via rcu_cpu_kthread().
This means that we need this_cpu_read() rather than __this_cpu_read(),
this_cpu_write() rather than __this_cpu_write(), and READ_ONCE()
rather than plain C-language loads. The exception is the boot-time
rcu_spawn_core_kthreads(), which cannot race with kthreads that have
not yet been spawned.
This commit therefore makes it so.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Ever since the kvfree_rcu() tracing moved out of the callback-enqueue
path, rcutree_enqueue() no longer looks at the callback function
pointer: By the time it is invoked, __call_rcu_common() has already
stored the function into rhp->func, and the enqueue path only adds
the rcu_head to the segmented callback list and emits tracepoints
that do not take the function pointer.
Nevertheless, the function pointer is still threaded through
call_rcu_core(), call_rcu_nocb(), and rcutree_enqueue(), forcing
each level to carry a dead argument.
Remove the parameter from all three functions, from the no-CBs stub,
and from the corresponding declarations. Anything needing the
callback function can still get it from rhp->func.
No functional change.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The ->expedited_need_qs counter was part of the old expedited
grace-period machinery that spun waiting for each CPU to check in.
The current implementation instead tracks holdout CPUs via the
rcu_node structures' ->expmask fields and waits on the ->exp_wq[]
wait queues, so nothing reads or writes ->expedited_need_qs any
longer.
Remove the field to avoid wasting space in rcu_state and to keep
readers of the expedited code from searching for nonexistent users.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently raw_cpu_write() is used to clear the ->rcu_need_heavy_qs
field of the per-CPU rcu_data structure. However, on x86 this is a
normal assignment, which does not play well with concurrent accesses.
This commit therefore upgrades the uses of raw_cpu_write() to its
concurrency-safe counterpart WRITE_ONCE() of an rdp pointer obtained
from this_cpu_ptr(&rcu_data).
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently __this_cpu_read() and __this_cpu_write() is used to access the
->cpu_no_qs.b.exp field of the per-CPU rcu_data structure. However,
this can fail when the accesses can happen in interrupt handlers, as
recently started being exercised by rcutorture. This commit therefore
upgrades the uses of __this_cpu_read() and __this_cpu_write() to their
interrupt-safe counterparts this_cpu_read() and this_cpu_write().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add nexp and exp_interval parameters to rcuscale that spawn kthreads
running synchronize_rcu_expedited() in a loop. This generates concurrent
expedited GP load while the normal writers measure GP or callback
latency.
When combined with gp_async=1 (which uses call_rcu() for writers), this
tests how effectively callbacks benefit from expedited grace periods.
With RCU callback expedited GP tracking, the async callbacks should
complete faster because they piggyback on the expedited GPs rather than
waiting for normal GPs.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Even when rcu_pending() triggers rcu_core(), the normal callback
advancement path through note_gp_changes() -> __note_gp_changes() bails
out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
called and callbacks remain stuck in RCU_WAIT_TAIL.
Add a direct callback advancement block in rcu_core() that checks for GP
completion via rcu_segcblist_nextgp() combined with
poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
GP kthread if rcu_advance_cbs() requests a new grace period.
Uses trylock to avoid adding contention on rnp->lock. If the lock is
contended, callbacks will be advanced on the next tick.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
rcu_pending() decides whether rcu_core() should run on the current CPU's
timer tick. It does not account for expedited grace periods: after an
expedited GP completes, a non-offloaded CPU's callbacks remain in
RCU_WAIT_TAIL (not yet advanced to RCU_DONE_TAIL) and rcu_core() is
never invoked to advance them.
Detect that case via rcu_segcblist_nextgp() combined with a new
memory-ordering-free poll variant,
poll_state_synchronize_rcu_full_unordered(). This keeps rcu_pending()
cheap: it runs on every tick that has pending callbacks, so it must
not pay for the two memory barriers in
poll_state_synchronize_rcu_full(). The check is only a hint to run
rcu_core(); the ordered re-check and the actual callback advancement
happen there.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When an expedited grace period completes, rcu_exp_wait_wake() wakes
waiters on rnp->exp_wq[] but does not notify the NOCB rcuog kthreads. An
rcuog kthread that is waiting for a grace period sleeps on the leaf
rcu_node's ->nocb_gp_wq[] with a wait condition based on the grace-period
state, so without a wakeup, callbacks on offloaded CPUs that could
benefit from the expedited GP wait until the rcuog kthread wakes for some
other reason (e.g. the next normal GP or a timer).
Make the rcuog grace-period wait honour expedited GPs and wake it when
one completes:
- nocb_gp_wait() now records the grace period to wait for as a struct
rcu_gp_seq (both normal and expedited), tracks the earliest pending
normal and expedited sequence across the group, and releases the wait
via poll_state_synchronize_rcu_full() so it wakes for whichever
completes first. ->nocb_gp_seq is widened to struct rcu_gp_seq
accordingly.
- rcu_exp_wait_wake() calls the new rcu_nocb_exp_cleanup() on leaf
nodes, which wakes both ->nocb_gp_wq[0] and ->nocb_gp_wq[1] (the
expedited sequence does not share parity with the normal ->gp_seq the
waiter indexed with). Both this path and rcu_nocb_gp_cleanup() use
the shared rcu_nocb_cleanup_wake() helper, which checks swait_active()
first; the smp_mb() in rcu_gp_cleanup()/rcu_exp_wait_wake() orders the
grace-period state update before that check.
A stub rcu_nocb_exp_cleanup() is provided for CONFIG_RCU_NOCB_CPU=n.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
