| Age | Commit message (Collapse) | Author |
|---|
|
alloc_pid() checks PIDNS_ADDING only on the leaf pid namespace before
making a new struct pid visible in every ancestor namespace. That is
insufficient when an unborn descendant pid namespace outlives an
ancestor whose init task has already exited. The descendant can still be
initialized later through setns(), and the new pid is then published
into the dead ancestor as well.
Keep the existing ENOMEM behavior, but require PIDNS_ADDING to be set in
every namespace that will receive the new pid before publishing any of
them. This preserves the invariant that free_pid() never decrements
pid_allocated in a namespace whose child_reaper is no longer live.
Fixes: a3bdc23ba8ea ("pid_namespace: allow opening pid_for_children before init was created")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Commit 16a1d968358a ("mm/slab: remove mm/slab.c and slab_def.h")
removed the SLAB allocator, the only allocator that implemented cpuset
slab spreading. Commit 61a182ab61a6 ("cgroup/cpuset: Remove
cpuset_do_slab_mem_spread()") then removed the last task_spread_slab()
caller. Commit 3ab67a9ce82f ("cgroup/cpuset: Mark memory_spread_slab as
obsolete") marked the legacy control obsolete.
cpuset still updates PFA_SPREAD_SLAB when tasks attach to a legacy
cpuset and walks all tasks in a cpuset when memory_spread_slab changes.
Remove the unused task flag and its helpers, and make spread task
updates depend only on memory_spread_page.
Keep the memory_spread_slab control and CS_SPREAD_SLAB state so legacy
users retain the existing write, readback and inheritance behavior.
Update the comments and documentation to describe only page-cache
spreading as functional.
Assisted-by: LLM
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Core scheduling's pick_next_task() picks what to run on every SMT sibling of
the core in a single pass under the shared core-wide rq lock. The selection
state is consistent only while the lock is held continuously, so
->pick_task() originally could not release it. However, since 4c95380701f5
("sched/ext: Fold balance_scx() into pick_task_scx()"), sched_ext runs
dispatch from inside the pick and dispatching can drop the rq lock. To
support this, pick_next_task() has been updated to restart the whole
selection when a pick returns RETRY_TASK after releasing the lock.
When selections on the same core interleave through the dropped lock, they
corrupt each other's state: one clears the other's rq->core_pick leading to
a NULL deref, or invalidates its keep-the-previous-task decision leaving a
dequeued task running, which deadlocks the next wakeup and matches the
reported hard hangs. A cookied ping-pong load on an SMT machine makes the
interleavings frequent and kills the kernel within seconds.
Fix it by making the pick return RETRY_TASK whenever dispatch released the
rq lock, so that a selection only ever commits picks made under a
continuously held lock. The previous patch's rq->scx.lock_drop_seq counts
the releases. A dispatch that touched nothing never releases the lock and
its verdict, including "nothing to run", stands: retries are bounded, each
following a dispatch that actually did something, and an idle CPU does not
loop.
If another dispatch is already in flight on the rq, skip dispatching and
pick from what is already queued locally - the in-flight dispatch has
released the lock, so its own selection will retry and re-pick this rq,
while returning RETRY_TASK here would only spin on the lock that dispatch
needs to finish.
Balance callbacks must run in the context that queued them, so they can only
be queued on the CPU's own rq. When dispatching for another rq, run the
deferred work directly instead - that rq may consume all its picks through
the core-sched fast path and never queue the callback itself.
The put_prev_task_scx() warning about a runnable task being left behind
assumed that dispatch ran as part of the very pick that is switching away.
That now only holds on the non-core path, so gate it and drop the
cookie-match test, which is always true without core scheduling, from its
condition.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Reported-by: ElXreno <elxreno@gmail.com>
Link: https://github.com/sched-ext/scx/issues/3715
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Under core scheduling, pick_next_task() selects for all SMT siblings under
one continuous hold of the shared core-wide rq lock, and sched_ext's
dispatch can release that lock from inside the pick. In preparation for
making the core-sched pick detect the releases and retry, add
rq->scx.lock_drop_seq and bump it at every site that can release an rq lock
while a dispatch may be in flight. The counter is only maintained while core
scheduling is enabled. No functional changes.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Under core scheduling, dispatch runs from within the core-wide pick and can
target a sibling rq, so ops.dispatch() may execute on a CPU different from
the dispatched rq's. Several kfunc paths assumed the two always coincide:
- scx_dsq_move() decided whether an rq lock is held by testing this_rq()'s
rq flags and lock-danced accordingly. A dispatch for a sibling took the
unlocked-context branch and acquired the source rq lock on top of the
already held dispatched rq lock which could deadlock.
- scx_bpf_sub_dispatch() dispatched this_rq() with its stashed
sub_dispatch_prev, which is NULL when dispatching for a sibling.
- finish_dispatch(), scx_bpf_dsq_reenq() and scx_bpf_dsq_nr_queued()
resolved SCX_DSQ_LOCAL to this CPU's local DSQ rather than the dispatched
rq's. The latter two are callable from other rq-locked operations too,
where SCX_DSQ_LOCAL now likewise resolves to the op's rq. This changes
behavior also without core scheduling, e.g. for ops.enqueue() running a
remote wakeup on the waking CPU, and is intended: which CPU happens to
execute an operation is incidental, the op's rq is what it is operating
on, and the resolution now matches the insert side where SCX_DSQ_LOCAL
dispatches land on the task's rq.
Use the rq tracked by scx_locked_rq(), which is set to the dispatched rq
around ops invocations and NULL in unlocked contexts.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
SCX_RQ_BAL_KEEP tells the pick to keep running the previous task, a leftover
from when balancing and picking were separate operations. An rq-level flag
only works while dispatches and picks pair up one to one, which core
scheduling breaks: selections interleave through dispatch's lock drops and a
pick can consume a stale flag, keeping a task that has since been dequeued.
Fixing core scheduling support requires the decision to travel with the
dispatch that made it. Make scx_dispatch_sched() and balance_one() return an
explicit verdict instead and drop the flag's plumbing from the tools autogen
enum headers.
Also factor the pick-side invocation, its follow-up queueing and the
post-dispatch checks out of do_pick_task_scx() into dispatch_pick(). No
functional changes intended.
v2: Drop the SCX_RQ_BAL_KEEP plumbing from the tools autogen enum headers
as well (Andrea).
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Core scheduling's pick_next_task() operates on all sibling rqs under one
acquisition of the shared core-wide lock. A ->pick_task() that releases the
rq lock leaves every sibling __lock momentarily free, letting
__sched_core_flip(false) complete mid-selection and rebind rq_lockp() under
it. The selection resumes on the split locks, touching sibling state it no
longer protects, and __schedule() finally releases a lock that was never
taken while leaking the one that was.
Count in-flight core-wide selections in the leader's rq->core_pick_in_flight
and make __sched_core_flip() wait for the count to drain. The count only
changes under the shared lock, which the flip holds while sampling, so no
other ordering is needed. The wait can repeat while selections overlap, but
the flip backs off between samples and flips are rare cookie-lifetime
events.
sched_core_cpu_deactivate() moves the count to the new leader - a stale copy
left behind would bias it forever if that CPU later returns as its own
leader.
Fixes: 539f65125d20 ("sched: Add core wide task selection and scheduling")
Cc: stable@vger.kernel.org # v5.14+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
|
|
Core scheduling's pick_next_task() breaks when a ->pick_task()
implementation can release the rq lock. The selection state derived on entry
is only valid while the lock is held continuously. Once a pick can drop the
lock, an interleaving selection can invalidate all of it: the single-CPU
fast path can commit an uncookied pick although the core went cookied during
the release, and forceidle committed by the interleaving selection skews the
restarted pass's accounting.
Fix it by restarting the whole selection when a pick returns RETRY_TASK
after releasing the lock: a single restart point above the state derivation
replaces the per-loop restart labels, so a retry picks up state committed by
interleaving selections and accounts and resets forceidle like a fresh
selection would.
need_sync and fi_before latch across retries. Clock validity can't be
re-derived - there is no program-ordered way to tell whether the own and
core rq clocks are still updated after the lock was released, as other
lockers' pin cycles may or may not have invalidated them. When restarting,
clear core_clock_updated so that the sibling loop re-updates the core rq,
and update the own rq clock if invalidated.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
|
|
In the cleanup_srcu_struct() function, when iterating over per-cpu's
srcu_data, timer_delete_sync(&sdp->delay_work) is called to cancel the
delayed work before doing flush_work(&sdp->work).
However, suppose that timer_delete_sync() returns 1, which means that it
successfully deleted an pending timer before it had a chance to fire.
But this also means that the sdp->work will not be queued, so that the
subsequent flush_work(&sdp->work) will returns immediately without waiting
for anything. Taken together, all of this means that any recently queued
SRCU callbacks to not be invoked, which can result in memory leaks,
hangs, or worse.
Fix this by checking the return value of timer_delete_sync(), if it
returns 1, explicitly queue sdp->work so that the callbacks will be
invoked and the following flush_work() will correctly wait for all of
those callbacks to finish executing.
[ Zqiang: Apply feedback from Breno Leitao and kernel test robot. ]
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states
must not be equal, or the verifier can prune an unsafe path.
Compare the pointer type for STACK_ITER slots.
Fixes: dfab99df147b ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS")
Signed-off-by: Ning Ding <dingning04@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
When a function-like macro expands to an expression, that expression
doesn't need a semicolon after it. All uses have been verified to
have their own semicolons.
This was found using the following Coccinelle semantic patch:
@r@
identifier i : script:ocaml() { String.lowercase_ascii i = i };
expression e;
@@
*#define i(...) e;
Link: https://patch.msgid.link/20260801191002.1383835-6-Julia.Lawall@inria.fr
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
hrtimers use ktime_t in their implementation and API.
The timer list performs a lot of unnecessary conversion to nanoseconds
which make the code harder to read and are also wrong in case the values
ever become negative.
Remove the conversions.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-5-910cbd485390@linutronix.de
|
|
'%Ld' and '%Lu' are GNU extensions. While they do work for kernel code,
checkpatch complains about them all the time. Replace them with the
standard placeholders for 'long long' types, namely '%lld' and '%llu'.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-4-910cbd485390@linutronix.de
|
|
Document and verify that the hrtimer_cpu_base::lock is held at this point.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-3-910cbd485390@linutronix.de
|
|
Use an explicitly sized type to make the code a bit more consistent
with other fields of the datastructure and other sequence counters.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-2-910cbd485390@linutronix.de
|
|
The current name is not clear about its behavior.
Rename it.
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-1-910cbd485390@linutronix.de
|
|
Re-arranging hrtimer_interrupt() switched the retry path to a local
counter and dropped the update of cpu_base->nr_retries, leaving the
field exported via /proc/timer_list stuck at zero.
Increment nr_retries only when another pass through the expiry loop
is started; the third attempt that falls through to hang handling is
still accounted by nr_hangs alone.
Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()")
Signed-off-by: Liang Hao <haohlliang@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260731150408.19554-1-haohlliang@gmail.com
|
|
The enable_data variable gets freed on most error paths in
event_enable_trigger_parse(). Use free() to free it and just before
returning normally, call retain_and_null_ptr(enable_data) just before a
successful exit to keep it from being freed. On success, the enable_data
is assigned to the trigger_data->private_data field.
Also add a comment to why event_trigger_free(trigger_data) is being called
before a successful exit.
Link: https://patch.msgid.link/20260807113558.0ff14e96@gandalf.local.home
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
WARN_ONCE() splats once per call site, so only the first offending event
registered is ever reported. The tree currently has six:
ice_{rx,tx}_dim_template, two hfi1 txq events, mtu3_ep and edma_log_io.
Whichever registers first hides the rest, and each has to be found again
on the next boot.
Add a pr_warn() next to the WARN_ONCE() so every offender is listed, the
same way test_event_printk() already pairs WARN_ON_ONCE() with pr_warn()
for unsafe %p* dereferences. The WARN_ONCE() stays so the condition still
fails tests and panics under panic_on_warn.
Link: https://patch.msgid.link/20260806215256.1680267-1-devnexen@gmail.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
On native sparc64, struct __kernel_old_timeval contains a four-byte hole
after tv_usec because tv_sec is 64-bit while __kernel_suseconds_t is 32-bit.
put_itimerval() fills only the named fields in a stack-allocated
__kernel_old_itimerval and copies the entire object to userspace, so
getitimer() can expose the two padding holes.
Zero-initialize the aggregate before assigning the fields so implicit
padding is deterministic before it crosses the user/kernel boundary.
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: Codex:gpt-5
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260809190428.1523014-1-Jeremy.Jean@oss.cyber.gouv.fr
|
|
'select' does not work on config options in a 'choice', so currently the
'select VIRT_CPU_ACCOUNTING_GEN' for NO_HZ_FULL is dead, with the choice
option VIRT_CPU_ACCOUNTING_GEN only being enabled when NO_HZ_FULL=y because
the other choice members depend on NO_HZ_FULL=n.
Remove the dead select, and encode this relationship as a default of the
choice, instead.
This dead select was found by kconfirm, a static analysis tool for Kconfig.
Signed-off-by: Julian Braha <julianbraha@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
Link: https://patch.msgid.link/20260801160140.2391000-1-julianbraha@gmail.com
|
|
The last user was removed in commit a6d799608e6a ("ptp: Switch to
ktime_get_snapshot_id() for pre/post timestamps").
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260731-timekeeping-aux-must-check-v1-1-11ae93068497@linutronix.de
|
|
Writing 0 to kernel.ftrace_enabled has not reliably disabled ftrace
for years (FTRACE_OPS_FL_PERMANENT users already block it, and more
callers rely on ftrace always being on). Refuse the write instead of
leaving it in an inconsistent "disables some, not all" state: return
-EOPNOTSUPP and log a message. Reads and enabling (writing 1) are
unaffected.
Update the docs to note the deprecation up front.
Link: https://patch.msgid.link/20260806153000.4184871-2-andrey.grodzovsky@crowdstrike.com
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next into for-7.3-arena-args
Pull bpf-next d114bb989367 ("Merge branch
'add-arena-argument-support-to-kfuncs-and-struct_ops'") to make the __arena
and __arena__nullable kfunc and struct_ops argument suffixes available. The
suffixed arguments will be used to convert sched_ext kfuncs and struct_ops
callbacks that currently pass arena pointers as scalars and rebase them by
hand.
|
|
apply_wqattrs_commit() updates node_nr_active->max unconditionally.
wq->node_nr_active[] is only allocated for unbound workqueues, so guard
the call before per-cpu workqueues start using this path.
No functional change: only unbound workqueues reach apply_wqattrs_*()
today.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
This allocates a pwq and binds it to the pool @attrs asks for.
Which pool that is becomes a property of the attrs (once per-cpu becomes
an affinity scope).
Remove the 'unbound" from the function name, given it will be bigger
than unbound.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The attrs are where the affinity scope lives, and a per-cpu workqueue
will need one once per-cpu becomes a scope rather than a separate
backend. Allocate them unconditionally.
wq_dump.py used a non-NULL wq->attrs as its test for an unbound
workqueue, which no longer holds; test WQ_UNBOUND there instead.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The unbound prefix says which workqueues currently have the field rather
than what it holds, and the next patch allocates it for every workqueue.
Rename it first so that change stays a single line.
tools/workqueue/wq_dump.py reads the field by name, so rename it there
too.
wq_sysfs_unbound_attrs[] keeps its name: it is the set of sysfs files
that only unbound workqueues expose.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
workqueue_online_cpu() and workqueue_offline_cpu() decide whether a
workqueue needs a pod affinity update by testing wq->unbound_attrs for
NULL, which is only meaningful because the attrs are allocated for
unbound workqueues alone.
Test the flag instead, so the attrs can later be allocated for every
workqueue.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
pwq_tryinc_nr_active() and pwq_dec_nr_active() choose between the shared
per-node nr_active and the plain per-pwq one by testing
wq_node_nr_active() for NULL.
Test the backing pool with is_percpu_pool() instead, so the accounting
follows the pool that runs the work rather than the workqueue type.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Add is_percpu_pool() and test the pool directly for per cpu. Convert the
other open-coded pool->cpu checks -- in put_unbound_pool(),
pool_allowed_cpus() and the workqueue watchdog -- to the same helper.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a
helper. The inner allocation-failure path now returns -ENOMEM and the
caller jumps to the existing enomem cleanup, equivalent to the previous
goto.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Move the static per-cpu worker_pool lookup in alloc_and_link_pwqs() into
a helper, get_percpu_pool(), so the lookup can be shared by other
pool-selection paths.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
If the auxiliary clock is disabled during tk_get_aux_ts64() but is enabled
before tks->clock_valid is checked, then uninitialized stackdata will be
used in the calculations and indirectly leaked to userspace.
The same race window also exists after this change and also for the core
timekeeper. But in these cases the only effect would be incorrect
adjustments and this is userspace's responsibility to avoid this.
Fixes: 4eca49d0b621 ("timekeeping: Prepare do_adtimex() for auxiliary clocks")
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260731-timekeeping-aux-adjtimex-return-v1-1-b7fea4692886@linutronix.de
|
|
cpuset_partition_show() reads cs->prs_err without cpuset_mutex using
READ_ONCE(). The field is documented as not lock protected, but several
updates to live cpusets still use plain stores.
Convert the remaining prs_err stores on live cpusets to WRITE_ONCE().
Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
Assisted-by: LLM
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The timekeeper update path uses a bulk memcpy() to synchronize the
timekeeper structure, which is not guaranteed to be atomic. This allows for
torn reads in ktime_get_real_seconds() on 64-bit systems, where the
sequence counter protection is bypassed for performance.
To prevent reading a torn 64-bit xtime_sec value, enforce atomic-like
access by using WRITE_ONCE() for the critical field before the bulk
memcpy() in timekeeping_update_from_shadow(). Correspondingly, use
READ_ONCE() in ktime_get_real_seconds() to ensure a fresh, consistent load
from memory.
[ tglx: Format changelog and add comment ]
Reported-by: syzbot+72789cd1697965e714ca@syzkaller.appspotmail.com
Signed-off-by: Dennis Moshegov <dennis@xzync.uk>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260724154405.70-1-dennis@xzync.uk
Closes: https://syzkaller.appspot.com/bug?extid=72789cd1697965e714ca
|
|
With the changes that enable preempt count to track IRQ disabling
nesting, we don't have enough bits in 32-bit preempt count
implementation, as a result we move NMI nesting bits out of the 32-bit
preempt count. However on the architectures that can support 64-bit
preempt count implementation, we can keep the NMI nesting bits in the
32-bit preempt count and avoid maintaining NMI nesting bits outside of
the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow
architectures to select this. Note that under this Kconfig, preempt
count is maintained in a 64-bit word however preempt_count() still
remains as an int because all the effective bits still fit in
(previously we mask out NEED_RESCHED bit in preempt_count()). This
should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit
preempt count since it may introduce {over,under}flow]
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
|
|
Currently preempt_count() is always a non-negative int on all archs
(PREEMPT_NEED_RESCHED archs will mask out the MSB when returning
preempt_count()), hence the checking in __cant_migrate() is in fact just
checking whether preempt_count() is 0 or not. In a future change, we are
going to use all the 32 bits of preempt_count(), which would make
negative int values possible from preempt_count(). Therefore convert the
"> 0" comparison into a zero check to prepare for the future change.
No functional changes are intended.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-10-boqun@kernel.org
|
|
The preempt_offset is always 0 in all the callsites of __cant_sleep(),
hence remove it. It also allows us to clear up the code a bit by
no longer using a "preempt_count() > .." comparison.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-9-boqun@kernel.org
|
|
The semantics of various IRQ disabling guards match what
*_irq_{disable,enable}() provide, i.e. the interrupt disabling is
properly nested, therefore it's OK to switch to use
*_irq_{disable,enable}() primitives.
[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided
by Peter and Lyude]
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-8-boqun@kernel.org
|
|
While making changes to the refcounted interrupt patch series, at some
point on my local branch I broke something and ended up writing some kunit
tests for testing refcounted interrupts as a result. So, let's include
these tests now that we have refcounted interrupts.
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-7-boqun@kernel.org
|
|
Currently the nested interrupt disabling and enabling is represented by
_irqsave() and _irqrestore() APIs, which are relatively unsafe, for
example:
<interrupts are enabled as beginning>
spin_lock_irqsave(l1, flag1);
spin_lock_irqsave(l2, flag2);
spin_unlock_irqrestore(l1, flags1);
<l2 is still held but interrupts are enabled>
// accesses to interrupt-disable protected data will cause races
This is even easier to trigger with guard facilities:
unsigned long flag2;
scoped_guard(spin_lock_irqsave, l1) {
spin_lock_irqsave(l2, flag2);
}
// l2 locked but interrupts are enabled.
spin_unlock_irqrestore(l2, flag2);
(Hand-to-hand locking critical sections are not uncommon for a
fine-grained lock design)
And because of this unsafety, Rust cannot easily wrap the
interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
* local_interrupt_disable();
* local_interrupt_enable();
They work like local_irq_save() and local_irq_restore() except that 1)
the outermost local_interrupt_disable() call saves the interrupt state
into a per-CPU variable, so that the outermost local_interrupt_enable()
can restore the state, and 2) a per-CPU counter is added to record the
nest level of these calls, so that interrupts are not accidentally
enabled inside the outermost critical section.
Also add the corresponding spin_lock primitives: spin_lock_irq_disable()
and spin_unlock_irq_enable(), as a result, code as follows:
spin_lock_irq_disable(l1);
spin_lock_irq_disable(l2);
spin_unlock_irq_enable(l1);
// Interrupts are still disabled.
spin_unlock_irq_enable(l2);
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier
to design.
[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
Co-developed-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804182657.87716-1-boqun@kernel.org
|
|
fprobe_remove_ips() manually duplicates the unregister and filter-removal
logic for both graph and ftrace ops. Simplify it by delegating to the
existing fprobe_graph_remove_ips() and fprobe_ftrace_remove_ips() helpers.
Link: https://lore.kernel.org/all/178528139798.102586.5349128066643420018.stgit@devnote2/
Assisted-by: Antigravity:gemini-3.6-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The futex state is used to prevent a waiter from attaching to the lock
owner while the owner runs the futex cleanup in exit() or exec().
Only the state transition from FUTEX_STATE_OK to FUTEX_STATE_EXITING must
be done with the task's pi_lock held, the transition away from
FUTEX_STATE_EXITING has no serialization requirements on the writer side,
but it's completely non obvious why. It's magically protected by
exit_pi_state(), which operates under tsk::pi_lock, as that's the state
which has to be correct when the waiter observes the new state.
OTOH, taking the pi_lock in futex_cleanup_end() is not a performance issue
because at that point the lock should be uncontended in the vast majority
of cases.
Aside of that the handling of FUTEX_STATE_EXITING in attach_to_pi_owner()
and handle_exit_race() is confusing at best.
Protect the store in futex_cleanup_end() with tsk::pi_lock, handle
FUTEX_STATE_EXITING in attach_to_pi_owner() explicitly and document how
this is supposed to work.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
|
|
A private futex key borrows the waiter's mm without taking an mm_users
reference. Nevertheless, attach_to_pi_owner() currently accepts an owner
from a different address space and copies the private key into the owner's
PI state.
When that owner exits, exit_pi_state_list() uses the saved key to find the
hash bucket and acquires a reference to the waiter's private hash. If the
last user of the waiter's mm exits concurrently, futex_hash_free() frees
the hash while the owner still uses its bucket and reference.
Prevent this by validating in attach_to_pi_owner() that, for private
futexes, the owner mm and waiter mm are the same. Perform the check with
the owner's pi_lock held and after validating owner::futex::state to
serialize against a concurrent PI-state exit cleanup.
[ tglx: Amended comment ]
Fixes: 80367ad01d93 ("futex: Add basic infrastructure for local task local hash")
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Assisted-by: Codex:gpt-5.6-sol
Cc: stable@vger.kernel.org
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Fix use-after-free in eventfs_remove_rec()
The freeing of the eventfs_inode children used list_for_each_entry()
where the child is freed via srcu, but there's still a chance that it
gets freed. It should be using list_for_each_entry_safe().
- Fix eventfs_inode SRCU use of list in freeing
The iterator uses an SRCU protected list walk on the eventfs inodes.
The eventfs inode uses its "list" field in a union with the RCU list
head. When the inode gets added to the SRCU list it immediately
corrupts the list pointer and can cause an issue with the iterator.
Move the RCU list head to be shared with the children list head which
allows the iterator to check the parent inode if is freed before
referencing the child. Have the iterator check the parent "is_freed"
field and break out if it is set. Also add memory barriers to make
sure the ordering is correct.
- Fix various RCU synchronization issues with direct_functions
Updates to direct_functions have some missing RCU protection and
synchronization. Restructure the code a bit to make sure updates to
the direct_functions are protected.
- Remove an unneeded comma from a scope_guard()
There's a spurious comma in a scope_guard(). Remove it.
- Fix race in per CPU buffer swap in the ring buffer
When a per CPU buffer swap happens, it must make sure that it doesn't
occur while a writer is active. Instead it returns an -EBUSY. But
there's a small race window when a writer moves from one sub-buffer
to the next that it resets the "committing" counter. If a swap
happens at that moment, the buffer used for the commit of an event
will not match the buffer the event is actually on. Instead of using
the "committing" counter, use the recursive detection counter that
does not get reset when the writer crosses sub-buffers.
- Fix off-by-one in ftrace_free_mem()
The function ftrace_free_mem() gets an "end_ptr" as a parameter that
is exclusive to the rang to be freed. But its value is used to search
for the records that expects an inclusive value. Subtract one from
the parameter to convert it to an inclusive range.
- Disable resizing of the ring buffer for persistent buffers
Resizing the persistent buffer has undefined behavior. Prevent it
from being resized.
- Disable changing ring buffer subbuf order when resizing is disabled
The ring buffer subbuffer order can not be changed during resizing.
Use that instead of just checking if the buffer is mapped as mapped
buffers also have resizing disabled.
- Initialize subbuf_order of reader pages when they are created
In rb_allocate_cpu_buffer() the bpage->order is not updated to the
current subbuf_order leaving it as zero. This value is used when the
page is freed.
- Fix test_ringbuffer() to test for ERR_PTR before calling
kthread_stop()
The rb_threads[] array is assigned the output of kthread_run_on_cpu()
which could return an ERR_PTR. At the end of the test, all threads in
the array are cleaned up by kthread_stop() passing in the value in
the array if it isn't zero. But if the array contains an ERR_PTR,
kthread_stop() will not be able to handle it properly.
* tag 'trace-v7.2-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
ring-buffer: Fix crash passing ERR_PTR to kthread_stop()
ring-buffer: Initialise reader page order in rb_allocate_cpu_buffer()
ring-buffer: Prevent subbuf order change when resizing is disabled
ring-buffer: Prevent resizing of persistent ring buffer
ftrace: Fix off-by-one fentry site disable in ftrace_free_mem()
ring-buffer: Use current_context for safe per-CPU buffer swap
ftrace: Drop extra comma in trace_buffered_event_enable
ftrace: Protect direct_functions in update_ftrace_direct_mod
ftrace: Protect direct_functions in update_ftrace_direct_del
ftrace: Protect direct_functions in ftrace_find_rec_direct
eventfs: Use children field for rcu head and add memory barriers
eventfs: Fix use-after-free in eventfs_remove_rec()
|
|
In test_ringbuffer()'s out_free cleanup loop, the check
`!rb_threads[cpu]` only catches NULL entries and misses entries that
hold an ERR_PTR.
rb_threads[] is static, so unassigned slots are NULL. But when
kthread_run_on_cpu() fails for a cpu, it stores ERR_PTR(-ENOMEM) (or
-EINTR) in rb_threads[cpu] before the creation loop jumps to out_free.
That entry is non-NULL, so the old `!ptr` check does not break, and the
cleanup proceeds to call kthread_stop() on the ERR_PTR. kthread_stop()
then dereferences the bogus pointer, crashing the kernel during the
late_initcall self-test.
crash logs:
BUG: kernel NULL pointer dereference, address: 000000000000001c
Oops: 0002 [#1] SMP NOPTI
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 7.2.0-rc6-dirty #7 PREEMPT(lazy)
RIP: 0010:kthread_stop+0x2e/0x220
RBX: fffffffffffffff4
CR2: 000000000000001c
Call Trace:
<TASK>
test_ringbuffer+0x1ec/0x650
do_one_initcall+0x6c/0x2c0
kernel_init_freeable+0x21d/0x420
kernel_init+0x15/0x1c0
ret_from_fork+0x21b/0x320
</TASK>
Kernel panic - not syncing: Fatal exception
Cc: stable@vger.kernel.org
Fixes: 64ed3a049e3e ("ring-buffer: make use of the helper function kthread_run_on_cpu()")
Link: https://patch.msgid.link/20260807154145.2846521-2-sh_def@163.com
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
In rb_allocate_cpu_buffer(), bpage->order was omitted, leaving it as 0.
This is an issue for a ring-buffer with subbufs bigger than PAGE_SIZE if
when freed: free_buffer_page() relies on this value. Align the value
with the actual allocation size (buffer::subbuf_order).
Cc: stable@vger.kernel.org
Fixes: f9b94daa542a ("ring-buffer: Set new size of the ring buffer sub page")
Link: https://patch.msgid.link/20260806211306.3704194-4-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Because ring_buffer_subbuf_order_set() frees buffer pages, we can't
allow it when resizing is disabled. A non-consuming reader is at risk of
use-after-free (rb_advance_iter()).
Return -EBUSY on resize_disabled, matching ring_buffer_resize()
behaviour.
Cc: stable@vger.kernel.org
Fixes: f9b94daa542a ("ring-buffer: Set new size of the ring buffer sub page")
Link: https://patch.msgid.link/20260806211306.3704194-3-vdonnefort@google.com
Reported-by: syzbot+e0cc44465d6bae735679@syzkaller.appspotmail.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Dynamically resizing a persistent ring buffer is not possible. Disable
the feature.
Cc: stable@vger.kernel.org
Fixes: be68d63a139b ("ring-buffer: Add ring_buffer_alloc_range()")
Link: https://patch.msgid.link/20260806211306.3704194-2-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
