| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
Pull workqueue fixes from Tejun Heo:
- An unbound worker pool could be freed while still reachable through
the pending-activation list, leading to a use-after-free. Unlink
before dropping the reference
- On PREEMPT_RT, the BH workqueue kick raised softirqs from preemptible
context, tripping a lockdep assertion and possibly losing
concurrently raised softirq bits
- Draining BH work off a dead CPU nests two pools' callback locks,
which lockdep misreported as recursive locking. The nesting cannot
deadlock. Annotate it
- Reject watchdog thresholds that overflow the conversion to jiffies
- Make the drgn workqueue dump script work again on kernels and vmcores
from before the workqueue attrs field rename
* tag 'wq-for-7.3-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
tools/workqueue/wq_dump.py: Support backward compatibility for wq->attrs rename
workqueue: reject watchdog thresholds that overflow jiffies
workqueue: Fix unbound pool lifetime for pending pwqs
workqueue: Use raise_softirq() to trigger softirq in irq_work handler
workqueue: Annotate cb_lock nesting when draining a dead BH pool
|
|
The watchdog threshold is supplied in seconds but is multiplied by HZ
before being used as a jiffies interval. Reject values that exceed
MAX_JIFFY_OFFSET / HZ so the multiplication cannot wrap and the
time_after() comparisons remain within their supported range.
The check is performed before changing the threshold or watchdog timer.
Zero remains the value used to disable the watchdog.
Fixes: 82607adcf9cdf ("workqueue: implement lockup detector")
Signed-off-by: Jiacheng Xu <stitch@zju.edu.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull NFS client updates from Trond Myklebust:
"Highlights include:
Stable fixes:
- Use-after-free fixes for the sunrpc client code
- Delegation hash table leak
- NULL dereference on lockowner allocation failure
- Fix a handshake completion race in the TLS code
- Fix an error sign checking issue when deciding whether the pNFS
layout is still in use, or can be returned
- Fix a layout segment leak in pnfs_layout_process()
Other bugfixes:
- Fix a missing NULL check in the rpcbind client
- annotate shared socket callbacks with READ_ONCE/WRITE_ONCE
- nfs_inode_set_delegation() error paths should return the delegation
- Use clear_and_wake_up_bit() in nfs_clear_invalid_mapping() and the
pNFS code.
- Fix the nfs4_alloc_client() error paths to free the IDR allocation
- fix folio dereference before NULL check in
nfs_inode_remove_request()
- Fix delayed delegation return
- Fix another state manager race with umount
- Fix device leaks on parse failure
- Avoid cancelling in-flight I/O during a layout recall if the server
doesn't require it
- flexfiles: report cancelled I/O as a layout error
- flexfiles: fix NULL dereference for NFSv4.0 data servers
- Fix incorrect argument passed to nfs4_delete_lease()
- Fix several symlink issues resulting from nfs_atomic_open_v23()
- Fix an uninitialised variable issue in the NFSv4.1 callback code
- fix LAYOUTSTATS send buffer exhaustion
Features and cleanups:
- NFSv4.2: Allow the server to specify that file data may not be cached
- localio: optimise I/O submission when when not doing memory reclaim
- localio: Remove duplicate wait code in nfs_local_commit
- flexfiles: support loosely coupled NFSv4.x data servers
- pNFS: key the data server cache on the NFS version"
* tag 'nfs-for-7.3-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs: (33 commits)
NFSv4.1: fix layout segment leak on the pnfs_layout_process() forget path
NFSv4/pnfs: key the data server cache on the NFS version
NFSv4.2: fix LAYOUTSTATS send buffer exhaustion
pNFS: Fix EBUSY check in pnfs_layout_need_return
NFSv4.1: zero referring call lists before decoding
nfs: fix ENXIO on O_CREAT open of existing symlink over NFSv3
SUNRPC: wait for in-flight client TLS handshake callback
NFSv4: Fix incorrect argument passed to nfs4_delete_lease() in nfs4_add_lease()
lockd: fix NULL dereference on lockowner allocation failure
NFS: fix delegation_hash_table leak when nfs4_server_common_setup() fails
NFSv4/flexfiles: support loosely coupled data servers
NFSv4/flexfiles: fix NULL dereference for NFSv4.0 data servers
NFSv4: pin the superblock for active state owners
sunrpc: fix use-after-free in __rpc_clnt_handle_event and __rpc_clnt_remove_pipedir
NFS/localio: issue commit inline when not in a memory-reclaim context
NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit
NFS/localio: issue IO inline when not in a memory-reclaim context
NFS: Fix delayed delegation return list handling
NFS: Verify symlink inode before caching target
NFS: fix folio dereference before NULL check in nfs_inode_remove_request()
...
|
|
KASAN reports a use-after-free of an unbound worker_pool in
node_activate_pending_pwq():
BUG: KASAN: slab-use-after-free in _raw_spin_trylock+0x6d/0x120
Read of size 4 at addr ffff8880089ce000 by task kworker/u22:0/318
CPU: 1 UID: 0 PID: 318 Comm: kworker/u22:0 Not tainted 7.2.0 #1 PREEMPT(lazy)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: 0x0 (flush-8:0)
Call Trace:
<TASK>
dump_stack_lvl+0x53/0x70
print_report+0xce/0x610
kasan_report+0xce/0x100
_raw_spin_trylock+0x6d/0x120
pwq_dec_nr_in_flight+0x4b4/0xcb0
process_one_work+0x921/0x11a0
worker_thread+0x4d0/0xd20
kthread+0x2de/0x3c0
ret_from_fork+0x3aa/0x620
ret_from_fork_asm+0x1a/0x30
</TASK>
Allocated by task 311:
alloc_pwq+0x439/0xca0
apply_wqattrs_prepare+0x75e/0xd10
apply_workqueue_attrs_locked+0x44/0xa0
wq_nice_store+0x350/0x450
Freed by task 0:
kfree+0x127/0x3b0
rcu_core+0x523/0x1780
handle_softirqs+0x1b3/0x610
Last potentially related work creation:
put_unbound_pool+0x3f3/0x7d0
pwq_release_workfn+0x494/0x8e0
kthread_worker_fn+0x1ff/0x790
Canceling the last inactive work skips pwq_dec_nr_active(), so an empty
pwq can remain on pending_pwqs when its refcnt reaches zero.
pwq_release_workfn() currently puts the pool before removing that pwq.
If this drops the last pool reference, the pool can be RCU-freed while
the pwq remains reachable, and node_activate_pending_pwq() may trylock
the freed pool->lock.
Remove the pwq from pending_pwqs before putting the pool.
Fixes: 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")
Cc: stable@vger.kernel.org
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
bh_pool_kick_normal() and bh_pool_kick_highpri() are registered via
init_irq_work() without the IRQ_WORK_HARD_IRQ flag. On PREEMPT_RT, such
irq_work items are processed by the per-CPU irq_workd kthread in preemptible
task context with IRQs enabled. However, raise_softirq_irqoff() requires
IRQs to be disabled. Calling it from irq_workd trips the lockdep assertion
in __raise_softirq_irqoff() and the non-atomic update of the softirq pending
mask can lose bits raised by an interrupt on the same CPU. Replace
raise_softirq_irqoff() with raise_softirq() in the irq_work handlers.
Fixes: 2f34d7337d98 ("workqueue: Fix queue_work_on() with BH workqueues")
Cc: stable@vger.kernel.org # v6.9+
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
On PREEMPT_RT, bh_worker() wraps work item execution in pool->cb_lock to
provide a handshake for canceling BH work items. When a CPU goes down,
drain_dead_softirq_workfn() runs the dead pool's bh_worker() nested inside
the local pool's bh_worker(), acquiring the cb_locks of two different pools
without a nesting annotation. lockdep reports possible recursive locking:
============================================
WARNING: possible recursive locking detected
--------------------------------------------
ktimers/0/16 is trying to acquire lock:
ffff8880b873a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
but task is already holding lock:
ffff8880b863a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
Call Trace:
bh_worker+0x7d/0x880 kernel/workqueue.c:3688
drain_dead_softirq_workfn+0x95/0x220 kernel/workqueue.c:3763
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3405
bh_worker+0x46a/0x880 kernel/workqueue.c:3708
tasklet_action+0xc/0x70 kernel/softirq.c:965
The nesting can't deadlock. A pool's bh_worker() runs nested only while the
pool's CPU is dead, entered from a live pool's bh_worker() on the draining
CPU, so the ordering is always live to dead. CPU hotplug operations are
serialized and the drain is synchronous, so the nesting depth never exceeds
two. Annotate the inner acquisition with SINGLE_DEPTH_NESTING.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: syzbot+1bd20115328f8254ed62@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=1bd20115328f8254ed62
Fixes: ad7c7f4b9c6c ("workqueue: Provide a handshake for canceling BH workers")
Cc: stable@vger.kernel.org # v6.18+
|
|
Every LOCALIO read and write is currently bounced through the dedicated
!WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually
required when the submitting context is a memory-reclaim context: LOCALIO
issues IO directly into a stacked local filesystem (e.g. XFS) which may in
turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a
WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or
an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and
risks a forward-progress deadlock, which is why commit b9f5dd57f4a5
("nfs/localio: use dedicated workqueues for filesystem read and write")
introduced the intermediate workqueue.
Outside of reclaim context -- ordinary application/task submission such as
O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and
merely adds a context switch and scheduling latency per IO while discarding
the NFS client's inherent application-context parallelism.
Add current_is_workqueue_mem_reclaim(), which reports whether %current is a
WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency()
warns on. Use it, together with the PF_MEMALLOC check, in the new
nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read()
and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue
the IO inline. Buffered writeback continues to bounce (wb_workfn is a
WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.
Running nfs_local_call_write() inline is safe: it already saves and
restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it
sets and scopes the file opener's creds. The async O_DIRECT completion
path is likewise unaffected: when the underlying filesystem returns
-EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete /
nfs_local_write_aio_complete) can run in bottom-half context and so must
still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to
nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is
independent of how the IO was submitted, and this change leaves it as-is;
only the submission side stops unconditionally hopping through
nfslocaliod_workqueue.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can
update it concurrently. KCSAN reports:
BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8
race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by
task 1817002 on cpu 40:
kick_pool_pick+0xf8/0x2d8
process_scheduled_works+0x2bc/0x888
worker_thread+0x394/0x548
kthread+0x1b8/0x1f0
ret_from_fork+0x10/0x20
value changed: 0x0000002b -> 0x0000002f
The race is harmless. wake_cpu is a best-effort placement hint: every
writer stores a valid CPU id and the wakeup path validates it through
select_task_rq(), so a stale value only affects which CPU the worker
wakes up on.
Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that
they are intentionally racy and to stop the compiler from reloading or
tearing them.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but
every caller dereferences the result right away, so the WARN_ON_ONCE()
only moves the oops one frame up, as raised by Tejun.
Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE();
Fixes: b72fdc651056 ("workqueue: account nr_active by the backing pool")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation
path fills it in with plain loads and stores, which sparse flags:
kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@
Allocate the array as __rcu pointers and publish each pwq with
rcu_assign_pointer() once it is initialized and linked, the order
install_unbound_pwq() uses.
The warnings are not new: commit 79f23600bc7b ("workqueue: factor out
get_percpu_pool()") only turned the flagged assignment into an
initializer.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain
load, so sparse complains about the dropped __rcu:
kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@
A pwq is released with kfree_rcu() and the read is protected by the
surrounding preempt_disable(), which is what
commit fd5081f4ef33 ("workqueue: Remove redundant rcu_read_lock/unlock() in
workqueue_congested()") relied on when it dropped the rcu_read_lock()
here.
Use the rcu_dereference_sched() helper to make that explicit.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
apply_wqattrs_commit() updates node_nr_active->max unconditionally.
wq->node_nr_active[] is only allocated for unbound workqueues, so guard
the call before per-cpu workqueues start using this path.
No functional change: only unbound workqueues reach apply_wqattrs_*()
today.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
This allocates a pwq and binds it to the pool @attrs asks for.
Which pool that is becomes a property of the attrs (once per-cpu becomes
an affinity scope).
Remove the 'unbound" from the function name, given it will be bigger
than unbound.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The attrs are where the affinity scope lives, and a per-cpu workqueue
will need one once per-cpu becomes a scope rather than a separate
backend. Allocate them unconditionally.
wq_dump.py used a non-NULL wq->attrs as its test for an unbound
workqueue, which no longer holds; test WQ_UNBOUND there instead.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The unbound prefix says which workqueues currently have the field rather
than what it holds, and the next patch allocates it for every workqueue.
Rename it first so that change stays a single line.
tools/workqueue/wq_dump.py reads the field by name, so rename it there
too.
wq_sysfs_unbound_attrs[] keeps its name: it is the set of sysfs files
that only unbound workqueues expose.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
workqueue_online_cpu() and workqueue_offline_cpu() decide whether a
workqueue needs a pod affinity update by testing wq->unbound_attrs for
NULL, which is only meaningful because the attrs are allocated for
unbound workqueues alone.
Test the flag instead, so the attrs can later be allocated for every
workqueue.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
pwq_tryinc_nr_active() and pwq_dec_nr_active() choose between the shared
per-node nr_active and the plain per-pwq one by testing
wq_node_nr_active() for NULL.
Test the backing pool with is_percpu_pool() instead, so the accounting
follows the pool that runs the work rather than the workqueue type.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Add is_percpu_pool() and test the pool directly for per cpu. Convert the
other open-coded pool->cpu checks -- in put_unbound_pool(),
pool_allowed_cpus() and the workqueue watchdog -- to the same helper.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a
helper. The inner allocation-failure path now returns -ENOMEM and the
caller jumps to the existing enomem cleanup, equivalent to the previous
goto.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Move the static per-cpu worker_pool lookup in alloc_and_link_pwqs() into
a helper, get_percpu_pool(), so the lookup can be shared by other
pool-selection paths.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The automatic CPU-intensive work item detection reads the worker task's
se.sum_exec_runtime without a lock in wq_worker_running(),
wq_worker_tick() and process_one_work(). The scheduler updates that
field under the rq lock (from the tick via update_curr(), or cross-CPU
via task_sched_runtime()), raising:
BUG: KCSAN: data-race in wq_worker_running+0xa8/0xe8
race at unknown origin, with read to 0xffff0009a11d1df8 of 8 bytes by task 238535 on cpu 68:
wq_worker_running
schedule
schedule_preempt_disabled
__mutex_lock
mutex_lock_nested
cgroup_bpf_release
process_one_work
worker_thread
kthread
ret_from_fork
value changed: 0x0000000088482ba0 -> 0x00000000884893c0
The value only feeds a heuristic, so the race is benign-ish. Unlike
commit ecf5aad9a441 ("workqueue: annotate racy PWQ_STAT_CPU_TIME update
in wq_worker_tick()") that only needs data_race(), these are plain reads
whose result drives a subtraction and comparison, so use READ_ONCE() for
a single, non-torn load, which also silences KCSAN.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
wq_worker_tick() bumps pwq->stats[PWQ_STAT_CPU_TIME] on every scheduler
tick before pool->lock is taken. For unbound workqueues the
pool_workqueue is shared by all workers of the pool across CPUs, so
concurrent ticks on different CPUs perform an unsynchronized 64-bit
read-modify-write on the same counter. KCSAN reports this as a
data-race:
BUG: KCSAN: data-race in wq_worker_tick / wq_worker_tick
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 29:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
update_process_times+0x200/0x260
tick_nohz_handler+0x230/0x2f8
__hrtimer_run_queues+0x1ec/0x6c8
hrtimer_interrupt+0x174/0x4b8
...
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 24:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
...
value changed: 0x000000000010a1d0 -> 0x000000000010a9a0
The counter is purely advisory, so an occasional lost update is
harmless, and every other stats[] update already runs under pool->lock.
Annotate the update with data_race().
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
To identify the task most likely responsible for a stall, add
last_woken_worker (L: pool->lock) to worker_pool and record it in
kick_pool() just before wake_up_process(). This captures the idle
worker that was kicked to take over when the last running worker went to
sleep; if the pool is now stuck with no running worker, that task is the
prime suspect and its backtrace is dumped by show_pool_no_running_worker().
Using struct worker * rather than struct task_struct * avoids any
lifetime concern: workers are only destroyed via set_worker_dying()
which requires pool->lock, and set_worker_dying() clears
last_woken_worker when the dying worker matches.
show_cpu_pool_busy_workers() holds pool->lock while calling
sched_show_task(), so last_woken_worker is either NULL or points to a
live worker with a valid task. More precisely, set_worker_dying() clears
last_woken_worker before setting WORKER_DIE, so a non-NULL
last_woken_worker means the kthread has not yet exited and worker->task
is still alive.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When a CPU pool is stalled with no running worker, the task occupying the
CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace
for the stalled CPU to capture what it is currently executing.
The CPU is snapshotted under pool->lock and the backtrace is triggered
after releasing the lock to avoid any potential issues with NMI delivery.
Skip the backtrace when the CPU is offline. A pool disassociated by CPU
hotplug keeps its pool->cpu, and an NMI to an offline CPU is never acked,
so nmi_trigger_cpumask_backtrace() would busy-wait for its full timeout
in the watchdog's timer context.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
show_cpu_pool_busy_workers() dumps every in-flight worker in the pool's
busy_hash, including workers that are not currently running on the CPU.
Restore the task_is_running() filter so only running workers are dumped.
When no running worker is found the pool may be stuck, unable to wake an
idle worker to process pending work, and the watchdog would otherwise
give no feedback. Add show_pool_no_running_worker() to report the pool
id, CPU, idle state, and worker counts in that case.
The pool info message is printed inside pool->lock using
printk_deferred_enter/exit, the same pattern used by the existing
busy-worker loop, to avoid deadlocks with console drivers that queue
work while holding locks also taken in their write paths.
This has been running on the Meta fleet for a while and caught some real
issues, for instance EFI stalls stalling the workqueue [1].
Link: https://lore.kernel.org/all/20260616-efi_timeout-v3-0-76dd1d26657b@debian.org/ [1]
Suggested-by: Petr Mladek <pmladek@suse.com>
Fixes: 8823eaef45da7 ("workqueue: Show all busy workers in stall diagnostics")
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() after the lock is dropped.
Unlike __queue_work(), this path has no surrounding RCU section, so take
rcu_read_lock() before dropping pool->lock to keep the picked worker's
task_struct valid across the wakeup.
Signed-off-by: Breno Leitao <leitao@debian.org>
Tested-by: Krishna Magar <kmagar@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
__queue_work() is the enqueue hot path: it inserts the work item and
calls kick_pool() while holding pool->lock. kick_pool() ends in a
wakeup, which takes the target task's rq->lock, so rq->lock nests under
pool->lock on every enqueue that wakes a worker on a contended unbound
pool.
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() right after dropping the lock.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Factor the worker selection out of kick_pool() into kick_pool_pick(),
which picks and claims the worker under pool->lock but, instead of waking
it, returns the worker's task via an out-param so the caller can issue
the wakeup after dropping pool->lock. BH kicks and wake_cpu setup still
happen under the lock.
kick_pool() becomes a thin wrapper that wakes the returned task, so all
existing callers keep waking under pool->lock.
Pure refactor, no functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull workqueue updates from Tejun Heo:
- Continued progress toward making alloc_workqueue() unbound by
default: more callers converted to WQ_PERCPU / system_percpu_wq /
system_dfl_wq, and new warnings for queues that use neither WQ_PERCPU
nor WQ_UNBOUND or the legacy system_wq / system_unbound_wq.
- Misc: drop the now-trivial apply_wqattrs_lock()/unlock() wrappers,
forbid the TEST_WORKQUEUE benchmark from being built-in, and fix a
spurious pointer level in the worker debug-dump path.
* tag 'wq-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
drm/bridge: anx7625: Add WQ_PERCPU add to alloc_workqueue
wifi: ath6kl: fix invalid workqueue flags in ath6kl_usb_create()
btrfs: Drop WQ_PERCPU from ordered_flags in btrfs_init_workqueues()
workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present
workqueue: Add warnings and fallback if system_{unbound}_wq is used
workqueue: drop spurious '*' from print_worker_info() fn declaration
workqueue: forbid TEST_WORKQUEUE from being built-in
workqueue: drop apply_wqattrs_lock()/unlock() wrappers
umh: replace use of system_unbound_wq with system_dfl_wq
rapidio: rio: add WQ_PERCPU to alloc_workqueue users
media: ddbridge: add WQ_PERCPU to alloc_workqueue users
platform: cznic: turris-omnia-mcu: replace use of system_wq with system_percpu_wq
media: synopsys: hdmirx: replace use of system_unbound_wq with system_dfl_wq
virt: acrn: Add WQ_PERCPU to alloc_workqueue users
|
|
gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"SMP load-balancing updates:
- A large series to introduce infrastructure for cache-aware load
balancing, with the goal of co-locating tasks that share data
within the same Last Level Cache (LLC) domain. By improving cache
locality, the scheduler can reduce cache bouncing and cache misses,
ultimately improving data access efficiency.
Implemented by Chen Yu and Tim Chen, based on early prototype work
by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and
Shrikanth Hegde.
- A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde)
Fair scheduler updates:
- A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing
SMT awareness (Andrea Righi, K Prateek Nayak)
- A series to optimize cfs_rq and sched_entity allocation for better
data locality (Zecheng Li)
- A preparatory series to change fair/cgroup scheduling to a single
runqueue, without the final change (Peter Zijlstra)
- Auto-manage ext/fair dl_server bandwidth (Andrea Righi)
- Fix cpu_util runnable_avg arithmetic (Hongyan Xia)
- Optimize update_tg_load_avg()'s rate-limiting code (Rik van Riel)
- Allow account_cfs_rq_runtime() to throttle current hierarchy
(K Prateek Nayak)
- Update util_est after updating util_avg during dequeue, to fix the
util signal update logic, which reduces signal noise (Vincent
Guittot)
Scheduler topology updates:
- Allow multiple domains to claim sched_domain_shared (K Prateek
Nayak)
- Add parameter to split LLC (Peter Zijlstra)
Core scheduler updates:
- Use trace_call__<tp>() to save a static branch (Gabriele Monaco)
Scheduler statistics updates:
- Drop now-stale mul_u64_u64_div_u64() cputime over-approximation
guard (Nicolas Pitre)
Deadline scheduler updates:
- Reject debugfs dl_server writes for offline CPUs (Andrea Righi)
- Fix replenishment logic for non-deferred servers (Yuri Andriaccio)
RT scheduling updates:
- Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt)
- Update default bandwidth for real-time tasks to 1.0 (Yuri
Andriaccio)
Proxy scheduling updates:
- A series to implement Optimized Donor Migration for Proxy Execution
(John Stultz, Peter Zijlstra)
- Various proxy scheduling cleanups and fixes (Peter Zijlstra,
K Prateek Nayak)
Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi,
Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde,
Peter Zijlstra, Liang Luo and Yiyang Chen"
* tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (91 commits)
sched/fair: Fix newidle vs core-sched
sched/deadline: Use task_on_rq_migrating() helper
sched/core: Combine separate 'else' and 'if' statements
sched/fair: Fix cpu_util runnable_avg arithmetic
sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()
sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up()
sched/fair: Call update_curr() before unthrottling the hierarchy
sched/fair: Use throttled_csd_list for local unthrottle
sched/fair: Convert cfs bandwidth throttling to use guards
sched/fair: Allocate cfs_tg_state with percpu allocator
sched/fair: Remove task_group->se pointer array
sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state
sched: restore timer_slack_ns when resetting RT policy on fork
MAINTAINERS: Fix spelling mistake in Peter's name
sched: Simplify ttwu_runnable()
sched/proxy: Remove superfluous clear_task_blocked_in()
sched/proxy: Remove PROXY_WAKING
sched/proxy: Switch proxy to use p->is_blocked
sched/proxy: Only return migrate when needed
sched: Be more strict about p->is_blocked
...
|
|
Currently there are no checks in order to enforce the use of one between
WQ_PERCPU or WQ_UNBOUND.
So act as following:
- if neither of them is present, set WQ_PERCPU
- if both are present, remove WQ_PERCPU
Along with this change, WARN_ONCE(), so that the code still uses both or
neither of them, can be changed.
Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Currently many users transitioned already to the new introduced workqueue
(system_percpu_wq, system_dfl_wq), but there are new users who still use the
older system_wq and system_unbound_wq.
This change try to push this transition forward, by warning whether the old
workqueues are used.
Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
print_worker_info() declares its local 'fn' as work_func_t * but
worker->current_func has type work_func_t (a function pointer). The
extra level of indirection is wrong and only happens to be harmless
today because every supported Linux architecture has
sizeof(work_func_t) == sizeof(work_func_t *):
copy_from_kernel_nofault() reads the correct number of bytes by
accident, and %ps still resolves the printed address because the
stored value is the function address regardless of declared type.
On any future ABI where sizeof(void (*)()) differs from
sizeof(void *), the nofault copy would transfer the wrong number of
bytes and the subsequent %ps would print an incorrect address.
Match the field type so the intent is explicit and the code does not
silently rely on equal pointer sizes.
Fixes: 3d1cb2059d93 ("workqueue: include workqueue info when printing debug dump of a worker task")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Now, that cpu_smt_mask is defined as cpumask_of(cpu) for
CONFIG_SCHED_SMT=n, it is possible to get rid of the ifdeffery.
Effectively,
- This makes sched_smt_present is defined always
- cpumask_weight(cpumask_of(cpu)) == 1. So sched_smt_present_inc/dec
will never enable the sched_smt_present. Which is expected.
- Paths that were compile-time eliminated become runtime guarded
using static keys.
- Defines set_idle_cores, test_idle_cores, etc which could likely benefit
the CONFIG_SCHED_SMT=n systems to use the same optimizations within the
LLC at wakeups.
- This will expose sched_smt_present symbol for CONFIG_SCHED_SMT=n.
Likely not a concern.
- There is a bloat of code CONFIG_SCHED_SMT=n. (NR_CPUS=2048)
add/remove: 24/18 grow/shrink: 26/28 up/down: 6396/-3188 (3208)
Total: Before=30629880, After=30633088, chg +0.01%
- No code bloat for CONFIG_SCHED_SMT=y, which is expected.
- Add comments around stop_core_cpuslocked on why ifdefs are not
removed.
- This leaves the remaining uses of CONFIG_SCHED_SMT mainly for
topology building bits which has a policy based decision.
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Phil Auld <pauld@redhat.com>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Acked-by: Tejun Heo <tj@kernel.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260515172456.542799-3-sshegde@linux.ibm.com
|
|
The apply_wqattrs_lock()/unlock() helpers were introduced by
commit a0111cf6710b ("workqueue: separate out and refactor the locking
of applying attrs") to encapsulate the get_online_cpus() (later
cpus_read_lock()) + mutex_lock(&wq_pool_mutex) acquire pair that was
duplicated across the apply-attrs paths.
Since commit 19af45757383 ("workqueue: Remove cpus_read_lock() from
apply_wqattrs_lock()") removed the cpus_read_lock() (pwq creation and
installation now operate on wq_online_cpumask, so CPU hotplug no longer
needs to be excluded), the wrappers have been one-line forwarders to
mutex_lock(&wq_pool_mutex)/mutex_unlock(&wq_pool_mutex).
They no longer encode any non-trivial locking rule and obscure the fact
that callers just take the existing wq_pool_mutex. This align with the
"unnecessary" helpers that got discussed in [1]
Inline the eight call sites and remove the wrappers. No functional
change.
Link: https://lore.kernel.org/all/afs_44-6ToJJVZTn@gmail.com/ [1]
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
For WQ_UNBOUND workqueues, alloc_and_link_pwqs() allocates wq->cpu_pwq
via alloc_percpu() and then calls apply_workqueue_attrs_locked(). On
failure it returns the error directly, bypassing the enomem: label
which holds the only free_percpu(wq->cpu_pwq) in this function.
The caller's error path kfree()s wq without touching wq->cpu_pwq,
leaking one percpu pointer table (nr_cpu_ids * sizeof(void *) bytes) per
failed call.
If kmemleak is enabled, we can see:
unreferenced object (percpu) 0xc0fffa5b121048 (size 8):
comm "insmod", pid 776, jiffies 4294682844
backtrace (crc 0):
pcpu_alloc_noprof+0x665/0xac0
__alloc_workqueue+0x33f/0xa20
alloc_workqueue_noprof+0x60/0x100
Route the error through the existing enomem: cleanup and any error
before this one.
Cc: stable@kernel.org
Fixes: 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The caller of __queue_work() owns WORK_STRUCT_PENDING, won via
test_and_set_bit() in queue_work_on()/__queue_delayed_work(). The
state machine documented above __queue_work() requires that owner
to either hand the token to a pwq (insert_work() -> set_work_pwq()),
hand it to a timer, or release it via set_work_pool_and_clear_pending().
try_to_grab_pending() relies on this: when it observes
"PENDING && off-queue" it busy-loops, trusting the current owner to
make progress.
The (__WQ_DESTROYING | __WQ_DRAINING) early-return path violates that
contract. It WARN_ONCE()s and bare-returns, leaving work->data with
PENDING set, WORK_STRUCT_PWQ clear, and work->entry empty.
The path is reachable without explicit API abuse: queue_delayed_work()
arms a timer with PENDING set; if drain_workqueue() runs while the
timer is still pending, delayed_work_timer_fn() -> __queue_work() in
softirq context hits the WARN, current is not a wq worker so
is_chained_work() is false, and the work is silently dropped with
PENDING leaked.
Mirror what clear_pending_if_disabled() already does on its analogous
reject path: unpack the off-queue data and call
set_work_pool_and_clear_pending() to release the token before
returning.
I was able to reproduce this by queueing several slow works on
a max_active=1 wq, arm a delayed_work whose timer fires while
drain_workqueue() is blocked, then call cancel_delayed_work_sync().
Without this patch the cancel livelocks at 100% CPU; with it the cancel
returns immediately.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
alloc_workqueue_va() forwards its va_list to __alloc_workqueue() which
ultimately feeds vsnprintf(). __alloc_workqueue() already carries
__printf(1, 0); the new wrapper needs the same annotation so format
string checking propagates through the forwarding.
Fixes: 0de4cb473aed ("workqueue: fix devm_alloc_workqueue() va_list misuse")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202604300347.2LgXyteh-lkp@intel.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
devm_alloc_workqueue() built a va_list and passed it as a single
positional argument to the variadic alloc_workqueue() macro:
va_start(args, max_active);
wq = alloc_workqueue(fmt, flags, max_active, args);
va_end(args);
C does not allow forwarding a va_list through a ... parameter.
alloc_workqueue() expands to alloc_workqueue_noprof(), which runs
its own va_start() over its ... params, so the inner
vsnprintf(wq->name, sizeof(wq->name), fmt, args) in
__alloc_workqueue() received the outer va_list object as the first
variadic slot rather than the caller's actual format arguments.
Add a new static helper alloc_workqueue_va() that wraps
__alloc_workqueue() and runs wq_init_lockdep() on success, and
fold both alloc_workqueue_noprof() and devm_alloc_workqueue_noprof()
onto it as suggested by Tejun.
The wq_init_lockdep() step is required on the devm path
too, otherwise __flush_workqueue()'s on-stack
COMPLETION_INITIALIZER_ONSTACK_MAP would NULL-deref wq->lockdep_map.
No caller changes are required. devm_alloc_ordered_workqueue() is
a macro forwarding to devm_alloc_workqueue() and inherits the fix.
Two in-tree callers actively trigger the broken path on every probe:
drivers/power/supply/mt6370-charger.c:889
drivers/power/supply/max77705_charger.c:649
both of which use devm_alloc_ordered_workqueue(dev, "%s", 0,
dev_name(dev)).
A standalone reproducer module is available at[1].
Link: https://github.com/leitao/debug/blob/main/workqueue/valist/wq_va_test.c [1]
Fixes: 1dfc9d60a69e ("workqueue: devres: Add device-managed allocate workqueue")
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull workqueue updates from Tejun Heo:
- New default WQ_AFFN_CACHE_SHARD affinity scope subdivides LLCs into
smaller shards to improve scalability on machines with many CPUs per
LLC
- Misc:
- system_dfl_long_wq for long unbound works
- devm_alloc_workqueue() for device-managed allocation
- sysfs exposure for ordered workqueues and the EFI workqueue
- removal of HK_TYPE_WQ from wq_unbound_cpumask
- various small fixes
* tag 'wq-for-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq: (21 commits)
workqueue: validate cpumask_first() result in llc_populate_cpu_shard_id()
workqueue: use NR_STD_WORKER_POOLS instead of hardcoded value
workqueue: avoid unguarded 64-bit division
docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope
workqueue: add test_workqueue benchmark module
tools/workqueue: add CACHE_SHARD support to wq_dump.py
workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope
workqueue: add WQ_AFFN_CACHE_SHARD affinity scope
workqueue: fix typo in WQ_AFFN_SMT comment
workqueue: Remove HK_TYPE_WQ from affecting wq_unbound_cpumask
workqueue: unlink pwqs from wq->pwqs list in alloc_and_link_pwqs() error path
workqueue: Remove NULL wq WARN in __queue_delayed_work()
workqueue: fix parse_affn_scope() prefix matching bug
workqueue: devres: Add device-managed allocate workqueue
workqueue: Add system_dfl_long_wq for long unbound works
tools/workqueue/wq_dump.py: add NODE prefix to all node columns
tools/workqueue/wq_dump.py: fix column alignment in node_nr/max_active section
tools/workqueue/wq_dump.py: remove backslash separator from node_nr/max_active header
efi: Allow to expose the workqueue via sysfs
workqueue: Allow to expose ordered workqueues via sysfs
...
|
|
On uniprocessor (UP) configs such as nios2, NR_CPUS is 1, so
cpu_shard_id[] is a single-element array (int[1]). In
llc_populate_cpu_shard_id(), cpumask_first(sibling_cpus) returns an
unsigned int that the compiler cannot prove is always 0, triggering
a -Warray-bounds warning when the result is used to index
cpu_shard_id[]:
kernel/workqueue.c:8321:55: warning: array subscript 1 is above
array bounds of 'int[1]' [-Warray-bounds]
8321 | cpu_shard_id[c] = cpu_shard_id[cpumask_first(sibling_cpus)];
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is a false positive: sibling_cpus can never be empty here because
'c' itself is always set in it, so cpumask_first() will always return a
valid CPU. However, the compiler cannot prove this statically, and the
warning only manifests on UP configs where the array size is 1.
Add a bounds check with WARN_ON_ONCE to silence the warning, and store
the result in a local variable to make the code clearer and avoid calling
cpumask_first() twice.
Fixes: 5920d046f7ae ("workqueue: add WQ_AFFN_CACHE_SHARD affinity scope")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202604022343.GQtkF2vO-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
use NR_STD_WORKER_POOLS for irq_work_fns[] array definition.
NR_STD_WORKER_POOLS is also 2, but better to use MACRO.
Initialization loop for_each_bh_worker_pool() also uses same MACRO.
Signed-off-by: Maninder Singh <maninder1.s@samsung.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Set WQ_AFFN_CACHE_SHARD as the default affinity scope for unbound
workqueues. On systems where many CPUs share one LLC, the previous
default (WQ_AFFN_CACHE) collapses all CPUs to a single worker pool,
causing heavy spinlock contention on pool->lock.
WQ_AFFN_CACHE_SHARD subdivides each LLC into smaller groups, providing
a better balance between locality and contention. Users can revert to
the previous behavior with workqueue.default_affinity_scope=cache.
On systems with 8 or fewer cores per LLC, CACHE_SHARD produces a single
shard covering the entire LLC, making it functionally identical to the
previous CACHE default. The sharding only activates when an LLC has more
than 8 cores.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
On systems where many CPUs share one LLC, unbound workqueues using
WQ_AFFN_CACHE collapse to a single worker pool, causing heavy spinlock
contention on pool->lock. For example, Chuck Lever measured 39% of
cycles lost to native_queued_spin_lock_slowpath on a 12-core shared-L3
NFS-over-RDMA system.
The existing affinity hierarchy (cpu, smt, cache, numa, system) offers
no intermediate option between per-LLC and per-SMT-core granularity.
Add WQ_AFFN_CACHE_SHARD, which subdivides each LLC into groups of at
most wq_cache_shard_size cores (default 8, tunable via boot parameter).
Shards are always split on core (SMT group) boundaries so that
Hyper-Threading siblings are never placed in different pods. Cores are
distributed across shards as evenly as possible -- for example, 36 cores
in a single LLC with max shard size 8 produces 5 shards of 8+7+7+7+7
cores.
The implementation follows the same comparator pattern as other affinity
scopes: precompute_cache_shard_ids() pre-fills the cpu_shard_id[] array
from the already-initialized WQ_AFFN_CACHE and WQ_AFFN_SMT topology,
and cpus_share_cache_shard() is passed to init_pod_type().
Benchmark on NVIDIA Grace (72 CPUs, single LLC, 50k items/thread), show
cache_shard delivers ~5x the throughput and ~6.5x lower p50 latency
compared to cache scope on this 72-core single-LLC system.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
inactive works
In unplug_oldest_pwq(), the first inactive work item on the
pool_workqueue is activated correctly. However, if multiple inactive
works exist on the same pool_workqueue, subsequent works fail to
activate because wq_node_nr_active.pending_pwqs is empty — the list
insertion is skipped when the pool_workqueue is plugged.
Fix this by checking for additional inactive works in
unplug_oldest_pwq() and updating wq_node_nr_active.pending_pwqs
accordingly.
Fixes: 4c065dbce1e8 ("workqueue: Enable unbound cpumask update on ordered workqueues")
Cc: stable@vger.kernel.org
Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Waiman Long <longman@redhat.com>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
|
|
For historical reason, wq_unbound_cpumask is initially set as
intersection of HK_TYPE_DOMAIN, HK_TYPE_WQ and workqueue.unbound_cpus
boot command line option.
At run time, users can update the unbound cpumask via the
/sys/devices/virtual/workqueue/cpumask sysfs file. Creation
and modification of cpuset isolated partitions will also update
wq_unbound_cpumask based on the latest HK_TYPE_DOMAIN cpumask.
The HK_TYPE_WQ cpumask is out of the picture with these runtime updates.
Complete the transition by taking HK_TYPE_WQ out from the workqueue code
and make it depends on HK_TYPE_DOMAIN only from the housekeeping side.
The final goal is to eliminate HK_TYPE_WQ as a housekeeping cpumask type.
Signed-off-by: Waiman Long <longman@redhat.com>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When alloc_and_link_pwqs() fails partway through the per-cpu allocation
loop, some pool_workqueues may have already been linked into wq->pwqs
via link_pwq(). The error path frees these pwqs with kmem_cache_free()
but never removes them from the wq->pwqs list, leaving dangling pointers
in the list.
Currently this is not exploitable because the workqueue was never added
to the global workqueues list and the caller frees the wq immediately
after. However, this makes sure that alloc_and_link_pwqs() doesn't leave
any half-baked structure, which may have side effects if not properly
cleaned up.
Fix this by unlinking each pwq from wq->pwqs before freeing it. No
locking is needed as the workqueue has not been published yet, thus
no concurrency is possible.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Try to be more explicit why the workqueue watchdog does not take
pool->lock by default. Spin locks are full memory barriers which
delay anything. Obviously, they would primary delay operations
on the related worker pools.
Explain why it is enough to prevent the false positive by re-checking
the timestamp under the pool->lock.
Finally, make it clear what would be the alternative solution in
__queue_work() which is a hotter path.
Signed-off-by: Petr Mladek <pmladek@suse.com>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
On weakly ordered architectures (e.g., arm64), the lockless check in
wq_watchdog_timer_fn() can observe a reordering between the worklist
insertion and the last_progress_ts update. Specifically, the watchdog
can see a non-empty worklist (from a list_add) while reading a stale
last_progress_ts value, causing a false positive stall report.
This was confirmed by reading pool->last_progress_ts again after holding
pool->lock in wq_watchdog_timer_fn():
workqueue watchdog: pool 7 false positive detected!
lockless_ts=4784580465 locked_ts=4785033728
diff=453263ms worklist_empty=0
To avoid slowing down the hot path (queue_work, etc.), recheck
last_progress_ts with pool->lock held. This will eliminate the false
positive with minimal overhead.
Remove two extra empty lines in wq_watchdog_timer_fn() as we are on it.
Fixes: 82607adcf9cd ("workqueue: implement lockup detector")
Cc: stable@vger.kernel.org # v4.5+
Assisted-by: claude-code:claude-opus-4-6
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Remove the WARN_ON_ONCE(!wq) which doesn't serve any useful purpose.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
