linux-stable.git/include/linux/padata.h, branch v6.6.141

padata: Put CPU offline callback in ONLINE section to allow failure

2026-05-23T11:03:11+00:00

[ Upstream commit c8c4a2972f83c8b68ff03b43cecdb898939ff851 ]

syzbot reported the following warning:

    DEAD callback error for CPU1
    WARNING: kernel/cpu.c:1463 at _cpu_down+0x759/0x1020 kernel/cpu.c:1463, CPU#0: syz.0.1960/14614

at commit 4ae12d8bd9a8 ("Merge tag 'kbuild-fixes-7.0-2' of git://git.kernel.org/pub/scm/linux/kernel/git/kbuild/linux")
which tglx traced to padata_cpu_dead() given it's the only
sub-CPUHP_TEARDOWN_CPU callback that returns an error.

Failure isn't allowed in hotplug states before CPUHP_TEARDOWN_CPU
so move the CPU offline callback to the ONLINE section where failure is
possible.

Fixes: 894c9ef9780c ("padata: validate cpumask without removed CPU during offline")
Reported-by: syzbot+123e1b70473ce213f3af@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/69af0a05.050a0220.310d8.002f.GAE@google.com/
Debugged-by: Thomas Gleixner 
Signed-off-by: Daniel Jordan 
Signed-off-by: Herbert Xu 
Signed-off-by: Sasha Levin

padata: Remove comment for reorder_work

2026-05-17T15:13:26+00:00

[ Upstream commit 82a0302e7167d0b7c6cde56613db3748f8dd806d ]

Remove comment for reorder_work which no longer exists.

Reported-by: Stephen Rothwell 
Fixes: 71203f68c774 ("padata: Fix pd UAF once and for all")
Signed-off-by: Herbert Xu 
Signed-off-by: Bin Lan 
Signed-off-by: Sasha Levin

padata: Fix pd UAF once and for all

2026-05-17T15:13:26+00:00

[ Upstream commit 71203f68c7749609d7fc8ae6ad054bdedeb24f91 ]

There is a race condition/UAF in padata_reorder that goes back
to the initial commit.  A reference count is taken at the start
of the process in padata_do_parallel, and released at the end in
padata_serial_worker.

This reference count is (and only is) required for padata_replace
to function correctly.  If padata_replace is never called then
there is no issue.

In the function padata_reorder which serves as the core of padata,
as soon as padata is added to queue->serial.list, and the associated
spin lock released, that padata may be processed and the reference
count on pd would go away.

Fix this by getting the next padata before the squeue->serial lock
is released.

In order to make this possible, simplify padata_reorder by only
calling it once the next padata arrives.

Fixes: 16295bec6398 ("padata: Generic parallelization/serialization interface")
Signed-off-by: Herbert Xu 
[ Adjust context of padata_find_next(). Replace
cpumask_next_wrap(cpu, pd->cpumask.pcpu) with
cpumask_next_wrap(cpu, pd->cpumask.pcpu, -1, false) in padata_reorder() in
v6.6 according to dc5bb9b769c9 ("cpumask: deprecate cpumask_next_wrap()") and
f954a2d37637 ("padata: switch padata_find_next() to using cpumask_next_wrap()")
. ]
Signed-off-by: Bin Lan 
Signed-off-by: Sasha Levin

padata: Convert from atomic_t to refcount_t on parallel_data->refcnt

2021-07-30T02:58:36+00:00

refcount_t type and corresponding API can protect refcounters from
accidental underflow and overflow and further use-after-free situations.

Signed-off-by: Xiyu Yang 
Signed-off-by: Xin Tan 
Acked-by: Daniel Jordan 
Signed-off-by: Herbert Xu

padata: remove padata_parallel_queue

2020-07-23T07:34:18+00:00

Only its reorder field is actually used now, so remove the struct and
embed @reorder directly in parallel_data.

No functional change, just a cleanup.

Signed-off-by: Daniel Jordan 
Cc: Herbert Xu 
Cc: Steffen Klassert 
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu

padata: fold padata_alloc_possible() into padata_alloc()

2020-07-23T07:34:18+00:00

There's no reason to have two interfaces when there's only one caller.
Removing _possible saves text and simplifies future changes.

Signed-off-by: Daniel Jordan 
Cc: Herbert Xu 
Cc: Steffen Klassert 
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu

padata: remove effective cpumasks from the instance

2020-07-23T07:34:18+00:00

A padata instance has effective cpumasks that store the user-supplied
masks ANDed with the online mask, but this middleman is unnecessary.
parallel_data keeps the same information around.  Removing this saves
text and code churn in future changes.

Signed-off-by: Daniel Jordan 
Cc: Herbert Xu 
Cc: Steffen Klassert 
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu

padata: remove stop function

2020-07-23T07:34:18+00:00

padata_stop() has two callers and is unnecessary in both cases.  When
pcrypt calls it before padata_free(), it's being unloaded so there are
no outstanding padata jobs[0].  When __padata_free() calls it, it's
either along the same path or else pcrypt initialization failed, which
of course means there are also no outstanding jobs.

Removing it simplifies padata and saves text.

[0] https://lore.kernel.org/linux-crypto/20191119225017.mjrak2fwa5vccazl@gondor.apana.org.au/

Signed-off-by: Daniel Jordan 
Cc: Herbert Xu 
Cc: Steffen Klassert 
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu

padata: remove start function

2020-07-23T07:34:17+00:00

padata_start() is only used right after pcrypt allocates an instance
with all possible CPUs, when PADATA_INVALID can't happen, so there's no
need for a separate "start" step.  It can be done during allocation to
save text, make using padata easier, and avoid unneeded calls in the
future.

Signed-off-by: Daniel Jordan 
Cc: Herbert Xu 
Cc: Steffen Klassert 
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu

padata: add basic support for multithreaded jobs

2020-06-04T03:09:45+00:00

Sometimes the kernel doesn't take full advantage of system memory
bandwidth, leading to a single CPU spending excessive time in
initialization paths where the data scales with memory size.

Multithreading naturally addresses this problem.

Extend padata, a framework that handles many parallel yet singlethreaded
jobs, to also handle multithreaded jobs by adding support for splitting up
the work evenly, specifying a minimum amount of work that's appropriate
for one helper thread to do, load balancing between helpers, and
coordinating them.

This is inspired by work from Pavel Tatashin and Steve Sistare.

Signed-off-by: Daniel Jordan 
Signed-off-by: Andrew Morton 
Tested-by: Josh Triplett 
Cc: Alexander Duyck 
Cc: Alex Williamson 
Cc: Dan Williams 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Herbert Xu 
Cc: Jason Gunthorpe 
Cc: Jonathan Corbet 
Cc: Kirill Tkhai 
Cc: Michal Hocko 
Cc: Pavel Machek 
Cc: Pavel Tatashin 
Cc: Peter Zijlstra 
Cc: Randy Dunlap 
Cc: Robert Elliott 
Cc: Shile Zhang 
Cc: Steffen Klassert 
Cc: Steven Sistare 
Cc: Tejun Heo 
Cc: Zi Yan 
Link: http://lkml.kernel.org/r/20200527173608.2885243-5-daniel.m.jordan@oracle.com
Signed-off-by: Linus Torvalds