linux.git/lib/raid/xor/xor-core.c, branch master

xor: use static_call for xor_gen

2026-04-03T06:36:21+00:00

Avoid the indirect call for xor_generation by using a static_call.

Link: https://lkml.kernel.org/r/20260327061704.3707577-28-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: pass the entire operation to the low-level ops

2026-04-03T06:36:21+00:00

Currently the high-level xor code chunks up all operations into small
units for only up to 1 + 4 vectors, and passes it to four different
methods.  This means the FPU/vector context is entered and left a lot for
wide stripes, and a lot of indirect expensive indirect calls are
performed.  Switch to passing the entire gen_xor request to the low-level
ops, and provide a macro to dispatch it to the existing helper.

This reduce the number of indirect calls and FPU/vector context switches
by a factor approaching nr_stripes / 4, and also reduces source and binary
code size.

Link: https://lkml.kernel.org/r/20260327061704.3707577-27-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: add a better public API

2026-04-03T06:36:20+00:00

xor_blocks is very annoying to use, because it is limited to 4 + 1 sources
/ destinations, has an odd argument order and is completely undocumented.

Lift the code that loops around it from btrfs and async_tx/async_xor into
common code under the name xor_gen and properly document it.

[hch@lst.de: make xor_blocks less annoying to use]
 Link: https://lkml.kernel.org/r/20260327061704.3707577-24-hch@lst.de
Link: https://lkml.kernel.org/r/20260327061704.3707577-23-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: make xor.ko self-contained in lib/raid/

2026-04-03T06:36:20+00:00

Move the asm/xor.h headers to lib/raid/xor/$(SRCARCH)/xor_arch.h and
include/linux/raid/xor_impl.h to lib/raid/xor/xor_impl.h so that the
xor.ko module implementation is self-contained in lib/raid/.

As this remove the asm-generic mechanism a new kconfig symbol is added to
indicate that a architecture-specific implementations exists, and
xor_arch.h should be included.

Link: https://lkml.kernel.org/r/20260327061704.3707577-22-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: remove macro abuse for XOR implementation registrations

2026-04-03T06:36:17+00:00

Drop the pretty confusing historic XOR_TRY_TEMPLATES and
XOR_SELECT_TEMPLATE, and instead let the architectures provide a
arch_xor_init that calls either xor_register to register candidates or
xor_force to force a specific implementation.

Link: https://lkml.kernel.org/r/20260327061704.3707577-10-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: split xor.h

2026-04-03T06:36:17+00:00

Keep xor.h for the public API, and split the struct xor_block_template
definition that is only needed by the xor.ko core and
architecture-specific optimizations into a separate xor_impl.h header.

Link: https://lkml.kernel.org/r/20260327061704.3707577-9-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: cleanup registration and probing

2026-04-03T06:36:16+00:00

Originally, the XOR code benchmarked all algorithms at load time, but it
has since then been hacked multiple times to allow forcing an algorithm,
and then commit 524ccdbdfb52 ("crypto: xor - defer load time benchmark to
a later time") changed the logic to a two-step process or registration and
benchmarking, but only when built-in.

Rework this, so that the XOR_TRY_TEMPLATES macro magic now always just
deals with adding the templates to the list, and benchmarking is always
done in a second pass; for modular builds from module_init, and for the
built-in case using a separate init call level.

Link: https://lkml.kernel.org/r/20260327061704.3707577-8-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: small cleanups

2026-04-03T06:36:16+00:00

Update the to of file comment to be correct and non-redundant, and drop
the unused BH_TRACE define.

Link: https://lkml.kernel.org/r/20260327061704.3707577-7-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

xor: move to lib/raid/

2026-04-03T06:36:16+00:00

Move the RAID XOR code to lib/raid/ as it has nothing to do with the
crypto API.

Link: https://lkml.kernel.org/r/20260327061704.3707577-6-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Eric Biggers 
Tested-by: Eric Biggers 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Anton Ivanov 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chris Mason 
Cc: Christian Borntraeger 
Cc: Dan Williams 
Cc: David S. Miller 
Cc: David Sterba 
Cc: Heiko Carstens 
Cc: Herbert Xu 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jason A. Donenfeld 
Cc: Johannes Berg 
Cc: Li Nan 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Richard Henderson 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Song Liu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton