linux-stable.git/kernel/bpf/fixups.c, branch v7.3-rc2

bpf: zero extend the result of an arena 32-bit cmpxchg

2026-09-04T01:54:45+00:00

bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from
BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, and it runs before
bpf_opt_subreg_zext_lo32_rnd_hi32().

That pass emits an explicit zero extension for a 32-bit cmpxchg even
when bpf_jit_needs_zext() is false. This is done because on some
architectures 32-bit cmpxchg requires explicit zero extension for the
dst register. E.g. on x86-64 'lock cmpxchg' does not change the %eax
if comparison is successful, while BPF semantics declare that each
operation on a 32-bit register zero extends it's upper half.

is_cmpxchg_insn() matches BPF_MODE == BPF_ATOMIC only, so an arena
cmpxchg misses said zero extension adjustment. This patch adjusts
is_cmpxchg_insn() to match BPF_PROBE_ATOMIC alongside BPF_ATOMIC.

Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Reported-by: Nicholas Carlini 
Suggested-by: Nicholas Carlini 
Signed-off-by: Eduard Zingerman 
Link: https://lore.kernel.org/r/20260903171542.1438050-1-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov

bpf: Rewrite any fault prone load out of a mem or btf_id pointer

2026-08-17T08:06:42+00:00

bpf_convert_ctx_accesses() turns a BPF_LDX into a BPF_PROBE_MEM one by
matching the type recorded for the insn against a list of exact pointer
types. The list cannot keep up with the flag combinations the verifier
produces, and a type which is missing from it ends up as a plain load
without an exception table entry, so a bad address panics the kernel
instead of being handled.

Two such types exist today and are reachable:

  - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_ALLOC | NON_OWN_REF
  - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU

Rather than adding the two, just drop the list and state the property
itself in the default case of the switch. This is a superset of what
the list matched, the untrusted PTR_TO_MEM does not have to carry
MEM_RDONLY for it anymore, and it stays in sync with the verifier side
which uses the same match in save_aux_ptr_type() and reg_type_mismatch_ok().

Assert that a fault prone type which does not get the rewrite for whatever
reason is rejected at load time rather than left to fault at runtime to
catch any future cases.

Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Fixes: 6fcd486b3a0a ("bpf: Refactor RCU enforcement in the verifier.")
Signed-off-by: Daniel Borkmann 
Acked-by: Eduard Zingerman 
Link: https://lore.kernel.org/bpf/20260814215301.709827-4-daniel@iogearbox.net

bpf: Fix func_info_aux desync after dead code elimination

2026-08-13T21:41:30+00:00

The verifier keeps per-subprogram metadata in three parallel arrays:
subprog_info, func_info, and func_info_aux. Dead code elimination can
remove whole subprograms, and adjust_subprog_starts_after_remove()
shifts subprog_info and func_info to close the gap, but leaves
func_info_aux in place. From that point on, func_info_aux[i] no longer
describes subprogram i.

Shift func_info_aux together with func_info so the three arrays stay
aligned after subprogram removal.

Reported-by: Sashiko 
Signed-off-by: Kumar Kartikeya Dwivedi 
Signed-off-by: Andrii Nakryiko 
Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org
Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com

bpf: Introduce global percpu data

2026-08-13T17:27:40+00:00

Introduce global percpu data, inspired by the commit
6316f78306c1 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.

For example, in BPF, it is able to define a global percpu variable like:

int data SEC(".percpu");

With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.

Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.

Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop

Signed-off-by: Leon Hwang 
Signed-off-by: Andrii Nakryiko 
Acked-by: Eduard Zingerman 
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev

bpf: Drop duplicate blank lines in kernel/bpf/

2026-08-13T17:27:40+00:00

There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.

Drop them for cleanup.

No functional changes intended.

Signed-off-by: Leon Hwang 
Signed-off-by: Andrii Nakryiko 
Reviewed-by: Emil Tsalapatis 
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev

bpf: Eliminate dup/restore of insn_aux_data

2026-08-13T02:36:30+00:00

The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.

After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.

However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.

So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.

Signed-off-by: Xu Kuohai 
Acked-by: Eduard Zingerman 
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Derive the atomic load register in one place

2026-08-12T17:33:53+00:00

check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.

Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.

Signed-off-by: Daniel Borkmann 
Acked-by: Eduard Zingerman 
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman

bpf: Simplify the bpf_is_reg64()

2026-08-08T09:06:20+00:00

After the previous commit bpf_is_reg64() is only used in a context
where destination register's property is queried, and only for
instructions for which insn_def_regno() >= 0. Hence, simplify the
function by:

- removing unused parameters;
- removing code paths considering BPF_JMP{,32} instructions;
- streamlining the condition expressions.

Signed-off-by: Eduard Zingerman 
Signed-off-by: Daniel Borkmann 
Acked-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-6-b6c270013c77@gmail.com

bpf: Infer zext_dst based on static register liveness analysis

2026-08-08T09:06:08+00:00

As reported in the thread [1], the verifier's 32-bit operations zero
extension logic is broken. This logic is responsible for correct
semantics of 32-bit operations on s390 architecture.

According to BPF semantics, operation `w1 += 1` is supposed to zero
extend the upper half of the register `r1`. On s390 the JIT relies on
the verifier emitting explicit zero extension before such operations.

The verifier attempts to minimize the amount of zero extensions
inserted by tracking whether upper halves of the 64-bit registers are
ever used. Previously such tracking worked as follows:

- bpf_reg_state->subreg_def field was set by do_check_insn()
for each operation defining lower but not the upper halves
of the register.
- Whenever an operation reading the whole register was verified,
the verifier checked register's subreg_def and set
bpf_insn_aux_data->zext_dst flag as true via a call to
mark_insn_zext() function.
- After the verification was complete, a special pass
bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations
with bpf_insn_aux_data->zext_dst set as true by adding
explicit zero extension.

Note that the logic above relies on bpf_reg_state->subreg_def,
which is a property of a current verifier state.
Before the commit [2] two additional steps happened:

- The verifier tracked upper and lower register halves' liveness as
flags REG_LIVE_READ{32,64} in bpf_reg_state->live.
- The function propagate_liveness() called mark_insn_zext()
in order to transfer the knowledge about which registers have
their upper halves alive (and thus might require zero extension).

The commit [2] removed the two steps described above,
hence making possible a situation like below:

- The register's upper half is set and is used on some verification
path P1 and the register happens not to be marked as precise.
- The checkpoint C is created while processing some instruction
between register initialization and usage.
- On some other verification path P2 the register's upper half is not
initialized and that path ends hitting the checkpoint C.
- In such a case the register's initialization on path P2 would lack
zext_dst mark, making it possible for the program to inject
an arbitrary value in the register's upper half.

This commit replaces subreg_def based logic with computing zext_dst
statically, as a part of the bpf_compute_live_registers() analysis:

- The analysis now tracks usage of upper and lower halves of the
registers separately.
- If some instruction defines a 32-bit subregister, but not the whole
register, *and* the upper half of the register is alive after that
instruction, the instruction is marked as zext_dst.

There is one notable drop in precision: whenever a BPF subprogram is
called, all 64 bits of parameter registers are presumed to be used.
The assumption is that such a drop in precision would not inflict
a noticeable performance penalty.

[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")

Fixes: 107e16979905 ("bpf: disable and remove registers chain based liveness")
Reported-by: Min-gyu Kim
Reported-by: STAR Labs SG
Signed-off-by: Eduard Zingerman
Signed-off-by: Daniel Borkmann
Acked-by: Daniel Borkmann
Link: https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-5-b6c270013c77@gmail.com

bpf: Move bpf_is_reg64() to fixups.c

2026-08-08T09:06:02+00:00

The following patches are going to remove bpf_is_reg64() users from
everywhere except fixups.c, and also make it dependent on functions
local to fixups.c. Move the function before hand to simplify the
review. Non functional change.

Signed-off-by: Eduard Zingerman 
Signed-off-by: Daniel Borkmann 
Acked-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-3-b6c270013c77@gmail.com