linux.git/kernel/bpf/liveness.c, branch v7.3-rc2

treewide: refresh kmalloc_obj() conversions

2026-09-05T04:37:00+00:00

This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci

This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.

Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.

Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook

bpf: Drop duplicate blank lines in kernel/bpf/

2026-08-13T17:27:40+00:00

There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.

Drop them for cleanup.

No functional changes intended.

Signed-off-by: Leon Hwang 
Signed-off-by: Andrii Nakryiko 
Reviewed-by: Emil Tsalapatis 
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev

bpf: Derive the atomic load register in one place

2026-08-12T17:33:53+00:00

check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.

Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.

Signed-off-by: Daniel Borkmann 
Acked-by: Eduard Zingerman 
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman

bpf: Infer zext_dst based on static register liveness analysis

2026-08-08T09:06:08+00:00

As reported in the thread [1], the verifier's 32-bit operations zero
extension logic is broken. This logic is responsible for correct
semantics of 32-bit operations on s390 architecture.

According to BPF semantics, operation `w1 += 1` is supposed to zero
extend the upper half of the register `r1`. On s390 the JIT relies on
the verifier emitting explicit zero extension before such operations.

The verifier attempts to minimize the amount of zero extensions
inserted by tracking whether upper halves of the 64-bit registers are
ever used. Previously such tracking worked as follows:

- bpf_reg_state->subreg_def field was set by do_check_insn()
for each operation defining lower but not the upper halves
of the register.
- Whenever an operation reading the whole register was verified,
the verifier checked register's subreg_def and set
bpf_insn_aux_data->zext_dst flag as true via a call to
mark_insn_zext() function.
- After the verification was complete, a special pass
bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations
with bpf_insn_aux_data->zext_dst set as true by adding
explicit zero extension.

Note that the logic above relies on bpf_reg_state->subreg_def,
which is a property of a current verifier state.
Before the commit [2] two additional steps happened:

- The verifier tracked upper and lower register halves' liveness as
flags REG_LIVE_READ{32,64} in bpf_reg_state->live.
- The function propagate_liveness() called mark_insn_zext()
in order to transfer the knowledge about which registers have
their upper halves alive (and thus might require zero extension).

The commit [2] removed the two steps described above,
hence making possible a situation like below:

- The register's upper half is set and is used on some verification
path P1 and the register happens not to be marked as precise.
- The checkpoint C is created while processing some instruction
between register initialization and usage.
- On some other verification path P2 the register's upper half is not
initialized and that path ends hitting the checkpoint C.
- In such a case the register's initialization on path P2 would lack
zext_dst mark, making it possible for the program to inject
an arbitrary value in the register's upper half.

This commit replaces subreg_def based logic with computing zext_dst
statically, as a part of the bpf_compute_live_registers() analysis:

- The analysis now tracks usage of upper and lower halves of the
registers separately.
- If some instruction defines a 32-bit subregister, but not the whole
register, *and* the upper half of the register is alive after that
instruction, the instruction is marked as zext_dst.

There is one notable drop in precision: whenever a BPF subprogram is
called, all 64 bits of parameter registers are presumed to be used.
The assumption is that such a drop in precision would not inflict
a noticeable performance penalty.

[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")

Fixes: 107e16979905 ("bpf: disable and remove registers chain based liveness")
Reported-by: Min-gyu Kim
Reported-by: STAR Labs SG
Signed-off-by: Eduard Zingerman
Signed-off-by: Daniel Borkmann
Acked-by: Daniel Borkmann
Link: https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-5-b6c270013c77@gmail.com

bpf: Track upper 32-bit register halves' liveness in compute_live_registers()

2026-08-08T09:06:06+00:00

Extend compute_live_registers() to track upper and lower register
halves' liveness separately. This is mostly straightforward:

- use/def masks are extended to track 2 bits per register;
- compute_insn_live_regs() is updated to properly track these
  2 bits according to the instruction semantics.

Signed-off-by: Eduard Zingerman 
Signed-off-by: Daniel Borkmann 
Acked-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-4-b6c270013c77@gmail.com

bpf: Do not print a newline after disassembly in bpf_verbose_insn()

2026-08-08T09:05:49+00:00

At the moment there are more callsites that want bpf_verbose_insn() to
not print a newline after the instruction, than callsites that want a
newline. Drop '\n' from disasm.c. Non-functional change.

The changes in bpftool are verified by writing a bpf program using a
variety of instructions and comparing `prog dump xlated` output in the
following modes: plain, opcodes, visual, visual opcodes. The output
before and after the changes is identical.

Signed-off-by: Eduard Zingerman 
Signed-off-by: Daniel Borkmann 
Reviewed-by: Quentin Monnet 
Acked-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-1-b6c270013c77@gmail.com

bpf: Fix arg_track_join log to use sa prefix for stack arg slots

2026-05-17T00:46:16+00:00

arg_track_join() logs state transitions at CFG merge points. For
stack arg slots (r >= MAX_BPF_REG), it printed "r11:", "r12:", etc.,
which is misleading since r11 is a special register (BPF_REG_PARAMS)
not meaningful to the user.

Fix it to print "sa0:", "sa1:", etc., matching the per-instruction
transition log in arg_track_log() which already uses the "sa" prefix.

Update the existing stack_arg_pruning_type_mismatch selftest to expect
the corrected format.

Fixes: 2af4e792773f ("bpf: Extend liveness analysis to track stack argument slots")
Signed-off-by: Yonghong Song 
Link: https://lore.kernel.org/r/20260515225056.823086-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov

bpf: Extend liveness analysis to track stack argument slots

2026-05-13T16:27:31+00:00

BPF_REG_PARAMS (R11) is at index MAX_BPF_REG, which is beyond the
register tracking arrays in const_fold.c and liveness.c. Handle it
explicitly to avoid out-of-bounds accesses.

Extend the arg tracking dataflow to cover stack arg slots. Otherwise,
pointers passed through stack args are invisible to liveness, causing
the pointed-to stack slots to be incorrectly poisoned.

Extend the at_out tracking array to MAX_AT_TRACK_REGS (registers
plus stack arg slots) so that outgoing stack arg stores are tracked
alongside registers. Add a separate at_stack_arg_entry array in
compute_subprog_args(), passed to arg_track_xfer(), to restore
FP-derived values on incoming stack arg reads.

Extend record_call_access() to check stack arg slots for FP-derived
pointers at kfunc call sites, reusing the record_arg_access() helper
extracted in the previous patch. Pass stack arg state from caller to
callee in analyze_subprog() so that callees can track pointers received
through stack args, hence avoid poisoning.

Skip stack arg instructions in record_load_store_access(). Stack arg
STX uses dst_reg=BPF_REG_PARAMS (index 11), but at[11] is repurposed
to track the value stored in stack arg slot 0. Without the skip, if a
prior stack arg STX stored an FP-derived pointer (e.g., fp-64) into
slot 0, a subsequent stack arg STX would read that FP-derived value as
the base pointer and spuriously mark a regular stack slot (e.g., fp-72
from -64 + -8) as accessed in the liveness bitmap.

Extend arg_track_log() to log state transitions for outgoing stack arg
slots at indices MAX_BPF_REG through MAX_AT_TRACK_REGS-1. Without this,
changes to at_out[11..17] caused by stack arg store instructions are
silently omitted from BPF_LOG_LEVEL2 output. For example, when a
caller passes fp-64 through a stack argument:

  subprog#0:
   10: (bf) r6 = r10
   11: (07) r6 += -64
   12: (7b) *(u64 *)(r11 -8) = r6
	sa0: none -> fp0-64
   13: (85) call pc+5

Without the fix, the "sa0: none -> fp0-64" transition at insn 12
would not appear.

Extend print_subprog_arg_access() to include stack arg slots in the
per-instruction FP-derived state dump. For example:

  subprog#0:
   12: (7b) *(u64 *)(r11 - 8) = r6  // r6=fp0-64
   13: (85) call pc+5              // r6=fp0-64 sa0=fp0-64

Without the fix, the "sa0=fp0-64" annotation at insn 13 would not
appear, making it harder to debug liveness analysis for programs
that pass FP-derived pointers through stack arguments.

Extend has_fp_args() to also check stack arg slots for FP-derived
pointers, so that callees receiving pointers only through stack args
are still recursively analyzed.

Signed-off-by: Yonghong Song 
Link: https://lore.kernel.org/r/20260513045043.2389049-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov

bpf: Use arg_is_fp() in has_fp_args()

2026-05-13T16:27:31+00:00

Replace "frame != ARG_NONE" with arg_is_fp() in has_fp_args().
The function's purpose is to check whether any argument is derived
from a frame pointer, which is exactly what arg_is_fp() tests
(frame >= 0 || frame == ARG_IMPRECISE). Using the dedicated
predicate is clearer and more consistent with the rest of the file.

Signed-off-by: Yonghong Song 
Link: https://lore.kernel.org/r/20260513045035.2388671-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov

bpf: Refactor record_call_access() to extract per-arg logic

2026-05-13T16:27:30+00:00

Extract the per-argument FP-derived pointer handling from
record_call_access() into a new record_arg_access() helper.

The existing loop body — checking arg_is_fp, querying stack access
bytes, and calling record_stack_access/record_imprecise — will be
reused for stack argument slots in the next patch. Factoring it out
now avoids duplicating the logic.

No functional change.

Signed-off-by: Yonghong Song 
Link: https://lore.kernel.org/r/20260513045030.2388067-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov