linux.git/kernel/bpf/core.c, branch v7.3-rc2

x86/bpf: Make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

2026-08-20T15:52:31+00:00

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge
number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and
execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA
are ROX after the allocation. execmem_alloc_rw() temporarily sets them to
W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size()
that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same
address range, so x86::arch_bpf_trampoline_size() can directly use
execmem_alloc(EXECMEM_MODULE_DATA)

Replace the call to bpf_jit_alloc_exec_rw() with a call to
execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and
drop bpf_jit_alloc_exec_rw() helper.

Fixes: 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable")
Reported-by: Jiri Olsa 
Signed-off-by: Mike Rapoport (Microsoft) 
Signed-off-by: Daniel Borkmann 
Tested-by: Jiri Olsa 
Link: https://lore.kernel.org/all/an8r7EODLIL-bZM3@krava
Link: https://lore.kernel.org/bpf/20260818130510.3110054-1-rppt@kernel.org

bpf: Add source and instruction diagnostic context

2026-08-15T18:11:16+00:00

Teach verifier diagnostics to annotate an instruction with BTF source
line information and nearby BPF instructions. The renderer keeps source
text in a fixed-width lane and prints instructions in a stable right-hand
gutter.

Wrap annotation text under the source line so long error labels remain
readable while the source and instruction lanes keep their fixed layout.

Keeping source and instruction context in one commit preserves the visual
layout contract that later diagnostic reports rely on.

Signed-off-by: Kumar Kartikeya Dwivedi 
Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman

bpf: Eliminate dup/restore of insn_aux_data

2026-08-13T02:36:30+00:00

The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.

After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.

However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.

So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.

Signed-off-by: Xu Kuohai 
Acked-by: Eduard Zingerman 
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Support arena and arena__nullable kfunc argument suffixes

2026-08-08T10:03:26+00:00

Passing an arena pointer to a kfunc takes two steps today. There is no
arena pointer argument type, so the pointer crosses the boundary as a
bare scalar, and the kfunc then offsets it by the arena base and casts
it before it can touch the memory. Every such kfunc open-codes the same
translation.

Add the __arena and __arena__nullable argument suffixes to make this more
convenient. The kfunc declares the parameter by its real pointer type
and dereferences it directly, with the JIT rebasing the value at the
call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the
u32 offset stays within the guard-padded arena kernel mapping, and a
fault on an unpopulated page recovers through the per-arena scratch
page. A suffixed argument accepts a PTR_TO_ARENA or scalar register,
matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a
value with zero in the low 32 bits arrives as the arena base.
__arena__nullable preserves NULL for optional arguments by skipping the
rebase when the truncated value, arena offset 0, is zero. Keeping the
plain form NULL-free saves the NULL test on every call.

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable and naturally follows the
common nullable argument path. Plain __arena follows that path too for
verifier type checking because both forms accept a constant zero; the
function-model flag still determines whether the JIT preserves NULL or
rebases it to the arena base.

This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args() and distilled into argument flags in the function
model stored in the kfunc descriptor. JITs retrieve the model while
emitting the call, avoiding per-call state in insn_aux_data.

JITs declare support with bpf_jit_supports_arena_args() and verification
fails with -ENOTSUPP elsewhere.

Co-developed-by: Kumar Kartikeya Dwivedi 
Signed-off-by: Tejun Heo 
Signed-off-by: Kumar Kartikeya Dwivedi 
Acked-by: Eduard Zingerman 
Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman

bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable

2026-07-22T15:27:10+00:00

arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.

This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates  rel32 offsets relatively to
that buffer.

In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.

Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().

Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Song Liu 
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: alloc_prog_pack(): Skip ROX management for already ROX memory

2026-07-22T15:27:05+00:00

execmem_alloc() can return ROX memory that is already filled with
architecture defined trapping instructions.

In preparation for enabling this mode for BPF on x86, make sure that there
is no redundant management of the ROX memory.

There is no need to fill allocated memory with trapping instructions, to
request permissions reset on free and to set ROX permissions as this all
is handled by execmem_alloc().

Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if
execmem_alloc() returns ROX memory and skip the redundant steps in that
case.

Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Song Liu 
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec()

2026-07-22T15:26:58+00:00

bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the
corresponding execmem APIs.

Architectures define the properties of the memory range needed by BPF in
their initialization of execmem and don't need to override neither of
them.

Drop the __weak qualifier from bpf_jit_alloc_exec() and
bpf_jit_free_exec().

Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Song Liu 
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Introduce jit_required flag and remove bpf_prog_has_kfunc_call()

2026-07-09T06:24:41+00:00

Introduce a 'jit_required' bitfield flag in struct bpf_prog to track
whether a BPF program strictly requires the JIT compiler to run. This
prevents a dangerous runtime fallback to the interpreter for features
that are only implemented in the JIT compiler.

Currently, bpf_prog_has_kfunc_call() is used only for kernel function
calls, replace the kfunc-specific helper with the new 'jit_required'
flag. This makes it easy to support other JIT-only BPF features, such
as inlined helpers.

Suggested-by: Alexei Starovoitov 
Suggested-by: KaFai Wan 
Suggested-by: Leon Hwang 
Acked-by: Leon Hwang 
Signed-off-by: Tiezhu Yang 
Signed-off-by: Eduard Zingerman

bpf: Account scratch buffer in bpf_prog_calc_tag

2026-07-09T05:00:02+00:00

bpf_prog_calc_tag() copies the instructions into a plain vmalloc() scratch
buffer to blind the map fds before hashing. The buffer scales with the
program, up to ~8MB at the 1M instruction limit, and is allocated on every
program load, but unlike the rest of the load-time scratch memory it is
not charged to the loader's memcg. Use GFP_KERNEL_ACCOUNT to account it
like the other allocations scoped to the verification/load.

Signed-off-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260708211537.371874-5-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Prefer dirty packs for eBPF allocations

2026-07-01T08:34:41+00:00

The pack allocator only flushes predictors when reusing a dirty pack for
cBPF, eBPF allocations never trigger a flush. Currently, eBPF picks the
first free pack, which could be a clean pack. As an optimization, leaving
a clean pack for cBPF can avoid flushes.

Prefer dirty packs for eBPF and keep clean packs free for cBPF. This
mirrors the existing cBPF preference for clean packs: each program kind
prefers the pack that avoids an extra flush, and falls back to the other
kind only when no preferred pack has room. eBPF reuse of a dirty pack is
harmless since eBPF being privileged does not flush.

Signed-off-by: Pawan Gupta 
Acked-by: Daniel Borkmann 
Signed-off-by: Daniel Borkmann