linux-stable.git/kernel/bpf/syscall.c, branch v7.3-rc2

bpf: Mark syscall helpers as sleepable

2026-09-03T16:44:55+00:00

bpf_sys_bpf() executes the bpf(2) syscall body, which can take mutexes,
allocate with GFP_KERNEL, and wait for an RCU grace period.
bpf_sys_close() reaches close_fd() and filp_close(), which can sleep as
well.

Both helpers are limited to BPF_PROG_TYPE_SYSCALL, whose main program is
sleepable. That does not make every callback sleepable: a syscall program
can register a bpf_timer callback, and the verifier checks that callback
in a non-sleepable context while retaining the syscall helper set.

Without .might_sleep on the prototypes, such a callback can invoke
bpf_sys_bpf() from hrtimer softirq context and trigger a
scheduling-while-atomic failure. bpf_sys_close() is exposed through the
same missing context check.

Set .might_sleep on both prototypes so the existing helper-context check
rejects them from timer callbacks and other atomic regions. Calls from the
sleepable main body remain valid.

Fixes: 79a7f8bdb159 ("bpf: Introduce bpf_sys_bpf() helper and program type.")
Fixes: 3abea089246f ("bpf: Add bpf_sys_close() helper.")
Reported-by: Nicholas Carlini 
Suggested-by: Nicholas Carlini 
Signed-off-by: Kumar Kartikeya Dwivedi 
Link: https://lore.kernel.org/r/20260903144433.1716731-10-memxor@gmail.com
Signed-off-by: Alexei Starovoitov

bpf: Populate mmap-able array map memory lazily

2026-08-14T22:31:30+00:00

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.

The handler is reached through a new optional ->map_mmap_fault callback.
Maps that provide it get a vm_operations_struct with a .fault handler;
maps that populate their mapping eagerly keep the one they had. Both
share the same open/close callbacks, so the existing VMA accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized
rather than each map installing its own vm_operations_struct.

Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.

Time for one mmap()+munmap() of an 8MiB mmap-able array map:

                                       before     after
  no MAP_POPULATE, no access            226us     1.1us
  no MAP_POPULATE, access all pages     236us    1341us
  MAP_POPULATE, no access               312us     493us
  MAP_POPULATE, access all pages        318us     519us

Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Song Liu 
Signed-off-by: Andrii Nakryiko 
Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org

bpf: Drop duplicate blank lines in kernel/bpf/

2026-08-13T17:27:40+00:00

There are many adjacent blank lines in kernel/bpf/ that have accumulated
over time.

Drop them for cleanup.

No functional changes intended.

Signed-off-by: Leon Hwang 
Signed-off-by: Andrii Nakryiko 
Reviewed-by: Emil Tsalapatis 
Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev

bpf: Generate kfunc argument prototype at add-call time

2026-08-02T22:31:26+00:00

Kfunc argument checking re-derives each argument's kfunc_ptr_arg_type
from BTF on every verification of a call in check_kfunc_args(). Now that
get_kfunc_arg_type() is a function of the kfunc's BTF alone, it no
longer inspects register state. The classification can be computed once
when the call is added and cached. This is a step toward describing
kfuncs with a bpf_func_proto and sharing the helper argument-checking
path.

Generate the classification at bpf_add_kfunc_call() time:

- Extend struct bpf_func_proto to be able to describe a kfunc: widen
  arg_type[] and the arg_btf_id[]/arg_size[] union from 5 to
  MAX_BPF_FUNC_ARGS, since a kfunc may take up to 12 arguments (5 in
  registers, 7 on the stack).

- Embed a bpf_func_proto in struct bpf_kfunc_desc, populated by
  gen_kfunc_arg_proto() which runs get_kfunc_arg_type() for each
  argument and stores the result in proto.arg_type[]. Grow the
  descriptor table's descs[] as a flexible array to not waste memory.

- check_kfunc_args() reads the cached classification from meta->fn

The KF_ARG_PTR_TO_CTX classification depends on the resolved program type,
and for BPF_PROG_TYPE_EXT that is the target program's type, which
resolve_prog_type() reads from prog->aux->saved_dst_prog_type. That field
is normally recorded later during verification in check_attach_btf_id(),
after bpf_add_kfunc_call() has run.

Record saved_dst_prog_type and saved_dst_attach_type from dst_prog at
program load time in bpf_prog_load() so the resolved type is available
at add-call time without reordering check_attach_btf_id(). This keeps
e.g. an freplace of an XDP program calling bpf_xdp_metadata_rx_hash()
classifying its struct xdp_md * argument as context.

The classification result is unchanged; it is only computed earlier and
cached.

Signed-off-by: Amery Hung 
Link: https://lore.kernel.org/bpf/20260801074633.1595644-19-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Rename ARG_CONST_SIZE{,_OR_ZERO} to ARG_MEM_SIZE{,_OR_ZERO}

2026-08-02T22:29:14+00:00

ARG_CONST_SIZE does not require a constant: check_mem_size_reg() accepts
any bounded scalar and verifies the memory access against its maximum
(reg_umax). Rename ARG_CONST_SIZE and ARG_CONST_SIZE_OR_ZERO to
ARG_MEM_SIZE and ARG_MEM_SIZE_OR_ZERO to reflect that. ARG_CONST_ALLOC_
SIZE_OR_ZERO, which does require a constant, is left unchanged.

Pure rename, no functional change.

Signed-off-by: Amery Hung 
Link: https://lore.kernel.org/bpf/20260801074633.1595644-10-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Log error code on trampoline unlink failure

2026-07-30T23:29:03+00:00

Replace silent WARN_ON_ONCE with WARN_ONCE that prints the actual error
code from bpf_trampoline_unlink_prog(). This aids debugging of race
conditions during link teardown, while keeping the warning rate limited
to avoid log flooding.

This will be very helpful for speeding up trouble-shooting of some crash
UAF due to bpf_trampoline_unlink_prog failures.

No change to unlink behavior.

Signed-off-by: Xu Xin 
Signed-off-by: Andrii Nakryiko 
Reviewed-by: Emil Tsalapatis 
Acked-by: Leon Hwang 
Link: https://lore.kernel.org/bpf/20260729141159128mEJmS_aujBKr-cBu1p_UI@zte.com.cn

bpf: Fix potential UAF when reading bpf link info

2026-07-30T22:29:51+00:00

In bpf_link_show_fdinfo and bpf_link_get_info_by_fd, link->prog is
accessed without holding any locks. If the prog is concurrently replaced
via bpf_link_update, the old prog can be freed, leading to a potential
UAF issue.

Fix this by accessing link->prog under RCU protection to safely fetch
the pointer and guarantee its lifetime while reading its fields.

Fixes: 0c991ebc8c69 ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link")
Reported-by: Sashiko 
Signed-off-by: Pu Lehui 
Signed-off-by: Andrii Nakryiko 
Reviewed-by: Emil Tsalapatis 
Reviewed-by: Amery Hung 
Acked-by: Leon Hwang 
Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0]
Link: https://lore.kernel.org/bpf/20260728025457.2814876-1-pulehui@huaweicloud.com

bpf: Fix security_bpf_map_create error handling

2026-07-09T08:16:34+00:00

Commit 5816bf4273ed ("lsm,selinux: Add LSM blob support for BPF objects")
made the LSM hook wrappers for BPF object creation clean up the LSM
state internally upon denial, e.g. security_bpf_map_create() internally
calls security_bpf_map_free() when the bpf_map_create hook returns an
error. map_create() however still routes a denial to its free_map_sec
label, which invokes security_bpf_map_free() a second time, so the
bpf_map_free hook fires twice for a single denied map.

In-tree LSMs are unaffected in practice since the blob kfree() inside
security_bpf_map_free() is NULL-safe and idempotent and none of them
implement bpf_map_free, but a BPF LSM program attached to that hook
observes double invocations. Route the denial to free_map instead.

Fixes: 5816bf4273ed ("lsm,selinux: Add LSM blob support for BPF objects")
Signed-off-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260709073422.379247-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi

libbpf: Drop in-loader metadata check for load-time verification

2026-07-08T18:04:46+00:00

The signed gen_loader used to police its own metadata map from within
BPF: emit_signature_match() read the kernel-cached map->sha[] back
through hardcoded struct bpf_map offsets and compared it against a hash
that compute_sha_update_offsets() baked into the signed instructions,
after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[].

The kernel now verifies the metadata at BPF_PROG_LOAD time by folding
the frozen contents of the loader's exclusive fd_array maps into the
signature, so the loader no longer checks anything itself. Generated
loaders thus carry no verification logic of their own anymore: Nothing
in the signing chain depends on emitted loader bytecode doing the right
thing.

On the loading side, skel_internal.h now sets fd_array_cnt for a signed
load so the kernel scans fd_array for the exclusive metadata map -
still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD
round-trip to populate map->sha[] is gone. The struct bpf_map layout
BUILD_BUG_ON()s on the kernel side are removed as well: they only
pinned the ABI for the in-BPF read of map->sha[] that is no longer
needed. Same for the map->excl member. Note: gen_hash is retained; it
still marks a loader as signed so an untrusted host cannot re-dimension
maps or override initial values now covered by the signature.

Signed-off-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Verify signed loader metadata at load time

2026-07-08T18:04:40+00:00

A signed gen_loader program carries the programs, maps and relocations
it installs in a metadata array map. The loader instructions are covered
by the PKCS#7 signature, but the metadata map is not: Today the loader
compares the map contents from within BPF against a hash baked into its
(signed) instructions, using the kernel-cached map hash. The kernel
itself never actually attests that the metadata the loader installs is
the metadata that was signed.

This split is the core of the long-standing objection to the BPF signing
scheme from the LSM / integrity side: the integrity check of a light
skeleton only completes once the loader program runs, that is, after the
security_bpf_prog_load() hook, so at admission time an LSM observes a
program whose payload has not yet been verified. Auditing the chain
link is also not a purely cryptographic operation: whoever signs or
reviews an lskel has to disassemble the loader's preamble to convince
themselves that the embedded hash check is present and correct [0][1].
Two acceptable fixes were identified in those threads: Complete the
integrity check before the admission hook fires, or add a second hook
that collects the verification result after the loader ran [2]. Covering
both the loader and its maps directly with the PKCS#7 signature is what
Blaise Boscaccy's patchsets proposed in several forms. Let's implement
the former, without growing the UAPI, and in particular as a single
unified scheme where the signature spans the raw bytes rather than
derived hashes.

A signed loader binds its metadata map(s) through the existing fd_array,
and an exclusive map is already bound to a program digest (excl_prog_hash).
So when a signature is present, collect the exclusive maps from fd_array
and append their frozen contents to the instructions before verification:
The signature now covers insns || metadata_0 || metadata_1 || [...] in
the fd_array order, and verification completes in bpf_check(), once the
fd_array maps are resolved into used_maps, before the LSM admission hook
and the rest of verification. A program is either BPF_SIG_UNSIGNED or
BPF_SIG_VERIFIED, with nothing in between. While folding the fd_array
maps, a non-exclusive map bound to a signed program is rejected, so every
map folded into the signature is exclusive. A signed loader that fails
to cover its metadata thus does not load, and BPF_SIG_VERIFIED always
means the instructions and every exclusive map are authentic. The maps
must be frozen so the hashed bytes cannot change before the loader runs;
the map <-> program digest binding is enforced by the verifier for every
used map. Binding maps through fd_array_cnt makes the verifier resolve
and excl-check them (excl_prog_sha vs prog->digest) before it would
otherwise compute the digest, so compute prog->digest up front in
bpf_check(), over the unmodified instructions the signature covers, for
a load that folds metadata.

Unsigned programs are not affected by the signature path; for them the
LSM admission hook merely moves below fd_array resolution, with minimal
bounded work in between. Note, signed loaders generated by older libbpf/
bpftool versions need to be regenerated; some of the recent fixes we've
had on the signed loader side require the latter already to close gaps.

Finally, some remarks around the security_bpf_prog_load() placement
given there was discussion on whether a new hook is needed or the existing
security_bpf_prog() hook should be reused [3]: For a new hook it would
mean that just for loading a single BPF program it has to pass through
four layers of LSM hooks:

  1) security_bpf (cmd=PROG_LOAD): for gating various bpf subcmds
  2) security_bpf_prog_load: historical admission hook (CAP/token,
     prog_type, attach point), pre-verification
  3) security_bpf_prog_verify_signature: newly asked admission hook,
     same role as 2), plus the BPF signature verdict
  4) security_bpf_prog: gate handing the prog fd back to userspace,
     verification done & signature verified

The use-cases of 2) and 3) conflate, thus BPF community prefers to just
keep a total of 3 LSM hooks (as-is today): 3) makes 2) incoherent given
they are the /same class/ of hook, that is, access-control admission on
the load and split only by _what_ they can see. Worse, with the split,
for a signed BPF program security_bpf_prog_load 2) admits a program whose
signature has not been checked, so a policy gating at 2) is structurally
unable to express "admit only verified" and every such policy is forced
onto 3) *anyway*. In other words, one doesn't get two complementary hooks,
but rather, one real admission hook aka 3) plus a now-degraded /legacy/
hook 2) that can't answer the question operators actually want to ask.

Reusing security_bpf_prog() 4) for admission is no alternative either:
it fires only after the entire verifier (and JIT) pipeline ran, so
denying a not-yet-verified program at that point burns exactly the
work a denial is supposed to avoid, and by then the program has an id
assigned and the kallsyms/perf/audit load events fired. Policies are
free to also consume the signature verdict at 4), but admission control
belongs into security_bpf_prog_load(). Hence the latter remains the only
admission hook, merely moved past signature verification; with moving
large allocations further down into the BPF verifier, there is now only
minimal work between the old and new location: The preparation work in
bpf_check() is reordered such that only the minimally necessary setup
happens up front: Allocating the env, initializing the verifier log and
resolving the fd_array that a signed BPF metadata map needs. The worst
case allocation up until security_bpf_prog_load() is ~90K which is the
env itself (~54K) plus the continuous fd_array cache (at most 32K). The
insn_aux_data array is moved into a later stage in the verification.

Signed-off-by: Daniel Borkmann 
Link: https://lore.kernel.org/bpf/2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com [0]
Link: https://lore.kernel.org/lkml/ecf0521ed302db672672ebfbc670ecfba36a6e00.camel@HansenPartnership.com [1]
Link: https://lore.kernel.org/bpf/88703f00d5b7a779728451008626efa45e42db3d.camel@HansenPartnership.com [2]
Link: https://lore.kernel.org/bpf/DJOFY21DYUI4.19WKQ3NPZ4H5R@gmail.com [3]
Link: https://lore.kernel.org/bpf/20260708075343.358712-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi