linux.git/mm/vmalloc.c, branch v7.3-rc1

mm/vmalloc: do not warn on -ENOMEM from va_alloc()

2026-08-25T01:43:02+00:00

Since vmalloc() accepts non-blocking GFP flags, allocation requests may
fail when callers pass restrictive GFP masks.

va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails
during NE_FIT_TYPE splitting.  This is an expected failure, so va_alloc()
should return the error without triggering a kernel splat.

Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com
Signed-off-by: Uladzislau Rezki (Sony) 
Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com
Reviewed-by: Anshuman Khandual 
Reviewed-by: Baoquan He 
Signed-off-by: Andrew Morton

mm/vmalloc: make vm_struct.nr_pages an unsigned long

2026-08-25T01:43:00+00:00

vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte
counts from it as nr_pages << PAGE_SHIFT.  A shift is evaluated in the
type of its promoted left operand, so those are 32-bit arithmetic and wrap
at 4 GiB of bytes, which is 2^20 pages.  Every site depends on a cast
being remembered; vmap() has one, two recent commits did not. 
vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and
/proc/kcore returns it as zeros while reporting a successful read, which
drgn, crash or gdb cannot tell from real memory, and the vrealloc()
grow-in-place check declines a request that would have fit.

Widen the field so the class of bug goes away instead of one site at a
time.  Everything feeding or consuming it widens too:
vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages
and old_nr_pages, the index range of vm_area_free_pages(), and three page
indexes that were plain int.  Five casts go.  Two prints needed fixing as
well, %u in vmalloc_dump_obj() and %d for the unsigned field in
vmalloc_info_show().

No bug report behind this, I found it reading the code.  The 4 GiB wrap
needs only a machine with over 4 GiB of memory.  Neither larger threshold
is a practical concern: 2^32 pages, where the field itself truncates, is
16 TiB and beyond what hardware can populate, and 2^31, where the plain
int indexes break, is 8 TiB and larger than anything in the tree asks for.
The int *nr cursor in the mapping path is unchanged and is separate work.
Users outside mm/vmalloc.c need no change either.  Those handing the
count to a narrower parameter cannot drive it near 2^31, and
kho_preserve_vmalloc() stores it into a 32-bit ABI field that still
receives the same low bits; above 2^32 pages the truncation just moves out
of vm_struct into that store.

sizeof(struct vm_struct) on x86-64 stays 72 bytes with
CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled,
both inside the kmalloc-96 bucket it already comes from.

Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com
Fixes: 0bca23804632 ("mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas")
Fixes: d57ac904ffdc ("mm/vmalloc: use physical page count for vrealloc() grow-in-place check")
Signed-off-by: Artem Lytkin 
Suggested-by: Andrew Morton 
Reviewed-by: Uladzislau Rezki (Sony) 
Assisted-by: Claude:claude-fable-5
Cc: Matthew Wilcox (Oracle) 
Cc: 
Cc: 
Signed-off-by: Andrew Morton

Merge branch 'mm-hotfixes-stable' into mm-stable to pick up

2026-08-25T01:40:27+00:00

already-upstream changes to memcontrol.c, needed by "memcg: move
mem_cgroup_swappiness and vm_swappiness to mm/swap.h".

arm64: remove redundant concurrent ptdump UAF mitigation

2026-08-05T03:02:00+00:00

This partially reverts commit fa93b45fd397 ("arm64: Enable vmalloc-huge
with ptdump"), retaining vmalloc-huge support but eliminating the now
redundant mitigation against a race between huge vmap page table freeing
and ptdump, as this issue has now been fixed at core.

We also simultaneously remove the arm64 if-deffery when acquiring the mmap
read lock upon vmap huge page table promotion as it is no longer required.

Note that this patch relies on the preceding vmalloc patch, and should not
be backported alone.

Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-5-8cc77dcc0018@kernel.org
Fixes: fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump")
Signed-off-by: Lorenzo Stoakes (ARM) 
Reviewed-by: Dev Jain 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Kiryl Shutsemau (Meta) 
Acked-by: Will Deacon 
Reviewed-by: David Hildenbrand (Arm) 
Cc: Andy Lutomirski 
Cc: "Borah, Chaitanya Kumar" 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Dave Hansen 
Cc: David Carlier 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Liam R. Howlett 
Cc: Michal Hocko 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Toshi Kani 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF

2026-08-05T03:02:00+00:00

Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable
freeing", v6.

Kernel page table walkers fall into two broad categories - those ranges
where no exclusion is required via walk_kernel_page_table_range_lockless()
and those where exclusion is required via walk_kernel_page_table_range()
or walk_page_range_debug().

The former category is used only by arm64 arch code operating on ranges it
both wholly owns and does not concurrently write.

The latter category consists of kernel page table walkers operating on
ranges that are wholly owned (but which need exclusion against concurrent
writers).

The lock used for exclusion is the mmap lock, and for kernel ranges this
is the mmap lock on init_mm.

ptdump is a special case being both the only user of
walk_page_range_debug(), and the only case in which it walks ranges it
does not own.

This presents a problem, as page tables may be freed under ptdump.  And
indeed there is a use-after-free bug in the kernel as a result, which this
series addresses.

vmap promotes page tables to huge leaf entries where possible, freeing the
lower page table when it does.  It does this with no meaningful locks held
against concurrent ptdump walks.

As a result, use-after-free can currently occur.  This series addresses
the issue by having the vmap huge promotion logic acquire the mmap read
lock while both setting the huge page table entry and freeing the prior
leaf page table.

The ptdump code already acquires the mmap write lock, so by doing so we
ensure that the ptdump walker only ever observes either the huge page
table entry or the existing page table entry, and nothing is freed
underneath it.

A mitigation for this issue was already applied for arm64 in commit
fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which this series
has to deal with carefully.

This mitigation resolves the issue by acquiring the mmap read lock on
init_mm on vmap page table free if a ptdump is in progress.

However the fix in this series would cause a deadlock if we were to simply
apply it for arm64 without also reverting the change.

This is because vmap may acquire the read lock before ptdump attempts to
acquire the write lock, which then gets queued, and rwsem starvation rules
mean that the (unacknowledged) nested mmap read lock in the arm64 code
would also block, meaning the original read lock is never released and
thus deadlock.

This series works around this by #ifndef CONFIG_ARM64'ing the mmap read
lock in vmap logic, then partially reverting commit fa93b45fd397 ("arm64:
Enable vmalloc-huge with ptdump"), keeping the enablement of huge vmap
support, and removing the ifdeffery with the partial revert patch.

There are related issues that are also addressed in this series:

* x86 page attribute logic, specifically Change Page Attributes (CPA),
  implements a feature whereby huge ranges can be collapsed into huge leaf
  entries. This can similarly cause a UAF when done in parallel with a
  ptdump walk, so similarly acquire the init_mm mmap lock to avoid this.

* The CPA logic allows concurrent page table manipulation and CPA
  collapse, meaning the former risks accessing a page table the latter
  frees. Fix this by acquiring mmap write lock on init_mm across the
  whole CPA collapse operation and read lock on the page table
  manipulation.

* x86 and arm64 permit walks of non-kernel mm's (both allowing efi mm
  walks, and in x86's case arbitrary mm's), so we ensure kernel mappings
  remain stable by locking the init_mm as well as the mm being walked.

The ordering of patches is established for both strict dependencies (the
arm64 partial revert in particular has to be done after the vmap changes)
and logical ones (the non-kernel mm fix only makes sense once the vmap/CPA
fixes are in place).


This patch (of 3):

Currently there is a nasty race between ptdump and vmap when attempting to
map a huge P4D, PUD or PMD entry:

* ptdump walks kernel page table ranges it doesn't own.

* When vmap maps ranges it tries to promotes existing ones to huge page
  tables in vmap_try_huge_[p4d,pud,pmd]() at P4D, PUD and PMD level,
  freeing the lower page table in [p4d,pud,pmd]_free_[pud,pmd,pte]_page()
  when it succeeds.

Both of these things can happen at the same time and as a result ptdump
can access a freed page table, resulting in a use-after-free and memory
corruption.

This is possible because while ptdump_walk_pgd() holds both the mem
hotplug lock and the mmap write lock before invoking
walk_page_range_debug(), vmap takes no relevant locks at all.

Fix this by holding the mmap read lock in vmap_try_huge_*() when freeing
page tables.

The read lock is sufficient: ptdump is the only walker that must be
excluded and it holds the mmap write lock.  Other holders of the read lock
may run concurrently, but each exclusively owns the range it operates on
and cannot reach the page tables freed here.

We also hold the lock while assigning the huge page table entry, which
means page table walkers observe only the huge or non-huge page table
entry.

We use a trylock to prevent ptdump from blocking vmap making forward
progress.  This is fine because it's an optimisation in any case, and thus
the vmap can safely proceed regardless.

All other kernel page table walkers that touch vmalloc ranges either
exclusively own the memory walked or acquire the mmap lock, so this
correctly excludes those walkers.

One wrinkle here is commit fa93b45fd397 ("arm64: Enable vmalloc-huge with
ptdump"), which addresses the issue for arm64 only by explicitly acquiring
the mmap read lock on kernel page table freeing should a concurrent ptdump
be in progress.

This is problematic as vmap may acquire the mmap read lock prior to ptdump
attempting to acquire an mmap write lock, leading to a deadlock when the
mmap read lock is slept upon on page table freeing due to rwsem
anti-starvation.

We work around this by predicating the mmap lock being taken on
!CONFIG_ARM64 for the time being.

With this patch applied, a follow up will partially revert commit
fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") and at that stage
remove the arm64 ifdeffery.

We also update walk_page_range_debug() to assert the mmap write lock
unconditionally and update the comment here to reflect this change.

The issue has existed as long as ptdump was available and vmap freed page
tables when promoting to a huge leaf entry, that is, since commit
b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table")
for huge ioremap, and commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc
mappings") for huge vmalloc.

Since the former is the earlier of the two we choose that for our Fixes
tag.

We also define a guard class for mmap_read_trylock() so we can use
cleanup.h to make the scope handling cleaner in the implementation.

This patch is based on work by David Carlier (linked), with gratitude!

Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-0-8cc77dcc0018@kernel.org
Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-1-8cc77dcc0018@kernel.org
Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table")
Signed-off-by: Lorenzo Stoakes (ARM) 
Reported-by: syzbot+fd95a72470f5a44e464c@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a287988.39669fcc.33b062.00a0.GAE@google.com/T/
Link: https://lore.kernel.org/linux-mm/20260706203128.162335-1-devnexen@gmail.com/
Reviewed-by: Mike Rapoport (Microsoft) 
Reviewed-by: Dev Jain 
Acked-by: David Hildenbrand (Arm) 
Reviewed-by: Kiryl Shutsemau 
Cc: 
Cc: Andy Lutomirski 
Cc: "Borah, Chaitanya Kumar" 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Dave Hansen 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Liam R. Howlett 
Cc: Michal Hocko 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Toshi Kani 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: split out vmalloc declarations from internal.h

2026-08-05T02:18:46+00:00

mm/internal.h becomes more and more bloated.

Move declarations related to vmalloc to a new mm/vmalloc.h header.

No functional changes.

Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Muchun Song 
Acked-by: Vlastimil Babka (SUSE) 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Lorenzo Stoakes 
Acked-by: Pratyush Yadav 
Acked-by: SJ Park 
Cc: Alexander Graf 
Cc: Alexander Potapenko 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: Dennis Zhou 
Cc: Dmitry Vyukov 
Cc: Johannes Weiner 
Cc: Liam R. Howlett 
Cc: Marco Elver 
Cc: Michal Hocko 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Suren Baghdasaryan 
Cc: Tejun Heo 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/vmalloc: add alignment info in warning print as possible failure reason

2026-07-31T02:40:39+00:00

When running 'fix_align_alloc_test' case of test_vmalloc module with
command:
    insmod ./test_vmalloc.ko run_test_mask=64

It will fail, which is the expected result, as the case increment
the alignment parameter gradually to 64bit limit. And the dmesg has
warning msg:
    "vmalloc_test/0: vmalloc error: size 4096, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0"

It doesn't give the alignment info, which is the real reason for the
failure (not the 'size').

Add alignment info to the warning print to give the necessary hint
for possible failure reason, and the message will be:
    "vmalloc_test/0: vmalloc error: size 4096, align 0x800000000000, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0"

Link: https://lore.kernel.org/20260702112610.21589-1-feng.tang@linux.alibaba.com
Signed-off-by: Feng Tang 
Reviewed-by: Uladzislau Rezki (Sony) 
Signed-off-by: Andrew Morton

mm/vmalloc: use more common error handling code in pcpu_get_vm_areas()

2026-07-29T04:11:48+00:00

Use an existing label once more so that a bit of exception handling can be
better reused at the end of this function implementation.

This issue was detected by using the Coccinelle software.

Link: https://lore.kernel.org/453375c4-c3ca-4e6f-8880-0e6ff3c74ee3@web.de
Signed-off-by: Markus Elfring 
Reviewed-by: Uladzislau Rezki (Sony) 
Cc: Alexander Potapenko 
Cc: Daniel Axtens 
Cc: Dmitry Vyukov 
Signed-off-by: Andrew Morton

mm/vmalloc: honor GFP constraints in pcpu_get_vm_areas()

2026-07-29T04:11:46+00:00

Patch series "mm/percpu: Fix possible NOFS/NOIO reclaim recursion", v4.

Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations
atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use
pcpu_alloc_mutex and the chunk creation slow path.  This restored the
allocation capability that was lost when those constrained allocations
were treated as atomic, but it also makes the percpu slow path visible to
callers from constrained reclaim contexts.

There are two related problems.

First, the create and populate slow paths do not fully preserve the
caller's allocation constraints.  pcpu_alloc_noprof() derives pcpu_gfp
from the caller supplied GFP mask and passes it down to the percpu backing
page allocator.  However, chunk creation calls pcpu_get_vm_areas(), and
chunk population can allocate temporary metadata or vmalloc page tables
while mapping backing pages.  Those internal allocations can still use
GFP_KERNEL, so a caller using GFP_NOFS or GFP_NOIO can enter unconstrained
FS or IO reclaim while holding pcpu_alloc_mutex.

One possible case is blk-cgroup after commit 5d726c4dbeed ("blk-cgroup:
fix possible deadlock while configuring policy").  blkg_conf_prep() now
serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and
blkg_alloc() uses GFP_NOIO because queue freeze and IO reclaim
dependencies can otherwise deadlock.  If the percpu slow path loses that
GFP_NOIO context, direct reclaim or writeback can issue IO to a frozen
queue while q->blkcg_mutex is held.

Second, allowing sleepable GFP_NOFS/GFP_NOIO allocations to take
pcpu_alloc_mutex means that unconstrained backing allocations made under
the mutex can create an FS/IO reclaim dependency against a constrained
caller which already holds an FS or IO lock and then waits for
pcpu_alloc_mutex.

This series fixes those issues in three steps:

  - pass the caller supplied GFP mask into pcpu_get_vm_areas() and use it
    for vmalloc metadata and KASAN shadow allocations;
  - pass the GFP mask through the chunk population path, including the
    temporary pages array and vmalloc page table allocation scope;
  - restrict percpu backing allocations performed while holding
    pcpu_alloc_mutex to GFP_NOIO, so they cannot recurse into IO or FS
    reclaim.

This keeps sleepable GFP_NOFS/GFP_NOIO percpu allocations working, while
avoiding the reclaim recursion risks introduced by making those
allocations eligible for the mutex-protected slow path.


This patch (of 4):

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and
passes it down to the backing percpu allocator.  However, when the percpu
vmalloc allocator has to create a new chunk, pcpu_create_chunk() calls
pcpu_get_vm_areas() to allocate the corresponding vmalloc areas.

pcpu_get_vm_areas() currently performs its internal allocations with
GFP_KERNEL, including vmap area metadata, vm_struct metadata and KASAN
vmalloc shadow population.  This means that a caller which deliberately
uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating
the vmalloc areas for a new percpu chunk.

One possible case is blk-cgroup after commit 5d726c4dbeed ("blk-cgroup:
fix possible deadlock while configuring policy").  blkg_conf_prep() now
serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and
blkg_alloc() was changed to GFP_NOIO for that reason:

  CPU0: blkg_conf_prep()
    mutex_lock(q->blkcg_mutex)
    blkg_alloc(..., GFP_NOIO)
      alloc_percpu_gfp(..., GFP_NOIO)
        pcpu_alloc_noprof(..., GFP_NOIO)
	  pcpu_create_chunk(GFP_NOIO)
	    pcpu_get_vm_areas()
              -> if percpu chunks are exhausted, chunk create may do
                 internal GFP_KERNEL allocations
              -> direct reclaim / writeback can issue IO to this queue
              -> IO waits because the queue is frozen

  CPU1: blkcg_deactivate_policy()
    blk_mq_freeze_queue(q)
    mutex_lock(q->blkcg_mutex)
      -> waits for CPU0
    ... unfreeze only happens after q->blkcg_mutex is acquired/released

So the concern is that the caller deliberately uses GFP_NOIO because it
may hold a lock which can be acquired after queue freeze, but the percpu
slow path can temporarily lose that allocation context.

Pass the caller supplied GFP mask from pcpu_create_chunk() to
pcpu_get_vm_areas(), and use it for the internal vmalloc metadata and
KASAN shadow allocations.

Link: https://lore.kernel.org/20260618130414.96383-1-kaitao.cheng@linux.dev
Link: https://lore.kernel.org/20260618130414.96383-2-kaitao.cheng@linux.dev
Fixes: 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Signed-off-by: Kaitao Cheng 
Reviewed-by: Uladzislau Rezki (Sony) 
Reviewed-by: Shivam Kalra 
Acked-by: Dennis Zhou 
Acked-by: Michal Hocko 
Cc: Christoph Lameter 
Cc: Pedro Falcato 
Cc: Tejun Heo 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/vmalloc: free unused pages on vrealloc() shrink

2026-06-02T22:22:32+00:00

When vrealloc() shrinks an allocation and the new size crosses a page
boundary, unmap and free the tail pages that are no longer needed.  This
reclaims physical memory that was previously wasted for the lifetime of
the allocation.

The heuristic is simple: always free when at least one full page becomes
unused.  Huge page allocations (page_order > 0) are skipped, as partial
freeing would require splitting.  Allocations with VM_FLUSH_RESET_PERMS
are also skipped, as their direct-map permissions must be reset before
pages are returned to the page allocator, which is handled by
vm_reset_perms() during vfree().

Additionally, allocations with VM_USERMAP are skipped because
remap_vmalloc_range_partial() validates mapping requests against the
unchanged vm->size; freeing tail pages would cause vmalloc_to_page() to
return NULL for the unmapped range.

To protect concurrent readers, the shrink path uses Node lock to
synchronize before freeing the pages.

Finally, we notify kmemleak of the reduced allocation size using
kmemleak_free_part() to prevent the kmemleak scanner from faulting on the
newly unmapped virtual addresses.

The virtual address reservation (vm->size / vmap_area) is intentionally
kept unchanged, preserving the address for potential future grow-in-place
support.

Link: https://lore.kernel.org/20260519-vmalloc-shrink-v14-4-70b96ee3e9c9@zohomail.in
Signed-off-by: Shivam Kalra 
Suggested-by: Danilo Krummrich 
Reviewed-by: Uladzislau Rezki (Sony) 
Cc: Alice Ryhl 
Signed-off-by: Andrew Morton