linux-stable.git/mm/sparse-vmemmap.c, branch v7.2.4

mm/hugetlb: initialize gigantic bootmem hugepage struct pages earlier

2026-09-07T15:36:33+00:00

commit b1b7c045e808c761b1cc8c19b3040fadedda3fef upstream.

Gigantic bootmem HugeTLB pages are currently initialized from
hugetlb_init(), but page_alloc_init_late() runs earlier and walks
pageblocks to determine zone contiguity.

If a bootmem HugeTLB region is marked noinit, set_zone_contiguous() can
observe still-uninitialized struct pages through
__pageblock_pfn_to_page().  This may not trigger an immediate failure, but
it can make set_zone_contiguous() compute the wrong zone contiguity state.
If extra poisoned-page checks are added in this path, such as
PF_POISONED_CHECK() in page_zone_id(), it can also trigger an early boot
panic.

Initialize gigantic bootmem HugeTLB struct pages from
page_alloc_init_late(), before zone contiguity is evaluated, so later page
allocator setup only sees valid struct page state.  This also makes the
initialization order more natural, as struct pages should be initialized
before later code inspects them.

Link: https://lore.kernel.org/20260612035903.2468601-5-songmuchun@bytedance.com
Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Frank van der Linden 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Oscar Salvador (SUSE) 
Cc: "Ritesh Harjani (IBM)" 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: fix boot panic with CONFIG_DEBUG_VM and HVO bootmem pages

2026-09-07T15:36:33+00:00

commit c0caeceb0c3899dc42844d3979093b27d1434108 upstream.

Patch series "mm: Refactor bootmem gigantic hugepage allocation", v4.

This series is split out from the earlier larger series "mm: Generalize
HVO for HugeTLB and device DAX" [1].  It collects the first 19 patches of
that series as a standalone set of fixes and preparatory cleanups around
bootmem HugeTLB handling, sparse initialization ordering, and related
vmemmap setup.

The first patches fix a few bugs found while reviewing the existing code,
including incorrect bootmem HVO handling, wrong vmemmap registration
arguments, a powerpc compound-vmemmap tracking bug, and too-late
initialization of gigantic bootmem HugeTLB struct pages.

The rest of the series reorders early memory initialization so the
relevant zone state is available before sparse and HugeTLB boot-time setup
runs, then simplifies the remaining bootmem gigantic hugepage allocation
path and removes code made obsolete by that rework.

At a high level:
  - patches [1-4] fix boot-time and arch-specific bugs
  - patches [5-12] reorder and simplify sparse/mm/hugetlb early init
  - patches [13-19] refactor bootmem gigantic hugepage allocation and
    remove obsolete helpers and state


This patch (of 19):

Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") switched
HVO to reuse per-zone shared tail pages from zone->vmemmap_tails[].

Those shared tail pages were initialized in hugetlb_vmemmap_init(), but
bootmem HugeTLB folios are prepared earlier from
gather_bootmem_prealloc().  With hugetlb_free_vmemmap=on,
prep_and_add_bootmem_folios() can access pageblock flags on bootmem
HugeTLB pages whose mirrored tail struct pages already point to the shared
tail page.  On CONFIG_DEBUG_VM kernels, get_pfnblock_bitmap_bitidx() then
dereferences the still-uninitialized shared tail page and can panic during
boot.

Initialize zone->vmemmap_tails[] from gather_bootmem_prealloc(), before
bootmem HugeTLB folios are processed, and drop the later initialization
from hugetlb_vmemmap_init().

This bug only affects CONFIG_DEBUG_VM kernels, where the relevant
assertion is evaluated.

Link: https://lore.kernel.org/20260612035903.2468601-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260612035903.2468601-2-songmuchun@bytedance.com
Fixes: 622026e87c40 ("mm/hugetlb: remove fake head pages")
Signed-off-by: Muchun Song 
Acked-by: Oscar Salvador 
Tested-by: Michal Clapinski 
Reviewed-by: Michal Clapinski 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: "Ritesh Harjani (IBM)" 
Cc: Vlastimil Babka 
Cc: Frank van der Linden 
Cc: Oscar Salvador (SUSE) 
Cc: Usama Arif 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm/sparse-vmemmap: flush_cache_vmap() after hotplugging vmemmap

2026-07-13T17:29:52+00:00

section_activate() does not flush TLB after populating new vmemmap
pages. On most architectures, this is okay. However it is a problem on
RISC-V since there the TLB caching non-present entries is permitted,
which causes spurious faults on some hardwares.

This seems to be most easily reproduced with DEBUG_VM=y and
PAGE_POISONING=y, which causes these newly mapped struct pages to be
poisoned i.e. written to immediately after mapping.

Extend the RISC-V flush_cache_vmap() to also handle the vmemmap range,
and call it after hotplugging vmemmap, which gets the possible spurious
fault handled in the exception handler.

At least for now, the only other architecture with both
SPARSEMEM_VMEMMAP and flush_cache_vmap() is PowerPC, which has a similar
problem with newly valid PTEs. But there flush_cache_vmap() is just a
ptesync. So it should be safe to do this for generic code while having
minimal performance impact.

Suggested-by: Muchun Song 
Signed-off-by: Vivian Wang 
Reviewed-by: Muchun Song 
Acked-by: David Hildenbrand (Arm) 
Link: https://patch.msgid.link/20260713-mark-after-vmemmap-populate-v6-2-b945ceba29d4@iscas.ac.cn
Signed-off-by: Paul Walmsley

mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()

2026-06-09T01:21:30+00:00

Patch series "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and
vmemmap_check_pmd()", v3.

The weak vmemmap_set_pmd() and vmemmap_check_pmd() hooks are currently
no-ops in the generic code, which leaves architectures that need PMD-level
handling to open-code the same logic locally.

This series provides generic implementations for both helpers in
mm/sparse-vmemmap.c.  vmemmap_set_pmd() installs a huge PMD with
PAGE_KERNEL protection, and vmemmap_check_pmd() verifies a present leaf
PMD before reusing the existing vmemmap_verify() helper.

With those generic helpers in place, patches 2-5 remove the now redundant
arch-specific implementations from arm64, riscv, loongarch, and sparc.


This patch (of 5):

The two weak functions are currently no-ops on every architecture, forcing
each platform that needs them to duplicate the same handful of lines. 
Provide a generic implementation:

- vmemmap_set_pmd() simply sets a huge PMD with PAGE_KERNEL protection.

- vmemmap_check_pmd() verifies that the PMD is present and leaf,
  then calls the existing vmemmap_verify() helper.

Architectures that need special handling can continue to override the weak
symbols; everyone else gets the standard version for free.

Link: https://lore.kernel.org/20260601084845.3792171-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260601084845.3792171-2-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Oscar Salvador (SUSE) 
Cc: Albert Ou 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Huacai Chen 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Palmer Dabbelt 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: fix DAX vmemmap accounting with optimization

2026-05-29T04:04:54+00:00

When vmemmap optimization is enabled for DAX, the nr_memmap_pages counter
in /proc/vmstat is incorrect.  The current code always accounts for the
full, non-optimized vmemmap size, but vmemmap optimization reduces the
actual number of vmemmap pages by reusing tail pages.  This causes the
system to overcount vmemmap usage, leading to inaccurate page statistics
in /proc/vmstat.

Fix this by introducing section_nr_vmemmap_pages(), which returns the
exact vmemmap page count for a given pfn range based on whether
optimization is in effect.

Link: https://lore.kernel.org/20260428081855.1249045-5-songmuchun@bytedance.com
Fixes: 15995a352474 ("mm: report per-page metadata information")
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Liam R. Howlett 
Cc: "Aneesh Kumar K.V" 
Cc: Joao Martins 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Nicholas Piggin 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: pass @pgmap argument to memory deactivation paths

2026-05-29T04:04:54+00:00

Currently, the memory hot-remove call chain -- arch_remove_memory(),
__remove_pages(), sparse_remove_section() and section_deactivate() -- does
not carry the struct dev_pagemap pointer.  This prevents the lower levels
from knowing whether the section was originally populated with vmemmap
optimizations (e.g., DAX with vmemmap optimization enabled).

Without this information, we cannot call vmemmap_can_optimize() to
determine if the vmemmap pages were optimized.  As a result, the vmemmap
page accounting during teardown will mistakenly assume a non-optimized
allocation, leading to incorrect memmap statistics.

To lay the groundwork for fixing the vmemmap page accounting, we need to
pass the @pgmap pointer down to the deactivation location.  Plumb the
@pgmap argument through the APIs of arch_remove_memory(), __remove_pages()
and sparse_remove_section(), mirroring the corresponding *_activate()
paths.

Link: https://lore.kernel.org/20260428081855.1249045-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Reviewed-by: Oscar Salvador 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Liam R. Howlett 
Cc: "Aneesh Kumar K.V" 
Cc: Joao Martins 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Nicholas Piggin 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: fix vmemmap accounting underflow

2026-05-29T04:04:53+00:00

Patch series "mm: Fix vmemmap optimization accounting and initialization",
v8.

The series fixes several bugs in vmemmap optimization, mainly around
incorrect page accounting and memmap initialization in DAX and memory
hotplug paths.  It also fixes pageblock migratetype initialization and
struct page initialization for ZONE_DEVICE compound pages.

Patches 1-4 fix vmemmap accounting issues.  Patch 1 fixes an accounting
underflow in the section activation failure path by moving vmemmap page
accounting into the lower-level allocation and freeing helpers.  Patch 2
fixes incorrect altmap passing in the memory hotplug error path.  Patch 3
passes pgmap through memory deactivation paths so the teardown side can
determine whether vmemmap optimization was in effect.  Patch 4 uses that
information to account the optimized DAX vmemmap size correctly.

Patches 5-6 fix initialization issues in mm/mm_init.  One makes sure all
pageblocks in ZONE_DEVICE compound pages get their migratetype
initialized.  The other fixes a case where DAX memory hotplug reuses an
unoptimized early-section memmap while compound_nr_pages() still assumes
vmemmap optimization, leaving tail struct pages uninitialized.


This patch (of 6):

In section_activate(), if populate_section_memmap() fails, the error
handling path calls section_deactivate() to roll back the state.  This
causes a vmemmap accounting imbalance.

Since commit c3576889d87b ("mm: fix accounting of memmap pages"), memmap
pages are accounted for only after populate_section_memmap() succeeds. 
However, the failure path unconditionally calls section_deactivate(),
which decreases the vmemmap count.  Consequently, a failure in
populate_section_memmap() leads to an accounting underflow, incorrectly
reducing the system's tracked vmemmap usage.

Fix this more thoroughly by moving all accounting calls into the lower
level functions that actually perform the vmemmap allocation and freeing:

  - populate_section_memmap() accounts for newly allocated vmemmap pages -
depopulate_section_memmap() unaccounts when vmemmap is freed

This ensures proper accounting in all code paths, including error handling
and early section cases.

Link: https://lore.kernel.org/20260428081855.1249045-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260428081855.1249045-2-songmuchun@bytedance.com
Fixes: c3576889d87b ("mm: fix accounting of memmap pages")
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Liam R. Howlett 
Cc: "Aneesh Kumar K.V" 
Cc: Joao Martins 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Nicholas Piggin 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/sparse: remove sparse buffer pre-allocation mechanism

2026-05-29T04:04:49+00:00

Commit 9bdac9142407 ("sparsemem: Put mem map for one node together.")
introduced a mechanism to pre-allocate a large memory block to hold all
memmaps for a NUMA node upfront.

However, the original commit message did not clearly state the actual
benefits or the necessity of explicitly pre-allocating a single chunk for
all memmap areas of a given node.

One of the concerns about removing this pre-allocation is that the
subsequent per-section memmap allocations could become scattered around,
and might turn too many memory blocks/sections into an "un-offlinable"
state.  However, tests show that even without the explicit node-wide
pre-allocation, memblock still allocates memory closely and back-to-back. 
When tracing vmemmap_set_pmd allocations, the physical chunks allocated by
memblock are strictly adjacent to each other in a single contiguous
physical range (mapped top-down).  Because they are packed tightly
together naturally, they will at most consume or pollute the exact same
number of memory blocks as the explicit pre-allocation did.

Another concern is the boot performance impact of calling memmap_alloc()
multiple times compared to one large node-wide allocation.  Tests on a
256GB VM showed that memmap allocation time increased from 199,555 ns to
741,292 ns.  Even though it is 3.7x slower, on a 1TB machine, the entire
memory allocation time would only take a few milliseconds.  This boot
performance difference is completely negligible.

Since no negative impact on memory offlining behavior or noticeable boot
performance regression was found, this patch proposes removing the
explicit node-wide memmap pre-allocation mechanism to reduce the
maintenance burden.

Link: https://lore.kernel.org/20260410092419.2446420-1-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: David Hildenbrand (Arm) 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: convert vmemmap_p?d_populate() to static functions

2026-05-29T04:04:40+00:00

Since the vmemmap_p?d_populate functions are unused outside the mm
subsystem, we can remove their external declarations and convert them to
static functions.

Link: https://lore.kernel.org/20260423101441.7089-1-kaitao.cheng@linux.dev
Signed-off-by: Chengkaitao 
Acked-by: David Hildenbrand (arm) 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Cc: David Hildenbrand 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Signed-off-by: Andrew Morton

mm: mark early-init static variables with __meminitdata

2026-04-05T20:53:34+00:00

Static variables defined inside __meminit functions should also be marked
with __meminitdata, so that their storage is placed in the .init.data
section and reclaimed with free_initmem(), thereby reducing permanent .bss
memory usage when CONFIG_MEMORY_HOTPLUG is disabled.

Link: https://lkml.kernel.org/r/20260321120847.8159-1-pilgrimtao@gmail.com
Signed-off-by: Kaitao Cheng 
Reviewed-by: Andrew Morton 
Cc: David Hildenbrand 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Suren Baghdasaryan 
Signed-off-by: Andrew Morton