linux.git/mm/sparse-vmemmap.c, branch v7.3-rc2

mm: split out sparse declarations from internal.h

2026-08-05T02:18:46+00:00

mm/internal.h becomes more and more bloated.

Move declarations related to SPARSE and SPARSE_VMEMMAP memory models to
a new mm/sparse.h header.

No functional changes.

Link: https://lore.kernel.org/20260709-internal-h-v2-2-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Muchun Song 
Acked-by: Vlastimil Babka (SUSE) 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Lorenzo Stoakes 
Acked-by: Pratyush Yadav 
Acked-by: SJ Park 
Cc: Alexander Graf 
Cc: Alexander Potapenko 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: Dennis Zhou 
Cc: Dmitry Vyukov 
Cc: Johannes Weiner 
Cc: Liam R. Howlett 
Cc: Marco Elver 
Cc: Michal Hocko 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Suren Baghdasaryan 
Cc: Tejun Heo 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: split out mm_init and memblock declarations from internal.h

2026-08-05T02:18:45+00:00

Patch series "mm: split a couple of headers from internal.h", v2.

mm/internal.h becomes more and more bloated.

Split declarations related to mm_init, memblock, vmalloc and sparse into
new headers.


This patch (of 3):

mm/internal.h becomes more and more bloated.

Move declarations for related to mm/mm_init.c and mm/memblock.c to a new
mm/mm_init.h header.

No functional changes.

[rppt@kernel.org: split stubfs from internal.h to mm_init.h]
  Link: https://lore.kernel.org/alJd1BLypyK9Mpaw@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-0-695631425968@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-1-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Muchun Song 
Acked-by: Vlastimil Babka (SUSE) 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Lorenzo Stoakes 
Acked-by: Pratyush Yadav 
Acked-by: SJ Park 
Cc: Alexander Graf 
Cc: Alexander Potapenko 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: Dennis Zhou 
Cc: Dmitry Vyukov 
Cc: Johannes Weiner 
Cc: Liam R. Howlett 
Cc: Marco Elver 
Cc: Michal Hocko 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Suren Baghdasaryan 
Cc: Tejun Heo 
Cc: "Uladzislau Rezki (Sony)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: remove sparse_vmemmap_init_nid_late()

2026-07-29T04:12:00+00:00

hugetlb_vmemmap_init_late() no longer has any users, so the remaining
late-init path in sparse_vmemmap_init_nid_late() is dead code.

Remove sparse_vmemmap_init_nid_late() and its declarations.

Link: https://lore.kernel.org/20260612035903.2468601-18-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Frank van der Linden 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Oscar Salvador 
Cc: Oscar Salvador (SUSE) 
Cc: "Ritesh Harjani (IBM)" 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/sparse: move subsection_map_init() into sparse_init()

2026-07-29T04:11:57+00:00

subsection_map_init() is part of sparse memory initialization, but it is
currently called from free_area_init().

Move it into sparse_init() so the sparse-specific setup stays together
instead of being split across the generic free_area_init() path.

Link: https://lore.kernel.org/20260612035903.2468601-8-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Frank van der Linden 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Oscar Salvador (SUSE) 
Cc: "Ritesh Harjani (IBM)" 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/hugetlb: initialize gigantic bootmem hugepage struct pages earlier

2026-07-29T04:11:57+00:00

Gigantic bootmem HugeTLB pages are currently initialized from
hugetlb_init(), but page_alloc_init_late() runs earlier and walks
pageblocks to determine zone contiguity.

If a bootmem HugeTLB region is marked noinit, set_zone_contiguous() can
observe still-uninitialized struct pages through
__pageblock_pfn_to_page().  This may not trigger an immediate failure, but
it can make set_zone_contiguous() compute the wrong zone contiguity state.
If extra poisoned-page checks are added in this path, such as
PF_POISONED_CHECK() in page_zone_id(), it can also trigger an early boot
panic.

Initialize gigantic bootmem HugeTLB struct pages from
page_alloc_init_late(), before zone contiguity is evaluated, so later page
allocator setup only sees valid struct page state.  This also makes the
initialization order more natural, as struct pages should be initialized
before later code inspects them.

Link: https://lore.kernel.org/20260612035903.2468601-5-songmuchun@bytedance.com
Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Frank van der Linden 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Oscar Salvador (SUSE) 
Cc: "Ritesh Harjani (IBM)" 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/hugetlb: fix boot panic with CONFIG_DEBUG_VM and HVO bootmem pages

2026-07-29T04:11:56+00:00

Patch series "mm: Refactor bootmem gigantic hugepage allocation", v4.

This series is split out from the earlier larger series "mm: Generalize
HVO for HugeTLB and device DAX" [1].  It collects the first 19 patches of
that series as a standalone set of fixes and preparatory cleanups around
bootmem HugeTLB handling, sparse initialization ordering, and related
vmemmap setup.

The first patches fix a few bugs found while reviewing the existing code,
including incorrect bootmem HVO handling, wrong vmemmap registration
arguments, a powerpc compound-vmemmap tracking bug, and too-late
initialization of gigantic bootmem HugeTLB struct pages.

The rest of the series reorders early memory initialization so the
relevant zone state is available before sparse and HugeTLB boot-time setup
runs, then simplifies the remaining bootmem gigantic hugepage allocation
path and removes code made obsolete by that rework.

At a high level:
  - patches [1-4] fix boot-time and arch-specific bugs
  - patches [5-12] reorder and simplify sparse/mm/hugetlb early init
  - patches [13-19] refactor bootmem gigantic hugepage allocation and
    remove obsolete helpers and state


This patch (of 19):

Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") switched
HVO to reuse per-zone shared tail pages from zone->vmemmap_tails[].

Those shared tail pages were initialized in hugetlb_vmemmap_init(), but
bootmem HugeTLB folios are prepared earlier from
gather_bootmem_prealloc().  With hugetlb_free_vmemmap=on,
prep_and_add_bootmem_folios() can access pageblock flags on bootmem
HugeTLB pages whose mirrored tail struct pages already point to the shared
tail page.  On CONFIG_DEBUG_VM kernels, get_pfnblock_bitmap_bitidx() then
dereferences the still-uninitialized shared tail page and can panic during
boot.

Initialize zone->vmemmap_tails[] from gather_bootmem_prealloc(), before
bootmem HugeTLB folios are processed, and drop the later initialization
from hugetlb_vmemmap_init().

This bug only affects CONFIG_DEBUG_VM kernels, where the relevant
assertion is evaluated.

Link: https://lore.kernel.org/20260612035903.2468601-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260612035903.2468601-2-songmuchun@bytedance.com
Fixes: 622026e87c40 ("mm/hugetlb: remove fake head pages")
Signed-off-by: Muchun Song 
Acked-by: Oscar Salvador 
Tested-by: Michal Clapinski 
Reviewed-by: Michal Clapinski 
Cc: "Aneesh Kumar K.V" 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: "Ritesh Harjani (IBM)" 
Cc: Vlastimil Babka 
Cc: Frank van der Linden 
Cc: Oscar Salvador (SUSE) 
Cc: Usama Arif 
Cc: 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: flush_cache_vmap() after hotplugging vmemmap

2026-07-13T17:29:52+00:00

section_activate() does not flush TLB after populating new vmemmap
pages. On most architectures, this is okay. However it is a problem on
RISC-V since there the TLB caching non-present entries is permitted,
which causes spurious faults on some hardwares.

This seems to be most easily reproduced with DEBUG_VM=y and
PAGE_POISONING=y, which causes these newly mapped struct pages to be
poisoned i.e. written to immediately after mapping.

Extend the RISC-V flush_cache_vmap() to also handle the vmemmap range,
and call it after hotplugging vmemmap, which gets the possible spurious
fault handled in the exception handler.

At least for now, the only other architecture with both
SPARSEMEM_VMEMMAP and flush_cache_vmap() is PowerPC, which has a similar
problem with newly valid PTEs. But there flush_cache_vmap() is just a
ptesync. So it should be safe to do this for generic code while having
minimal performance impact.

Suggested-by: Muchun Song 
Signed-off-by: Vivian Wang 
Reviewed-by: Muchun Song 
Acked-by: David Hildenbrand (Arm) 
Link: https://patch.msgid.link/20260713-mark-after-vmemmap-populate-v6-2-b945ceba29d4@iscas.ac.cn
Signed-off-by: Paul Walmsley

mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()

2026-06-09T01:21:30+00:00

Patch series "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and
vmemmap_check_pmd()", v3.

The weak vmemmap_set_pmd() and vmemmap_check_pmd() hooks are currently
no-ops in the generic code, which leaves architectures that need PMD-level
handling to open-code the same logic locally.

This series provides generic implementations for both helpers in
mm/sparse-vmemmap.c.  vmemmap_set_pmd() installs a huge PMD with
PAGE_KERNEL protection, and vmemmap_check_pmd() verifies a present leaf
PMD before reusing the existing vmemmap_verify() helper.

With those generic helpers in place, patches 2-5 remove the now redundant
arch-specific implementations from arm64, riscv, loongarch, and sparc.


This patch (of 5):

The two weak functions are currently no-ops on every architecture, forcing
each platform that needs them to duplicate the same handful of lines. 
Provide a generic implementation:

- vmemmap_set_pmd() simply sets a huge PMD with PAGE_KERNEL protection.

- vmemmap_check_pmd() verifies that the PMD is present and leaf,
  then calls the existing vmemmap_verify() helper.

Architectures that need special handling can continue to override the weak
symbols; everyone else gets the standard version for free.

Link: https://lore.kernel.org/20260601084845.3792171-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260601084845.3792171-2-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Oscar Salvador (SUSE) 
Cc: Albert Ou 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Huacai Chen 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Palmer Dabbelt 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: fix DAX vmemmap accounting with optimization

2026-05-29T04:04:54+00:00

When vmemmap optimization is enabled for DAX, the nr_memmap_pages counter
in /proc/vmstat is incorrect.  The current code always accounts for the
full, non-optimized vmemmap size, but vmemmap optimization reduces the
actual number of vmemmap pages by reusing tail pages.  This causes the
system to overcount vmemmap usage, leading to inaccurate page statistics
in /proc/vmstat.

Fix this by introducing section_nr_vmemmap_pages(), which returns the
exact vmemmap page count for a given pfn range based on whether
optimization is in effect.

Link: https://lore.kernel.org/20260428081855.1249045-5-songmuchun@bytedance.com
Fixes: 15995a352474 ("mm: report per-page metadata information")
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Oscar Salvador 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Liam R. Howlett 
Cc: "Aneesh Kumar K.V" 
Cc: Joao Martins 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Nicholas Piggin 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/sparse-vmemmap: pass @pgmap argument to memory deactivation paths

2026-05-29T04:04:54+00:00

Currently, the memory hot-remove call chain -- arch_remove_memory(),
__remove_pages(), sparse_remove_section() and section_deactivate() -- does
not carry the struct dev_pagemap pointer.  This prevents the lower levels
from knowing whether the section was originally populated with vmemmap
optimizations (e.g., DAX with vmemmap optimization enabled).

Without this information, we cannot call vmemmap_can_optimize() to
determine if the vmemmap pages were optimized.  As a result, the vmemmap
page accounting during teardown will mistakenly assume a non-optimized
allocation, leading to incorrect memmap statistics.

To lay the groundwork for fixing the vmemmap page accounting, we need to
pass the @pgmap pointer down to the deactivation location.  Plumb the
@pgmap argument through the APIs of arch_remove_memory(), __remove_pages()
and sparse_remove_section(), mirroring the corresponding *_activate()
paths.

Link: https://lore.kernel.org/20260428081855.1249045-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song 
Acked-by: Mike Rapoport (Microsoft) 
Reviewed-by: Oscar Salvador 
Acked-by: David Hildenbrand (Arm) 
Acked-by: Liam R. Howlett 
Cc: "Aneesh Kumar K.V" 
Cc: Joao Martins 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Nicholas Piggin 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton