linux.git/mm/mm_init.c, branch v6.15

mm/page_alloc: fix race condition in unaccepted memory handling

2025-05-12T00:26:07+00:00

The page allocator tracks the number of zones that have unaccepted memory
using static_branch_enc/dec() and uses that static branch in hot paths to
determine if it needs to deal with unaccepted memory.

Borislav and Thomas pointed out that the tracking is racy: operations on
static_branch are not serialized against adding/removing unaccepted pages
to/from the zone.

Sanity checks inside static_branch machinery detects it:

WARNING: CPU: 0 PID: 10 at kernel/jump_label.c:276 __static_key_slow_dec_cpuslocked+0x8e/0xa0

The comment around the WARN() explains the problem:

	/*
	 * Warn about the '-1' case though; since that means a
	 * decrement is concurrent with a first (0->1) increment. IOW
	 * people are trying to disable something that wasn't yet fully
	 * enabled. This suggests an ordering problem on the user side.
	 */

The effect of this static_branch optimization is only visible on
microbenchmark.

Instead of adding more complexity around it, remove it altogether.

Link: https://lkml.kernel.org/r/20250506133207.1009676-1-kirill.shutemov@linux.intel.com
Signed-off-by: Kirill A. Shutemov 
Fixes: dcdfdd40fa82 ("mm: Add support for unaccepted memory")
Link: https://lore.kernel.org/all/20250506092445.GBaBnVXXyvnazly6iF@fat_crate.local
Reported-by: Borislav Petkov 
Tested-by: Borislav Petkov (AMD) 
Reported-by: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Suren Baghdasaryan 
Cc: Michal Hocko 
Cc: Brendan Jackman 
Cc: Johannes Weiner 
Cc: 	[6.5+]
Signed-off-by: Andrew Morton

mm,mm_init: Mark set_high_memory as __init

2025-05-06T17:07:25+00:00

set_high_memory() touches arch_zone_lowest_possible_pfn which is
marked as __initdata, which creates a section mismatch.
Since the only user of the function is free_area_init() which is also marked
as __init, mark set_high_memory() as __init as well.

Signed-off-by: Oscar Salvador 
Reported-by: kernel test robot 
Closes: https://lore.kernel.org/oe-kbuild-all/202505060901.Qcs06UoB-lkp@intel.com/
Link: https://lore.kernel.org/r/20250506111012.108743-1-osalvador@suse.de
Signed-off-by: Mike Rapoport (Microsoft)

mm/page_alloc: fix deadlock on cpu_hotplug_lock in __accept_page()

2025-04-18T03:10:05+00:00

When the last page in the zone is accepted, __accept_page() calls
static_branch_dec().  This function takes cpu_hotplug_lock, which can lead
to a deadlock if the allocation occurs during CPU bringup path as
_cpu_up() also takes the lock.

To prevent this deadlock, defer static_branch_dec() to a workqueue.

Call static_branch_dec() only when the workqueue is not yet initialized. 
Workqueues are initialized before CPU bring up, so this will not conflict
with the first scenario.

Link: https://lkml.kernel.org/r/20250329171030.3942298-1-kirill.shutemov@linux.intel.com
Fixes: 55ad43e8ba0f ("mm: add a helper to accept page")
Signed-off-by: Kirill A. Shutemov 
Reported-by: Srikanth Aithal 
Tested-by: Srikanth Aithal 
Cc: Dave Hansen 
Cc: Ashish Kalra 
Cc: David Hildenbrand 
Cc: "Edgecombe, Rick P" 
Cc: Mel Gorman 
Cc: "Mike Rapoport (IBM)" 
Cc: Thomas Lendacky 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/mm_init: init holes in the end of the memory map for FLATMEM

2025-04-01T22:17:11+00:00

Patch series "mm: fixes for fallouts from mem_init() cleanup".

These are the fixes for fallouts from mem_init() cleanup reported by
Nathan Chancellor and kbuild.  The details are in the commit messages.


This patch (of 2):

Kernel test robot reports the following crash on 32-bit system with
FLATMEM and DEBUG_VM_PGFLAGS enabled:

[    0.478822][    T0] kernel BUG at include/linux/page-flags.h:536!
[    0.479312][    T0] Oops: invalid opcode: 0000 [#1] PREEMPT SMP
[    0.479768][    T0] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.14.0-rc6-00357-g8268af309d07 #1
[    0.480470][    T0] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 0.481260][ T0] EIP: reserve_bootmem_region (include/linux/page-flags.h:536)
[ 0.481683][ T0] Code: 5d c3 01 f1 89 c8 ba e1 38 f4 c3 e8 1e 37 8e fc 0f 0b b8 90 e2 62 c4 e8 e2 05 5e fc 01 f1 89 c8 ba be 85 f7 c3 e8 04 37 8e fc <0f> 0b b8 80 e2 62 c4 e8 c8 05 5e fc 55 89 e5 53 57 56 83 ec 10 89
[    0.483177][    T0] EAX: 00000000 EBX: c425df50 ECX: 00000000 EDX: 00000000
[    0.483712][    T0] ESI: 017ffc00 EDI: ffffffff EBP: c425df34 ESP: c425df2c
[    0.484248][    T0] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210046
[    0.484846][    T0] CR0: 80050033 CR2: 00000000 CR3: 04b48000 CR4: 00000090
[    0.485376][    T0] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[    0.485907][    T0] DR6: fffe0ff0 DR7: 00000400
[    0.486253][    T0] Call Trace:
[ 0.486494][ T0] ? __die_body (arch/x86/kernel/dumpstack.c:478)
[ 0.486822][ T0] ? die (arch/x86/kernel/dumpstack.c:?)
[ 0.487099][ T0] ? do_trap (arch/x86/kernel/traps.c:? arch/x86/kernel/traps.c:197)
[ 0.487409][ T0] ? do_error_trap (arch/x86/kernel/traps.c:217)
[ 0.487752][ T0] ? reserve_bootmem_region (include/linux/page-flags.h:536)
[ 0.488153][ T0] ? exc_overflow (arch/x86/kernel/traps.c:301)
[ 0.488490][ T0] ? handle_invalid_op (arch/x86/kernel/traps.c:254)
[ 0.488869][ T0] ? reserve_bootmem_region (include/linux/page-flags.h:536)
[ 0.489271][ T0] ? exc_invalid_op (arch/x86/kernel/traps.c:316)
[ 0.489619][ T0] ? handle_exception (arch/x86/entry/entry_32.S:1055)
[ 0.489996][ T0] ? exc_overflow (arch/x86/kernel/traps.c:301)
[ 0.490332][ T0] ? reserve_bootmem_region (include/linux/page-flags.h:536)
[ 0.490733][ T0] ? exc_overflow (arch/x86/kernel/traps.c:301)
[ 0.491068][ T0] ? reserve_bootmem_region (include/linux/page-flags.h:536)
[ 0.491470][ T0] memmap_init_reserved_pages (mm/memblock.c:2203)
[ 0.491887][ T0] free_low_memory_core_early (mm/memblock.c:?)
[ 0.492302][ T0] memblock_free_all (mm/memblock.c:2272 include/linux/atomic/atomic-arch-fallback.h:546 include/linux/atomic/atomic-long.h:123 include/linux/atomic/atomic-instrumented.h:3261 include/linux/mm.h:67 mm/memblock.c:2273)
[ 0.492659][ T0] mem_init (arch/x86/mm/init_32.c:735)
[ 0.492952][ T0] mm_core_init (mm/mm_init.c:2730)
[ 0.493271][ T0] start_kernel (init/main.c:958)
[ 0.493604][ T0] i386_start_kernel (arch/x86/kernel/head32.c:79)
[ 0.493969][ T0] startup_32_smp (arch/x86/kernel/head_32.S:292)

The crash happens because after commit 8268af309d07 ("arch, mm: set
max_mapnr when allocating memory map for FLATMEM") max_mapnr is rounded up
to MAX_ORDER_NR_PAGES and the pages in the end of the memory map are
passing pfn_valid() check in reserve_bootmem_region().

Make sure that that pages in the end of the memory map are initialized,
just like the pages in the end of the last section for SPARSEMEM.

Link: https://lkml.kernel.org/r/20250325114928.1791109-1-rppt@kernel.org
Link: https://lkml.kernel.org/r/20250325114928.1791109-2-rppt@kernel.org
Fixes: 8268af309d07 ("arch, mm: set max_mapnr when allocating memory map for FLATMEM")
Signed-off-by: Mike Rapoport (Microsoft) 
Reported-by: kernel test robot 
Closes: https://lore.kernel.org/oe-lkp/202503241424.d16223ec-lkp@intel.com
Cc: Andy Lutomirski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: Nathan Chancellor 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleinxer 
Signed-off-by: Andrew Morton

mm/mm_init: rename init_reserved_page to init_deferred_page

2025-03-22T05:03:11+00:00

When CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled, init_reserved_page()
function performs initialization of a struct page that would have been
deferred normally.

Rename it to init_deferred_page() to better reflect what the function does.

Link: https://lkml.kernel.org/r/20250225083017.567649-3-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: Wei Yang 
Cc: Frank van der Linden 
Cc: Muchun Song 
Cc: Changyuan Lyu 
Signed-off-by: Andrew Morton

mm/mm_init: rename __init_reserved_page_zone to __init_page_from_nid

2025-03-22T05:03:11+00:00

__init_reserved_page_zone() function finds the zone for pfn and nid and
performs initialization of a struct page with that zone and nid.  There is
nothing in that function about reserved pages and it is misnamed.

Rename it to __init_page_from_nid() to better reflect what the function
does.

Link: https://lkml.kernel.org/r/20250225083017.567649-2-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: Wei Yang 
Cc: Frank van der Linden 
Cc: Muchun Song 
Signed-off-by: Andrew Morton

arch, mm: make releasing of memory to page allocator more explicit

2025-03-18T05:06:53+00:00

The point where the memory is released from memblock to the buddy
allocator is hidden inside arch-specific mem_init()s and the call to
memblock_free_all() is needlessly duplicated in every artiste cure and
after introduction of arch_mm_preinit() hook, mem_init() implementation on
many architecture only contains the call to memblock_free_all().

Pull memblock_free_all() call into mm_core_init() and drop mem_init() on
relevant architectures to make it more explicit where the free memory is
released from memblock to the buddy allocator and to reduce code
duplication in architecture specific code.

Link: https://lkml.kernel.org/r/20250313135003.836600-14-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Dave Hansen 	[x86]
Acked-by: Geert Uytterhoeven 	[m68k]
Tested-by: Mark Brown 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Andy Lutomirski 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Betkov 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Dinh Nguyen 
Cc: Gerald Schaefer 
Cc: Guo Ren (csky) 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: Johannes Berg 
Cc: John Paul Adrian Glaubitz 
Cc: Madhavan Srinivasan 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Michal Simek 
Cc: Palmer Dabbelt 
Cc: Richard Weinberger 
Cc: Russel King 
Cc: Stafford Horne 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleinxer 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

arch, mm: introduce arch_mm_preinit

2025-03-18T05:06:53+00:00

Currently, implementation of mem_init() in every architecture consists of
one or more of the following:

* initializations that must run before page allocator is active, for
  instance swiotlb_init()
* a call to memblock_free_all() to release all the memory to the buddy
  allocator
* initializations that must run after page allocator is ready and there is
  no arch-specific hook other than mem_init() for that, like for example
  register_page_bootmem_info() in x86 and sparc64 or simple setting of
  mem_init_done = 1 in several architectures
* a bunch of semi-related stuff that apparently had no better place to
  live, for example a ton of BUILD_BUG_ON()s in parisc.

Introduce arch_mm_preinit() that will be the first thing called from
mm_core_init(). On architectures that have initializations that must happen
before the page allocator is ready, move those into arch_mm_preinit() along
with the code that does not depend on ordering with page allocator setup.

On several architectures this results in reduction of mem_init() to a
single call to memblock_free_all() that allows its consolidation next.

Link: https://lkml.kernel.org/r/20250313135003.836600-13-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Dave Hansen 	[x86]
Tested-by: Mark Brown 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Andy Lutomirski 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Betkov 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Guo Ren (csky) 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: Johannes Berg 
Cc: John Paul Adrian Glaubitz 
Cc: Madhavan Srinivasan 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Michal Simek 
Cc: Palmer Dabbelt 
Cc: Richard Weinberger 
Cc: Russel King 
Cc: Stafford Horne 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleinxer 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

arch, mm: set high_memory in free_area_init()

2025-03-18T05:06:52+00:00

high_memory defines upper bound on the directly mapped memory.  This bound
is defined by the beginning of ZONE_HIGHMEM when a system has high memory
and by the end of memory otherwise.

All this is known to generic memory management initialization code that
can set high_memory while initializing core mm structures.

Add a generic calculation of high_memory to free_area_init() and remove
per-architecture calculation except for the architectures that set and use
high_memory earlier than that.

Link: https://lkml.kernel.org/r/20250313135003.836600-11-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Dave Hansen 	[x86]
Tested-by: Mark Brown 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Andy Lutomirski 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Betkov 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Guo Ren (csky) 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: Johannes Berg 
Cc: John Paul Adrian Glaubitz 
Cc: Madhavan Srinivasan 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Michal Simek 
Cc: Palmer Dabbelt 
Cc: Richard Weinberger 
Cc: Russel King 
Cc: Stafford Horne 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleinxer 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

arch, mm: set max_mapnr when allocating memory map for FLATMEM

2025-03-18T05:06:52+00:00

max_mapnr is essentially the size of the memory map for systems that use
FLATMEM. There is no reason to calculate it in each and every architecture
when it's anyway calculated in alloc_node_mem_map().

Drop setting of max_mapnr from architecture code and set it once in
alloc_node_mem_map().

While on it, move definition of mem_map and max_mapnr to mm/mm_init.c so
there won't be two copies for MMU and !MMU variants.

Link: https://lkml.kernel.org/r/20250313135003.836600-10-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: Dave Hansen 	[x86]
Tested-by: Mark Brown 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Andy Lutomirski 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Betkov 
Cc: Catalin Marinas 
Cc: David S. Miller 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Guo Ren (csky) 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: Johannes Berg 
Cc: John Paul Adrian Glaubitz 
Cc: Madhavan Srinivasan 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Michal Simek 
Cc: Palmer Dabbelt 
Cc: Richard Weinberger 
Cc: Russel King 
Cc: Stafford Horne 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleinxer 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Signed-off-by: Andrew Morton