linux.git/mm/compaction.c, branch v7.3-rc2

mm: page_alloc: fix non-movable reclaim storm in defrag_mode

2026-08-07T01:57:24+00:00

As we deployed defrag_mode into Meta production, pressure spikes and
excessive swapping were observed on some workloads.  Tracing confirmed
that this is unmovable/reclaimable requests spinning in the allocator and
direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to
produce blocks, and if those are overwhelmed under high pressure, let the
allocator fall back (__rmqueue_steal()) after its retry loops.  However,
that retrying results in more reclaim on some of these workloads than we'd
hoped, sometimes excessively so, spurred on by the !costly order
conditions in should_reclaim_retry().

The storms are dependent on the request type.  Reclaim will inevitably
make room in existing movable blocks, since that's where the LRU pages
live.  So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive.  They
cannot use the individually freed pages, and the process is unlikely to
accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.  They
spin and overreclaim excessively, which tanks performance and triggers
userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into
pageblock reclaim/compaction.  This way, they help move things along to
meet the ALLOC_NOFRAGMENT bar.  After this patch, the reclaim storms and
excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable
ones, help make blocks to spread the cost of defragmenting more evenly and
fairly; combined with proper watermarking to reduce allocation latencies
in the common case.  However, doing this naively unearths scaling and
concurrency limitations in compaction that need to be addressed first. 
Promoting just non-movables for now is the minimally viable bug fix for
the above issue.

[brendan.jackman@linux.dev: fix try_to_compact_pages() kerneldoc]
  Link: https://lore.kernel.org/DK7NM9RPUJOD.11PNJJ5N2OBED@linux.dev
Link: https://lore.kernel.org/20260722150006.3848560-5-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner 
Signed-off-by: "Brendan Jackman" 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Gregory Price 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton

mm: page_alloc: move capture_control to the page allocator

2026-08-07T01:57:24+00:00

The compaction capturing code assumes the allocation request order and
compaction target order are the same.  That won't be true once defrag_mode
promotes sub-block allocations to pageblock-order compaction: compaction
targets the larger order, while capture should remain at the original
allocation order.

Move the capture_control to the page allocator and give it its own copies
of what the page freeing path matches against - zone, migratetype and the
allocation order - rather than reaching into compaction's live
compact_control.  __alloc_pages_direct_compact() fills in migratetype and
order, and installs and hides current->capture_control around the whole
compaction call; try_to_compact_pages() aims capc->zone at each zone while
it is being compacted.  compact_zone_order() no longer deals with capture
at all.

Pass the capture_control through try_to_compact_pages() /
compact_zone_order() in place of the bare struct page **.

No functional change.

Link: https://lore.kernel.org/20260722150006.3848560-4-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Vlastimil Babka (SUSE) 
Co-developed-by: Johannes Weiner 
Signed-off-by: Johannes Weiner 
Reviewed-by: Gregory Price 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton

mm: compaction: support non-movable compaction for pageblock requests

2026-08-07T01:57:23+00:00

While trying to fix a reclaim storm in defrag_mode, I noticed that
non-movable direct compaction is extremely inefficient.

When searching for space to evacuate, compaction only allows blocks of the
same type as the incoming request.  This is to prevent migratetype
pollution, where a small non-movable request frees space in a movable
block and provokes the allocator to fall back and pollute it.

This protection is reasonable on one hand, but the downside is that it
makes non-movable direct compaction nearly useless: if we get the type
annotations right, by definition there aren't any movable pages inside the
non-movable blocks it is allowed to scan.

With defrag_mode, the goal is the production of whole blocks, which are
essentially type neutral: __rmqueue_claim() will convert them wholesale on
alloc.  This makes type mixing and pollution a non-issue.

Fix the pollution gates to take the requested order into account, and
allow whole-block requests to scan blocks of other types.

The only exception is CMA blocks.  That type is sticky and these blocks
cannot be claimed to other types.  Continue to be strict with them, and
allow only explicit ALLOC_CMA requests and kcompactd to evacuate them.

Link: https://lore.kernel.org/20260722150006.3848560-3-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner 
Reviewed-by: Vlastimil Babka (SUSE) 
Reviewed-by: Gregory Price 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton

mm/compaction: stop recording free page order in page->private

2026-08-05T02:18:32+00:00

Commit 733aea0b3a7bb ("mm/compaction: add support for >0 order folio
memory compaction.") stores isolated free pages in an array indexed by
free page orders, it is no longer needed to store the order in each page's
->private field.  And there is no code using the stored order.  Stop doing
that.

It also prepares for an upcoming change that ensures subpage->private is
zero at page free time and the removal of set_page_private(0) from
prep_compound_tail().

Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-2-7e4fe155f5b9@nvidia.com
Signed-off-by: Zi Yan 
Reviewed-by: David Hildenbrand (Arm) 
Reviewed-by: Lance Yang 
Reviewed-by: Baolin Wang 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Alistair Popple 
Cc: Barry Song 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: Dennis Zhou 
Cc: Dev Jain 
Cc: Johannes Weiner 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nico Pache 
Cc: Ryan Roberts 
Cc: Suren Baghdasaryan 
Cc: Tejun Heo 
Signed-off-by: Andrew Morton

mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG

2026-07-31T02:40:44+00:00

Now that alloc_pages has an entrypoint that allows passing alloc_flags, we
can take advantage of this to start removing GFP flags that are only used
for mm-internal stuff.

This requires also plumbing the alloc_flags into some more of the
allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags
arg to go along with its callees, and we now need to pass those flags
deeper into the allocator so they can reach the alloc_tag code.

While moving the flag definition into page_alloc.h, also update the
comment per Hao's suggestion.

No functional change intended.

Link: https://lore.kernel.org/all/b4916118-3537-4e19-8bc8-1d103dd0d225@linux.dev/
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-15-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman 
Reviewed-by: Vlastimil Babka (SUSE) 
Tested-by: Hao Ge 
Acked-by: Hao Ge 
Signed-off-by: Andrew Morton

mm: split out internal page_alloc.h

2026-07-31T02:40:41+00:00

internal.h is a bit bloated, seems like time for a page_alloc.h.

Where it wasn't obvious, the heuristic for deciding what goes into this
new header was "does it support/correspond to a definition in
mm/page_alloc.c?"

Only need to include it from ~20 .c files out of ~150 so this does seem
like a genuine reduction in scopes, which is nice. And there's no
circular internal.h<->page_alloc.h dependency, so it seems worthwhile to
split this up before that inevitably emerges!

Link: https://lore.kernel.org/20260703-alloc-trylock-v5-4-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman 
Suggested-by: "David Hildenbrand (Arm)" 
Link: https://lore.kernel.org/all/41e92bab-6882-401a-8de9-154adbdcfb36@kernel.org/
Reviewed-by: Vlastimil Babka (SUSE) 
Acked-by: Zi Yan 
Signed-off-by: Andrew Morton

mm/compaction: handle free_pages_prepare() properly in compaction_free()

2026-07-02T02:02:53+00:00

free_pages_prepare() can fail but compaction_free() does not handle the
failure case.  Failed pages should not be added back to cc->freepages for
future use, since they can be either PageHWPoison or free_page_is_bad()
and might cause data corruption.

Link: https://lore.kernel.org/20260622-handle_free_pages_prepare_in_compaction_free-v1-1-fcf3b14abcf7@nvidia.com
Fixes: 733aea0b3a7b ("mm/compaction: add support for >0 order folio memory compaction.")
Signed-off-by: Zi Yan 
Reviewed-by: Vlastimil Babka (SUSE) 
Acked-by: Johannes Weiner 
Reviewed-by: Baolin Wang 
Reviewed-by: Lance Yang 
Cc: Brendan Jackman 
Cc: Jiaqi Yan 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: 
Signed-off-by: Andrew Morton

mm/compaction: respect cpusets when checking retry suitability

2026-06-09T01:21:25+00:00

should_compact_retry() handles COMPACT_SKIPPED by asking
compaction_zonelist_suitable() whether reclaim can make a later compaction
attempt worthwhile.  That answer is used for the current allocation, so it
should follow the same zone eligibility rules as the allocation itself.

When cpusets are enabled, allocator slowpath decisions are marked with
ALLOC_CPUSET.  The allocation path, direct compaction and reclaim retry
all skip zones rejected by __cpuset_zone_allowed().

compaction_zonelist_suitable() does not apply that filter.  It only walks
ac->zonelist/ac->nodemask, so it can return true because a zone that is
not usable for the current allocation would pass __compaction_suitable().

That does not let the allocation use the disallowed zone.  Later
allocation and direct compaction paths still apply cpuset filtering. 
However, it can make should_compact_retry() retry based on memory that
this allocation cannot use.

Pass gfp_mask down and apply the same ALLOC_CPUSET check in
compaction_zonelist_suitable().  This keeps the retry decision aligned
with the zones that the allocation is allowed to use.

A temporary debugfs probe was also used to call the old and new
compaction_zonelist_suitable() predicates in the same two-node NUMA guest.
The task was restricted to mems=0 while ac->nodemask covered nodes 0-1. 
After putting pressure on node0, node0 failed __compaction_suitable() for
order-10 and node1 passed it, but node1 was rejected by
__cpuset_zone_allowed().  In that state the old predicate returned true
and the patched predicate returned false.

Link: https://lore.kernel.org/tencent_F59F2BA2CC5779308E10DF54593C736D3E0A@qq.com
Fixes: 435b3894e742 ("mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()")
Signed-off-by: fujunjie 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Brendan Jackman 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: remove mentions of PageWriteback

2026-06-04T21:45:07+00:00

Update two comments to refer to writeback in general instead of the
specific flag.  Convert the large comment in memory.c to be entirely
folio-based.

Link: https://lore.kernel.org/20260526195650.353196-1-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton

mm/page_alloc: don't overload migratetype in find_suitable_fallback()

2026-06-02T22:22:19+00:00

This function currently returns a signed integer that encodes status
in-band, as negative numbers, along with a migratetype.  Switch to a more
explicit/verbose style that encodes the status and migratetype separately.

In the spirit of making things more explicit, also create an enum to avoid
using magic integer literals with special meanings.  This enables
documenting the values at their definition instead of in one of the
callers.

Link: https://lore.kernel.org/20260513-page_alloc-unmapped-prep-v1-2-dacdf5402be8@google.com
Signed-off-by: Brendan Jackman 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Axel Rasmussen 
Cc: Barry Song 
Cc: David Hildenbrand 
Cc: Johannes Weiner 
Cc: Kairui Song 
Cc: Len Brown 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport (Microsoft) 
Cc: "Rafael J. Wysocki" 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Wei Xu 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton