linux-stable.git/mm/compaction.c, branch v7.2.4

mm: page_alloc: fix non-movable reclaim storm in defrag_mode

2026-09-07T15:36:36+00:00

commit 7e8756d7ad22655b935c384f123071aa9de07a27 upstream.

As we deployed defrag_mode into Meta production, pressure spikes and
excessive swapping were observed on some workloads.  Tracing confirmed
that this is unmovable/reclaimable requests spinning in the allocator and
direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to
produce blocks, and if those are overwhelmed under high pressure, let the
allocator fall back (__rmqueue_steal()) after its retry loops.  However,
that retrying results in more reclaim on some of these workloads than we'd
hoped, sometimes excessively so, spurred on by the !costly order
conditions in should_reclaim_retry().

The storms are dependent on the request type.  Reclaim will inevitably
make room in existing movable blocks, since that's where the LRU pages
live.  So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive.  They
cannot use the individually freed pages, and the process is unlikely to
accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar.  They
spin and overreclaim excessively, which tanks performance and triggers
userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into
pageblock reclaim/compaction.  This way, they help move things along to
meet the ALLOC_NOFRAGMENT bar.  After this patch, the reclaim storms and
excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable
ones, help make blocks to spread the cost of defragmenting more evenly and
fairly; combined with proper watermarking to reduce allocation latencies
in the common case.  However, doing this naively unearths scaling and
concurrency limitations in compaction that need to be addressed first.
Promoting just non-movables for now is the minimally viable bug fix for
the above issue.

[brendan.jackman@linux.dev: fix try_to_compact_pages() kerneldoc]
  Link: https://lore.kernel.org/DK7NM9RPUJOD.11PNJJ5N2OBED@linux.dev
Link: https://lore.kernel.org/20260722150006.3848560-5-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner 
Signed-off-by: "Brendan Jackman" 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Gregory Price 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm: page_alloc: move capture_control to the page allocator

2026-09-07T15:36:35+00:00

commit aee220f565cce38f0efcff940ae2b44bdc495408 upstream.

The compaction capturing code assumes the allocation request order and
compaction target order are the same.  That won't be true once defrag_mode
promotes sub-block allocations to pageblock-order compaction: compaction
targets the larger order, while capture should remain at the original
allocation order.

Move the capture_control to the page allocator and give it its own copies
of what the page freeing path matches against - zone, migratetype and the
allocation order - rather than reaching into compaction's live
compact_control.  __alloc_pages_direct_compact() fills in migratetype and
order, and installs and hides current->capture_control around the whole
compaction call; try_to_compact_pages() aims capc->zone at each zone while
it is being compacted.  compact_zone_order() no longer deals with capture
at all.

Pass the capture_control through try_to_compact_pages() /
compact_zone_order() in place of the bare struct page **.

No functional change.

Link: https://lore.kernel.org/20260722150006.3848560-4-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Vlastimil Babka (SUSE) 
Co-developed-by: Johannes Weiner 
Signed-off-by: Johannes Weiner 
Reviewed-by: Gregory Price 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm: compaction: support non-movable compaction for pageblock requests

2026-09-07T15:36:35+00:00

commit 1b4b697a5743e624cc259d6c31a8d8716b796305 upstream.

While trying to fix a reclaim storm in defrag_mode, I noticed that
non-movable direct compaction is extremely inefficient.

When searching for space to evacuate, compaction only allows blocks of the
same type as the incoming request.  This is to prevent migratetype
pollution, where a small non-movable request frees space in a movable
block and provokes the allocator to fall back and pollute it.

This protection is reasonable on one hand, but the downside is that it
makes non-movable direct compaction nearly useless: if we get the type
annotations right, by definition there aren't any movable pages inside the
non-movable blocks it is allowed to scan.

With defrag_mode, the goal is the production of whole blocks, which are
essentially type neutral: __rmqueue_claim() will convert them wholesale on
alloc.  This makes type mixing and pollution a non-issue.

Fix the pollution gates to take the requested order into account, and
allow whole-block requests to scan blocks of other types.

The only exception is CMA blocks.  That type is sticky and these blocks
cannot be claimed to other types.  Continue to be strict with them, and
allow only explicit ALLOC_CMA requests and kcompactd to evacuate them.

Link: https://lore.kernel.org/20260722150006.3848560-3-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner 
Reviewed-by: Vlastimil Babka (SUSE) 
Reviewed-by: Gregory Price 
Cc: Brendan Jackman 
Cc: Brendan Jackman 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm/compaction: handle free_pages_prepare() properly in compaction_free()

2026-07-02T02:02:53+00:00

free_pages_prepare() can fail but compaction_free() does not handle the
failure case.  Failed pages should not be added back to cc->freepages for
future use, since they can be either PageHWPoison or free_page_is_bad()
and might cause data corruption.

Link: https://lore.kernel.org/20260622-handle_free_pages_prepare_in_compaction_free-v1-1-fcf3b14abcf7@nvidia.com
Fixes: 733aea0b3a7b ("mm/compaction: add support for >0 order folio memory compaction.")
Signed-off-by: Zi Yan 
Reviewed-by: Vlastimil Babka (SUSE) 
Acked-by: Johannes Weiner 
Reviewed-by: Baolin Wang 
Reviewed-by: Lance Yang 
Cc: Brendan Jackman 
Cc: Jiaqi Yan 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: 
Signed-off-by: Andrew Morton

mm/compaction: respect cpusets when checking retry suitability

2026-06-09T01:21:25+00:00

should_compact_retry() handles COMPACT_SKIPPED by asking
compaction_zonelist_suitable() whether reclaim can make a later compaction
attempt worthwhile.  That answer is used for the current allocation, so it
should follow the same zone eligibility rules as the allocation itself.

When cpusets are enabled, allocator slowpath decisions are marked with
ALLOC_CPUSET.  The allocation path, direct compaction and reclaim retry
all skip zones rejected by __cpuset_zone_allowed().

compaction_zonelist_suitable() does not apply that filter.  It only walks
ac->zonelist/ac->nodemask, so it can return true because a zone that is
not usable for the current allocation would pass __compaction_suitable().

That does not let the allocation use the disallowed zone.  Later
allocation and direct compaction paths still apply cpuset filtering. 
However, it can make should_compact_retry() retry based on memory that
this allocation cannot use.

Pass gfp_mask down and apply the same ALLOC_CPUSET check in
compaction_zonelist_suitable().  This keeps the retry decision aligned
with the zones that the allocation is allowed to use.

A temporary debugfs probe was also used to call the old and new
compaction_zonelist_suitable() predicates in the same two-node NUMA guest.
The task was restricted to mems=0 while ac->nodemask covered nodes 0-1. 
After putting pressure on node0, node0 failed __compaction_suitable() for
order-10 and node1 passed it, but node1 was rejected by
__cpuset_zone_allowed().  In that state the old predicate returned true
and the patched predicate returned false.

Link: https://lore.kernel.org/tencent_F59F2BA2CC5779308E10DF54593C736D3E0A@qq.com
Fixes: 435b3894e742 ("mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()")
Signed-off-by: fujunjie 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Brendan Jackman 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: remove mentions of PageWriteback

2026-06-04T21:45:07+00:00

Update two comments to refer to writeback in general instead of the
specific flag.  Convert the large comment in memory.c to be entirely
folio-based.

Link: https://lore.kernel.org/20260526195650.353196-1-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton

mm/page_alloc: don't overload migratetype in find_suitable_fallback()

2026-06-02T22:22:19+00:00

This function currently returns a signed integer that encodes status
in-band, as negative numbers, along with a migratetype.  Switch to a more
explicit/verbose style that encodes the status and migratetype separately.

In the spirit of making things more explicit, also create an enum to avoid
using magic integer literals with special meanings.  This enables
documenting the values at their definition instead of in one of the
callers.

Link: https://lore.kernel.org/20260513-page_alloc-unmapped-prep-v1-2-dacdf5402be8@google.com
Signed-off-by: Brendan Jackman 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Axel Rasmussen 
Cc: Barry Song 
Cc: David Hildenbrand 
Cc: Johannes Weiner 
Cc: Kairui Song 
Cc: Len Brown 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport (Microsoft) 
Cc: "Rafael J. Wysocki" 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Wei Xu 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

2026-04-18T07:10:46+00:00

The following diagram illustrates how to ensure the safety of the folio
lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

    rcu_read_lock();
retry:
    lruvec = folio_lruvec(folio);
    /* There is a possibility of folio reparenting at this point. */
    spin_lock(&lruvec->lru_lock);
    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
        /*
         * The wrong lruvec lock was acquired, and a retry is required.
         * This is because the folio resides on the parent memcg lruvec
         * list.
         */
        spin_unlock(&lruvec->lru_lock);
        goto retry;
    }

    /* Reaching here indicates that folio_memcg() is stable. */


In the memcg_reparent_objcgs(memcg) function:

    spin_lock(&lruvec->lru_lock);
    spin_lock(&lruvec_parent->lru_lock);
    /* Transfer folios from the lruvec list to the parent's. */
    spin_unlock(&lruvec_parent->lru_lock);
    spin_unlock(&lruvec->lru_lock);

After acquiring the lruvec lock, it is necessary to verify whether the
folio has been reparented.  If reparenting has occurred, the new lruvec
lock must be reacquired.  During the LRU folio reparenting process, the
lruvec lock will also be acquired (this will be implemented in a
subsequent patch).  Therefore, folio_memcg() remains unchanged while the
lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio)
after the lruvec lock is acquired, the lruvec_memcg_debug() check is
redundant.  Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

Link: https://lore.kernel.org/23f22cbb1419f277a3483018b32158ae2b86c666.1772711148.git.zhengqi.arch@bytedance.com
Signed-off-by: Muchun Song 
Signed-off-by: Qi Zheng 
Acked-by: Johannes Weiner 
Acked-by: Shakeel Butt 
Cc: Allen Pais 
Cc: Axel Rasmussen 
Cc: Baoquan He 
Cc: Chengming Zhou 
Cc: Chen Ridong 
Cc: David Hildenbrand 
Cc: Hamza Mahfooz 
Cc: Harry Yoo 
Cc: Hugh Dickins 
Cc: Imran Khan 
Cc: Kamalesh Babulal 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Michal Hocko 
Cc: Michal Koutný 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Nhat Pham 
Cc: Roman Gushchin 
Cc: Suren Baghdasaryan 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Cc: Wei Xu 
Cc: Yosry Ahmed 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: rename unlock_page_lruvec_irq and its variants

2026-04-18T07:10:44+00:00

It is inappropriate to use folio_lruvec_lock() variants in conjunction
with unlock_page_lruvec() variants, as this involves the inconsistent
operation of locking a folio while unlocking a page.  To rectify this, the
functions unlock_page_lruvec{_irq, _irqrestore} are renamed to
lruvec_unlock{_irq,_irqrestore}.

Link: https://lore.kernel.org/4e5e05271a250df4d1812e1832be65636a78c957.1772711148.git.zhengqi.arch@bytedance.com
Signed-off-by: Muchun Song 
Signed-off-by: Qi Zheng 
Acked-by: Roman Gushchin 
Acked-by: Johannes Weiner 
Reviewed-by: Harry Yoo 
Reviewed-by: Chen Ridong 
Acked-by: David Hildenbrand (Red Hat) 
Acked-by: Shakeel Butt 
Cc: Allen Pais 
Cc: Axel Rasmussen 
Cc: Baoquan He 
Cc: Chengming Zhou 
Cc: Hamza Mahfooz 
Cc: Hugh Dickins 
Cc: Imran Khan 
Cc: Kamalesh Babulal 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Michal Hocko 
Cc: Michal Koutný 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Nhat Pham 
Cc: Suren Baghdasaryan 
Cc: Usama Arif 
Cc: Vlastimil Babka 
Cc: Wei Xu 
Cc: Yosry Ahmed 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/compaction: fix low_pfn advance on isolating hugetlb

2025-09-28T18:51:29+00:00

Commit 56ae0bb349b4 ("mm: compaction: convert to use a folio in
isolate_migratepages_block()") converts api from page to folio.  But the
low_pfn advance for hugetlb page seems wrong when low_pfn doesn't point to
head page.

Originally, if page is a hugetlb tail page, compound_nr() return 1, which
means low_pfn only advance one in next iteration.  After the change,
low_pfn would advance more than the hugetlb range, since folio_nr_pages()
always return total number of the large page.  This results in skipping
some range to isolate and then to migrate.

The worst case for alloc_contig is it does all the isolation and
migration, but finally find some range is still not isolated.  And then
undo all the work and try a new range.

Advance low_pfn to the end of hugetlb.

Link: https://lkml.kernel.org/r/20250910092240.3981-1-richard.weiyang@gmail.com
Fixes: 56ae0bb349b4 ("mm: compaction: convert to use a folio in isolate_migratepages_block()")
Signed-off-by: Wei Yang 
Acked-by: Zi Yan 
Cc: "Vishal Moola (Oracle)" 
Cc: Kefeng Wang 
Cc: Oscar Salvador 
Cc: Brendan Jackman 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Wei Yang 
Signed-off-by: Andrew Morton