linux-stable.git/mm/page_alloc.c, branch v6.6.2

mm/page_alloc: correct start page when guard page debug is enabled

2023-10-06T21:11:38+00:00

When guard page debug is enabled and set_page_guard returns success, we
miss to forward page to point to start of next split range and we will do
split unexpectedly in page range without target page.  Move start page
update before set_page_guard to fix this.

As we split to wrong target page, then splited pages are not able to merge
back to original order when target page is put back and splited pages
except target page is not usable.  To be specific:

Consider target page is the third page in buddy page with order 2.
| buddy-2 | Page | Target | Page |

After break down to target page, we will only set first page to Guard
because of bug.
| Guard   | Page | Target | Page |

When we try put_page_back_buddy with target page, the buddy page of target
if neither guard nor buddy, Then it's not able to construct original page
with order 2
| Guard | Page | buddy-0 | Page |

All pages except target page is not in free list and is not usable.

Link: https://lkml.kernel.org/r/20230927094401.68205-1-shikemeng@huaweicloud.com
Fixes: 06be6ff3d2ec ("mm,hwpoison: rework soft offline for free pages")
Signed-off-by: Kemeng Shi 
Acked-by: Naoya Horiguchi 
Cc: Matthew Wilcox (Oracle) 
Cc: Oscar Salvador 
Cc: 
Signed-off-by: Andrew Morton

mm: page_alloc: fix CMA and HIGHATOMIC landing on the wrong buddy list

2023-09-19T20:21:32+00:00

Commit 4b23a68f9536 ("mm/page_alloc: protect PCP lists with a spinlock")
bypasses the pcplist on lock contention and returns the page directly to
the buddy list of the page's migratetype.

For pages that don't have their own pcplist, such as CMA and HIGHATOMIC,
the migratetype is temporarily updated such that the page can hitch a ride
on the MOVABLE pcplist.  Their true type is later reassessed when flushing
in free_pcppages_bulk().  However, when lock contention is detected after
the type was already overridden, the bypass will then put the page on the
wrong buddy list.

Once on the MOVABLE buddy list, the page becomes eligible for fallbacks
and even stealing.  In the case of HIGHATOMIC, otherwise ineligible
allocations can dip into the highatomic reserves.  In the case of CMA, the
page can be lost from the CMA region permanently.

Use a separate pcpmigratetype variable for the pcplist override.  Use the
original migratetype when going directly to the buddy.  This fixes the bug
and should make the intentions more obvious in the code.

Originally sent here to address the HIGHATOMIC case:
https://lore.kernel.org/lkml/20230821183733.106619-4-hannes@cmpxchg.org/

Changelog updated in response to the CMA-specific bug report.

[mgorman@techsingularity.net: updated changelog]
Link: https://lkml.kernel.org/r/20230911181108.GA104295@cmpxchg.org
Fixes: 4b23a68f9536 ("mm/page_alloc: protect PCP lists with a spinlock")
Signed-off-by: Johannes Weiner 
Reported-by: Joe Liu 
Reviewed-by: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm: page_alloc: remove stale CMA guard code

2023-09-02T22:17:34+00:00

In the past, movable allocations could be disallowed from CMA through
PF_MEMALLOC_PIN.  As CMA pages are funneled through the MOVABLE pcplist,
this required filtering that cornercase during allocations, such that
pinnable allocations wouldn't accidentally get a CMA page.

However, since 8e3560d963d2 ("mm: honor PF_MEMALLOC_PIN for all movable
pages"), PF_MEMALLOC_PIN automatically excludes __GFP_MOVABLE.  Once
again, MOVABLE implies CMA is allowed.

Remove the stale filtering code.  Also remove a stale comment that was
introduced as part of the filtering code, because the filtering let
order-0 pages fall through to the buddy allocator.  See 1d91df85f399
("mm/page_alloc: handle a missing case for memalloc_nocma_{save/restore}
APIs") for context.  The comment's been obsolete since the introduction of
the explicit ALLOC_HIGHATOMIC flag in eb2e2b425c69 ("mm/page_alloc:
explicitly record high-order atomic allocations in alloc_flags").

Link: https://lkml.kernel.org/r/20230824153821.243148-1-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner 
Acked-by: Mel Gorman 
Cc: David Hildenbrand 
Cc: Joonsoo Kim 
Cc: Miaohe Lin 
Cc: Pasha Tatashin 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: add large_rmappable page flag

2023-08-21T21:28:44+00:00

Stored in the first tail page's flags, this flag replaces the destructor. 
That removes the last of the destructors, so remove all references to
folio_dtor and compound_dtor.

Link: https://lkml.kernel.org/r/20230816151201.3655946-9-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: remove HUGETLB_PAGE_DTOR

2023-08-21T21:28:44+00:00

We can use a bit in page[1].flags to indicate that this folio belongs to
hugetlb instead of using a value in page[1].dtors.  That lets
folio_test_hugetlb() become an inline function like it should be.  We can
also get rid of NULL_COMPOUND_DTOR.

Link: https://lkml.kernel.org/r/20230816151201.3655946-8-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: remove free_compound_page() and the compound_page_dtors array

2023-08-21T21:28:44+00:00

The only remaining destructor is free_compound_page().  Inline it into
destroy_large_folio() and remove the array it used to live in.

Link: https://lkml.kernel.org/r/20230816151201.3655946-7-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: convert prep_transhuge_page() to folio_prep_large_rmappable()

2023-08-21T21:28:43+00:00

Match folio_undo_large_rmappable(), and move the casting from page to
folio into the callers (which they were largely doing anyway).

Link: https://lkml.kernel.org/r/20230816151201.3655946-6-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: convert free_transhuge_folio() to folio_undo_large_rmappable()

2023-08-21T21:28:43+00:00

Indirect calls are expensive, thanks to Spectre.  Test for
TRANSHUGE_PAGE_DTOR and destroy the folio appropriately.  Move the
free_compound_page() call into destroy_large_folio() to simplify later
patches.

Link: https://lkml.kernel.org/r/20230816151201.3655946-5-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: convert free_huge_page() to free_huge_folio()

2023-08-21T21:28:43+00:00

Pass a folio instead of the head page to save a few instructions.  Update
the documentation, at least in English.

Link: https://lkml.kernel.org/r/20230816151201.3655946-4-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Sidhartha Kumar 
Cc: Yanteng Si 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Signed-off-by: Andrew Morton

mm: call free_huge_page() directly

2023-08-21T21:28:43+00:00

Indirect calls are expensive, thanks to Spectre.  Call free_huge_page()
directly if the folio belongs to hugetlb.

Link: https://lkml.kernel.org/r/20230816151201.3655946-3-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton