linux.git/mm/huge_memory.c, branch v6.15

mm/huge_memory: fix dereferencing invalid pmd migration entry

2025-05-08T06:39:38+00:00

When migrating a THP, concurrent access to the PMD migration entry during
a deferred split scan can lead to an invalid address access, as
illustrated below.  To prevent this invalid access, it is necessary to
check the PMD migration entry and return early.  In this context, there is
no need to use pmd_to_swp_entry and pfn_swap_entry_to_page to verify the
equality of the target folio.  Since the PMD migration entry is locked, it
cannot be served as the target.

Mailing list discussion and explanation from Hugh Dickins: "An anon_vma
lookup points to a location which may contain the folio of interest, but
might instead contain another folio: and weeding out those other folios is
precisely what the "folio != pmd_folio((*pmd)" check (and the "risk of
replacing the wrong folio" comment a few lines above it) is for."

BUG: unable to handle page fault for address: ffffea60001db008
CPU: 0 UID: 0 PID: 2199114 Comm: tee Not tainted 6.14.0+ #4 NONE
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:split_huge_pmd_locked+0x3b5/0x2b60
Call Trace:

try_to_migrate_one+0x28c/0x3730
rmap_walk_anon+0x4f6/0x770
unmap_folio+0x196/0x1f0
split_huge_page_to_list_to_order+0x9f6/0x1560
deferred_split_scan+0xac5/0x12a0
shrinker_debugfs_scan_write+0x376/0x470
full_proxy_write+0x15c/0x220
vfs_write+0x2fc/0xcb0
ksys_write+0x146/0x250
do_syscall_64+0x6a/0x120
entry_SYSCALL_64_after_hwframe+0x76/0x7e

The bug is found by syzkaller on an internal kernel, then confirmed on
upstream.

Link: https://lkml.kernel.org/r/20250421113536.3682201-1-gavinguo@igalia.com
Link: https://lore.kernel.org/all/20250414072737.1698513-1-gavinguo@igalia.com/
Link: https://lore.kernel.org/all/20250418085802.2973519-1-gavinguo@igalia.com/
Fixes: 84c3fc4e9c56 ("mm: thp: check pmd migration entry in common path")
Signed-off-by: Gavin Guo 
Acked-by: David Hildenbrand 
Acked-by: Hugh Dickins 
Acked-by: Zi Yan 
Reviewed-by: Gavin Shan 
Cc: Florent Revest 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: 
Signed-off-by: Andrew Morton

mm: separate folio_split_memcg_refs() from split_page_memcg()

2025-03-22T05:03:11+00:00

Patch series "Minor memcg cleanups & prep for memdescs", v2.

Separate the handling of accounted folios and GFP_ACCOUNT pages for easier
to understand code.  For more detail, see
https://lore.kernel.org/linux-mm/Z9LwTOudOlCGny3f@casper.infradead.org/


This patch (of 5):

Folios always use memcg_data to refer to the mem_cgroup while pages
allocated with GFP_ACCOUNT have a pointer to the obj_cgroup.  Since the
caller already knows what it has, split the function into two and then we
don't need to check.

Move the assignment of split folio memcg_data to the point where we set up
the other parts of the new folio.  That leaves folio_split_memcg_refs()
just handling the memcg accounting.

Link: https://lkml.kernel.org/r/20250314133617.138071-1-willy@infradead.org
Link: https://lkml.kernel.org/r/20250314133617.138071-2-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Johannes Weiner 
Acked-by: Shakeel Butt 
Acked-by: Zi Yan 
Acked-by: Roman Gushchin 
Cc: David Hildenbrand 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Signed-off-by: Andrew Morton

mm: convert lru_add_page_tail() to lru_add_split_folio()

2025-03-18T05:07:06+00:00

Remove three hidden calls to compound_head() and accesses to page->lru.

Link: https://lkml.kernel.org/r/20250313151458.4145978-1-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Zi Yan 
Signed-off-by: Andrew Morton

mm/truncate: use folio_split() in truncate operation

2025-03-18T05:07:00+00:00

Instead of splitting the large folio uniformly during truncation, try to
use buddy allocator like folio_split() at the start and the end of a
truncation range to minimize the number of resulting folios if it is
supported.  try_folio_split() is introduced to use folio_split() if
supported and it falls back to uniform split otherwise.

For example, to truncate a order-4 folio
[0, 1, 2, 3, 4, 5, ..., 15]
between [3, 10] (inclusive), folio_split() splits the folio at 3 to
[0,1], [2], [3], [4..7], [8..15] and [3], [4..7] can be dropped and
[8..15] is kept with zeros in [8..10], then another folio_split() is
done at 10, so [8..10] can be dropped.

One possible optimization is to make folio_split() to split a folio based
on a given range, like [3..10] above.  But that complicates folio_split(),
so it will be investigated when necessary.

Link: https://lkml.kernel.org/r/20250226210032.2044041-8-ziy@nvidia.com
Link: https://lkml.kernel.org/r/20250307174001.242794-8-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: add folio_split() to debugfs testing interface

2025-03-18T05:07:00+00:00

This allows to test folio_split() by specifying an additional in folio
page offset parameter to split_huge_page debugfs interface.

Link: https://lkml.kernel.org/r/20250307174001.242794-7-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: remove the old, unused __split_huge_page()

2025-03-18T05:07:00+00:00

Now split_huge_page_to_list_to_order() uses the new backend split code in
__split_unmapped_folio(), the old __split_huge_page() and
__split_huge_page_tail() can be removed.

Link: https://lkml.kernel.org/r/20250307174001.242794-6-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: add buddy allocator like (non-uniform) folio_split()

2025-03-18T05:06:59+00:00

folio_split() splits a large folio in the same way as buddy allocator
splits a large free page for allocation.  The purpose is to minimize the
number of folios after the split.  For example, if user wants to free the
3rd subpage in a order-9 folio, folio_split() will split the order-9 folio
as:

O-0, O-0, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-8 if it is anon,
since anon folio does not support order-1 yet.
-----------------------------------------------------------------
|   |   |   |   |     |   |       |                             |
|O-0|O-0|O-0|O-0| O-2 |...|  O-7  |             O-8             |
|   |   |   |   |     |   |       |                             |
-----------------------------------------------------------------

O-1,      O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-9 if it is pagecache
---------------------------------------------------------------
|     |   |   |     |   |       |                             |
| O-1 |O-0|O-0| O-2 |...|  O-7  |             O-8             |
|     |   |   |     |   |       |                             |
---------------------------------------------------------------

It generates fewer folios (i.e., 11 or 10) than existing page split
approach, which splits the order-9 to 512 order-0 folios.  It also reduces
the number of new xa_node needed during a pagecache folio split from 8 to
1, potentially decreasing the folio split failure rate due to memory
constraints.

folio_split() and existing split_huge_page_to_list_to_order() share the
folio unmapping and remapping code in __folio_split() and the common
backend split code in __split_unmapped_folio() using uniform_split
variable to distinguish their operations.

uniform_split_supported() and non_uniform_split_supported() are added to
factor out check code and will be used outside __folio_split() in the
following commit.

Link: https://lkml.kernel.org/r/20250307174001.242794-5-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: move folio split common code to __folio_split()

2025-03-18T05:06:59+00:00

This is a preparation patch for folio_split().

In the upcoming patch folio_split() will share folio unmapping and
remapping code with split_huge_page_to_list_to_order(), so move the code
to a common function __folio_split() first.

Add a TODO for splitting large shmem folio in swap cache.

Link: https://lkml.kernel.org/r/20250307174001.242794-4-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: add two new (not yet used) functions for folio_split()

2025-03-18T05:06:59+00:00

This is a preparation patch, both added functions are not used yet.

The added __split_unmapped_folio() is able to split a folio with its
mapping removed in two manners: 1) uniform split (the existing way), and
2) buddy allocator like (or non-uniform) split.

The added __split_folio_to_order() can split a folio into any lower order.
For uniform split, __split_unmapped_folio() calls it once to split the
given folio to the new order.  For buddy allocator like (non-uniform)
split, __split_unmapped_folio() calls it (folio_order - new_order) times
and each time splits the folio containing the given page to one lower
order.

[ziy@nvidia.com: unfreeze head folio after page cache entries are updated]
  Link: https://lkml.kernel.org/r/0F15DA7F-1977-412F-9A3E-F06B515D4BD2@nvidia.com
[ziy@nvidia.com: use NULL instead of 0 for folio->private assignment]
  Link: https://lkml.kernel.org/r/1E11B9DD-3A87-4C9C-8FB4-E1324FB6A21A@nvidia.com
Link: https://lkml.kernel.org/r/20250307174001.242794-3-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm: remove redundant return in set_huge_zero_folio()

2025-03-18T05:06:57+00:00

It is the responsibility of the caller to check pmd_none(); in any case,
we are not achieving anything by returning since there is no return value
to tell the caller that we succeeded or not.  So remove this check.

Link: https://lkml.kernel.org/r/20250306144315.21907-1-dev.jain@arm.com
Signed-off-by: Dev Jain 
Reviewed-by: David Hildenbrand 
Cc: Matthew Wilcow (Oracle) 
Cc: Ryan Roberts 
Signed-off-by: Andrew Morton