linux.git/mm/rmap.c, branch v6.16

mm/rmap: fix potential out-of-bounds page table access during batched unmap

2025-07-10T04:07:53+00:00

As pointed out by David[1], the batched unmap logic in
try_to_unmap_one() may read past the end of a PTE table when a large
folio's PTE mappings are not fully contained within a single page
table.

While this scenario might be rare, an issue triggerable from userspace
must be fixed regardless of its likelihood.  This patch fixes the
out-of-bounds access by refactoring the logic into a new helper,
folio_unmap_pte_batch().

The new helper correctly calculates the safe batch size by capping the
scan at both the VMA and PMD boundaries.  To simplify the code, it also
supports partial batching (i.e., any number of pages from 1 up to the
calculated safe maximum), as there is no strong reason to special-case
for fully mapped folios.

Link: https://lkml.kernel.org/r/20250701143100.6970-1-lance.yang@linux.dev
Link: https://lkml.kernel.org/r/20250630011305.23754-1-lance.yang@linux.dev
Link: https://lkml.kernel.org/r/20250627062319.84936-1-lance.yang@linux.dev
Link: https://lore.kernel.org/linux-mm/a694398c-9f03-4737-81b9-7e49c857fcbe@redhat.com [1]
Fixes: 354dffd29575 ("mm: support batched unmap for lazyfree large folios during reclamation")
Signed-off-by: Lance Yang 
Suggested-by: David Hildenbrand 
Reported-by: David Hildenbrand 
Closes: https://lore.kernel.org/linux-mm/a694398c-9f03-4737-81b9-7e49c857fcbe@redhat.com
Suggested-by: Barry Song 
Acked-by: Barry Song 
Reviewed-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Harry Yoo 
Cc: Baolin Wang 
Cc: Chris Li 
Cc: "Huang, Ying" 
Cc: Kairui Song 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Mingzhe Yang 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: Tangquan Zheng 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/rmap: fix typo in comment in page_address_in_vma

2025-05-13T06:50:37+00:00

Fix a minor typo in the comment above page_address_in_vma():
"responsibililty" → "responsibility"

Link: https://lkml.kernel.org/r/20250421085729.127845-3-ye.liu@linux.dev
Signed-off-by: Ye Liu 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: Harry Yoo 
Cc: David Hildenbrand 
Cc: Liam Howlett 
Cc: Miaohe Lin 
Cc: Naoya Horiguchi 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/rmap: rename page__anon_vma to anon_vma for consistency

2025-05-13T06:50:36+00:00

Patch series "mm: minor cleanups in rmap", v3.

Minor cleanups in mm/rmap.c:

- Rename a local variable for consistency
- Fix a typo in a comment

No functional changes.


This patch (of 2):

Rename local variable page__anon_vma in page_address_in_vma() to anon_vma.
The previous naming convention of using double underscores (__) is
unnecessary and inconsistent with typical kernel style, which uses single
underscores to denote local variables.  Also updated comments to reflect
the new variable name.

Functionality unchanged.

Link: https://lkml.kernel.org/r/20250421085729.127845-1-ye.liu@linux.dev
Link: https://lkml.kernel.org/r/20250421085729.127845-2-ye.liu@linux.dev
Signed-off-by: Ye Liu 
Reviewed-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Harry Yoo 
Cc: Liam Howlett 
Cc: Liu Ye 
Cc: Miaohe Lin 
Cc: Naoya Horiguchi 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/huge_memory: remove useless folio pointers passing

2025-05-13T06:50:34+00:00

Since the previous commit "mm/huge_memory: Adjust try_to_migrate_one() and
split_huge_pmd_locked()" has simplified the logic by leveraging the folio
verification in page_vma_mapped_walk(), this patch removes the unnecessary
folio pointers passing.

Link: https://lkml.kernel.org/r/20250425103859.825879-3-gavinguo@igalia.com
Link: https://lore.kernel.org/all/98d1d195-7821-4627-b518-83103ade56c0@redhat.com/
Link: https://lore.kernel.org/all/91599a3c-e69e-4d79-bac5-5013c96203d7@redhat.com/
Signed-off-by: Gavin Guo 
Suggested-by: David Hildenbrand 
Acked-by: David Hildenbrand 
Reviewed-by: Zi Yan 
Reviewed-by: Baolin Wang 
Cc: Florent Revest 
Cc: Gavin Shan 
Cc: Hugh Dickins 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Signed-off-by: Andrew Morton

mm/huge_memory: adjust try_to_migrate_one() and split_huge_pmd_locked()

2025-05-13T06:50:33+00:00

Patch series "Clean up split_huge_pmd_locked() and remove unnecessary
folio pointers", v2.

The patch series enhances the folio verification by leveraging the
existing page_vma_mapped_walk() mechanism and removing redundant folio
pointer passing.


This patch (of 2):

split_huge_pmd_locked() currently performs redundant checks for migration
entries and folio validation that are already handled by the
page_vma_mapped_walk mechanism in try_to_migrate_one.

Specifically, page_vma_mapped_walk already ensures that:
- The folio is properly mapped in the given VMA area
- pmd_trans_huge, pmd_devmap, and migration entry validation are
  performed

To leverage page_vma_mapped_walk's work, moving TTU_SPLIT_HUGE_PMD
handling to the while loop checking and removing these duplicate checks
from split_huge_pmd_locked.

Link: https://lkml.kernel.org/r/20250425103859.825879-1-gavinguo@igalia.com
Link: https://lkml.kernel.org/r/20250425103859.825879-2-gavinguo@igalia.com
Link: https://lore.kernel.org/all/98d1d195-7821-4627-b518-83103ade56c0@redhat.com/
Link: https://lore.kernel.org/all/91599a3c-e69e-4d79-bac5-5013c96203d7@redhat.com/
Signed-off-by: Gavin Guo 
Suggested-by: David Hildenbrand 
Acked-by: David Hildenbrand 
Reviewed-by: Zi Yan 
Reviewed-by: Baolin Wang 
Cc: Florent Revest 
Cc: Gavin Shan 
Cc: Hugh Dickins 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Signed-off-by: Andrew Morton

mm: stop maintaining the per-page mapcount of large folios (CONFIG_NO_PAGE_MAPCOUNT)

2025-03-18T05:06:48+00:00

Everything is in place to stop using the per-page mapcounts in large
folios: the mapcount of tail pages will always be logically 0 (-1 value),
just like it currently is for hugetlb folios already, and the page
mapcount of the head page is either 0 (-1 value) or contains a page type
(e.g., hugetlb).

Maintaining _nr_pages_mapped without per-page mapcounts is impossible, so
that one also has to go with CONFIG_NO_PAGE_MAPCOUNT.

There are two remaining implications:

(1) Per-node, per-cgroup and per-lruvec stats of "NR_ANON_MAPPED"
    ("mapped anonymous memory") and "NR_FILE_MAPPED"
    ("mapped file memory"):

    As soon as any page of the folio is mapped -- folio_mapped() -- we
    now account the complete folio as mapped. Once the last page is
    unmapped -- !folio_mapped() -- we account the complete folio as
    unmapped.

    This implies that ...

    * "AnonPages" and "Mapped" in /proc/meminfo and
      /sys/devices/system/node/*/meminfo
    * cgroup v2: "anon" and "file_mapped" in "memory.stat" and
      "memory.numa_stat"
    * cgroup v1: "rss" and "mapped_file" in "memory.stat" and
      "memory.numa_stat

    ... can now appear higher than before. But note that these folios do
    consume that memory, simply not all pages are actually currently
    mapped.

    It's worth nothing that other accounting in the kernel (esp. cgroup
    charging on allocation) is not affected by this change.

    [why oh why is "anon" called "rss" in cgroup v1]

 (2) Detecting partial mappings

     Detecting whether anon THPs are partially mapped gets a bit more
     unreliable. As long as a single MM maps such a large folio
     ("exclusively mapped"), we can reliably detect it. Especially before
     fork() / after a short-lived child process quit, we will detect
     partial mappings reliably, which is the common case.

     In essence, if the average per-page mapcount in an anon THP is < 1,
     we know for sure that we have a partial mapping.

     However, as soon as multiple MMs are involved, we might miss detecting
     partial mappings: this might be relevant with long-lived child
     processes. If we have a fully-mapped anon folio before fork(), once
     our child processes and our parent all unmap (zap/COW) the same pages
     (but not the complete folio), we might not detect the partial mapping.
     However, once the child processes quit we would detect the partial
     mapping.

     How relevant this case is in practice remains to be seen.
     Swapout/migration will likely mitigate this.

     In the future, RMAP walkers could check for that for that case
     (e.g., when collecting access bits during reclaim) and simply flag
     them for deferred-splitting.

Link: https://lkml.kernel.org/r/20250303163014.1128035-21-david@redhat.com
Signed-off-by: David Hildenbrand 
Cc: Andy Lutomirks^H^Hski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Johannes Weiner 
Cc: Jonathan Corbet 
Cc: Kirill A. Shutemov 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Koutn 
Cc: Muchun Song 
Cc: tejun heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Zefan Li 
Signed-off-by: Andrew Morton

mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared()

2025-03-18T05:06:46+00:00

Let's reuse our new MM ownership tracking infrastructure for large folios
to make folio_likely_mapped_shared() never return false negatives -- never
indicating "not mapped shared" although the folio *is* mapped shared. 
With that, we can rename it to folio_maybe_mapped_shared() and get rid of
the dependency on the mapcount of the first folio page.

The semantics are now arguably clearer: no mixture of "false negatives"
and "false positives", only the remaining possibility for "false
positives".

Thoroughly document the new semantics.  We might now detect that a large
folio is "maybe mapped shared" although it *no longer* is -- but once was.
Now, if more than two MMs mapped a folio at the same time, and the MM
mapping the folio exclusively at the end is not one tracked in the two
folio MM slots, we will detect the folio as "maybe mapped shared".

For anonymous folios, usually (except weird corner cases) all PTEs that
target a "maybe mapped shared" folio are R/O.  As soon as a child process
would write to them (iow, actively use them), we would CoW and effectively
replace these PTEs.  Most cases (below) are not expected to really matter
with large anonymous folios for this reason.

Most importantly, there will be no change at all for:
* small folios
* hugetlb folios
* PMD-mapped PMD-sized THPs (single mapping)

This change has the potential to affect existing callers of
folio_likely_mapped_shared() -> folio_maybe_mapped_shared():

(1) fs/proc/task_mmu.c: no change (hugetlb)

(2) khugepaged counts PTEs that target shared folios towards
    max_ptes_shared (default: HPAGE_PMD_NR / 2), meaning we could skip a
    collapse where we would have previously collapsed.  This only applies
    to anonymous folios and is not expected to matter in practice.

    Worth noting that this change sorts out case (A) documented in
    commit 1bafe96e89f0 ("mm/khugepaged: replace page_mapcount() check by
    folio_likely_mapped_shared()") by removing the possibility for "false
    negatives".

(3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting
    PTE-mapped THPs that are considered shared but not fully covered by
    the requested range, consequently not processing them.

    PMD-mapped PMD-sized THP are not affected, or when all PTEs are
    covered.  These functions are usually only called on anon/file folios
    that are exclusively mapped most of the time (no other file mappings
    or no fork()), so the "false negatives" are not expected to matter in
    practice.

(4) mbind() / migrate_pages() / move_pages() will refuse to migrate
    shared folios unless MPOL_MF_MOVE_ALL is effective (requires
    CAP_SYS_NICE).  We will now reject some folios that could be migrated.

    Similar to (3), especially with MPOL_MF_MOVE_ALL, so this is not
    expected to matter in practice.

    Note that cpuset_migrate_mm_workfn() calls do_migrate_pages() with
    MPOL_MF_MOVE_ALL.

(5) NUMA hinting

    mm/migrate.c:migrate_misplaced_folio_prepare() will skip file
    folios that are probably shared libraries (-> "mapped shared" and
    executable).  This check would have detected it as a shared library at
    some point (at least 3 MMs mapping it), so detecting it afterwards
    does not sound wrong (still a shared library).  Not expected to
    matter.

    mm/memory.c:numa_migrate_check() will indicate TNF_SHARED in
    MAP_SHARED file mappings when encountering a shared folio.  Similar
    reasoning, not expected to matter.

    mm/mprotect.c:change_pte_range() will skip folios detected as
    shared in CoW mappings.  Similarly, this is not expected to matter in
    practice, but if it would ever be a problem we could relax that check
    a bit (e.g., basing it on the average page-mapcount in a folio),
    because it was only an optimization when many (e.g., 288) processes
    were mapping the same folios -- see commit 859d4adc3415 ("mm: numa: do
    not trap faults on shared data section pages.")

(6) mm/rmap.c:folio_referenced_one() will skip exclusive swapbacked
    folios in dying processes.  Applies to anonymous folios only.  Without
    "false negatives", we'll now skip all actually shared ones.  Skipping
    ones that are actually exclusive won't really matter, it's a pure
    optimization, and is not expected to matter in practice.

In theory, one can detect the problematic scenario: folio_mapcount() > 0
and no folio MM slot is occupied ("state unknown").  One could reset the
MM slots while doing an rmap walk, which migration / folio split already
do when setting everything up.  Further, when batching PTEs we might
naturally learn about a owner (e.g., folio_mapcount() == nr_ptes) and
could update the owner.  However, we'll defer that until the scenarios
where it would really matter are clear.

Link: https://lkml.kernel.org/r/20250303163014.1128035-15-david@redhat.com
Signed-off-by: David Hildenbrand 
Cc: Andy Lutomirks^H^Hski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Johannes Weiner 
Cc: Jonathan Corbet 
Cc: Kirill A. Shutemov 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Koutn 
Cc: Muchun Song 
Cc: tejun heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Zefan Li 
Signed-off-by: Andrew Morton

mm/rmap: use folio_large_nr_pages() in add/remove functions

2025-03-18T05:06:45+00:00

Let's just use the "large" variant in code where we are sure that we have
a large folio in our hands: this way we are sure that we don't perform any
unnecessary "large" checks.

While at it, convert the VM_BUG_ON_VMA to a VM_WARN_ON_ONCE.

Maybe in the future there will not be a difference in that regard between
large and small folios; in that case, unifying the handling again will be
easy.  E.g., folio_large_nr_pages() will simply translate to
folio_nr_pages() until we replace all instances.

Link: https://lkml.kernel.org/r/20250303163014.1128035-12-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Kirill A. Shutemov 
Cc: Andy Lutomirks^H^Hski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Johannes Weiner 
Cc: Jonathan Corbet 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Koutn 
Cc: Muchun Song 
Cc: tejun heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Zefan Li 
Signed-off-by: Andrew Morton

mm/rmap: abstract large mapcount operations for large folios (!hugetlb)

2025-03-18T05:06:45+00:00

Let's abstract the operations so we can extend these operations easily.

Link: https://lkml.kernel.org/r/20250303163014.1128035-10-david@redhat.com
Signed-off-by: David Hildenbrand 
Cc: Andy Lutomirks^H^Hski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Johannes Weiner 
Cc: Jonathan Corbet 
Cc: Kirill A. Shutemov 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Koutn 
Cc: Muchun Song 
Cc: tejun heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Zefan Li 
Signed-off-by: Andrew Morton

mm/rmap: pass vma to __folio_add_rmap()

2025-03-18T05:06:45+00:00

We'll need access to the destination MM when modifying the mapcount large
folios next.  So pass in the VMA.

Link: https://lkml.kernel.org/r/20250303163014.1128035-9-david@redhat.com
Signed-off-by: David Hildenbrand 
Cc: Andy Lutomirks^H^Hski 
Cc: Borislav Betkov 
Cc: Dave Hansen 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Johannes Weiner 
Cc: Jonathan Corbet 
Cc: Kirill A. Shutemov 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcow (Oracle) 
Cc: Michal Koutn 
Cc: Muchun Song 
Cc: tejun heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Zefan Li 
Signed-off-by: Andrew Morton