linux.git/mm/rmap.c, branch v6.17

mm: add get_and_clear_ptes() and clear_ptes()

2025-08-02T19:06:10+00:00

Patch series "Optimizations for khugepaged", v4.

If the underlying folio mapped by the ptes is large, we can process those
ptes in a batch using folio_pte_batch().

For arm64 specifically, this results in a 16x reduction in the number of
ptep_get() calls, since on a contig block, ptep_get() on arm64 will
iterate through all 16 entries to collect a/d bits.  Next, ptep_clear()
will cause a TLBI for every contig block in the range via
contpte_try_unfold().  Instead, use clear_ptes() to only do the TLBI at
the first and last contig block of the range.

For split folios, there will be no pte batching; the batch size returned
by folio_pte_batch() will be 1.  For pagetable split folios, the ptes will
still point to the same large folio; for arm64, this results in the
optimization described above, and for other arches, a minor improvement is
expected due to a reduction in the number of function calls and batching
atomic operations.


This patch (of 3):

Let's add variants to be used where "full" does not apply -- which will
be the majority of cases in the future. "full" really only applies if
we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will
be added next.

Link: https://lkml.kernel.org/r/20250724052301.23844-2-dev.jain@arm.com
Signed-off-by: David Hildenbrand 
Signed-off-by: Dev Jain 
Reviewed-by: Baolin Wang 
Reviewed-by: Barry Song 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: Zi Yan 
Cc: Liam Howlett 
Cc: Mariano Pache 
Cc: Ryan Roberts 
Signed-off-by: Andrew Morton

mm: remove arch_flush_tlb_batched_pending() arch helper

2025-07-25T02:12:32+00:00

Since commit 4b634918384c ("arm64/mm: Close theoretical race where stale
TLB entry remains valid"), all arches that use tlbbatch for reclaim
(arm64, riscv, x86) implement arch_flush_tlb_batched_pending() with a
flush_tlb_mm().

So let's simplify by removing the unnecessary abstraction and doing the
flush_tlb_mm() directly in flush_tlb_batched_pending().  This effectively
reverts commit db6c1f6f236d ("mm/tlbbatch: introduce
arch_flush_tlb_batched_pending()").

Link: https://lkml.kernel.org/r/20250609103132.447370-1-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts 
Suggested-by: Will Deacon 
Reviewed-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Anshuman Khandual 
Acked-by: Will Deacon 
Cc: Albert Ou 
Cc: Alexandre Ghiti 
Cc: Borislav Betkov 
Cc: Catalin Marinas 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: Thomas Gleinxer 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: split folio_pte_batch() into folio_pte_batch() and folio_pte_batch_flags()

2025-07-20T01:59:45+00:00

Many users (including upcoming ones) don't really need the flags etc, and
can live with the possible overhead of a function call.

So let's provide a basic, non-inlined folio_pte_batch(), to avoid code
bloat while still providing a variant that optimizes out all flag checks
at runtime.  folio_pte_batch_flags() will get inlined into
folio_pte_batch(), optimizing out any conditionals that depend on input
flags.

folio_pte_batch() will behave like folio_pte_batch_flags() when no flags
are specified.  It's okay to add new users of folio_pte_batch_flags(), but
using folio_pte_batch() if applicable is preferred.

So, before this change, folio_pte_batch() was inlined into the C file
optimized by propagating constants within the resulting object file.

With this change, we now also have a folio_pte_batch() that is optimized
by propagating all constants.  But instead of having one instance per
object file, we have a single shared one.

In zap_present_ptes(), where we care about performance, the compiler
already seem to generate a call to a common inlined folio_pte_batch()
variant, shared with fork() code.  So calling the new non-inlined variant
should not make a difference.

While at it, drop the "addr" parameter that is unused.

Link: https://lkml.kernel.org/r/20250702104926.212243-4-david@redhat.com
Signed-off-by: David Hildenbrand 
Suggested-by: Andrew Morton 
Link: https://lore.kernel.org/linux-mm/20250503182858.5a02729fcffd6d4723afcfc2@linux-foundation.org/
Reviewed-by: Oscar Salvador 
Reviewed-by: Zi Yan 
Reviewed-by: Dev Jain 
Cc: Alistair Popple 
Cc: Byungchul Park 
Cc: Gregory Price 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joshua Hahn 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Mathew Brost 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: convert FPB_IGNORE_* into FPB_RESPECT_*

2025-07-20T01:59:45+00:00

Patch series "mm: folio_pte_batch() improvements", v2.

Ever since we added folio_pte_batch() for fork() + munmap() purposes, a
lot more users appeared (and more are being proposed), and more
functionality was added.

Most of the users only need basic functionality, and could benefit from a
non-inlined version.

So let's clean up folio_pte_batch() and split it into a basic
folio_pte_batch() (no flags) and a more advanced folio_pte_batch_ext(). 
Using either variant will now look much cleaner.

This series will likely conflict with some changes in some (old+new)
folio_pte_batch() users, but conflicts should be trivial to resolve.


This patch (of 4):

Respecting these PTE bits is the exception, so let's invert the meaning.

With this change, most callers don't have to pass any flags.  This is a
preparation for splitting folio_pte_batch() into a non-inlined variant
that doesn't consume any flags.

Long-term, we want folio_pte_batch() to probably ignore most common PTE
bits (e.g., write/dirty/young/soft-dirty) that are not relevant for most
page table walkers: uffd-wp and protnone might be bits to consider in the
future.  Only walkers that care about them can opt-in to respect them.

No functional change intended.

Link: https://lkml.kernel.org/r/20250702104926.212243-2-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Lance Yang 
Reviewed-by: Zi Yan 
Reviewed-by: Oscar Salvador 
Reviewed-by: Dev Jain 
Cc: Alistair Popple 
Cc: Byungchul Park 
Cc: Gregory Price 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joshua Hahn 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Mathew Brost 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: rename PAGE_MAPPING_* to FOLIO_MAPPING_*

2025-07-13T23:38:31+00:00

Now that the mapping flags are only used for folios, let's rename the
defines.

Link: https://lkml.kernel.org/r/20250704102524.326966-27-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Zi Yan 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: Harry Yoo 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Brendan Jackman 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Christian Brauner 
Cc: Christophe Leroy 
Cc: Eugenio Pé rez 
Cc: Greg Kroah-Hartman 
Cc: Gregory Price 
Cc: "Huang, Ying" 
Cc: Jan Kara 
Cc: Jason Gunthorpe 
Cc: Jason Wang 
Cc: Jerrin Shaji George 
Cc: Johannes Weiner 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Joshua Hahn 
Cc: Liam Howlett 
Cc: Madhavan Srinivasan 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michael Ellerman 
Cc: "Michael S. Tsirkin" 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Minchan Kim 
Cc: Naoya Horiguchi 
Cc: Nicholas Piggin 
Cc: Oscar Salvador 
Cc: Peter Xu 
Cc: Qi Zheng 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Sergey Senozhatsky 
Cc: Shakeel Butt 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Xuan Zhuo 
Cc: xu xin 
Signed-off-by: Andrew Morton

Merge branch 'mm-hotfixes-stable' into mm-stable to pick up changes which

2025-07-12T21:48:26+00:00

are required for a merge of the series "mm: folio_pte_batch()
improvements".

mm: update core kernel code to use vm_flags_t consistently

2025-07-10T05:42:13+00:00

The core kernel code is currently very inconsistent in its use of
vm_flags_t vs.  unsigned long.  This prevents us from changing the type of
vm_flags_t in the future and is simply not correct, so correct this.

While this results in rather a lot of churn, it is a critical
pre-requisite for a future planned change to VMA flag type.

Additionally, update VMA userland tests to account for the changes.

To make review easier and to break things into smaller parts, driver and
architecture-specific changes is left for a subsequent commit.

The code has been adjusted to cascade the changes across all calling code
as far as is needed.

We will adjust architecture-specific and driver code in a subsequent patch.

Overall, this patch does not introduce any functional change.

Link: https://lkml.kernel.org/r/d1588e7bb96d1ea3fe7b9df2c699d5b4592d901d.1750274467.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Acked-by: Kees Cook 
Acked-by: Mike Rapoport (Microsoft) 
Acked-by: Jan Kara 
Acked-by: Christian Brauner 
Reviewed-by: Vlastimil Babka 
Acked-by: Oscar Salvador 
Reviewed-by: Pedro Falcato 
Acked-by: Zi Yan 
Acked-by: David Hildenbrand 
Reviewed-by: Anshuman Khandual 
Cc: Jann Horn 
Cc: Liam R. Howlett 
Cc: Catalin Marinas 
Cc: Jarkko Sakkinen 
Signed-off-by: Andrew Morton

mm/rmap: fix potential out-of-bounds page table access during batched unmap

2025-07-10T04:07:53+00:00

As pointed out by David[1], the batched unmap logic in
try_to_unmap_one() may read past the end of a PTE table when a large
folio's PTE mappings are not fully contained within a single page
table.

While this scenario might be rare, an issue triggerable from userspace
must be fixed regardless of its likelihood.  This patch fixes the
out-of-bounds access by refactoring the logic into a new helper,
folio_unmap_pte_batch().

The new helper correctly calculates the safe batch size by capping the
scan at both the VMA and PMD boundaries.  To simplify the code, it also
supports partial batching (i.e., any number of pages from 1 up to the
calculated safe maximum), as there is no strong reason to special-case
for fully mapped folios.

Link: https://lkml.kernel.org/r/20250701143100.6970-1-lance.yang@linux.dev
Link: https://lkml.kernel.org/r/20250630011305.23754-1-lance.yang@linux.dev
Link: https://lkml.kernel.org/r/20250627062319.84936-1-lance.yang@linux.dev
Link: https://lore.kernel.org/linux-mm/a694398c-9f03-4737-81b9-7e49c857fcbe@redhat.com [1]
Fixes: 354dffd29575 ("mm: support batched unmap for lazyfree large folios during reclamation")
Signed-off-by: Lance Yang 
Suggested-by: David Hildenbrand 
Reported-by: David Hildenbrand 
Closes: https://lore.kernel.org/linux-mm/a694398c-9f03-4737-81b9-7e49c857fcbe@redhat.com
Suggested-by: Barry Song 
Acked-by: Barry Song 
Reviewed-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Harry Yoo 
Cc: Baolin Wang 
Cc: Chris Li 
Cc: "Huang, Ying" 
Cc: Kairui Song 
Cc: Lance Yang 
Cc: Liam Howlett 
Cc: Mingzhe Yang 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: Tangquan Zheng 
Cc: Vlastimil Babka 
Cc: 
Signed-off-by: Andrew Morton

mm/rmap: fix typo in comment in page_address_in_vma

2025-05-13T06:50:37+00:00

Fix a minor typo in the comment above page_address_in_vma():
"responsibililty" → "responsibility"

Link: https://lkml.kernel.org/r/20250421085729.127845-3-ye.liu@linux.dev
Signed-off-by: Ye Liu 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: Harry Yoo 
Cc: David Hildenbrand 
Cc: Liam Howlett 
Cc: Miaohe Lin 
Cc: Naoya Horiguchi 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/rmap: rename page__anon_vma to anon_vma for consistency

2025-05-13T06:50:36+00:00

Patch series "mm: minor cleanups in rmap", v3.

Minor cleanups in mm/rmap.c:

- Rename a local variable for consistency
- Fix a typo in a comment

No functional changes.


This patch (of 2):

Rename local variable page__anon_vma in page_address_in_vma() to anon_vma.
The previous naming convention of using double underscores (__) is
unnecessary and inconsistent with typical kernel style, which uses single
underscores to denote local variables.  Also updated comments to reflect
the new variable name.

Functionality unchanged.

Link: https://lkml.kernel.org/r/20250421085729.127845-1-ye.liu@linux.dev
Link: https://lkml.kernel.org/r/20250421085729.127845-2-ye.liu@linux.dev
Signed-off-by: Ye Liu 
Reviewed-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Harry Yoo 
Cc: Liam Howlett 
Cc: Liu Ye 
Cc: Miaohe Lin 
Cc: Naoya Horiguchi 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton