linux.git/include/asm-generic/tlb.h, branch v6.15

mm: pgtable: remove tlb_remove_page_ptdesc()

2025-04-01T22:17:14+00:00

The tlb_remove_ptdesc()/tlb_remove_table() is specially designed for page
table pages, and now all architectures have been converted to use it to
remove page table pages.  So let's remove tlb_remove_page_ptdesc(), it
currently has no users and should not be used for page table pages.

Link: https://lkml.kernel.org/r/3df04c8494339073b71be4acb2d92e108ecd1b60.1740454179.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Suggested-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Cc: Alexandre Ghiti 
Cc: "Aneesh Kumar K.V" 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Hugh Dickens 
Cc: Jann Horn 
Cc: Matthew Wilcow (Oracle) 
Cc: "Mike Rapoport (IBM)" 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Rik van Riel 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: change pt parameter of tlb_remove_ptdesc() to struct ptdesc*

2025-04-01T22:17:13+00:00

All callers of tlb_remove_ptdesc() pass it a pointer of struct ptdesc, so
let's change the pt parameter from void * to struct ptdesc * to perform a
type safety check.

Link: https://lkml.kernel.org/r/60bb44299cf2d731df6592e446e7f694054d0dbe.1740454179.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Originally-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Cc: Alexandre Ghiti 
Cc: "Aneesh Kumar K.V" 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Hugh Dickens 
Cc: Jann Horn 
Cc: Matthew Wilcow (Oracle) 
Cc: "Mike Rapoport (IBM)" 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Rik van Riel 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: make generic tlb_remove_table() use struct ptdesc

2025-04-01T22:17:13+00:00

Patch series "remove tlb_remove_page_ptdesc()", v2.

As suggested by Peter Zijlstra below [1], this series aims to remove
tlb_remove_page_ptdesc().

: Fundamentally tlb_remove_page() is about removing *pages* as from a PTE,
: there should not be a page-table anywhere near here *ever*.
:
: Yes, some architectures use tlb_remove_page() for page-tables too, but
: that is more or less an implementation detail that can be fixed.

After this series, all architectures use tlb_remove_table() or
tlb_remove_ptdesc() to remove the page table pages.  In the future, once
all architectures using tlb_remove_table() have also converted to using
struct ptdesc (eg.  powerpc), it may be possible to use only
tlb_remove_ptdesc().

[1] https://lore.kernel.org/linux-mm/20250103111457.GC22934@noisy.programming.kicks-ass.net/


This patch (of 6):

Now only arm will call tlb_remove_ptdesc()/tlb_remove_table() when
CONFIG_MMU_GATHER_TABLE_FREE is disabled.  In this case, the type of the
table parameter is actually struct ptdesc * instead of struct page *.

Since struct ptdesc still overlaps with struct page and has not been
separated from it, forcing the table parameter to struct page * will not
cause any problems at this time.  But this is definitely incorrect and
needs to be fixed.  So just like the generic __tlb_remove_table(), let
generic tlb_remove_table() use struct ptdesc by default when
CONFIG_MMU_GATHER_TABLE_FREE is disabled.

Link: https://lkml.kernel.org/r/cover.1740454179.git.zhengqi.arch@bytedance.com
Link: https://lkml.kernel.org/r/5be8c3ab7bd68510bf0db4cf84010f4dfe372917.1740454179.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Reviewed-by: Kevin Brodsky 
Cc: Alexandre Ghiti 
Cc: "Aneesh Kumar K.V" 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Hugh Dickens 
Cc: Jann Horn 
Cc: Matthew Wilcow (Oracle) 
Cc: "Mike Rapoport (IBM)" 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Rik van Riel 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/mmu_gather: clean up the stale code comment

2025-03-17T05:06:14+00:00

In commit d7f861b9c43a ("mm/mmu_gather: add __tlb_remove_folio_pages()"),
helper function __tlb_remove_folio_pages_size() was added.  And based on
the helper, wrapper functions __tlb_remove_folio_pages() and
__tlb_remove_page_size() are created and used by upper level functions.

So let's update the code comment to reflect the current code about
tlb_remove_page()/tlb_remove_page_size(), etc.

Link: https://lkml.kernel.org/r/20250211034348.39531-2-bhe@redhat.com
Signed-off-by: Baoquan He 
Cc: "Aneesh Kumar K.V" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Qi Zheng 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm/mmu_gather: remove unused __tlb_remove_page()

2025-03-17T05:06:14+00:00

Nobody is using __tlb_remove_page() now, clean it up.

And also remove the code comment above tlb_remove_page() because it's not
meaningful any more.

Link: https://lkml.kernel.org/r/Z6si0/A/zzEF/bFJ@MiWiFi-R3L-srv
Signed-off-by: Baoquan He 
Reviewed-by: Qi Zheng 
Cc: "Aneesh Kumar K.V" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: pgtable: introduce generic pagetable_dtor_free()

2025-01-26T04:22:23+00:00

The pte_free(), pmd_free(), __pud_free() and __p4d_free() in
asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically
the same, so let's introduce pagetable_dtor_free() to deduplicate them.

In addition, the pagetable_dtor_free() in s390 does the same thing, so
let's s390 also calls generic pagetable_dtor_free().

Link: https://lkml.kernel.org/r/1663a0565aca881d1338ceb7d1db4aa9c333abd6.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Suggested-by: Peter Zijlstra (Intel) 
Reviewed-by: Kevin Brodsky 
Acked-by: Alexander Gordeev 	[s390]
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table()

2025-01-26T04:22:23+00:00

For the generic tlb_remove_table(), it is implemented in the following two
forms:

1) CONFIG_MMU_GATHER_TABLE_FREE is enabled

tlb_remove_table
--> generic __tlb_remove_table()

2) CONFIG_MMU_GATHER_TABLE_FREE is disabled

tlb_remove_table
--> tlb_remove_page

For case 1), the pagetable_dtor() has already been moved to generic
__tlb_remove_table().

For case 2), now only arm will call tlb_remove_table()/tlb_remove_ptdesc()
when CONFIG_MMU_GATHER_TABLE_FREE is disabled.  Let's move
pagetable_dtor() completely to generic tlb_remove_table(), so that the
architectures can follow more easily.

Link: https://lkml.kernel.org/r/0c733ac867b287ec08190676496d1decebf49da2.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Suggested-by: Kevin Brodsky 
Reviewed-by: Kevin Brodsky 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Andreas Larsson 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Arnd Bergmann 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Peter Zijlstra (Intel) 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: pgtable: introduce generic __tlb_remove_table()

2025-01-26T04:22:23+00:00

Several architectures (arm, arm64, riscv and x86) define exactly the same
__tlb_remove_table(), just introduce generic __tlb_remove_table() to
eliminate these duplications.

The s390 __tlb_remove_table() is nearly the same, so also make s390
__tlb_remove_table() version generic.

Link: https://lkml.kernel.org/r/ea372633d94f4d3f9f56a7ec5994bf050bf77e39.1736317725.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Reviewed-by: Kevin Brodsky 
Acked-by: Andreas Larsson 		[sparc]
Acked-by: Alexander Gordeev 	[s390]
Acked-by: Arnd Bergmann 			[asm-generic]
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Aneesh Kumar K.V (Arm) 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Rapoport (Microsoft) 
Cc: Muchun Song 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Peter Zijlstra (Intel) 
Cc: Ryan Roberts 
Cc: Thomas Gleixner 
Cc: Vishal Moola (Oracle) 
Cc: Will Deacon 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/mmu_gather: add __tlb_remove_folio_pages()

2024-02-22T23:27:17+00:00

Add __tlb_remove_folio_pages(), which will remove multiple consecutive
pages that belong to the same large folio, instead of only a single page. 
We'll be using this function when optimizing unmapping/zapping of large
folios that are mapped by PTEs.

We're using the remaining spare bit in an encoded_page to indicate that
the next enoced page in an array contains actually shifted "nr_pages". 
Teach swap/freeing code about putting multiple folio references, and
delayed rmap handling to remove page ranges of a folio.

This extension allows for still gathering almost as many small folios as
we used to (-1, because we have to prepare for a possibly bigger next
entry), but still allows for gathering consecutive pages that belong to
the same large folio.

Note that we don't pass the folio pointer, because it is not required for
now.  Further, we don't support page_size != PAGE_SIZE, it won't be
required for simple PTE batching.

We have to provide a separate s390 implementation, but it's fairly
straight forward.

Another, more invasive and likely more expensive, approach would be to use
folio+range or a PFN range instead of page+nr_pages.  But, we should do
that consistently for the whole mmu_gather.  For now, let's keep it simple
and add "nr_pages" only.

Note that it is now possible to gather significantly more pages: In the
past, we were able to gather ~10000 pages, now we can also gather ~5000
folio fragments that span multiple pages.  A folio fragment on x86-64 can
span up to 512 pages (2 MiB THP) and on arm64 with 64k in theory 8192
pages (512 MiB THP).  Gathering more memory is not considered something we
should worry about, especially because these are already corner cases.

While we can gather more total memory, we won't free more folio fragments.
As long as page freeing time primarily only depends on the number of
involved folios, there is no effective change for !preempt configurations.
However, we'll adjust tlb_batch_pages_flush() separately to handle corner
cases where page freeing time grows proportionally with the actual memory
size.

Link: https://lkml.kernel.org/r/20240214204435.167852-9-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Ryan Roberts 
Cc: Alexander Gordeev 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: "Naveen N. Rao" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Cc: Yin Fengwei 
Signed-off-by: Andrew Morton

mm/mmu_gather: add tlb_remove_tlb_entries()

2024-02-22T23:27:17+00:00

Let's add a helper that lets us batch-process multiple consecutive PTEs.

Note that the loop will get optimized out on all architectures except on
powerpc.  We have to add an early define of __tlb_remove_tlb_entry() on
ppc to make the compiler happy (and avoid making tlb_remove_tlb_entries()
a macro).

[arnd@kernel.org: change __tlb_remove_tlb_entry() to an inline function]
  Link: https://lkml.kernel.org/r/20240221154549.2026073-1-arnd@kernel.org
Link: https://lkml.kernel.org/r/20240214204435.167852-8-david@redhat.com
Signed-off-by: David Hildenbrand 
Signed-off-by: Arnd Bergmann 
Reviewed-by: Ryan Roberts 
Cc: Alexander Gordeev 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: "Naveen N. Rao" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Cc: Yin Fengwei 
Signed-off-by: Andrew Morton