linux.git/include/asm-generic/tlb.h, branch vsnprintf

mm/mmu_gather: add __tlb_remove_folio_pages()

2024-02-22T23:27:17+00:00

Add __tlb_remove_folio_pages(), which will remove multiple consecutive
pages that belong to the same large folio, instead of only a single page. 
We'll be using this function when optimizing unmapping/zapping of large
folios that are mapped by PTEs.

We're using the remaining spare bit in an encoded_page to indicate that
the next enoced page in an array contains actually shifted "nr_pages". 
Teach swap/freeing code about putting multiple folio references, and
delayed rmap handling to remove page ranges of a folio.

This extension allows for still gathering almost as many small folios as
we used to (-1, because we have to prepare for a possibly bigger next
entry), but still allows for gathering consecutive pages that belong to
the same large folio.

Note that we don't pass the folio pointer, because it is not required for
now.  Further, we don't support page_size != PAGE_SIZE, it won't be
required for simple PTE batching.

We have to provide a separate s390 implementation, but it's fairly
straight forward.

Another, more invasive and likely more expensive, approach would be to use
folio+range or a PFN range instead of page+nr_pages.  But, we should do
that consistently for the whole mmu_gather.  For now, let's keep it simple
and add "nr_pages" only.

Note that it is now possible to gather significantly more pages: In the
past, we were able to gather ~10000 pages, now we can also gather ~5000
folio fragments that span multiple pages.  A folio fragment on x86-64 can
span up to 512 pages (2 MiB THP) and on arm64 with 64k in theory 8192
pages (512 MiB THP).  Gathering more memory is not considered something we
should worry about, especially because these are already corner cases.

While we can gather more total memory, we won't free more folio fragments.
As long as page freeing time primarily only depends on the number of
involved folios, there is no effective change for !preempt configurations.
However, we'll adjust tlb_batch_pages_flush() separately to handle corner
cases where page freeing time grows proportionally with the actual memory
size.

Link: https://lkml.kernel.org/r/20240214204435.167852-9-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Ryan Roberts 
Cc: Alexander Gordeev 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: "Naveen N. Rao" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Cc: Yin Fengwei 
Signed-off-by: Andrew Morton

mm/mmu_gather: add tlb_remove_tlb_entries()

2024-02-22T23:27:17+00:00

Let's add a helper that lets us batch-process multiple consecutive PTEs.

Note that the loop will get optimized out on all architectures except on
powerpc.  We have to add an early define of __tlb_remove_tlb_entry() on
ppc to make the compiler happy (and avoid making tlb_remove_tlb_entries()
a macro).

[arnd@kernel.org: change __tlb_remove_tlb_entry() to an inline function]
  Link: https://lkml.kernel.org/r/20240221154549.2026073-1-arnd@kernel.org
Link: https://lkml.kernel.org/r/20240214204435.167852-8-david@redhat.com
Signed-off-by: David Hildenbrand 
Signed-off-by: Arnd Bergmann 
Reviewed-by: Ryan Roberts 
Cc: Alexander Gordeev 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: "Naveen N. Rao" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Cc: Yin Fengwei 
Signed-off-by: Andrew Morton

mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size()

2024-02-22T23:27:17+00:00

We have two bits available in the encoded page pointer to store additional
information.  Currently, we use one bit to request delay of the rmap
removal until after a TLB flush.

We want to make use of the remaining bit internally for batching of
multiple pages of the same folio, specifying that the next encoded page
pointer in an array is actually "nr_pages".  So pass page + delay_rmap
flag instead of an encoded page, to handle the encoding internally.

Link: https://lkml.kernel.org/r/20240214204435.167852-6-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Ryan Roberts 
Cc: Alexander Gordeev 
Cc: Aneesh Kumar K.V 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Matthew Wilcox (Oracle) 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: "Naveen N. Rao" 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra (Intel) 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Cc: Yin Fengwei 
Signed-off-by: Andrew Morton

mm: add utility functions for ptdesc

2023-08-21T20:37:52+00:00

Introduce utility functions setting the foundation for ptdescs.  These
will also assist in the splitting out of ptdesc from struct page.

Functions that focus on the descriptor are prefixed with ptdesc_* while
functions that focus on the pagetable are prefixed with pagetable_*.

pagetable_alloc() is defined to allocate new ptdesc pages as compound
pages.  This is to standardize ptdescs by allowing for one allocation and
one free function, in contrast to 2 allocation and 2 free functions.

Link: https://lkml.kernel.org/r/20230807230513.102486-4-vishal.moola@gmail.com
Signed-off-by: Vishal Moola (Oracle) 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Claudio Imbrenda 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: "David S. Miller" 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Geert Uytterhoeven 
Cc: Guo Ren 
Cc: Huacai Chen 
Cc: Hugh Dickins 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Mike Rapoport (IBM) 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Weinberger 
Cc: Thomas Bogendoerfer 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

mmu_notifiers: call invalidate_range() when invalidating TLBs

2023-08-18T17:12:41+00:00

The invalidate_range() is going to become an architecture specific mmu
notifier used to keep the TLB of secondary MMUs such as an IOMMU in sync
with the CPU page tables.  Currently it is called from separate code paths
to the main CPU TLB invalidations.  This can lead to a secondary TLB not
getting invalidated when required and makes it hard to reason about when
exactly the secondary TLB is invalidated.

To fix this move the notifier call to the architecture specific TLB
maintenance functions for architectures that have secondary MMUs requiring
explicit software invalidations.

This fixes a SMMU bug on ARM64.  On ARM64 PTE permission upgrades require
a TLB invalidation.  This invalidation is done by the architecture
specific ptep_set_access_flags() which calls flush_tlb_page() if required.
However this doesn't call the notifier resulting in infinite faults being
generated by devices using the SMMU if it has previously cached a
read-only PTE in it's TLB.

Moving the invalidations into the TLB invalidation functions ensures all
invalidations happen at the same time as the CPU invalidation.  The
architecture specific flush_tlb_all() routines do not call the notifier as
none of the IOMMUs require this.

Link: https://lkml.kernel.org/r/0287ae32d91393a582897d6c4db6f7456b1001f2.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Suggested-by: Jason Gunthorpe 
Tested-by: SeongJae Park 
Acked-by: Catalin Marinas 
Reviewed-by: Jason Gunthorpe 
Tested-by: Luis Chamberlain 
Cc: Andrew Donnellan 
Cc: Chaitanya Kumar Borah 
Cc: Frederic Barrat 
Cc: John Hubbard 
Cc: Kevin Tian 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Nicolin Chen 
Cc: Robin Murphy 
Cc: Sean Christopherson 
Cc: Tvrtko Ursulin 
Cc: Will Deacon 
Cc: Zhi Wang 
Signed-off-by: Andrew Morton

mm: delay page_remove_rmap() until after the TLB has been flushed

2022-11-30T23:58:50+00:00

When we remove a page table entry, we are very careful to only free the
page after we have flushed the TLB, because other CPUs could still be
using the page through stale TLB entries until after the flush.

However, we have removed the rmap entry for that page early, which means
that functions like folio_mkclean() would end up not serializing with the
page table lock because the page had already been made invisible to rmap.

And that is a problem, because while the TLB entry exists, we could end up
with the following situation:

 (a) one CPU could come in and clean it, never seeing our mapping of the
     page

 (b) another CPU could continue to use the stale and dirty TLB entry and
     continue to write to said page

resulting in a page that has been dirtied, but then marked clean again,
all while another CPU might have dirtied it some more.

End result: possibly lost dirty data.

This extends our current TLB gather infrastructure to optionally track a
"should I do a delayed page_remove_rmap() for this page after flushing the
TLB".  It uses the newly introduced 'encoded page pointer' to do that
without having to keep separate data around.

Note, this is complicated by a couple of issues:

 - we want to delay the rmap removal, but not past the page table lock,
   because that simplifies the memcg accounting

 - only SMP configurations want to delay TLB flushing, since on UP
   there are obviously no remote TLBs to worry about, and the page
   table lock means there are no preemption issues either

 - s390 has its own mmu_gather model that doesn't delay TLB flushing,
   and as a result also does not want the delayed rmap. As such, we can
   treat S390 like the UP case and use a common fallback for the "no
   delays" case.

 - we can track an enormous number of pages in our mmu_gather structure,
   with MAX_GATHER_BATCH_COUNT batches of MAX_TABLE_BATCH pages each,
   all set up to be approximately 10k pending pages.

   We do not want to have a huge number of batched pages that we then
   need to check for delayed rmap handling inside the page table lock.

Particularly that last point results in a noteworthy detail, where the
normal page batch gathering is limited once we have delayed rmaps pending,
in such a way that only the last batch (the so-called "active batch") in
the mmu_gather structure can have any delayed entries.

NOTE!  While the "possibly lost dirty data" sounds catastrophic, for this
all to happen you need to have a user thread doing either madvise() with
MADV_DONTNEED or a full re-mmap() of the area concurrently with another
thread continuing to use said mapping.

So arguably this is about user space doing crazy things, but from a VM
consistency standpoint it's better if we track the dirty bit properly even
when user space goes off the rails.

[akpm@linux-foundation.org: fix UP build, per Linus]
Link: https://lore.kernel.org/all/B88D3073-440A-41C7-95F4-895D3F657EF2@gmail.com/
Link: https://lkml.kernel.org/r/20221109203051.1835763-4-torvalds@linux-foundation.org
Signed-off-by: Linus Torvalds 
Acked-by: Johannes Weiner 
Acked-by: Hugh Dickins 
Reported-by: Nadav Amit 
Tested-by: Nadav Amit 
Signed-off-by: Andrew Morton

mm: mmu_gather: prepare to gather encoded page pointers with flags

2022-11-30T23:58:50+00:00

This is purely a preparatory patch that makes all the data structures
ready for encoding flags with the mmu_gather page pointers.

The code currently always sets the flag to zero and doesn't use it yet,
but now it's tracking the type state along.  The next step will be to
actually start using it.

Link: https://lkml.kernel.org/r/20221109203051.1835763-3-torvalds@linux-foundation.org
Signed-off-by: Linus Torvalds 
Acked-by: Johannes Weiner 
Acked-by: Hugh Dickins 
Signed-off-by: Andrew Morton

mm/khugepaged: fix GUP-fast interaction by sending IPI

2022-11-30T22:49:42+00:00

Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP
collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to
ensure that the page table was not removed by khugepaged in between.

However, lockless_pages_from_mm() still requires that the page table is
not concurrently freed.  Fix it by sending IPIs (if the architecture uses
semi-RCU-style page table freeing) before freeing/reusing page tables.

Link: https://lkml.kernel.org/r/20221129154730.2274278-2-jannh@google.com
Link: https://lkml.kernel.org/r/20221128180252.1684965-2-jannh@google.com
Link: https://lkml.kernel.org/r/20221125213714.4115729-2-jannh@google.com
Fixes: ba76149f47d8 ("thp: khugepaged")
Signed-off-by: Jann Horn 
Reviewed-by: Yang Shi 
Acked-by: David Hildenbrand 
Cc: John Hubbard 
Cc: Peter Xu 
Cc: 
Signed-off-by: Andrew Morton

mmu_gather: fix the CONFIG_MMU_GATHER_NO_RANGE case

2022-07-22T16:28:34+00:00

Sudip reports that alpha doesn't build properly, with errors like

  include/asm-generic/tlb.h:401:1: error: redefinition of 'tlb_update_vma_flags'
    401 | tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma)
        | ^~~~~~~~~~~~~~~~~~~~
  include/asm-generic/tlb.h:372:1: note: previous definition of 'tlb_update_vma_flags' with type 'void(struct mmu_gather *, struct vm_area_struct *)'
    372 | tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) { }

the cause being that We have this odd situation where some architectures
were never converted to the newer TLB flushing interfaces that have a
range for the flush.  Instead people left them alone, and we have them
select the MMU_GATHER_NO_RANGE config option to make the tlb header
files account for this.

Peter Zijlstra cleaned some of these nasty header file games up in
commits

  1e9fdf21a433 ("mmu_gather: Remove per arch tlb_{start,end}_vma()")
  18ba064e42df ("mmu_gather: Let there be one tlb_{start,end}_vma() implementation")

but tlb_update_vma_flags() was left alone, and then commit b67fbebd4cf9
("mmu_gather: Force tlb-flush VM_PFNMAP vmas") ended up removing only
_one_ of the two stale duplicate dummy inline functions.

This removes the other stale one.

Somebody braver than me should try to remove MMU_GATHER_NO_RANGE
entirely, but it requires fixing up the oddball architectures that use
it: alpha, m68k, microblaze, nios2 and openrisc.

The fixups should be fairly straightforward ("fix the build errors it
exposes by adding the appropriate range arguments"), but the reason this
wasn't done in the first place is that so few people end up working on
those architectures.  But it could be done one architecture at a time,
hint, hint.

Reported-by: Sudip Mukherjee (Codethink) 
Fixes: b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas")
Link: https://lore.kernel.org/all/YtpXh0QHWwaEWVAY@debian/
Cc: Peter Zijlstra 
Cc: Will Deacon 
Cc: "Aneesh Kumar K.V" 
Cc: Andrew Morton 
Cc: Nick Piggin 
Cc: Arnd Bergmann 
Signed-off-by: Linus Torvalds

mmu_gather: Force tlb-flush VM_PFNMAP vmas

2022-07-21T17:50:13+00:00

Jann reported a race between munmap() and unmap_mapping_range(), where
unmap_mapping_range() will no-op once unmap_vmas() has unlinked the
VMA; however munmap() will not yet have invalidated the TLBs.

Therefore unmap_mapping_range() will complete while there are still
(stale) TLB entries for the specified range.

Mitigate this by force flushing TLBs for VM_PFNMAP ranges.

Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Signed-off-by: Linus Torvalds