linux-stable.git/mm/vmalloc.c, branch v6.5.2

mm: add a call to flush_cache_vmap() in vmap_pfn()

2023-08-21T20:07:21+00:00

flush_cache_vmap() must be called after new vmalloc mappings are installed
in the page table in order to allow architectures to make sure the new
mapping is visible.

It could lead to a panic since on some architectures (like powerpc),
the page table walker could see the wrong pte value and trigger a
spurious page fault that can not be resolved (see commit f1cb8f9beba8
("powerpc/64s/radix: avoid ptesync after set_pte and
ptep_set_access_flags")).

But actually the patch is aiming at riscv: the riscv specification
allows the caching of invalid entries in the TLB, and since we recently
removed the vmalloc page fault handling, we now need to emit a tlb
shootdown whenever a new vmalloc mapping is emitted
(https://lore.kernel.org/linux-riscv/20230725132246.817726-1-alexghiti@rivosinc.com/).
That's a temporary solution, there are ways to avoid that :)

Link: https://lkml.kernel.org/r/20230809164633.1556126-1-alexghiti@rivosinc.com
Fixes: 3e9a9e256b1e ("mm: add a vmap_pfn function")
Reported-by: Dylan Jhong 
Closes: https://lore.kernel.org/linux-riscv/ZMytNY2J8iyjbPPy@atctrx.andestech.com/
Signed-off-by: Alexandre Ghiti 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Palmer Dabbelt 
Acked-by: Palmer Dabbelt 
Reviewed-by: Dylan Jhong 
Cc: 
Signed-off-by: Andrew Morton

Merge mm-hotfixes-stable into mm-stable to pick up depended-upon changes.

2023-06-23T23:58:19+00:00

mm: ptep_get() conversion

2023-06-19T23:19:25+00:00

Convert all instances of direct pte_t* dereferencing to instead use
ptep_get() helper.  This means that by default, the accesses change from a
C dereference to a READ_ONCE().  This is technically the correct thing to
do since where pgtables are modified by HW (for access/dirty) they are
volatile and therefore we should always ensure READ_ONCE() semantics.

But more importantly, by always using the helper, it can be overridden by
the architecture to fully encapsulate the contents of the pte.  Arch code
is deliberately not converted, as the arch code knows best.  It is
intended that arch code (arm64) will override the default with its own
implementation that can (e.g.) hide certain bits from the core code, or
determine young/dirty status by mixing in state from another source.

Conversion was done using Coccinelle:

----

// $ make coccicheck \
//          COCCI=ptepget.cocci \
//          SPFLAGS="--include-headers" \
//          MODE=patch

virtual patch

@ depends on patch @
pte_t *v;
@@

- *v
+ ptep_get(v)

----

Then reviewed and hand-edited to avoid multiple unnecessary calls to
ptep_get(), instead opting to store the result of a single call in a
variable, where it is correct to do so.  This aims to negate any cost of
READ_ONCE() and will benefit arch-overrides that may be more complex.

Included is a fix for an issue in an earlier version of this patch that
was pointed out by kernel test robot.  The issue arose because config
MMU=n elides definition of the ptep helper functions, including
ptep_get().  HUGETLB_PAGE=n configs still define a simple
huge_ptep_clear_flush() for linking purposes, which dereferences the ptep.
So when both configs are disabled, this caused a build error because
ptep_get() is not defined.  Fix by continuing to do a direct dereference
when MMU=n.  This is safe because for this config the arch code cannot be
trying to virtualize the ptes because none of the ptep helpers are
defined.

Link: https://lkml.kernel.org/r/20230612151545.3317766-4-ryan.roberts@arm.com
Reported-by: kernel test robot 
Link: https://lore.kernel.org/oe-kbuild-all/202305120142.yXsNEo6H-lkp@intel.com/
Signed-off-by: Ryan Roberts 
Cc: Adrian Hunter 
Cc: Alexander Potapenko 
Cc: Alexander Shishkin 
Cc: Alex Williamson 
Cc: Al Viro 
Cc: Andrey Konovalov 
Cc: Andrey Ryabinin 
Cc: Christian Brauner 
Cc: Christoph Hellwig 
Cc: Daniel Vetter 
Cc: Dave Airlie 
Cc: Dimitri Sivanich 
Cc: Dmitry Vyukov 
Cc: Ian Rogers 
Cc: Jason Gunthorpe 
Cc: Jérôme Glisse 
Cc: Jiri Olsa 
Cc: Johannes Weiner 
Cc: Kirill A. Shutemov 
Cc: Lorenzo Stoakes 
Cc: Mark Rutland 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Kravetz 
Cc: Mike Rapoport (IBM) 
Cc: Muchun Song 
Cc: Namhyung Kim 
Cc: Naoya Horiguchi 
Cc: Oleksandr Tyshchenko 
Cc: Pavel Tatashin 
Cc: Roman Gushchin 
Cc: SeongJae Park 
Cc: Shakeel Butt 
Cc: Uladzislau Rezki (Sony) 
Cc: Vincenzo Frascino 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/vmalloc: vmalloc_to_page() use pte_offset_kernel()

2023-06-19T23:19:14+00:00

vmalloc_to_page() was using pte_offset_map() (followed by pte_unmap()),
but it's intended for userspace page tables: prefer pte_offset_kernel().

Link: https://lkml.kernel.org/r/696386a-84f8-b33c-82e5-f865ed6eb39@google.com
Signed-off-by: Hugh Dickins 
Reviewed-by: Lorenzo Stoakes 
Cc: Alistair Popple 
Cc: Anshuman Khandual 
Cc: Axel Rasmussen 
Cc: Christophe Leroy 
Cc: Christoph Hellwig 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Ira Weiny 
Cc: Jason Gunthorpe 
Cc: Kirill A. Shutemov 
Cc: Matthew Wilcox 
Cc: Mel Gorman 
Cc: Miaohe Lin 
Cc: Mike Kravetz 
Cc: Mike Rapoport (IBM) 
Cc: Minchan Kim 
Cc: Naoya Horiguchi 
Cc: Pavel Tatashin 
Cc: Peter Xu 
Cc: Peter Zijlstra 
Cc: Qi Zheng 
Cc: Ralph Campbell 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Song Liu 
Cc: Steven Price 
Cc: Suren Baghdasaryan 
Cc: Thomas Hellström 
Cc: Will Deacon 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Zack Rusin 
Signed-off-by: Andrew Morton

mm/vmalloc: replace the ternary conditional operator with min()

2023-06-19T23:19:11+00:00

It would be better to replace the traditional ternary conditional
operator with min() in zero_iter

Link: https://lkml.kernel.org/r/20230609093057.27777-1-luhongfei@vivo.com
Signed-off-by: Lu Hongfei 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: David Hildenbrand 
Cc: Christoph Hellwig 
Cc: Uladzislau Rezki (Sony) 
Signed-off-by: Andrew Morton

mm/vmalloc: do not output a spurious warning when huge vmalloc() fails

2023-06-19T20:19:31+00:00

In __vmalloc_area_node() we always warn_alloc() when an allocation
performed by vm_area_alloc_pages() fails unless it was due to a pending
fatal signal.

However, huge page allocations instigated either by vmalloc_huge() or
__vmalloc_node_range() (or a caller that invokes this like kvmalloc() or
kvmalloc_node()) always falls back to order-0 allocations if the huge page
allocation fails.

This renders the warning useless and noisy, especially as all callers
appear to be aware that this may fallback.  This has already resulted in
at least one bug report from a user who was confused by this (see link).

Therefore, simply update the code to only output this warning for order-0
pages when no fatal signal is pending.

Link: https://bugzilla.suse.com/show_bug.cgi?id=1211410
Link: https://lkml.kernel.org/r/20230605201107.83298-1-lstoakes@gmail.com
Fixes: 80b1d8fdfad1 ("mm: vmalloc: correct use of __GFP_NOWARN mask in __vmalloc_area_node()")
Signed-off-by: Lorenzo Stoakes 
Acked-by: Vlastimil Babka 
Reviewed-by: Baoquan He 
Acked-by: Michal Hocko 
Reviewed-by: Uladzislau Rezki (Sony) 
Reviewed-by: David Hildenbrand 
Cc: Christoph Hellwig 
Cc: 
Signed-off-by: Andrew Morton

mm: vmalloc must set pte via arch code

2023-06-09T23:25:55+00:00

Patch series "Fixes for pte encapsulation bypasses", v3.

A series to improve the encapsulation of pte entries by disallowing
non-arch code from directly dereferencing pte_t pointers.


This patch (of 4):

It is bad practice to directly set pte entries within a pte table. 
Instead all modifications must go through arch-provided helpers such as
set_pte_at() to give the arch code visibility and allow it to check (and
potentially modify) the operation.

Link: https://lkml.kernel.org/r/20230602092949.545577-1-ryan.roberts@arm.com
Link: https://lkml.kernel.org/r/20230602092949.545577-2-ryan.roberts@arm.com
Fixes: 3e9a9e256b1e ("mm: add a vmap_pfn function")
Signed-off-by: Ryan Roberts 
Reviewed-by: Zi Yan 
Acked-by: Lorenzo Stoakes 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Uladzislau Rezki (Sony) 
Reviewed-by: Mike Rapoport (IBM) 
Cc: Kirill A. Shutemov 
Cc: Matthew Wilcox (Oracle) 
Cc: SeongJae Park 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/vmalloc: dont purge usable blocks unnecessarily

2023-06-09T23:25:41+00:00

Purging fragmented blocks is done unconditionally in several contexts:

  1) From drain_vmap_area_work(), when the number of lazy to be freed
     vmap_areas reached the threshold

  2) Reclaiming vmalloc address space from pcpu_get_vm_areas()

  3) _vm_unmap_aliases()

#1 There is no reason to zap fragmented vmap blocks unconditionally, simply
   because reclaiming all lazy areas drains at least

      32MB * fls(num_online_cpus())

   per invocation which is plenty.

#2 Reclaiming when running out of space or due to memory pressure makes a
   lot of sense

#3 _unmap_aliases() requires to touch everything because the caller has no
   clue which vmap_area used a particular page last and the vmap_area lost
   that information too.

   Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the
   vmap area first and then cares about the flush. That in turn requires
   a full walk of _all_ vmap areas including the one which was just
   added to the purge list.

   But as this has to be flushed anyway this is an opportunity to combine
   outstanding TLB flushes and do the housekeeping of purging freed areas,
   but like #1 there is no real good reason to zap usable vmap blocks
   unconditionally.

Add a @force_purge argument to the newly split out block purge function and
if not true only purge fragmented blocks which have less than 1/4 of their
capacity left.

Rename purge_vmap_area_lazy() to reclaim_and_purge_vmap_areas() to make it
clear what the function does.

[lstoakes@gmail.com: correct VMAP_PURGE_THRESHOLD check]
  Link: https://lkml.kernel.org/r/3e92ef61-b910-4576-88e7-cf43211fd4e7@lucifer.local
Link: https://lkml.kernel.org/r/20230525124504.864005691@linutronix.de
Signed-off-by: Thomas Gleixner 
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Baoquan He 
Cc: Christoph Hellwig 
Cc: Lorenzo Stoakes 
Cc: Uladzislau Rezki (Sony) 
Signed-off-by: Andrew Morton

mm/vmalloc: add missing READ/WRITE_ONCE() annotations

2023-06-09T23:25:41+00:00

purge_fragmented_blocks() accesses vmap_block::free and vmap_block::dirty
lockless for a quick check.

Add the missing READ/WRITE_ONCE() annotations.

Link: https://lkml.kernel.org/r/20230525124504.807356682@linutronix.de
Signed-off-by: Thomas Gleixner 
Reviewed-by: Uladzislau Rezki (Sony) 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Baoquan He 
Cc: Lorenzo Stoakes 
Signed-off-by: Andrew Morton

mm/vmalloc: check free space in vmap_block lockless

2023-06-09T23:25:41+00:00

vb_alloc() unconditionally locks a vmap_block on the free list to check
the free space.

This can be done locklessly because vmap_block::free never increases, it's
only decreased on allocations.

Check the free space lockless and only if that succeeds, recheck under the
lock.

Link: https://lkml.kernel.org/r/20230525124504.750481992@linutronix.de
Signed-off-by: Thomas Gleixner 
Reviewed-by: Uladzislau Rezki (Sony) 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: Baoquan He 
Signed-off-by: Andrew Morton