linux.git/arch/riscv/include/asm/pgalloc.h, branch v6.9

riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU

2024-01-24T23:55:55+00:00

In order to implement fast gup we need to ensure that the page
table walker is protected from page table pages being freed from
under it.

riscv situation is more complicated than other architectures: some
riscv platforms may use IPI to perform TLB shootdown, for example,
those platforms which support AIA, usually the riscv_ipi_for_rfence is
true on these platforms; some riscv platforms may rely on the SBI to
perform TLB shootdown, usually the riscv_ipi_for_rfence is false on
these platforms. To keep software pagetable walkers safe in this case
we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the
comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in
include/asm-generic/tlb.h for more details.

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform
TLB shootdown;

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB
shootdown;

Both case mean that disabling interrupts will block the free and
protect the fast gup page walker.

Signed-off-by: Jisheng Zhang 
Reviewed-by: Alexandre Ghiti 
Link: https://lore.kernel.org/r/20231219175046.2496-4-jszhang@kernel.org
Signed-off-by: Palmer Dabbelt

riscv: tlb: convert __p*d_free_tlb() to inline functions

2024-01-24T23:55:54+00:00

This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE.
No functionality changes.

Signed-off-by: Jisheng Zhang 
Reviewed-by: Alexandre Ghiti 
Link: https://lore.kernel.org/r/20231219175046.2496-3-jszhang@kernel.org
Signed-off-by: Palmer Dabbelt

riscv: tlb: fix __p*d_free_tlb()

2024-01-24T23:55:53+00:00

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is
a must for safe, imagine if an implementation caches the non-leaf
translation in TLB, although I didn't meet this HW so far, but it's
possible in theory.

Signed-off-by: Jisheng Zhang 
Fixes: c5e9b2c2ae82 ("riscv: Improve tlb_flush()")
Link: https://lore.kernel.org/r/20231219175046.2496-2-jszhang@kernel.org
Signed-off-by: Palmer Dabbelt

riscv: convert alloc_{pmd, pte}_late() to use ptdescs

2023-08-21T20:37:57+00:00

As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.

Some of the functions use the *get*page*() helper functions.  Convert
these to use pagetable_alloc() and ptdesc_address() instead to help
standardize page tables further.

Link: https://lkml.kernel.org/r/20230807230513.102486-27-vishal.moola@gmail.com
Signed-off-by: Vishal Moola (Oracle) 
Acked-by: Palmer Dabbelt 
Acked-by: Mike Rapoport (IBM) 
Cc: Arnd Bergmann 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Claudio Imbrenda 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: "David S. Miller" 
Cc: Dinh Nguyen 
Cc: Geert Uytterhoeven 
Cc: Geert Uytterhoeven 
Cc: Guo Ren 
Cc: Huacai Chen 
Cc: Hugh Dickins 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Paul Walmsley 
Cc: Richard Weinberger 
Cc: Thomas Bogendoerfer 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

riscv: Sync efi page table's kernel mappings before switching

2022-11-29T00:36:34+00:00

The EFI page table is initially created as a copy of the kernel page table.
With VMAP_STACK enabled, kernel stacks are allocated in the vmalloc area:
if the stack is allocated in a new PGD (one that was not present at the
moment of the efi page table creation or not synced in a previous vmalloc
fault), the kernel will take a trap when switching to the efi page table
when the vmalloc kernel stack is accessed, resulting in a kernel panic.

Fix that by updating the efi kernel mappings before switching to the efi
page table.

Signed-off-by: Alexandre Ghiti 
Fixes: b91540d52a08 ("RISC-V: Add EFI runtime services")
Tested-by: Emil Renner Berthing 
Reviewed-by: Atish Patra 
Link: https://lore.kernel.org/r/20221121133303.1782246-1-alexghiti@rivosinc.com
Signed-off-by: Palmer Dabbelt

riscv: mm: Control p4d's folding by pgtable_l5_enabled

2022-02-15T00:32:39+00:00

To determine pgtable level at boot time, we can not use helper functions
in include/asm-generic/pgtable-nop4d.h and must implement these
functions. This patch uses pgtable_l5_enabled variable instead of
including pgtable-nop4d.h to controle p4d's folding, and implements
corresponding helper functions.

Signed-off-by: Qinglin Pan 
Signed-off-by: Palmer Dabbelt

riscv: Implement sv48 support

2022-01-20T01:54:09+00:00

By adding a new 4th level of page table, give the possibility to 64bit
kernel to address 2^48 bytes of virtual address: in practice, that offers
128TB of virtual address space to userspace and allows up to 64TB of
physical memory.

If the underlying hardware does not support sv48, we will automatically
fallback to a standard 3-level page table by folding the new PUD level into
PGDIR level. In order to detect HW capabilities at runtime, we
use SATP feature that ignores writes with an unsupported mode.

Signed-off-by: Alexandre Ghiti 
Signed-off-by: Palmer Dabbelt

mm/thp: define default pmd_pgtable()

2021-07-01T18:06:03+00:00

Currently most platforms define pmd_pgtable() as pmd_page() duplicating
the same code all over.  Instead just define a default value i.e
pmd_page() for pmd_pgtable() and let platforms override when required via
.  All the existing platform that override pmd_pgtable()
have been moved into their respective  header in order to
precede before the new generic definition.  This makes it much cleaner
with reduced code.

Link: https://lkml.kernel.org/r/1623646133-20306-1-git-send-email-anshuman.khandual@arm.com
Signed-off-by: Anshuman Khandual 
Acked-by: Geert Uytterhoeven 
Acked-by: Mike Rapoport 
Cc: Nick Hu 
Cc: Richard Henderson 
Cc: Vineet Gupta 
Cc: Catalin Marinas 
Cc: Will Deacon 
Cc: Guo Ren 
Cc: Brian Cain 
Cc: Geert Uytterhoeven 
Cc: Michal Simek 
Cc: Thomas Bogendoerfer 
Cc: Ley Foon Tan 
Cc: Jonas Bonn 
Cc: Stefan Kristiansson 
Cc: Stafford Horne 
Cc: "James E.J. Bottomley" 
Cc: Michael Ellerman 
Cc: Christophe Leroy 
Cc: Paul Walmsley 
Cc: Palmer Dabbelt 
Cc: Heiko Carstens 
Cc: Yoshinori Sato 
Cc: "David S. Miller" 
Cc: Jeff Dike 
Cc: Thomas Gleixner 
Cc: Chris Zankel 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-generic: pgalloc: provide generic pgd_free()

2020-08-07T18:33:26+00:00

Most architectures define pgd_free() as a wrapper for free_page().

Provide a generic version in asm-generic/pgalloc.h and enable its use for
most architectures.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Acked-by: Geert Uytterhoeven 	[m68k]
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra (Intel) 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Matthew Wilcox 
Link: http://lkml.kernel.org/r/20200627143453.31835-7-rppt@kernel.org
Signed-off-by: Linus Torvalds

asm-generic: pgalloc: provide generic pmd_alloc_one() and pmd_free_one()

2020-08-07T18:33:26+00:00

For most architectures that support >2 levels of page tables,
pmd_alloc_one() is a wrapper for __get_free_pages(), sometimes with
__GFP_ZERO and sometimes followed by memset(0) instead.

More elaborate versions on arm64 and x86 account memory for the user page
tables and call to pgtable_pmd_page_ctor() as the part of PMD page
initialization.

Move the arm64 version to include/asm-generic/pgalloc.h and use the
generic version on several architectures.

The pgtable_pmd_page_ctor() is a NOP when ARCH_ENABLE_SPLIT_PMD_PTLOCK is
not enabled, so there is no functional change for most architectures
except of the addition of __GFP_ACCOUNT for allocation of user page
tables.

The pmd_free() is a wrapper for free_page() in all the cases, so no
functional change here.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Cc: Matthew Wilcox 
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra (Intel) 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Geert Uytterhoeven 
Link: http://lkml.kernel.org/r/20200627143453.31835-5-rppt@kernel.org
Signed-off-by: Linus Torvalds