linux-stable.git/include/linux/huge_mm.h, branch v6.8.3

mm: remove VM_EXEC requirement for THP eligibility

2023-12-29T19:58:42+00:00

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking
in transparent_hugepage_enabled()") introduced the VM_EXEC requirement,
which is not strictly needed.

lld's default --rosegment option and GNU ld's -z separate-code option
(default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD
segment without the PF_X flag, which should be eligible for THP.

Certain architectures support medium and large code models, where .lrodata
may be placed in a separate read-only PT_LOAD segment, which should be
eligible for THP as well.

Link: https://lkml.kernel.org/r/20231220054123.1266001-1-maskray@google.com
Signed-off-by: Fangrui Song 
Acked-by: Yang Shi 
Cc: Miaohe Lin 
Cc: Song Liu 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton

mm: thp: support allocation of anonymous multi-size THP

2023-12-20T22:48:12+00:00

Introduce the logic to allow THP to be configured (through the new sysfs
interface we just added) to allocate large folios to back anonymous
memory, which are larger than the base page size but smaller than
PMD-size.  We call this new THP extension "multi-size THP" (mTHP).

mTHP continues to be PTE-mapped, but in many cases can still provide
similar benefits to traditional PMD-sized THP: Page faults are
significantly reduced (by a factor of e.g.  4, 8, 16, etc.  depending on
the configured order), but latency spikes are much less prominent because
the size of each page isn't as huge as the PMD-sized variant and there is
less memory to clear in each page fault.  The number of per-page
operations (e.g.  ref counting, rmap management, lru list management) are
also significantly reduced since those ops now become per-folio.

Some architectures also employ TLB compression mechanisms to squeeze more
entries in when a set of PTEs are virtually and physically contiguous and
approporiately aligned.  In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by
writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see
documentation in previous commit).  The long term aim is to change the
default to include suitable lower orders, but there are some risks around
internal fragmentation that need to be better understood first.

[ryan.roberts@arm.com: resolve some multi-size THP review nits]
  Link: https://lkml.kernel.org/r/20231214160251.3574571-1-ryan.roberts@arm.com
Link: https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts 
Tested-by: Kefeng Wang 
Tested-by: John Hubbard 
Acked-by: David Hildenbrand 
Cc: Alistair Popple 
Cc: Anshuman Khandual 
Cc: Barry Song 
Cc: Catalin Marinas 
Cc: David Rientjes 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Itaru Kitayama 
Cc: Kirill A. Shutemov 
Cc: Luis Chamberlain 
Cc: Matthew Wilcox (Oracle) 
Cc: Vlastimil Babka 
Cc: Yang Shi 
Cc: Yin Fengwei 
Cc: Yu Zhao 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: thp: introduce multi-size THP sysfs interface

2023-12-20T22:48:12+00:00

In preparation for adding support for anonymous multi-size THP, introduce
new sysfs structure that will be used to control the new behaviours.  A
new directory is added under transparent_hugepage for each supported THP
size, and contains an `enabled` file, which can be set to "inherit" (to
inherit the global setting), "always", "madvise" or "never".  For now, the
kernel still only supports PMD-sized anonymous THP, so only 1 directory is
populated.

The first half of the change converts transhuge_vma_suitable() and
hugepage_vma_check() so that they take a bitfield of orders for which the
user wants to determine support, and the functions filter out all the
orders that can't be supported, given the current sysfs configuration and
the VMA dimensions.  The resulting functions are renamed to
thp_vma_suitable_orders() and thp_vma_allowable_orders() respectively. 
Convenience functions that take a single, unencoded order and return a
boolean are also defined as thp_vma_suitable_order() and
thp_vma_allowable_order().

The second half of the change implements the new sysfs interface.  It has
been done so that each supported THP size has a `struct thpsize`, which
describes the relevant metadata and is itself a kobject.  This is pretty
minimal for now, but should make it easy to add new per-thpsize files to
the interface if needed in future (e.g.  per-size defrag).  Rather than
keep the `enabled` state directly in the struct thpsize, I've elected to
directly encode it into huge_anon_orders_[always|madvise|inherit]
bitfields since this reduces the amount of work required in
thp_vma_allowable_orders() which is called for every page fault.

See Documentation/admin-guide/mm/transhuge.rst, as modified by this
commit, for details of how the new sysfs interface works.

[ryan.roberts@arm.com: fix build warning when CONFIG_SYSFS is disabled]
  Link: https://lkml.kernel.org/r/20231211125320.3997543-1-ryan.roberts@arm.com
Link: https://lkml.kernel.org/r/20231207161211.2374093-4-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts 
Reviewed-by: Barry Song 
Tested-by: Kefeng Wang 
Tested-by: John Hubbard 
Acked-by: David Hildenbrand 
Cc: Alistair Popple 
Cc: Anshuman Khandual 
Cc: Catalin Marinas 
Cc: David Rientjes 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Itaru Kitayama 
Cc: Kirill A. Shutemov 
Cc: Luis Chamberlain 
Cc: Matthew Wilcox (Oracle) 
Cc: Vlastimil Babka 
Cc: Yang Shi 
Cc: Yin Fengwei 
Cc: Yu Zhao 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: convert prep_transhuge_page() to folio_prep_large_rmappable()

2023-08-21T21:28:43+00:00

Match folio_undo_large_rmappable(), and move the casting from page to
folio into the callers (which they were largely doing anyway).

Link: https://lkml.kernel.org/r/20230816151201.3655946-6-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm: convert free_transhuge_folio() to folio_undo_large_rmappable()

2023-08-21T21:28:43+00:00

Indirect calls are expensive, thanks to Spectre.  Test for
TRANSHUGE_PAGE_DTOR and destroy the folio appropriately.  Move the
free_compound_page() call into destroy_large_folio() to simplify later
patches.

Link: https://lkml.kernel.org/r/20230816151201.3655946-5-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

smaps: use vm_normal_page_pmd() instead of follow_trans_huge_pmd()

2023-08-21T20:07:20+00:00

We shouldn't be using a GUP-internal helper if it can be avoided.

Similar to smaps_pte_entry() that uses vm_normal_page(), let's use
vm_normal_page_pmd() that similarly refuses to return the huge zeropage.

In contrast to follow_trans_huge_pmd(), vm_normal_page_pmd():

(1) Will always return the head page, not a tail page of a THP.

 If we'd ever call smaps_account with a tail page while setting "compound
 = true", we could be in trouble, because smaps_account() would look at
 the memmap of unrelated pages.

 If we're unlucky, that memmap does not exist at all. Before we removed
 PG_doublemap, we could have triggered something similar as in
 commit 24d7275ce279 ("fs/proc: task_mmu.c: don't read mapcount for
 migration entry").

 This can theoretically happen ever since commit ff9f47f6f00c ("mm: proc:
 smaps_rollup: do not stall write attempts on mmap_lock"):

  (a) We're in show_smaps_rollup() and processed a VMA
  (b) We release the mmap lock in show_smaps_rollup() because it is
      contended
  (c) We merged that VMA with another VMA
  (d) We collapsed a THP in that merged VMA at that position

 If the end address of the original VMA falls into the middle of a THP
 area, we would call smap_gather_stats() with a start address that falls
 into a PMD-mapped THP. It's probably very rare to trigger when not
 really forced.

(2) Will succeed on a is_pci_p2pdma_page(), like vm_normal_page()

 Treat such PMDs here just like smaps_pte_entry() would treat such PTEs.
 If such pages would be anonymous, we most certainly would want to
 account them.

(3) Will skip over pmd_devmap(), like vm_normal_page() for pte_devmap()

 As noted in vm_normal_page(), that is only for handling legacy ZONE_DEVICE
 pages. So just like smaps_pte_entry(), we'll now also ignore such PMD
 entries.

 Especially, follow_pmd_mask() never ends up calling
 follow_trans_huge_pmd() on pmd_devmap(). Instead it calls
 follow_devmap_pmd() -- which will fail if neither FOLL_GET nor FOLL_PIN
 is set.

 So skipping pmd_devmap() pages seems to be the right thing to do.

(4) Will properly handle VM_MIXEDMAP/VM_PFNMAP, like vm_normal_page()

 We won't be returning a memmap that should be ignored by core-mm, or
 worse, a memmap that does not even exist. Note that while
 walk_page_range() will skip VM_PFNMAP mappings, walk_page_vma() won't.

 Most probably this case doesn't currently really happen on the PMD level,
 otherwise we'd already be able to trigger kernel crashes when reading
 smaps / smaps_rollup.

So most probably only (1) is relevant in practice as of now, but could only
cause trouble in extreme corner cases.

Let's move follow_trans_huge_pmd() to mm/internal.h to discourage future
reuse in wrong context.

Link: https://lkml.kernel.org/r/20230803143208.383663-3-david@redhat.com
Fixes: ff9f47f6f00c ("mm: proc: smaps_rollup: do not stall write attempts on mmap_lock")
Signed-off-by: David Hildenbrand 
Acked-by: Mel Gorman 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Linus Torvalds 
Cc: liubo 
Cc: Matthew Wilcox (Oracle) 
Cc: Mel Gorman 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Shuah Khan 
Signed-off-by: Andrew Morton

mm: remove vmf_insert_pfn_xxx_prot() for huge page-table entries

2023-04-06T02:42:56+00:00

This functionality's sole user, the drm ttm module, removed support for it
in commit 0d979509539e ("drm/ttm: remove ttm_bo_vm_insert_huge()") as the
whole approach is currently unworkable without a PMD/PUD special bit and
updates to GUP.

Link: https://lkml.kernel.org/r/604c2ad79659d4b8a6e3e1611c6219d5d3233988.1678661628.git.lstoakes@gmail.com
Signed-off-by: Lorenzo Stoakes 
Cc: Christian König 
Cc: Dan Williams 
Cc: Jason Gunthorpe 
Cc: Kirill A. Shutemov 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Thomas Hellström 
Cc: Aaron Tomlin 
Cc: Christoph Lameter 
Cc: Frederic Weisbecker 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Marcelo Tosatti 
Cc: Peter Xu 
Cc: "Russell King (Oracle)" 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/thp: rename TRANSPARENT_HUGEPAGE_NEVER_DAX to _UNSUPPORTED

2023-03-28T23:20:18+00:00

TRANSPARENT_HUGEPAGE_NEVER_DAX has nothing to do with DAX.  It's set when
has_transparent_hugepage() returns false, checked in hugepage_vma_check()
and will disable THP completely if false.  Rename it to
TRANSPARENT_HUGEPAGE_UNSUPPORTED to reflect its real purpose.

[peterx@redhat.com: fix comment, per David]
  Link: https://lkml.kernel.org/r/ZBMzQW674oHQJV7F@x1n
Link: https://lkml.kernel.org/r/20230315171642.1244625-1-peterx@redhat.com
Signed-off-by: Peter Xu 
Acked-by: David Hildenbrand 
Cc: Aneesh Kumar K.V 
Cc: Yang Shi 
Signed-off-by: Andrew Morton

mm: convert deferred_split_huge_page() to deferred_split_folio()

2023-02-03T06:33:00+00:00

Now that both callers use a folio, pass the folio in and save a call to
compound_head().

Link: https://lkml.kernel.org/r/20230111142915.1001531-28-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton

mm/huge_memory: remove page_deferred_list()

2023-02-03T06:33:00+00:00

Use folio->_deferred_list directly.

Link: https://lkml.kernel.org/r/20230111142915.1001531-26-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton