linux-stable.git/include/linux/huge_mm.h, branch v6.15.2

mm/truncate: use folio_split() in truncate operation

2025-03-18T05:07:00+00:00

Instead of splitting the large folio uniformly during truncation, try to
use buddy allocator like folio_split() at the start and the end of a
truncation range to minimize the number of resulting folios if it is
supported.  try_folio_split() is introduced to use folio_split() if
supported and it falls back to uniform split otherwise.

For example, to truncate a order-4 folio
[0, 1, 2, 3, 4, 5, ..., 15]
between [3, 10] (inclusive), folio_split() splits the folio at 3 to
[0,1], [2], [3], [4..7], [8..15] and [3], [4..7] can be dropped and
[8..15] is kept with zeros in [8..10], then another folio_split() is
done at 10, so [8..10] can be dropped.

One possible optimization is to make folio_split() to split a folio based
on a given range, like [3..10] above.  But that complicates folio_split(),
so it will be investigated when necessary.

Link: https://lkml.kernel.org/r/20250226210032.2044041-8-ziy@nvidia.com
Link: https://lkml.kernel.org/r/20250307174001.242794-8-ziy@nvidia.com
Signed-off-by: Zi Yan 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: Kefeng Wang 
Cc: Kirill A. Shuemov 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Kairui Song 
Signed-off-by: Andrew Morton

mm/huge_memory: add vmf_insert_folio_pmd()

2025-03-18T05:06:40+00:00

Currently DAX folio/page reference counts are managed differently to
normal pages.  To allow these to be managed the same as normal pages
introduce vmf_insert_folio_pmd.  This will map the entire PMD-sized folio
and take references as it would for a normally mapped page.

This is distinct from the current mechanism, vmf_insert_pfn_pmd, which
simply inserts a special devmap PMD entry into the page table without
holding a reference to the page for the mapping.

It is not currently useful to implement a more generic vmf_insert_folio()
which selects the correct behaviour based on folio_order().  This is
because PTE faults require only a subpage of the folio to be PTE mapped
rather than the entire folio.  It would be possible to add this context
somewhere but callers already need to handle PTE faults and PMD faults
separately so a more generic function is not useful.

Link: https://lkml.kernel.org/r/7bf92a2e68225d13ea368d53bbfee327314d1c40.1740713401.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Acked-by: David Hildenbrand 
Tested-by: Alison Schofield 
Cc: Alexander Gordeev 
Cc: Asahi Lina 
Cc: Balbir Singh 
Cc: Bjorn Helgaas 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christoph Hellwig 
Cc: Chunyan Zhang 
Cc: Dan Wiliams 
Cc: "Darrick J. Wong" 
Cc: Dave Chinner 
Cc: Dave Hansen 
Cc: Dave Jiang 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ira Weiny 
Cc: Jan Kara 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: linmiaohe 
Cc: Logan Gunthorpe 
Cc: Matthew Wilcow (Oracle) 
Cc: Michael "Camp Drill Sergeant" Ellerman 
Cc: Nicholas Piggin 
Cc: Peter Xu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: Vishal Verma 
Cc: Vivek Goyal 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm/huge_memory: add vmf_insert_folio_pud()

2025-03-18T05:06:40+00:00

Currently DAX folio/page reference counts are managed differently to
normal pages.  To allow these to be managed the same as normal pages
introduce vmf_insert_folio_pud.  This will map the entire PUD-sized folio
and take references as it would for a normally mapped page.

This is distinct from the current mechanism, vmf_insert_pfn_pud, which
simply inserts a special devmap PUD entry into the page table without
holding a reference to the page for the mapping.

Link: https://lkml.kernel.org/r/649a1ef91d556593948351e94f51ef73a14f6794.1740713401.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Reviewed-by: Dan Williams 
Acked-by: David Hildenbrand 
Tested-by: Alison Schofield 
Cc: Alexander Gordeev 
Cc: Asahi Lina 
Cc: Balbir Singh 
Cc: Bjorn Helgaas 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christoph Hellwig 
Cc: Chunyan Zhang 
Cc: "Darrick J. Wong" 
Cc: Dave Chinner 
Cc: Dave Hansen 
Cc: Dave Jiang 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ira Weiny 
Cc: Jan Kara 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: linmiaohe 
Cc: Logan Gunthorpe 
Cc: Matthew Wilcow (Oracle) 
Cc: Michael "Camp Drill Sergeant" Ellerman 
Cc: Nicholas Piggin 
Cc: Peter Xu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: Vishal Verma 
Cc: Vivek Goyal 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: completely abstract unnecessary adj_start calculation

2025-03-17T05:06:02+00:00

The adj_start calculation has been a constant source of confusion in the
VMA merge code.

There are two cases to consider, one where we adjust the start of the
vmg->middle VMA (i.e.  the vmg->__adjust_middle_start merge flag is set),
in which case adj_start is calculated as:

(1) adj_start = vmg->end - vmg->middle->vm_start

And the case where we adjust the start of the vmg->next VMA (i.e.  the
vmg->__adjust_next_start merge flag is set), in which case adj_start is
calculated as:

(2) adj_start = -(vmg->middle->vm_end - vmg->end)

We apply (1) thusly:

vmg->middle->vm_start =
	vmg->middle->vm_start + vmg->end - vmg->middle->vm_start

Which simplifies to:

vmg->middle->vm_start = vmg->end

Similarly, we apply (2) as:

vmg->next->vm_start =
	vmg->next->vm_start + -(vmg->middle->vm_end - vmg->end)

Noting that for these VMAs to be mergeable vmg->middle->vm_end ==
vmg->next->vm_start and so this simplifies to:

vmg->next->vm_start =
	vmg->next->vm_start + -(vmg->next->vm_start - vmg->end)

Which simplifies to:

vmg->next->vm_start = vmg->end

Therefore in each case, we simply need to adjust the start of the VMA to
vmg->end (!) and can do away with this adj_start calculation.  The only
caveat is that we must ensure we update the vm_pgoff field correctly.

We therefore abstract this entire calculation to a new function
vmg_adjust_set_range() which performs this calculation and sets the
adjusted VMA's new range using the general vma_set_range() function.

We also must update vma_adjust_trans_huge() which expects the
now-abstracted adj_start parameter.  It turns out this is wholly
unnecessary.

In vma_adjust_trans_huge() the relevant code is:

	if (adjust_next > 0) {
		struct vm_area_struct *next = find_vma(vma->vm_mm, vma->vm_end);
		unsigned long nstart = next->vm_start;
		nstart += adjust_next;
		split_huge_pmd_if_needed(next, nstart);
	}

The only case where this is relevant is when vmg->__adjust_middle_start is
specified (in which case adj_next would have been positive), i.e.  the one
in which the vma specified is vmg->prev and this the sought 'next' VMA
would be vmg->middle.

We can therefore eliminate the find_vma() invocation altogether and simply
provide the vmg->middle VMA in this instance, or NULL otherwise.

Again we have an adj_next offset calculation:

next->vm_start + vmg->end - vmg->middle->vm_start

Where next == vmg->middle this simplifies to vmg->end as previously
demonstrated.

Therefore nstart is equal to vmg->end, which is already passed to
vma_adjust_trans_huge() via the 'end' parameter and so this code (rather
delightfully) simplifies to:

	if (next)
		split_huge_pmd_if_needed(next, end);

With these changes in place, it becomes silly for commit_merge() to return
vmg->target, as it is always the same and threaded through vmg, so we
finally change commit_merge() to return an error value once again.

This patch has no change in functional behaviour.

Link: https://lkml.kernel.org/r/7bce2cd4b5afb56211822835d145471280c3dccc.1738326519.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Vlastimil Babka 
Cc: Jann Horn 
Cc: Liam Howlett 
Signed-off-by: Andrew Morton

mm: add per-order mTHP swap-in fallback/fallback_charge counters

2025-01-14T06:40:49+00:00

Currently, large folio swap-in is supported, but we lack a method to
analyze their success ratio.  Similar to anon_fault_fallback, we introduce
per-order mTHP swpin_fallback and swpin_fallback_charge counters for
calculating their success ratio.  The new counters are located at:

/sys/kernel/mm/transparent_hugepage/hugepages-/stats/
	swpin_fallback
	swpin_fallback_charge

Link: https://lkml.kernel.org/r/20241202124730.2407037-1-haowenchao22@gmail.com
Signed-off-by: Wenchao Hao 
Reviewed-by: Barry Song 
Reviewed-by: Lance Yang 
Cc: Baolin Wang 
Cc: David Hildenbrand 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox 
Cc: Peter Xu 
Cc: Ryan Roberts 
Cc: Usama Arif 
Signed-off-by: Andrew Morton

mm: add per-order mTHP swpin counters

2024-11-11T08:26:43+00:00

This helps profile the sizes of folios being swapped in. Currently,
only mTHP swap-out is being counted.
The new interface can be found at:
/sys/kernel/mm/transparent_hugepage/hugepages-/stats
         swpin
For example,
cat /sys/kernel/mm/transparent_hugepage/hugepages-64kB/stats/swpin
12809
cat /sys/kernel/mm/transparent_hugepage/hugepages-32kB/stats/swpin
4763

[v-songbaohua@oppo.com: add a blank line in doc]
  Link: https://lkml.kernel.org/r/20241030233423.80759-1-21cnbao@gmail.com
Link: https://lkml.kernel.org/r/20241026082423.26298-1-21cnbao@gmail.com
Signed-off-by: Barry Song 
Reviewed-by: Baolin Wang 
Acked-by: David Hildenbrand 
Cc: Chris Li 
Cc: Yosry Ahmed 
Cc: "Huang, Ying" 
Cc: Kairui Song 
Cc: Ryan Roberts 
Cc: Kanchana P Sridhar 
Cc: Usama Arif 
Signed-off-by: Andrew Morton

mm: swap: count successful large folio zswap stores in hugepage zswpout stats

2024-11-11T08:26:43+00:00

Added a new MTHP_STAT_ZSWPOUT entry to the sysfs transparent_hugepage
stats so that successful large folio zswap stores can be accounted under
the per-order sysfs "zswpout" stats:

/sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpout

Other non-zswap swap device swap-out events will be counted under
the existing sysfs "swpout" stats:

/sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/swpout

Also, added documentation for the newly added sysfs per-order hugepage
"zswpout" stats. The documentation clarifies that only non-zswap swapouts
will be accounted in the existing "swpout" stats.

Link: https://lkml.kernel.org/r/20241001053222.6944-8-kanchana.p.sridhar@intel.com
Signed-off-by: Kanchana P Sridhar 
Reviewed-by: Nhat Pham 
Cc: Chengming Zhou 
Cc: "Huang, Ying" 
Cc: Johannes Weiner 
Cc: Matthew Wilcox 
Cc: Ryan Roberts 
Cc: Shakeel Butt 
Cc: Usama Arif 
Cc: Wajdi Feghali 
Cc: Yosry Ahmed 
Cc: "Zou, Nanhai" 
Cc: Barry Song <21cnbao@gmail.com>
Signed-off-by: Andrew Morton

mm: huge_memory: move file_thp_enabled() into huge_memory.c

2024-11-07T04:11:15+00:00

file_thp_enabled() is only used in __thp_vma_allowable_orders(), so move
it into huge_memory.c, also check READ_ONLY_THP_FOR_FS ahead to avoid
unnecessary code if config disabled.

Link: https://lkml.kernel.org/r/20241017141457.1169092-1-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Acked-by: David Hildenbrand 
Reviewed-by: Baolin Wang 
Cc: Barry Song 
Cc: Hugh Dickins 
Cc: Matthew Wilcox 
Cc: Ryan Roberts 
Signed-off-by: Andrew Morton

mm: move mm flags to mm_types.h

2024-11-06T00:56:26+00:00

The types of mm flags are now far beyond the core dump related features. 
This patch moves mm flags from linux/sched/coredump.h to linux/mm_types.h.
The linux/sched/coredump.h has include the mm_types.h, so the C files
related to coredump does not need to change head file inclusion.  In
addition, the inclusion of sched/coredump.h now can be deleted from the C
files that irrelevant to core dump.

Link: https://lkml.kernel.org/r/20240926074922.2721274-1-sunnanyong@huawei.com
Signed-off-by: Nanyong Sun 
Cc: Kefeng Wang 
Cc: Masami Hiramatsu 
Cc: Matthew Wilcox 
Cc: Oleg Nesterov 
Cc: Peter Zijlstra 
Signed-off-by: Andrew Morton

mm: huge_memory: add vma_thp_disabled() and thp_disabled_by_hw()

2024-10-17T07:28:10+00:00

Patch series "mm: don't install PMD mappings when THPs are disabled by the
hw/process/vma".

During testing, it was found that we can get PMD mappings in processes
where THP (and more precisely, PMD mappings) are supposed to be disabled. 
While it works as expected for anon+shmem, the pagecache is the
problematic bit.

For s390 KVM this currently means that a VM backed by a file located on
filesystem with large folio support can crash when KVM tries accessing the
problematic page, because the readahead logic might decide to use a
PMD-sized THP and faulting it into the page tables will install a PMD
mapping, something that s390 KVM cannot tolerate.

This might also be a problem with HW that does not support PMD mappings,
but I did not try reproducing it.

Fix it by respecting the ways to disable THPs when deciding whether we can
install a PMD mapping.  khugepaged should already be taking care of not
collapsing if THPs are effectively disabled for the hw/process/vma.


This patch (of 2):

Add vma_thp_disabled() and thp_disabled_by_hw() helpers to be shared by
shmem_allowable_huge_orders() and __thp_vma_allowable_orders().

[david@redhat.com: rename to vma_thp_disabled(), split out thp_disabled_by_hw() ]
Link: https://lkml.kernel.org/r/20241011102445.934409-2-david@redhat.com
Fixes: 793917d997df ("mm/readahead: Add large folio readahead")
Signed-off-by: Kefeng Wang 
Signed-off-by: David Hildenbrand 
Reported-by: Leo Fu 
Tested-by: Thomas Huth 
Reviewed-by: Ryan Roberts 
Cc: Boqiao Fu 
Cc: Christian Borntraeger 
Cc: Claudio Imbrenda 
Cc: Hugh Dickins 
Cc: Janosch Frank 
Cc: Matthew Wilcox 
Cc: 
Signed-off-by: Andrew Morton