linux.git/mm/memory.c, branch v6.4-rc2

mm: do not increment pgfault stats when page fault handler retries

2023-04-21T21:52:04+00:00

If the page fault handler requests a retry, we will count the fault
multiple times.  This is a relatively harmless problem as the retry paths
are not often requested, and the only user-visible problem is that the
fault counter will be slightly higher than it should be.  Nevertheless,
userspace only took one fault, and should not see the fact that the kernel
had to retry the fault multiple times.

Move page fault accounting into mm_account_fault() and skip incomplete
faults which will be accounted upon completion.

Link: https://lkml.kernel.org/r/20230419175836.3857458-1-surenb@google.com
Fixes: d065bd810b6d ("mm: retry page fault when blocking on disk transfer")
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Matthew Wilcox (Oracle) 
Acked-by: Peter Xu 
Cc: Davidlohr Bueso 
Cc: Jan Kara 
Cc: Johannes Weiner 
Cc: Josef Bacik 
Cc: Laurent Dufour 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Michel Lespinasse 
Cc: Minchan Kim 
Cc: Punit Agrawal 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: hwpoison: support recovery from HugePage copy-on-write faults

2023-04-18T23:30:09+00:00

copy-on-write of hugetlb user pages with uncorrectable errors will result
in a kernel crash.  This is because the copy is performed in kernel mode
and in general we can not handle accessing memory with such errors while
in kernel mode.  Commit a873dfe1032a ("mm, hwpoison: try to recover from
copy-on write faults") introduced the routine copy_user_highpage_mc() to
gracefully handle copying of user pages with uncorrectable errors. 
However, the separate hugetlb copy-on-write code paths were not modified
as part of commit a873dfe1032a.

Modify hugetlb copy-on-write code paths to use copy_mc_user_highpage() so
that they can also gracefully handle uncorrectable errors in user pages. 
This involves changing the hugetlb specific routine
copy_user_large_folio() from type void to int so that it can return an
error.  Modify the hugetlb userfaultfd code in the same way so that it can
return -EHWPOISON if it encounters an uncorrectable error.

Link: https://lkml.kernel.org/r/20230413131349.2524210-1-liushixin2@huawei.com
Signed-off-by: Liu Shixin 
Acked-by: Mike Kravetz 
Reviewed-by: Naoya Horiguchi 
Cc: Miaohe Lin 
Cc: Muchun Song 
Cc: Tony Luck 
Signed-off-by: Andrew Morton

mm: convert copy_user_huge_page() to copy_user_large_folio()

2023-04-18T23:29:55+00:00

Replace copy_user_huge_page() with copy_user_large_folio(). 
copy_user_large_folio() does the same as copy_user_huge_page(), but takes
in folios instead of pages.  Remove pages_per_huge_page from
copy_user_large_folio(), because we can get that from folio_nr_pages(dst).

Convert copy_user_gigantic_page() to take in folios.

Link: https://lkml.kernel.org/r/20230410133932.32288-6-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Sidhartha Kumar 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

userfaultfd: convert copy_huge_page_from_user() to copy_folio_from_user()

2023-04-18T23:29:55+00:00

Replace copy_huge_page_from_user() with copy_folio_from_user(). 
copy_folio_from_user() does the same as copy_huge_page_from_user(), but
takes in a folio instead of a page.

Convert page_kaddr to kaddr in copy_folio_from_user() to do indenting
cleanup.

Link: https://lkml.kernel.org/r/20230410133932.32288-4-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Reviewed-by: Sidhartha Kumar 
Reviewed-by: Mike Kravetz 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

userfaultfd: use kmap_local_page() in copy_huge_page_from_user()

2023-04-18T23:29:54+00:00

kmap() and kmap_atomic() are being deprecated in favor of
kmap_local_page() which is appropriate for any thread local context.[1]

Let's replace the kmap() and kmap_atomic() with kmap_local_page() in
copy_huge_page_from_user().  When allow_pagefault is false, disable page
faults to prevent potential deadlock.[2]

[1] https://lore.kernel.org/all/20220813220034.806698-1-ira.weiny@intel.com/
[2] https://lkml.kernel.org/r/20221025220136.2366143-1-ira.weiny@intel.com

Link: https://lkml.kernel.org/r/20230410133932.32288-3-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Reviewed-by: Mike Kravetz 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Sidhartha Kumar 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

sync mm-stable with mm-hotfixes-stable to pick up depended-upon upstream changes

2023-04-16T19:31:58+00:00

sched/numa: enhance vma scanning logic

2023-04-06T03:03:03+00:00

During Numa scanning make sure only relevant vmas of the tasks are
scanned.

Before:
 All the tasks of a process participate in scanning the vma even if they
 do not access vma in it's lifespan.

Now:
 Except cases of first few unconditional scans, if a process do
 not touch vma (exluding false positive cases of PID collisions)
 tasks no longer scan all vma

Logic used:

1) 6 bits of PID used to mark active bit in vma numab status during
   fault to remember PIDs accessing vma.  (Thanks Mel)

2) Subsequently in scan path, vma scanning is skipped if current PID
   had not accessed vma.

3) First two times we do allow unconditional scan to preserve earlier
   behaviour of scanning.

Acknowledgement to Bharata B Rao  for initial patch to
store pid information and Peter Zijlstra  (Usage of
test and set bit)

Link: https://lkml.kernel.org/r/092f03105c7c1d3450f4636b1ea350407f07640e.1677672277.git.raghavendra.kt@amd.com
Signed-off-by: Raghavendra K T 
Suggested-by: Mel Gorman 
Cc: David Hildenbrand 
Cc: Disha Talreja 
Cc: Ingo Molnar 
Cc: Mike Rapoport 
Signed-off-by: Andrew Morton

mm: introduce per-VMA lock statistics

2023-04-06T03:03:01+00:00

Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics
about handling page fault under VMA lock.

Link: https://lkml.kernel.org/r/20230227173632.3292573-29-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Signed-off-by: Andrew Morton

mm: prevent userfaults to be handled under per-vma lock

2023-04-06T03:03:01+00:00

Due to the possibility of handle_userfault dropping mmap_lock, avoid fault
handling under VMA lock and retry holding mmap_lock.  This can be handled
more gracefully in the future.

Link: https://lkml.kernel.org/r/20230227173632.3292573-28-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Suggested-by: Peter Xu 
Signed-off-by: Andrew Morton

mm: prevent do_swap_page from handling page faults under VMA lock

2023-04-06T03:03:00+00:00

Due to the possibility of do_swap_page dropping mmap_lock, abort fault
handling under VMA lock and retry holding mmap_lock.  This can be handled
more gracefully in the future.

Link: https://lkml.kernel.org/r/20230227173632.3292573-27-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Laurent Dufour 
Signed-off-by: Andrew Morton