linux-stable.git/mm/memory.c, branch v6.4.3

mm: call arch_swap_restore() from do_swap_page()

2023-07-11T04:31:04+00:00

commit 6dca4ac6fc91fd41ea4d6c4511838d37f4e0eab2 upstream.

Commit c145e0b47c77 ("mm: streamline COW logic in do_swap_page()") moved
the call to swap_free() before the call to set_pte_at(), which meant that
the MTE tags could end up being freed before set_pte_at() had a chance to
restore them.  Fix it by adding a call to the arch_swap_restore() hook
before the call to swap_free().

Link: https://lkml.kernel.org/r/20230523004312.1807357-2-pcc@google.com
Link: https://linux-review.googlesource.com/id/I6470efa669e8bd2f841049b8c61020c510678965
Fixes: c145e0b47c77 ("mm: streamline COW logic in do_swap_page()")
Signed-off-by: Peter Collingbourne 
Reported-by: Qun-wei Lin 
Closes: https://lore.kernel.org/all/5050805753ac469e8d727c797c2218a9d780d434.camel@mediatek.com/
Acked-by: David Hildenbrand 
Acked-by: "Huang, Ying" 
Reviewed-by: Steven Price 
Acked-by: Catalin Marinas 
Cc: 	[6.1+]
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman

mm: always expand the stack with the mmap write lock held

2023-07-01T11:12:40+00:00

commit 8d7071af890768438c14db6172cc8f9f4d04e184 upstream.

This finishes the job of always holding the mmap write lock when
extending the user stack vma, and removes the 'write_locked' argument
from the vm helper functions again.

For some cases, we just avoid expanding the stack at all: drivers and
page pinning really shouldn't be extending any stacks.  Let's see if any
strange users really wanted that.

It's worth noting that architectures that weren't converted to the new
lock_mm_and_find_vma() helper function are left using the legacy
"expand_stack()" function, but it has been changed to drop the mmap_lock
and take it for writing while expanding the vma.  This makes it fairly
straightforward to convert the remaining architectures.

As a result of dropping and re-taking the lock, the calling conventions
for this function have also changed, since the old vma may no longer be
valid.  So it will now return the new vma if successful, and NULL - and
the lock dropped - if the area could not be extended.

Tested-by: Vegard Nossum 
Tested-by: John Paul Adrian Glaubitz  # ia64
Tested-by: Frank Scheiner  # ia64
Signed-off-by: Linus Torvalds 
Signed-off-by: Greg Kroah-Hartman

mm: make find_extend_vma() fail if write lock not held

2023-07-01T11:12:40+00:00

commit f440fa1ac955e2898893f9301568435eb5cdfc4b upstream.

Make calls to extend_vma() and find_extend_vma() fail if the write lock
is required.

To avoid making this a flag-day event, this still allows the old
read-locking case for the trivial situations, and passes in a flag to
say "is it write-locked".  That way write-lockers can say "yes, I'm
being careful", and legacy users will continue to work in all the common
cases until they have been fully converted to the new world order.

Co-Developed-by: Matthew Wilcox (Oracle) 
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Liam R. Howlett 
Signed-off-by: Linus Torvalds 
Signed-off-by: Greg Kroah-Hartman

mm: make the page fault mmap locking killable

2023-07-01T11:12:38+00:00

commit eda0047296a16d65a7f2bc60a408f70d178b2014 upstream.

This is done as a separate patch from introducing the new
lock_mm_and_find_vma() helper, because while it's an obvious change,
it's not what x86 used to do in this area.

We already abort the page fault on fatal signals anyway, so why should
we wait for the mmap lock only to then abort later? With the new helper
function that returns without the lock held on failure anyway, this is
particularly easy and straightforward.

Signed-off-by: Linus Torvalds 
Signed-off-by: Greg Kroah-Hartman

mm: introduce new 'lock_mm_and_find_vma()' page fault helper

2023-07-01T11:12:38+00:00

commit c2508ec5a58db67093f4fb8bf89a9a7c53a109e9 upstream.

.. and make x86 use it.

This basically extracts the existing x86 "find and expand faulting vma"
code, but extends it to also take the mmap lock for writing in case we
actually do need to expand the vma.

We've historically short-circuited that case, and have some rather ugly
special logic to serialize the stack segment expansion (since we only
hold the mmap lock for reading) that doesn't match the normal VM
locking.

That slight violation of locking worked well, right up until it didn't:
the maple tree code really does want proper locking even for simple
extension of an existing vma.

So extract the code for "look up the vma of the fault" from x86, fix it
up to do the necessary write locking, and make it available as a helper
function for other architectures that can use the common helper.

Note: I say "common helper", but it really only handles the normal
stack-grows-down case.  Which is all architectures except for PA-RISC
and IA64.  So some rare architectures can't use the helper, but if they
care they'll just need to open-code this logic.

It's also worth pointing out that this code really would like to have an
optimistic "mmap_upgrade_trylock()" to make it quicker to go from a
read-lock (for the common case) to taking the write lock (for having to
extend the vma) in the normal single-threaded situation where there is
no other locking activity.

But that _is_ all the very uncommon special case, so while it would be
nice to have such an operation, it probably doesn't matter in reality.
I did put in the skeleton code for such a possible future expansion,
even if it only acts as pseudo-documentation for what we're doing.

Signed-off-by: Linus Torvalds 
Signed-off-by: Greg Kroah-Hartman

mm: do not increment pgfault stats when page fault handler retries

2023-04-21T21:52:04+00:00

If the page fault handler requests a retry, we will count the fault
multiple times.  This is a relatively harmless problem as the retry paths
are not often requested, and the only user-visible problem is that the
fault counter will be slightly higher than it should be.  Nevertheless,
userspace only took one fault, and should not see the fact that the kernel
had to retry the fault multiple times.

Move page fault accounting into mm_account_fault() and skip incomplete
faults which will be accounted upon completion.

Link: https://lkml.kernel.org/r/20230419175836.3857458-1-surenb@google.com
Fixes: d065bd810b6d ("mm: retry page fault when blocking on disk transfer")
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Matthew Wilcox (Oracle) 
Acked-by: Peter Xu 
Cc: Davidlohr Bueso 
Cc: Jan Kara 
Cc: Johannes Weiner 
Cc: Josef Bacik 
Cc: Laurent Dufour 
Cc: Liam R. Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Michel Lespinasse 
Cc: Minchan Kim 
Cc: Punit Agrawal 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm: hwpoison: support recovery from HugePage copy-on-write faults

2023-04-18T23:30:09+00:00

copy-on-write of hugetlb user pages with uncorrectable errors will result
in a kernel crash.  This is because the copy is performed in kernel mode
and in general we can not handle accessing memory with such errors while
in kernel mode.  Commit a873dfe1032a ("mm, hwpoison: try to recover from
copy-on write faults") introduced the routine copy_user_highpage_mc() to
gracefully handle copying of user pages with uncorrectable errors. 
However, the separate hugetlb copy-on-write code paths were not modified
as part of commit a873dfe1032a.

Modify hugetlb copy-on-write code paths to use copy_mc_user_highpage() so
that they can also gracefully handle uncorrectable errors in user pages. 
This involves changing the hugetlb specific routine
copy_user_large_folio() from type void to int so that it can return an
error.  Modify the hugetlb userfaultfd code in the same way so that it can
return -EHWPOISON if it encounters an uncorrectable error.

Link: https://lkml.kernel.org/r/20230413131349.2524210-1-liushixin2@huawei.com
Signed-off-by: Liu Shixin 
Acked-by: Mike Kravetz 
Reviewed-by: Naoya Horiguchi 
Cc: Miaohe Lin 
Cc: Muchun Song 
Cc: Tony Luck 
Signed-off-by: Andrew Morton

mm: convert copy_user_huge_page() to copy_user_large_folio()

2023-04-18T23:29:55+00:00

Replace copy_user_huge_page() with copy_user_large_folio(). 
copy_user_large_folio() does the same as copy_user_huge_page(), but takes
in folios instead of pages.  Remove pages_per_huge_page from
copy_user_large_folio(), because we can get that from folio_nr_pages(dst).

Convert copy_user_gigantic_page() to take in folios.

Link: https://lkml.kernel.org/r/20230410133932.32288-6-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Sidhartha Kumar 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

userfaultfd: convert copy_huge_page_from_user() to copy_folio_from_user()

2023-04-18T23:29:55+00:00

Replace copy_huge_page_from_user() with copy_folio_from_user(). 
copy_folio_from_user() does the same as copy_huge_page_from_user(), but
takes in a folio instead of a page.

Convert page_kaddr to kaddr in copy_folio_from_user() to do indenting
cleanup.

Link: https://lkml.kernel.org/r/20230410133932.32288-4-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Reviewed-by: Sidhartha Kumar 
Reviewed-by: Mike Kravetz 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

userfaultfd: use kmap_local_page() in copy_huge_page_from_user()

2023-04-18T23:29:54+00:00

kmap() and kmap_atomic() are being deprecated in favor of
kmap_local_page() which is appropriate for any thread local context.[1]

Let's replace the kmap() and kmap_atomic() with kmap_local_page() in
copy_huge_page_from_user().  When allow_pagefault is false, disable page
faults to prevent potential deadlock.[2]

[1] https://lore.kernel.org/all/20220813220034.806698-1-ira.weiny@intel.com/
[2] https://lkml.kernel.org/r/20221025220136.2366143-1-ira.weiny@intel.com

Link: https://lkml.kernel.org/r/20230410133932.32288-3-zhangpeng362@huawei.com
Signed-off-by: ZhangPeng 
Reviewed-by: Mike Kravetz 
Cc: Kefeng Wang 
Cc: Matthew Wilcox 
Cc: Muchun Song 
Cc: Nanyong Sun 
Cc: Sidhartha Kumar 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton