linux.git/mm/rmap.c, branch v5.14

mm/rmap: fix munlocking Anon THP with mlocked ptes

2021-07-12T18:30:56+00:00

Many thanks to Kirill for reminding that PageDoubleMap cannot be relied on
to warn of pte mappings in the Anon THP case; and a scan of subpages does
not seem appropriate here.  Note how follow_trans_huge_pmd() does not even
mark an Anon THP as mlocked when compound_mapcount != 1: multiple mlocking
of Anon THP is avoided, so simply return from page_mlock() in this case.

Link: https://lore.kernel.org/lkml/cfa154c-d595-406-eb7d-eb9df730f944@google.com/
Fixes: d9770fcc1c0c ("mm/rmap: fix old bug: munlocking THP missed other mlocks")
Reported-by: Kirill A. Shutemov 
Signed-off-by: Hugh Dickins 
Acked-by: Kirill A. Shutemov 
Cc: Andrew Morton 
Cc: Alistair Popple 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: Christoph Hellwig 
Cc: Yang Shi 
Cc: Shakeel Butt 
Signed-off-by: Linus Torvalds

mm/rmap: try_to_migrate() skip zone_device !device_private

2021-07-11T22:05:15+00:00

I know nothing about zone_device pages and !device_private pages; but if
try_to_migrate_one() will do nothing for them, then it's better that
try_to_migrate() filter them first, than trawl through all their vmas.

Signed-off-by: Hugh Dickins 
Reviewed-by: Shakeel Butt 
Reviewed-by: Alistair Popple 
Link: https://lore.kernel.org/lkml/1241d356-8ec9-f47b-a5ec-9b2bf66d242@google.com/
Cc: Andrew Morton 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: Christoph Hellwig 
Cc: Yang Shi 
Cc: Kirill A. Shutemov 
Signed-off-by: Linus Torvalds

mm/rmap: fix new bug: premature return from page_mlock_one()

2021-07-11T22:05:15+00:00

In the unlikely race case that page_mlock_one() finds VM_LOCKED has been
cleared by the time it got page table lock, page_vma_mapped_walk_done()
must be called before returning, either explicitly, or by a final call
to page_vma_mapped_walk() - otherwise the page table remains locked.

Fixes: cd62734ca60d ("mm/rmap: split try_to_munlock from try_to_unmap")
Signed-off-by: Hugh Dickins 
Reviewed-by: Alistair Popple 
Reviewed-by: Shakeel Butt 
Reported-by: kernel test robot 
Link: https://lore.kernel.org/lkml/20210711151446.GB4070@xsang-OptiPlex-9020/
Link: https://lore.kernel.org/lkml/f71f8523-cba7-3342-40a7-114abc5d1f51@google.com/
Cc: Andrew Morton 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: Christoph Hellwig 
Cc: Yang Shi 
Cc: Kirill A. Shutemov 
Signed-off-by: Linus Torvalds

mm/rmap: fix old bug: munlocking THP missed other mlocks

2021-07-11T22:05:15+00:00

The kernel recovers in due course from missing Mlocked pages: but there
was no point in calling page_mlock() (formerly known as
try_to_munlock()) on a THP, because nothing got done even when it was
found to be mapped in another VM_LOCKED vma.

It's true that we need to be careful: Mlocked accounting of pte-mapped
THPs is too difficult (so consistently avoided); but Mlocked accounting
of only-pmd-mapped THPs is supposed to work, even when multiple mappings
are mlocked and munlocked or munmapped.  Refine the tests.

There is already a VM_BUG_ON_PAGE(PageDoubleMap) in page_mlock(), so
page_mlock_one() does not even have to worry about that complication.

(I said the kernel recovers: but would page reclaim be likely to split
THP before rediscovering that it's VM_LOCKED? I've not followed that up)

Fixes: 9a73f61bdb8a ("thp, mlock: do not mlock PTE-mapped file huge pages")
Signed-off-by: Hugh Dickins 
Reviewed-by: Shakeel Butt 
Acked-by: Kirill A. Shutemov 
Link: https://lore.kernel.org/lkml/cfa154c-d595-406-eb7d-eb9df730f944@google.com/
Cc: Andrew Morton 
Cc: Alistair Popple 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: Christoph Hellwig 
Cc: Yang Shi 
Signed-off-by: Linus Torvalds

mm/rmap: fix comments left over from recent changes

2021-07-11T22:05:15+00:00

Parallel developments in mm/rmap.c have left behind some out-of-date
comments: try_to_migrate_one() also accepts TTU_SYNC (already commented
in try_to_migrate() itself), and try_to_migrate() returns nothing at
all.

TTU_SPLIT_FREEZE has just been deleted, so reword the comment about it
in mm/huge_memory.c; and TTU_IGNORE_ACCESS was removed in 5.11, so
delete the "recently referenced" comment from try_to_unmap_one() (once
upon a time the comment was near the removed codeblock, but they drifted
apart).

Signed-off-by: Hugh Dickins 
Reviewed-by: Shakeel Butt 
Reviewed-by: Alistair Popple 
Link: https://lore.kernel.org/lkml/563ce5b2-7a44-5b4d-1dfd-59a0e65932a9@google.com/
Cc: Andrew Morton 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: Christoph Hellwig 
Cc: Yang Shi 
Cc: Kirill A. Shutemov 
Signed-off-by: Linus Torvalds

mm: device exclusive memory access

2021-07-01T18:06:03+00:00

Some devices require exclusive write access to shared virtual memory (SVM)
ranges to perform atomic operations on that memory.  This requires CPU
page tables to be updated to deny access whilst atomic operations are
occurring.

In order to do this introduce a new swap entry type
(SWP_DEVICE_EXCLUSIVE).  When a SVM range needs to be marked for exclusive
access by a device all page table mappings for the particular range are
replaced with device exclusive swap entries.  This causes any CPU access
to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original
mapping.  This results in MMU notifiers being called which a driver uses
to update access permissions such as revoking atomic access.  After
notifiers have been called the device will no longer have exclusive access
to the region.

Walking of the page tables to find the target pages is handled by
get_user_pages() rather than a direct page table walk.  A direct page
table walk similar to what migrate_vma_collect()/unmap() does could also
have been utilised.  However this resulted in more code similar in
functionality to what get_user_pages() provides as page faulting is
required to make the PTEs present and to break COW.

[dan.carpenter@oracle.com: fix signedness bug in make_device_exclusive_range()]
  Link: https://lkml.kernel.org/r/YNIz5NVnZ5GiZ3u1@mwanda

Link: https://lkml.kernel.org/r/20210616105937.23201-8-apopple@nvidia.com
Signed-off-by: Alistair Popple 
Signed-off-by: Dan Carpenter 
Reviewed-by: Christoph Hellwig 
Cc: Ben Skeggs 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Peter Xu 
Cc: Ralph Campbell 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/rmap: split migration into its own function

2021-07-01T18:06:03+00:00

Migration is currently implemented as a mode of operation for
try_to_unmap_one() generally specified by passing the TTU_MIGRATION flag
or in the case of splitting a huge anonymous page TTU_SPLIT_FREEZE.

However it does not have much in common with the rest of the unmap
functionality of try_to_unmap_one() and thus splitting it into a separate
function reduces the complexity of try_to_unmap_one() making it more
readable.

Several simplifications can also be made in try_to_migrate_one() based on
the following observations:

 - All users of TTU_MIGRATION also set TTU_IGNORE_MLOCK.
 - No users of TTU_MIGRATION ever set TTU_IGNORE_HWPOISON.
 - No users of TTU_MIGRATION ever set TTU_BATCH_FLUSH.

TTU_SPLIT_FREEZE is a special case of migration used when splitting an
anonymous page.  This is most easily dealt with by calling the correct
function from unmap_page() in mm/huge_memory.c - either try_to_migrate()
for PageAnon or try_to_unmap().

Link: https://lkml.kernel.org/r/20210616105937.23201-5-apopple@nvidia.com
Signed-off-by: Alistair Popple 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Ralph Campbell 
Cc: Ben Skeggs 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Peter Xu 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/rmap: split try_to_munlock from try_to_unmap

2021-07-01T18:06:03+00:00

The behaviour of try_to_unmap_one() is difficult to follow because it
performs different operations based on a fairly large set of flags used in
different combinations.

TTU_MUNLOCK is one such flag.  However it is exclusively used by
try_to_munlock() which specifies no other flags.  Therefore rather than
overload try_to_unmap_one() with unrelated behaviour split this out into
it's own function and remove the flag.

Link: https://lkml.kernel.org/r/20210616105937.23201-4-apopple@nvidia.com
Signed-off-by: Alistair Popple 
Reviewed-by: Ralph Campbell 
Reviewed-by: Christoph Hellwig 
Cc: Ben Skeggs 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Peter Xu 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/swapops: rework swap entry manipulation code

2021-07-01T18:06:03+00:00

Both migration and device private pages use special swap entries that are
manipluated by a range of inline functions.  The arguments to these are
somewhat inconsistent so rework them to remove flag type arguments and to
make the arguments similar for both read and write entry creation.

Link: https://lkml.kernel.org/r/20210616105937.23201-3-apopple@nvidia.com
Signed-off-by: Alistair Popple 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Ralph Campbell 
Cc: Ben Skeggs 
Cc: Hugh Dickins 
Cc: John Hubbard 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Peter Xu 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: rmap: make try_to_unmap() void function

2021-07-01T03:47:30+00:00

Currently try_to_unmap() return bool value by checking page_mapcount(),
however this may return false positive since page_mapcount() doesn't check
all subpages of compound page.  The total_mapcount() could be used
instead, but its cost is higher since it traverses all subpages.

Actually the most callers of try_to_unmap() don't care about the return
value at all.  So just need check if page is still mapped by page_mapped()
when necessary.  And page_mapped() does bail out early when it finds
mapped subpage.

Link: https://lkml.kernel.org/r/bb27e3fe-6036-b637-5086-272befbfe3da@google.com
Suggested-by: Hugh Dickins 
Signed-off-by: Yang Shi 
Acked-by: Minchan Kim 
Reviewed-by: Shakeel Butt 
Acked-by: Kirill A. Shutemov 
Signed-off-by: Hugh Dickins 
Acked-by: Naoya Horiguchi 
Cc: Alistair Popple 
Cc: Jan Kara 
Cc: Jue Wang 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Miaohe Lin 
Cc: Oscar Salvador 
Cc: Peter Xu 
Cc: Ralph Campbell 
Cc: Wang Yugui 
Cc: Zi Yan 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds