linux-stable.git/lib/test_hmm.c, branch linux-6.16.y

mm: allow compound zone device pages

2025-03-18T05:06:39+00:00

Zone device pages are used to represent various type of device memory
managed by device drivers.  Currently compound zone device pages are not
supported.  This is because MEMORY_DEVICE_FS_DAX pages are the only user
of higher order zone device pages and have their own page reference
counting.

A future change will unify FS DAX reference counting with normal page
reference counting rules and remove the special FS DAX reference counting.
Supporting that requires compound zone device pages.

Supporting compound zone device pages requires compound_head() to
distinguish between head and tail pages whilst still preserving the
special struct page fields that are specific to zone device pages.

A tail page is distinguished by having bit zero being set in
page->compound_head, with the remaining bits pointing to the head page. 
For zone device pages page->compound_head is shared with page->pgmap.

The page->pgmap field must be common to all pages within a folio, even if
the folio spans memory sections.  Therefore pgmap is the same for both
head and tail pages and can be moved into the folio and we can use the
standard scheme to find compound_head from a tail page.

Link: https://lkml.kernel.org/r/67055d772e6102accf85161d0b57b0b3944292bf.1740713401.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Signed-off-by: Balbir Singh 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Dan Williams 
Acked-by: David Hildenbrand 
Tested-by: Alison Schofield 
Cc: Alexander Gordeev 
Cc: Asahi Lina 
Cc: Bjorn Helgaas 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christoph Hellwig 
Cc: Chunyan Zhang 
Cc: "Darrick J. Wong" 
Cc: Dave Chinner 
Cc: Dave Hansen 
Cc: Dave Jiang 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ira Weiny 
Cc: Jan Kara 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: linmiaohe 
Cc: Logan Gunthorpe 
Cc: Matthew Wilcow (Oracle) 
Cc: Michael "Camp Drill Sergeant" Ellerman 
Cc: Nicholas Piggin 
Cc: Peter Xu 
Cc: Sven Schnelle 
Cc: Ted Ts'o 
Cc: Vasily Gorbik 
Cc: Vishal Verma 
Cc: Vivek Goyal 
Cc: WANG Xuerui 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

lib/test_hmm: make dmirror_atomic_map() consume a single page

2025-03-17T07:05:29+00:00

Patch series "mm: cleanups for device-exclusive entries (hmm)", v2.

Some smaller device-exclusive cleanups I have lying around.


This patch (of 5):

The caller now always passes a single page; let's simplify, and return "0"
on success.

Link: https://lkml.kernel.org/r/20250226132257.2826043-1-david@redhat.com
Link: https://lkml.kernel.org/r/20250226132257.2826043-2-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Alistair Popple 
Cc: Jason Gunthorpe 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton

mm/rmap: convert make_device_exclusive_range() to make_device_exclusive()

2025-03-17T05:05:57+00:00

The single "real" user in the tree of make_device_exclusive_range() always
requests making only a single address exclusive.  The current
implementation is hard to fix for properly supporting anonymous THP /
large folios and for avoiding messing with rmap walks in weird ways.

So let's always process a single address/page and return folio + page to
minimize page -> folio lookups.  This is a preparation for further
changes.

Reject any non-anonymous or hugetlb folios early, directly after GUP.

While at it, extend the documentation of make_device_exclusive() to
clarify some things.

Link: https://lkml.kernel.org/r/20250210193801.781278-4-david@redhat.com
Signed-off-by: David Hildenbrand 
Acked-by: Simona Vetter 
Reviewed-by: Alistair Popple 
Tested-by: Alistair Popple 
Cc: Alex Shi 
Cc: Danilo Krummrich 
Cc: Dave Airlie 
Cc: Jann Horn 
Cc: Jason Gunthorpe 
Cc: Jerome Glisse 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Karol Herbst 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Lyude 
Cc: "Masami Hiramatsu (Google)" 
Cc: Oleg Nesterov 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Peter Zijlstra (Intel) 
Cc: SeongJae Park 
Cc: Vlastimil Babka 
Cc: Yanteng Si 
Cc: Barry Song 
Signed-off-by: Andrew Morton

lib: test_hmm: use min() to improve dmirror_exclusive()

2024-09-02T03:25:52+00:00

Use min() to simplify the dmirror_exclusive() function and improve its
readability.

Link: https://lkml.kernel.org/r/20240726131245.161695-1-thorsten.blum@toblux.com
Signed-off-by: Thorsten Blum 
Reviewed-by: David Hildenbrand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton

lib: test_hmm: add missing MODULE_DESCRIPTION() macro

2024-07-04T02:30:05+00:00

make allmodconfig && make W=1 C=1 reports:
WARNING: modpost: missing MODULE_DESCRIPTION() in lib/test_hmm.o

Add the missing invocation of the MODULE_DESCRIPTION() macro.

Link: https://lkml.kernel.org/r/20240531-lib-md-test_hmm-v1-1-e4aa17daa57b@quicinc.com
Signed-off-by: Jeff Johnson 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton

lib/test_hmm.c: handle src_pfns and dst_pfns allocation failure

2024-04-26T03:55:48+00:00

The kcalloc() in dmirror_device_evict_chunk() will return null if the
physical memory has run out.  As a result, if src_pfns or dst_pfns is
dereferenced, the null pointer dereference bug will happen.

Moreover, the device is going away.  If the kcalloc() fails, the pages
mapping a chunk could not be evicted.  So add a __GFP_NOFAIL flag in
kcalloc().

Finally, as there is no need to have physically contiguous memory, Switch
kcalloc() to kvcalloc() in order to avoid failing allocations.

Link: https://lkml.kernel.org/r/20240312005905.9939-1-duoming@zju.edu.cn
Fixes: b2ef9f5a5cb3 ("mm/hmm/test: add selftest driver for HMM")
Signed-off-by: Duoming Zhou 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton

lib: replace kmap() with kmap_local_page()

2023-08-18T17:18:50+00:00

kmap() has been deprecated in favor of the kmap_local_page() due to high
cost, restricted mapping space, the overhead of a global lock for
synchronization, and making the process sleep in the absence of free
slots.

kmap_local_page() is faster than kmap() and offers thread-local and
CPU-local mappings, take pagefaults in a local kmap region and preserves
preemption by saving the mappings of outgoing tasks and restoring those of
the incoming one during a context switch.

The mappings are kept thread local in the functions “dmirror_do_read”
and “dmirror_do_write” in test_hmm.c

Therefore, replace kmap() with kmap_local_page() and use
mempcy_from/to_page() to avoid open coding kmap_local_page() + memcpy() +
kunmap_local().

Remove the unused variable “tmp”.

Link: https://lkml.kernel.org/r/20230610175712.GA348514@sumitra.com
Signed-off-by: Sumitra Sharma 
Suggested-by: Fabio M. De Francesco 
Reviewed-by: Fabio M. De Francesco 
Reviewed-by: Ira Weiny 
Cc: Deepak R Varma 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton

hmm-tests: add test for migrate_device_range()

2022-10-13T01:51:50+00:00

Link: https://lkml.kernel.org/r/a73cf109de0224cfd118d22be58ddebac3ae2897.1664366292.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Cc: Jason Gunthorpe 
Cc: Ralph Campbell 
Cc: John Hubbard 
Cc: Alex Sierra 
Cc: Felix Kuehling 
Cc: Alex Deucher 
Cc: Ben Skeggs 
Cc: Christian König 
Cc: Dan Williams 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Lyude Paul 
Cc: Matthew Wilcox 
Cc: Michael Ellerman 
Cc: Yang Shi 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: free device private pages have zero refcount

2022-10-13T01:51:49+00:00

Since 27674ef6c73f ("mm: remove the extra ZONE_DEVICE struct page
refcount") device private pages have no longer had an extra reference
count when the page is in use.  However before handing them back to the
owning device driver we add an extra reference count such that free pages
have a reference count of one.

This makes it difficult to tell if a page is free or not because both free
and in use pages will have a non-zero refcount.  Instead we should return
pages to the drivers page allocator with a zero reference count.  Kernel
code can then safely use kernel functions such as get_page_unless_zero().

Link: https://lkml.kernel.org/r/cf70cf6f8c0bdb8aaebdbfb0d790aea4c683c3c6.1664366292.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Acked-by: Felix Kuehling 
Cc: Jason Gunthorpe 
Cc: Michael Ellerman 
Cc: Alex Deucher 
Cc: Christian König 
Cc: Ben Skeggs 
Cc: Lyude Paul 
Cc: Ralph Campbell 
Cc: Alex Sierra 
Cc: John Hubbard 
Cc: Dan Williams 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Matthew Wilcox 
Cc: Yang Shi 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/memory.c: fix race when faulting a device private page

2022-10-13T01:51:49+00:00

Patch series "Fix several device private page reference counting issues",
v2

This series aims to fix a number of page reference counting issues in
drivers dealing with device private ZONE_DEVICE pages.  These result in
use-after-free type bugs, either from accessing a struct page which no
longer exists because it has been removed or accessing fields within the
struct page which are no longer valid because the page has been freed.

During normal usage it is unlikely these will cause any problems.  However
without these fixes it is possible to crash the kernel from userspace. 
These crashes can be triggered either by unloading the kernel module or
unbinding the device from the driver prior to a userspace task exiting. 
In modules such as Nouveau it is also possible to trigger some of these
issues by explicitly closing the device file-descriptor prior to the task
exiting and then accessing device private memory.

This involves some minor changes to both PowerPC and AMD GPU code. 
Unfortunately I lack hardware to test either of those so any help there
would be appreciated.  The changes mimic what is done in for both Nouveau
and hmm-tests though so I doubt they will cause problems.


This patch (of 8):

When the CPU tries to access a device private page the migrate_to_ram()
callback associated with the pgmap for the page is called.  However no
reference is taken on the faulting page.  Therefore a concurrent migration
of the device private page can free the page and possibly the underlying
pgmap.  This results in a race which can crash the kernel due to the
migrate_to_ram() function pointer becoming invalid.  It also means drivers
can't reliably read the zone_device_data field because the page may have
been freed with memunmap_pages().

Close the race by getting a reference on the page while holding the ptl to
ensure it has not been freed.  Unfortunately the elevated reference count
will cause the migration required to handle the fault to fail.  To avoid
this failure pass the faulting page into the migrate_vma functions so that
if an elevated reference count is found it can be checked to see if it's
expected or not.

[mpe@ellerman.id.au: fix build]
  Link: https://lkml.kernel.org/r/87fsgbf3gh.fsf@mpe.ellerman.id.au
Link: https://lkml.kernel.org/r/cover.60659b549d8509ddecafad4f498ee7f03bb23c69.1664366292.git-series.apopple@nvidia.com
Link: https://lkml.kernel.org/r/d3e813178a59e565e8d78d9b9a4e2562f6494f90.1664366292.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Acked-by: Felix Kuehling 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Ralph Campbell 
Cc: Michael Ellerman 
Cc: Lyude Paul 
Cc: Alex Deucher 
Cc: Alex Sierra 
Cc: Ben Skeggs 
Cc: Christian König 
Cc: Dan Williams 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Matthew Wilcox 
Cc: Yang Shi 
Cc: Zi Yan 
Signed-off-by: Andrew Morton