linux-stable.git/include/linux/hmm.h, branch v7.3-rc2

mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support

2026-08-05T02:18:54+00:00

hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call.  This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock.  Drivers that need to populate device page tables
for such mappings have no way to do so today.

Add hmm_range_fault_unlocked_timeout() for callers that do not need to
hold mmap_lock across any work outside the HMM fault itself.  The helper
takes mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still
held.  The timeout is specified in jiffies; passing 0 retries
indefinitely, while a non-zero timeout makes the helper return -EBUSY when
the retry budget expires.  The retry deadline is set before refreshing the
notifier sequence and acquiring mmap_lock, so contended mmap_lock
acquisition is included in the retry budget.  After acquiring mmap_lock,
the helper also rejects unstable address spaces before walking page
tables.

When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally.  If the lock was dropped, the retry deadline
is also restarted because a lock-dropping fault handler made progress. 
Ordinary -EBUSY retries keep the existing deadline, preserving the
caller's timeout policy for repeated mmu-notifier invalidations.

The caller only needs to perform the usual post-success
mmu_interval_read_retry() check while holding its update lock before
consuming the pfns.  If mmap_lock acquisition is interrupted or a fatal
signal is pending during retry handling, -EINTR is returned instead.

The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout().  The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.

The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small.  Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock. 
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.

Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.

Link: https://lore.kernel.org/20260723-hmm-v10-v11-2-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii 
Cc: Danilo Krummrich 
Cc: Dave Airlie 
Cc: David Hildenbrand 
Cc: Dexuan Cui 
Cc: Haiyang Zhang 
Cc: Jason Gunthorpe 
Cc: Jonathan Corbet 
Cc: K. Y. Srinivasan 
Cc: Leon Romanovsky 
Cc: Liam R. Howlett 
Cc: Lizhi Hou 
Cc: Long Li 
Cc: Lorenzo Stoakes 
Cc: Lyude 
Cc: Maarten Lankhorst 
Cc: Maxime Ripard 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Oded Gabbay 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Thomas Zimemrmann 
Cc: Vlastimil Babka 
Cc: Wei Liu 
Signed-off-by: Andrew Morton

mm/hmm: provide generic DMA managing logic

2025-05-12T10:06:42+00:00

HMM callers use PFN list to populate range while calling
to hmm_range_fault(), the conversion from PFN to DMA address
is done by the callers with help of another DMA list. However,
it is wasteful on any modern platform and by doing the right
logic, that DMA list can be avoided.

Provide generic logic to manage these lists and gave an interface
to map/unmap PFNs to DMA addresses, without requiring from the callers
to be an experts in DMA core API.

Tested-by: Jens Axboe 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Leon Romanovsky

mm/hmm: let users to tag specific PFN with DMA mapped bit

2025-05-12T10:06:37+00:00

Introduce new sticky flag (HMM_PFN_DMA_MAPPED), which isn't overwritten
by HMM range fault. Such flag allows users to tag specific PFNs with
information if this specific PFN was already DMA mapped.

Tested-by: Jens Axboe 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Leon Romanovsky

docs: rename Documentation/vm to Documentation/mm

2022-06-27T19:52:53+00:00

so it will be consistent with code mm directory and with
Documentation/admin-guide/mm and won't be confused with virtual machines.

Signed-off-by: Mike Rapoport 
Suggested-by: Matthew Wilcox 
Tested-by: Ira Weiny 
Acked-by: Jonathan Corbet 
Acked-by: Wu XiangCheng

mm: remove pointless includes from

2022-03-03T17:47:33+00:00

hmm.h pulls in the world for no good reason at all.  Remove the
includes and push a few ones into the users instead.

Link: https://lkml.kernel.org/r/20220210072828.2930359-4-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Chaitanya Kulkarni 
Reviewed-by: Muchun Song 
Tested-by: "Sierra Guiza, Alejandro (Alex)" 

Cc: Alex Deucher 
Cc: Alistair Popple 
Cc: Ben Skeggs 
Cc: Christian Knig 
Cc: Dan Williams 
Cc: Felix Kuehling 
Cc: Karol Herbst 
Cc: Lyude Paul 
Cc: Miaohe Lin 
Cc: "Pan, Xinhui" 
Cc: Ralph Campbell 
Signed-off-by: Andrew Morton 
Signed-off-by: Matthew Wilcox (Oracle)

mm: fix spelling mistakes in header files

2021-07-08T18:48:21+00:00

Fix some spelling mistakes in comments:
successfull ==> successful
potentialy ==> potentially
alloced ==> allocated
indicies ==> indices
wont ==> won't
resposible ==> responsible
dirtyness ==> dirtiness
droppped ==> dropped
alread ==> already
occured ==> occurred
interupts ==> interrupts
extention ==> extension
slighly ==> slightly
Dont't ==> Don't

Link: https://lkml.kernel.org/r/20210531034849.9549-2-thunder.leizhen@huawei.com
Signed-off-by: Zhen Lei 
Cc: Jerome Glisse 
Cc: Mike Kravetz 
Cc: Dennis Zhou 
Cc: Tejun Heo 
Cc: Christoph Lameter 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/hmm: provide the page mapping order in hmm_range_fault()

2020-07-10T19:24:28+00:00

hmm_range_fault() returns an array of page frame numbers and flags for how
the pages are mapped in the requested process' page tables. The PFN can be
used to get the struct page with hmm_pfn_to_page() and the page size order
can be determined with compound_order(page).

However, if the page is larger than order 0 (PAGE_SIZE), there is no
indication that a compound page is mapped by the CPU using a larger page
size. Without this information, the caller can't safely use a large device
PTE to map the compound page because the CPU might be using smaller PTEs
with different read/write permissions.

Add a new function hmm_pfn_to_map_order() to return the mapping size order
so that callers know the pages are being mapped with consistent
permissions and a large device page table mapping can be used if one is
available.

This will allow devices to optimize mapping the page into HW by avoiding
or batching work for huge pages. For instance the dma_map can be done with
a high order directly.

Link: https://lore.kernel.org/r/20200701225352.9649-3-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell 
Signed-off-by: Jason Gunthorpe

mm: introduce include/linux/pgtable.h

2020-06-09T16:39:13+00:00

The include/linux/pgtable.h is going to be the home of generic page table
manipulation functions.

Start with moving asm-generic/pgtable.h to include/linux/pgtable.h and
make the latter include asm/pgtable.h.

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Brian Cain 
Cc: Catalin Marinas 
Cc: Chris Zankel 
Cc: "David S. Miller" 
Cc: Geert Uytterhoeven 
Cc: Greentime Hu 
Cc: Greg Ungerer 
Cc: Guan Xuetao 
Cc: Guo Ren 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Ingo Molnar 
Cc: Ley Foon Tan 
Cc: Mark Salter 
Cc: Matthew Wilcox 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Michal Simek 
Cc: Nick Hu 
Cc: Paul Walmsley 
Cc: Richard Weinberger 
Cc: Rich Felker 
Cc: Russell King 
Cc: Stafford Horne 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleixner 
Cc: Tony Luck 
Cc: Vincent Chen 
Cc: Vineet Gupta 
Cc: Will Deacon 
Cc: Yoshinori Sato 
Link: http://lkml.kernel.org/r/20200514170327.31389-3-rppt@kernel.org
Signed-off-by: Linus Torvalds

mm/hmm: remove the customizable pfn format from hmm_range_fault

2020-05-11T13:47:29+00:00

Presumably the intent here was that hmm_range_fault() could put the data
into some HW specific format and thus avoid some work. However, nothing
actually does that, and it isn't clear how anything actually could do that
as hmm_range_fault() provides CPU addresses which must be DMA mapped.

Perhaps there is some special HW that does not need DMA mapping, but we
don't have any examples of this, and the theoretical performance win of
avoiding an extra scan over the pfns array doesn't seem worth the
complexity. Plus pfns needs to be scanned anyhow to sort out any
DEVICE_PRIVATE pages.

This version replaces the uint64_t with an usigned long containing a pfn
and fixed flags. On input flags is filled with the HMM_PFN_REQ_* values,
on successful output it is filled with HMM_PFN_* values, describing the
state of the pages.

amdgpu is simple to convert, it doesn't use snapshot and doesn't use
per-page flags.

nouveau uses only 16 hmm_pte entries at most (ie fits in a few cache
lines), and it sweeps over its pfns array a couple of times anyhow. It
also has a nasty call chain before it reaches the dma map and hardware
suggesting performance isn't important:

   nouveau_svm_fault():
     args.i.m.method = NVIF_VMM_V0_PFNMAP
     nouveau_range_fault()
      nvif_object_ioctl()
       client->driver->ioctl()
	  struct nvif_driver nvif_driver_nvkm:
	    .ioctl = nvkm_client_ioctl
	   nvkm_ioctl()
	    nvkm_ioctl_path()
	      nvkm_ioctl_v0[type].func(..)
	      nvkm_ioctl_mthd()
	       nvkm_object_mthd()
		  struct nvkm_object_func nvkm_uvmm:
		    .mthd = nvkm_uvmm_mthd
		   nvkm_uvmm_mthd()
		    nvkm_uvmm_mthd_pfnmap()
		     nvkm_vmm_pfn_map()
		      nvkm_vmm_ptes_get_map()
		       func == gp100_vmm_pgt_pfn
			struct nvkm_vmm_desc_func gp100_vmm_desc_spt:
			  .pfn = gp100_vmm_pgt_pfn
			 nvkm_vmm_iter()
			  REF_PTES == func == gp100_vmm_pgt_pfn()
			    dma_map_page()

Link: https://lore.kernel.org/r/5-v2-b4e84f444c7d+24f57-hmm_no_flags_jgg@mellanox.com
Acked-by: Felix Kuehling 
Tested-by: Ralph Campbell 
Signed-off-by: Christoph Hellwig 
Reviewed-by: Christoph Hellwig 
Signed-off-by: Jason Gunthorpe

mm/hmm: remove HMM_PFN_SPECIAL

2020-05-11T13:47:29+00:00

This is just an alias for HMM_PFN_ERROR, nothing cares that the error was
because of a special page vs any other error case.

Link: https://lore.kernel.org/r/4-v2-b4e84f444c7d+24f57-hmm_no_flags_jgg@mellanox.com
Acked-by: Felix Kuehling 
Reviewed-by: Christoph Hellwig 
Reviewed-by: John Hubbard 
Signed-off-by: Jason Gunthorpe