linux.git/mm/userfaultfd.c, branch v7.3-rc2

userfaultfd: reset err to be 0 when move_pages_ptes succeeded

2026-09-02T03:26:21+00:00

During move_pages() operation, when move_pages_ptes() returns EAGAIN, the
error code is not cleared even after we processed it.  This leads to a
successful retry but then the same pages are retried again due to the
stale error code.  This time move fails because pages are already moved,
loop is terminated and move_pages() reports a failure.  Clear the error
code once we processes EAGAIN.

Link: https://lore.kernel.org/e1e0b5f8-c3c6-0537-670b-4397f822f980@gmail.com
Fixes: 50944692052b ("userfaultfd: opportunistic TLB-flush batching for present pages in MOVE")
Assisted-by: ChatGPT:GPT-5.6-Luna
Signed-off-by: Bryan Lim 
Reviewed-by: Suren Baghdasaryan 
Acked-by: Mike Rapoport (Microsoft) 
Cc: Peter Xu 
Cc: 
Signed-off-by: Andrew Morton

mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios

2026-08-25T01:42:53+00:00

Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed
mappings are indexed by their page offset within the file in which they
were originally mapped.

This differs from anonymous folios belonging to pure anon mappings which
are indexed by their anonymous page offset (the address at which they'd
belong in the VMA when first faulted).

This change fixes this inconsistency, always indexing anonymous folios by
their anonymous page offset regardless of the VMA to which they belong.

The foundations have been laid such that we need only switch this
functionality on such by:

* Using linear_anon_page_index() in __folio_set_anon() to assign the
  folio's index to the anonymous linear index rather than the file-backed
  one.

* Otherwise using linear_anon_page_index() in all instances where
  anonymous folios are being referenced or manipulated.

* Replacing vma_address() with vma_filebacked_address() or
  vma_anon_address() as appropriate.

* Updating the merging logic to check that anonymous page offsets are
  aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs,
  introducing needs_adjacent_anon_pgoff() to figure out when this is
  required.

* Updating linear_folio_page_index() to invoke linear_anon_page_index()
  if the folio is anonymous.

* Updating vma_address_end() to use the VMA's anonymous page offset when
  pvmw->pgoff is anonymous.

* Correcting folio_within_range() to use anonymous page offset for
  anonymous folios.

This will have no impact on merging of anonymous VMAs, whose page offset
and anonymous page offset are identical, nor will it impact shared
file-backed VMAs, which will continue to be merged based on the
file-backed page offset.

However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous
page offset as well.

In most instances this should have no impact on merging of file-backed
mappings, which are usually not merged all that often, let alone
MAP_PRIVATE mapped ones, and rarely remapped and faulted before being
moved back in place (the case in which a merge may now fail).

One subtle impact of this change is in NUMA interleaving - since commit
88c91dc58582 ("mempolicy: migration attempt to match interleave nodes"),
migration heuristically tries to maintain interleaving behaviour matching
the policy using folio indices.

When doing migration of CoW'd MAP_PRIVATE-file backed ranges, the 'base'
upon which the interleaving behaviour is performed will vary for these
ranges.  However the commit notes that ranges spanning multiple VMAs will
already cause varying bases, and that this is an acceptable approximation.

It is very unlikely real world use-cases will be impacted by this
(MAP_PRIVATE file-backed mappings are already an edge case), and all that
will happen is that such ranges will cause interleaving to be rotated over
the CoW'd range, with little to no impact.

This commit lays the foundations for future scalable CoW work which needs
to track some remaps, meaning that most remap tracking can be avoided, and
in nearly all cases the anonymous page offset will be able to be used to
quickly find the VMA in an mm.

Note that the need_rmap_locks check doesn't need to be updated, as any
remapping will offset both the anonymous and file-backed page offset, so
it suffices to check only one.

Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-14-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) 
Acked-by: David Hildenbrand (Arm) 
Cc: Adrian Hunter 
Cc: Alexander Deucher 
Cc: Alexander Gordeev 
Cc: Alexander Shishkin 
Cc: Alistair Popple 
Cc: Arnaldo Carvalho de Melo 
Cc: Arnd Bergmann 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Boris Brezillon 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christan König 
Cc: Christian Borntraeger 
Cc: Claudio Imbrenda 
Cc: Dave Airlie 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Greg Kroah-Hartman 
Cc: Gregory Price (Meta) 
Cc: Harry Yoo 
Cc: Heiko Carstens 
Cc: Huang Ray 
Cc: "Huang, Ying" 
Cc: Ian Rogers 
Cc: Ingo Molnar 
Cc: James Clark 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Jiri Olsa 
Cc: John Hubbard 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kees Cook 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Liam R. Howlett 
Cc: Liviu Dudau 
Cc: Maarten Lankhorst 
Cc: Marc Rutland 
Cc: "Masami Hiramatsu (Google)" 
Cc: Matthew Auld 
Cc: Matthew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Maxime Ripard 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Namhyung kim 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oleg Nesterov 
Cc: Oscar Salvador 
Cc: Pedro Falcato 
Cc: Peter Xu 
Cc: Peter Zijlstra 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Rodrigo Vivi 
Cc: Ryan Roberts 
Cc: Steven Price 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Thomas Hellström 
Cc: Thomas Zimemrmann 
Cc: Vasily Gorbik 
Cc: Vlastimil Babka 
Cc: xu xin 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/vma: use vma_start_pgoff(), linear_page_index() in mm code

2026-08-05T02:19:00+00:00

There are many instances in which linear_page_index() (as well as
linear_page_delta()) is open-coded, which is confusing and inconsistent.

Additionally, vma->vm_pgoff doesn't necessarily make it clear that this is
the page offset of the start of the VMA range.

Doing so also aids greppability.

So use vma_start_pgoff() in favour of directly accessing vma->vm_pgoff, and
linear_page_index() where we can.

This also lays the ground for future changes which will add an anonymous
page offset in order to be able to index MAP_PRIVATE-file backed anon
folios in terms of their virtual page offset.

No functional change intended.

Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-18-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Gregory Price 
Reviewed-by: SJ Park 
Reviewed-by: Pedro Falcato 
Reviewed-by: Vlastimil Babka (SUSE) 
Cc: Ackerley Tng 
Cc: David Hildenbrand (Arm) 
Cc: Kai Huang 
Cc: Marek Szyprowski 
Cc: Thomas Zimmermann 
Cc: Liam R. Howlett (Oracle) 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle

2026-08-05T02:18:39+00:00

Add an ioctl to toggle async mode at runtime without re-registering the
userfaultfd.  This allows a VMM to switch between sync and async RWP modes
on-the-fly -- for example, starting in async mode for working set
scanning, then switching to sync mode to intercept faults during page
eviction.

UFFDIO_SET_MODE takes an enable/disable bitmask of UFFD_FEATURE_* flags. 
Only UFFD_FEATURE_RWP_ASYNC is toggleable today; the ioctl rejects any
other bit with -EINVAL.  Enabling RWP_ASYNC also requires RWP to have been
negotiated at UFFDIO_API time, mirroring the UFFDIO_API invariant.

Fault-path readers of ctx->features run under mmap_read_lock or a per-VMA
lock; the RMW takes mmap_write_lock and calls vma_start_write() on every
UFFD-armed VMA, so those readers are fully excluded. 
userfaultfd_show_fdinfo(), however, reads ctx->features without any lock,
so the RMW is written as a single WRITE_ONCE and fdinfo reads it with
READ_ONCE.  That keeps the lockless observer from seeing a mid-RMW
intermediate and removes the audit burden when new toggleable bits are
added later.

When switching to async, pending sync waiters are woken so they retry and
auto-resolve under the new mode.

Link: https://lore.kernel.org/20260708111417.173443-14-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) 
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

userfaultfd: add UFFD_FEATURE_RWP_ASYNC for async fault resolution

2026-08-05T02:18:39+00:00

Sync RWP delivers a message and blocks the faulting thread until the
handler resolves the fault.  For working-set tracking the VMM does not
need the message: it just needs to know, at scan time, which pages were
touched.  Async RWP serves that use case — the kernel restores access
in-place and the faulting thread continues without blocking.

The VMM reconstructs the access pattern after the fact via PAGEMAP_SCAN:
pages whose uffd bit is still set (inverted PAGE_IS_ACCESSED) were not
re-accessed since the last RWP cycle.

Worth calling out: async resolution upgrades writable private anon PTEs
via pte_mkwrite() when can_change_pte_writable() allows, mirroring
do_numa_page().  Without it, every re-access of an RWP'd writable page
would COW-fault a second time.

UFFD_FEATURE_RWP_ASYNC requires UFFD_FEATURE_RWP.

Link: https://lore.kernel.org/20260708111417.173443-13-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau 
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP

2026-08-05T02:18:38+00:00

Wire the fault side of read-write protection tracking and turn the
userspace interface on.

An RWP-protected PTE is PAGE_NONE with the uffd bit set.  The PROT_NONE
triggers a fault on any access; the uffd bit distinguishes it from plain
mprotect(PROT_NONE) or NUMA hinting.

Fault dispatch, per level:

  PTE     handle_pte_fault()    -> do_uffd_rwp()
  PMD     __handle_mm_fault()   -> do_huge_pmd_uffd_rwp()
  hugetlb hugetlb_fault()       -> hugetlb_handle_userfault()

The RWP branches gate on userfaultfd_pte_rwp() /
userfaultfd_huge_pmd_rwp() (VM_UFFD_RWP plus the uffd bit) and fall
through to do_numa_page() / do_huge_pmd_numa_page() otherwise.  Each
delivers a UFFD_PAGEFAULT_FLAG_RWP message through handle_userfault(); the
handler resolves it with UFFDIO_RWPROTECT clearing MODE_RWP.

userfaultfd_must_wait() and userfaultfd_huge_must_wait() add matching
protnone+uffd waiters so sync-mode fault handlers block correctly.

Expose the UAPI:

  UFFDIO_REGISTER_MODE_RWP   -> UFFD_API_REGISTER_MODES
  UFFD_FEATURE_RWP           -> UFFD_API_FEATURES
  _UFFDIO_RWPROTECT          -> UFFD_API_RANGE_IOCTLS
                                UFFD_API_RANGE_IOCTLS_BASIC

UFFD_FEATURE_RWP is masked out at UFFDIO_API time when PROT_NONE is not
available or VM_UFFD_RWP aliases VM_NONE (32-bit), so userspace never sees
an advertised-but-broken feature.

Works on anonymous, shmem, and hugetlb memory.

Link: https://lore.kernel.org/20260708111417.173443-11-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau 
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

userfaultfd: add UFFDIO_REGISTER_MODE_RWP and UFFDIO_RWPROTECT plumbing

2026-08-05T02:18:38+00:00

Add the userspace interface for read-write protection tracking:

  - UFFDIO_REGISTER_MODE_RWP      register a range for RWP tracking
  - UFFD_FEATURE_RWP              capability bit
  - UFFDIO_RWPROTECT              install / remove RWP on a range

Introduce CONFIG_USERFAULTFD_RWP, auto-selected on 64-bit kernels with
ARCH_HAS_PTE_PROTNONE and HAVE_ARCH_USERFAULTFD_WP.  The symbol gates
VM_UFFD_RWP (previously aliased to VM_NONE) and the smaps/trace-flag hooks
added in the preparatory patches; without it the UAPI bits added here have
nothing to drive and would be unreachable.

Registration sets VM_UFFD_RWP on the VMA.  Combining MODE_WP with MODE_RWP
is rejected because both modes claim the uffd PTE bit.

UFFDIO_RWPROTECT is the bidirectional counterpart of
UFFDIO_WRITEPROTECT:

  - MODE_RWP              change_protection() with MM_CP_UFFD_RWP
                          installs PAGE_NONE and sets the uffd bit on
                          present PTEs
  - !MODE_RWP             change_protection() with MM_CP_UFFD_RWP_RESOLVE
                          restores vma->vm_page_prot and clears the bit

userfaultfd_clear_vma() runs the same resolve pass on unregister so RWP
state cannot outlive the uffd.

Re-registering a range must not drop a mode that installs per-PTE markers
(WP or RWP); doing so returns -EBUSY.  This also closes a pre-existing
window where re-registering without MODE_WP would strand uffd-wp markers:
before, those caused extra write-faults but were otherwise benign; with
RWP preservation in place, a subsequent mprotect() on a VM_UFFD_RWP VMA
would silently promote the stale markers to RWP.

The feature is not yet advertised.  UFFDIO_REGISTER_MODE_RWP,
UFFD_FEATURE_RWP, and _UFFDIO_RWPROTECT are intentionally absent from
UFFD_API_REGISTER_MODES, UFFD_API_FEATURES, and UFFD_API_RANGE_IOCTLS, so
UFFDIO_API masks them out and the register-mode validator rejects the bit.
The follow-up patch adds fault dispatch and exposes the UAPI.

Link: https://lore.kernel.org/20260708111417.173443-10-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau 
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: preserve RWP marker across PTE rewrites

2026-08-05T02:18:37+00:00

The uffd PTE bit must survive any kernel path that rewrites a PTE on a
VM_UFFD_RWP VMA, otherwise the marker that carries PAGE_NONE semantics is
silently dropped and the next access leaks past RWP tracking.  Wire the
preservation through every path that rewrites a VM_UFFD_RWP PTE.

Swap and device-exclusive: do_swap_page(), restore_exclusive_pte(), and
unuse_pte() (swapoff()) re-apply PAGE_NONE when the swap PTE carries the
uffd bit and the VMA has VM_UFFD_RWP.

Migration: remove_migration_pte() and remove_migration_pmd() do the same
after the migration entry is replaced with a real PTE/PMD.

Fork: __copy_present_ptes(), copy_present_page(), copy_nonpresent_pte(),
copy_huge_pmd(), copy_huge_non_present_pmd(), and
copy_hugetlb_page_range() keep the uffd bit on the child when the
destination VMA has VM_UFFD_RWP, matching the existing VM_UFFD_WP
handling.  Add VM_UFFD_RWP to VM_COPY_ON_FORK so the flag itself
propagates.

mprotect(): change_pte_range() and change_huge_pmd() restore PAGE_NONE
after pte_modify()/pmd_modify() have recomputed the base protection from a
(possibly user-changed) vm_page_prot.  pte_modify() preserves _PAGE_UFFD,
so the bit stays; we just have to force PAGE_NONE back on top.

Link: https://lore.kernel.org/20260708111417.173443-8-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau 
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: rename uffd-wp PTE accessors to uffd

2026-08-05T02:18:36+00:00

Userfaultfd RWP will reuse the uffd-wp PTE bit to mark access-tracking
PTEs, alongside the write-protected ones it already marks.  The bit's
meaning now depends on the VMA flag (WP or RWP), not on its name.

Rename the kernel-internal names that describe the bit:

  - pte/pmd/huge_pte accessors (and swap variants)
  - pgtable_supports_uffd() capability query
  - SCAN_PTE_UFFD khugepaged enum

The ftrace string emitted by mm_khugepaged_scan_pmd for this enum is
kept as "pte_uffd_wp" so existing trace-based tooling keeps matching.

Pure mechanical rename -- no behavior change.

Link: https://lore.kernel.org/20260708111417.173443-4-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau 
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) 
Reviewed-by: SeongJae Park 
Cc: Andrea Arcangeli 
Cc: David Hildenbrand 
Cc: James Houghton 
Cc: Jonathan Corbet 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

userfaultfd: prevent registration of special VMAs

2026-07-09T22:48:53+00:00

Vova Tokarev says:

  userfaultfd allows registration on shadow stack VMAs.  With userfaultfd
  access, you can register on the shadow stack, discard a page ... and
  inject a page with chosen return addresses via UFFDIO_COPY.

Update vma_can_userfault() to reject VM_SHADOW_STACK.

While on it, also reject VM_SPECIAL so that if a driver would implement
vm_uffd_ops, it wouldn't be possible to register special VMAs with
userfaultfd.

Since VM_SPECIAL includes VM_DONTEXPAND which is set but hugetlb, exclude
hugetlb VMAs from the check for VM_SPECIAL.

Link: https://lore.kernel.org/20260618095017.2553004-1-rppt@kernel.org
Fixes: 54007f818206 ("mm: Introduce VM_SHADOW_STACK for shadow stack memory")
Signed-off-by: Mike Rapoport (Microsoft) 
Reported-by: vova tokarev 
Acked-by: David Hildenbrand (Arm) 
Reviewed-by: Lorenzo Stoakes 
Cc: Al Viro 
Cc: Christian Brauner 
Cc: Jan Kara 
Cc: Linus Torvalds 
Cc: Mike Rapoport 
Cc: Oleg Nesterov 
Cc: Peter Xu 
Cc: 
Signed-off-by: Andrew Morton