linux.git/mm/userfaultfd.c, branch v7.1

userfaultfd: remove redundant check in vm_uffd_ops()

2026-06-01T04:50:26+00:00

Lorenzo says:

  static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma)
  {
          if (vma_is_anonymous(vma))
                  return &anon_uffd_ops;
          return vma->vm_ops ? vma->vm_ops->uffd_ops : NULL;
  }

  This is doing a redundant check _and_ making life confusing, as if
  !vma->vm_ops is a condition that can be reached there, it can't, as
  vma_is_anonymous() is literally a !vma->vm_ops check :)

Remove the redundant check.

Link: https://lore.kernel.org/20260527184751.4147364-4-rppt@kernel.org
Fixes: 0f48947c4232 ("userfaultfd: introduce vm_uffd_ops")
Signed-off-by: Mike Rapoport (Microsoft) 
Suggested-by: Lorenzo Stoakes 
Reviewed-by: Lorenzo Stoakes 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Peter Xu 
Cc: David Carlier 
Cc: Michael Bommarito 
Signed-off-by: Andrew Morton

userfaultfd: refuse to __mfill_atomic_pte() for unsupported VMAs

2026-06-01T04:50:25+00:00

__mfill_atomic_pte() unconditionally dereferences ops because there is an
assumption that VMAs that can undergo mfill_* operations are vetted on
registration and must have valid vm_uffd_ops.

Add a guard against potential bugs and make sure __mfill_atomic_pte()
bails out if ops is NULL.

Link: https://lore.kernel.org/20260527184751.4147364-3-rppt@kernel.org
Fixes: ad9ac3081332 ("userfaultfd: introduce vm_uffd_ops->alloc_folio()")
Signed-off-by: Mike Rapoport (Microsoft) 
Suggested-by: Lorenzo Stoakes 
Reviewed-by: Lorenzo Stoakes 
Reviewed-by: David CARLIER 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Cc: Michael Bommarito 
Cc: Peter Xu 
Signed-off-by: Andrew Morton

userfaultfd: verify VMA state across UFFDIO_COPY retry

2026-06-01T04:50:25+00:00

Patch series "userfaultfd: verify VMA state across UFFDIO_COPY retry", v2.

... and two more small fixes.


This patch (of 3):

mfill_copy_folio_retry() drops the VMA lock for copy_from_user() and
reacquires it afterwards.  The destination VMA can be replaced during that
window.

The existing check compares vma_uffd_ops() before and after the retry, but
if a shmem VMA with MAP_SHARED is replaced with a shmem VMA with
MAP_PRIVATE (or vice versa) the replacement goes undetected.

The change from MAP_PRIVATE to MAP_SHARED will treat the folio allocated
with shmem_alloc_folio() as anonymous and this will cause BUG() when
mfill_atomic_install_pte() will try to folio_add_new_anon_rmap().

The change from MAP_SHARED to MAP_PRIVATE allows injection of folios into
the page cache of the original VMA.

There is no need to change for hugetlb because it never uses
mfill_copy_folio_retry().

Introduce helpers for more comprehensive comparison of VMA state:
- mfill_retry_state_save() to save the relevant VMA state into a struct
  mfill_retry_state (original uffd_ops, relevant VMA flags, vm_file and
  pgoff) before dropping the lock
- mfill_retry_state_changed() to compare the saved state with the state
  of the VMA acquired after retaking the locks
- mfill_retry_state_put() to release vm_file pinning.

Use DEFINE_FREE() cleanup to wrap mfill_retry_state_put() to avoid
complicating error handling paths in mfill_copy_folio_retry().

Link: https://lore.kernel.org/20260527184751.4147364-1-rppt@kernel.org
Link: https://lore.kernel.org/20260527184751.4147364-2-rppt@kernel.org
Fixes: 292411fda25b ("mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()")
Fixes: 6ab703034f14 ("userfaultfd: mfill_atomic(): remove retry logic")
Co-developed-by: Michael Bommarito 
Signed-off-by: Michael Bommarito 
Signed-off-by: Mike Rapoport (Microsoft) 
Suggested-by: Peter Xu 
Co-developed-by: David Carlier 
Signed-off-by: David Carlier 
Reviewed-by: Lorenzo Stoakes 
Cc: David Hildenbrand 
Cc: Liam R. Howlett 
Signed-off-by: Andrew Morton

mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()

2026-04-27T12:54:27+00:00

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call.
During this window, the VMA can be replaced with a different type (e.g.
hugetlb), making the caller's ops pointer stale.  Subsequent use of the
stale ops would dispatch into the wrong per-vma handlers.

Capture the VMA's ops via vma_uffd_ops() before dropping the lock and
compare against the current vma_uffd_ops() after re-acquiring it.
Return -EAGAIN if they differ so the operation can be retried.  This
avoids comparing against the caller's ops which may have been
overridden to anon_uffd_ops for MAP_PRIVATE file-backed mappings.

Link: https://lore.kernel.org/20260424183638.196227-1-devnexen@gmail.com
Fixes: 6ab703034f14 ("userfaultfd: mfill_atomic(): remove retry logic")
Reported-by: Usama Arif 
Closes: https://lore.kernel.org/all/20260410114809.3592720-1-usama.arif@linux.dev/
Signed-off-by: David Carlier 
Acked-by: Mike Rapoport (Microsoft) 
Cc: Jann Horn 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Peter Xu 
Signed-off-by: Andrew Morton

userfaultfd: mfill_atomic(): remove retry logic

2026-04-18T07:10:54+00:00

Since __mfill_atomic_pte() handles the retry for both anonymous and shmem,
there is no need to retry copying the date from the userspace in the loop
in mfill_atomic().

Drop the retry logic from mfill_atomic().

[rppt@kernel.org: remove safety measure of not returning ENOENT from _copy]
  Link: https://lore.kernel.org/ac5zcDUY8CFHr6Lw@kernel.org
Link: https://lore.kernel.org/20260402041156.1377214-12-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

2026-04-18T07:10:54+00:00

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them
in __mfill_atomic_pte() to add shmem folios to page cache and remove them
in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and
drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop
include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd,
make it static.

Link: https://lore.kernel.org/20260402041156.1377214-11-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

userfaultfd: introduce vm_uffd_ops->alloc_folio()

2026-04-18T07:10:54+00:00

and use it to refactor mfill_atomic_pte_zeroed_folio() and
mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform
almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly
introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Pass the ops structure from the callers to __mfill_atomic_pte() to later
allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Note, that the new ops method is called alloc_folio() rather than
folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

Link: https://lore.kernel.org/20260402041156.1377214-10-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

2026-04-18T07:10:54+00:00

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE
it needs to get a folio that already exists in the pagecache backing that
VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc()
method to 'struct vm_uffd_ops' that will return a folio if it exists in
the VMA's pagecache at given pgoff.

Implement get_folio_noalloc() method for shmem and slightly refactor
userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support
this new API.

Link: https://lore.kernel.org/20260402041156.1377214-9-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

userfaultfd: introduce vm_uffd_ops

2026-04-18T07:10:53+00:00

Current userfaultfd implementation works only with memory managed by core
MM: anonymous, shmem and hugetlb.

First, there is no fundamental reason to limit userfaultfd support only to
the core memory types and userfaults can be handled similarly to regular
page faults provided a VMA owner implements appropriate callbacks.

Second, historically various code paths were conditioned on
vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of
these conditions can be expressed as operations implemented by a
particular memory type.

Introduce vm_uffd_ops extension to vm_operations_struct that will delegate
memory type specific operations to a VMA owner.

Operations for anonymous memory are handled internally in userfaultfd
using anon_uffd_ops that implicitly assigned to anonymous VMAs.

Start with a single operation, ->can_userfault() that will verify that a
VMA meets requirements for userfaultfd support at registration time.

Implement that method for anonymous, shmem and hugetlb and move relevant
parts of vma_can_userfault() into the new callbacks.

[rppt@kernel.org: relocate VM_DROPPABLE test, per Tal]
  Link: https://lore.kernel.org/adffgfM5ANxtPIEF@kernel.org
Link: https://lore.kernel.org/20260402041156.1377214-8-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Cc: Tal Zussman 
Signed-off-by: Andrew Morton

userfaultfd: move vma_can_userfault out of line

2026-04-18T07:10:53+00:00

vma_can_userfault() has grown pretty big and it's not called on
performance critical path.

Move it out of line.

No functional changes.

Link: https://lore.kernel.org/20260402041156.1377214-7-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: David Hildenbrand (Red Hat) 
Reviewed-by: Liam R. Howlett 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton