linux.git/include/linux/userfaultfd_k.h, branch v7.2-rc1

userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks

2026-06-09T01:21:29+00:00

The VMA flags bitmap is a single word today: NUM_VMA_FLAG_BITS is
BITS_PER_LONG, so on 32-bit vma_flags_t holds only 32 bits.  (The bitmap
type exists so this can grow past BITS_PER_LONG later; until it does,
anything declared above the first word is out of range on 32-bit.) The bit
enum nevertheless declares some bits unconditionally above BITS_PER_LONG
-- VMA_UFFD_MINOR_BIT is 41, with VM_UFFD_MINOR == VM_NONE on 32-bit so no
VMA actually carries the bit.

__VMA_UFFD_FLAGS feeds VMA_UFFD_MINOR_BIT to mk_vma_flags()
unconditionally.  On 32-bit that becomes __set_bit(41, &one_long), a write
one word past the end of the single-word bitmap.  The compiler folds the
out-of-bounds store with wraparound (1UL << (41 % 32) == bit 9) into the
first word; bit 9 is already in __VMA_UFFD_FLAGS so the mask happens to
come out right today, but it is an out-of-bounds write all the same, and
any high-numbered bit whose mod-BITS_PER_LONG position is otherwise unused
would silently OR an extra bit into the mask.

Rather than feed bit numbers that may not exist on the current build to
mk_vma_flags(), build the mask from whole per-mode masks that collapse to
EMPTY_VMA_FLAGS when their feature is unavailable.  Add
mk_vma_flags_from_masks() for that, and define VMA_UFFD_MISSING / _WP /
_MINOR alongside the VM_UFFD_* flags, gating VMA_UFFD_MINOR on the same
config as VM_UFFD_MINOR (which implies 64BIT, where bit 41 fits).  An
out-of-range bit is then never materialised, on any arch, and the in-range
fast path stays a compile-time constant.

Link: https://lore.kernel.org/20260529172331.356655-7-kas@kernel.org
Fixes: 9ea35a25d51b ("mm: introduce VMA flags bitmap type")
Signed-off-by: Kiryl Shutsemau 
Reported-by: Sashiko AI review 
Suggested-by: Lorenzo Stoakes 
Reviewed-by: Lorenzo Stoakes 
Assisted-by: Claude:claude-opus-4-8
Cc: David Hildenbrand 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Peter Xu 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Balbir Singh 
Cc: 
Signed-off-by: Andrew Morton

userfaultfd: make functions that are not used outside uffd static

2026-06-04T21:45:03+00:00

After merging fs/userfaultfd.c into mm/userfaultfd.c, several functions
that were previously shared between the two files are now only used within
mm/userfaultfd.c.

Make them static and remove their declarations from
include/linux/userfaultfd_k.h.

Link: https://lore.kernel.org/20260523173759.3964908-3-rppt@kernel.org
Assisted-by: Copilot:claude-opus-4-6
Signed-off-by: Mike Rapoport (Microsoft) 
Cc: Al Viro 
Cc: Christian Brauner 
Cc: David Hildenbrand 
Cc: Jan Kara 
Cc: "Kirill A. Shutemov" 
Cc: Peter Xu 
Signed-off-by: Andrew Morton

shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops

2026-04-18T07:10:54+00:00

Add filemap_add() and filemap_remove() methods to vm_uffd_ops and use them
in __mfill_atomic_pte() to add shmem folios to page cache and remove them
in case of error.

Implement these methods in shmem along with vm_uffd_ops->alloc_folio() and
drop shmem_mfill_atomic_pte().

Since userfaultfd now does not reference any functions from shmem, drop
include if linux/shmem_fs.h from mm/userfaultfd.c

mfill_atomic_install_pte() is not used anywhere outside of mm/userfaultfd,
make it static.

Link: https://lore.kernel.org/20260402041156.1377214-11-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

userfaultfd: introduce vm_uffd_ops->alloc_folio()

2026-04-18T07:10:54+00:00

and use it to refactor mfill_atomic_pte_zeroed_folio() and
mfill_atomic_pte_copy().

mfill_atomic_pte_zeroed_folio() and mfill_atomic_pte_copy() perform
almost identical actions:
* allocate a folio
* update folio contents (either copy from userspace of fill with zeros)
* update page tables with the new folio

Split a __mfill_atomic_pte() helper that handles both cases and uses newly
introduced vm_uffd_ops->alloc_folio() to allocate the folio.

Pass the ops structure from the callers to __mfill_atomic_pte() to later
allow using anon_uffd_ops for MAP_PRIVATE mappings of file-backed VMAs.

Note, that the new ops method is called alloc_folio() rather than
folio_alloc() to avoid clash with alloc_tag macro folio_alloc().

Link: https://lore.kernel.org/20260402041156.1377214-10-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

shmem, userfaultfd: use a VMA callback to handle UFFDIO_CONTINUE

2026-04-18T07:10:54+00:00

When userspace resolves a page fault in a shmem VMA with UFFDIO_CONTINUE
it needs to get a folio that already exists in the pagecache backing that
VMA.

Instead of using shmem_get_folio() for that, add a get_folio_noalloc()
method to 'struct vm_uffd_ops' that will return a folio if it exists in
the VMA's pagecache at given pgoff.

Implement get_folio_noalloc() method for shmem and slightly refactor
userfaultfd's mfill_get_vma() and mfill_atomic_pte_continue() to support
this new API.

Link: https://lore.kernel.org/20260402041156.1377214-9-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: James Houghton 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

userfaultfd: introduce vm_uffd_ops

2026-04-18T07:10:53+00:00

Current userfaultfd implementation works only with memory managed by core
MM: anonymous, shmem and hugetlb.

First, there is no fundamental reason to limit userfaultfd support only to
the core memory types and userfaults can be handled similarly to regular
page faults provided a VMA owner implements appropriate callbacks.

Second, historically various code paths were conditioned on
vma_is_anonymous(), vma_is_shmem() and is_vm_hugetlb_page() and some of
these conditions can be expressed as operations implemented by a
particular memory type.

Introduce vm_uffd_ops extension to vm_operations_struct that will delegate
memory type specific operations to a VMA owner.

Operations for anonymous memory are handled internally in userfaultfd
using anon_uffd_ops that implicitly assigned to anonymous VMAs.

Start with a single operation, ->can_userfault() that will verify that a
VMA meets requirements for userfaultfd support at registration time.

Implement that method for anonymous, shmem and hugetlb and move relevant
parts of vma_can_userfault() into the new callbacks.

[rppt@kernel.org: relocate VM_DROPPABLE test, per Tal]
  Link: https://lore.kernel.org/adffgfM5ANxtPIEF@kernel.org
Link: https://lore.kernel.org/20260402041156.1377214-8-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: David Hildenbrand (Arm) 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Cc: Tal Zussman 
Signed-off-by: Andrew Morton

userfaultfd: move vma_can_userfault out of line

2026-04-18T07:10:53+00:00

vma_can_userfault() has grown pretty big and it's not called on
performance critical path.

Move it out of line.

No functional changes.

Link: https://lore.kernel.org/20260402041156.1377214-7-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Reviewed-by: David Hildenbrand (Red Hat) 
Reviewed-by: Liam R. Howlett 
Cc: Andrea Arcangeli 
Cc: Andrei Vagin 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Harry Yoo 
Cc: Harry Yoo (Oracle) 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Lorenzo Stoakes (Oracle) 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Nikita Kalyazin 
Cc: Oscar Salvador 
Cc: Paolo Bonzini 
Cc: Peter Xu 
Cc: Sean Christopherson 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: David Carlier 
Signed-off-by: Andrew Morton

mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t

2026-04-05T20:53:41+00:00

Update the vma_modify_flags() and vma_modify_flags_uffd() functions to
accept a vma_flags_t parameter rather than a vm_flags_t one, and propagate
the changes as needed to implement this change.

Also add vma_flags_reset_once() in replacement of vm_flags_reset_once(). We
still need to be careful here because we need to avoid tearing, so maintain
the assumption that the first system word set of flags are the only ones
that require protection from tearing, and retain this functionality.

We can copy the remainder of VMA flags above 64 bits normally. But
hopefully by the time that happens, we will have replaced the logic that
requires these WRITE_ONCE()'s with something else.

We also replace instances of vm_flags_reset() with a simple write of VMA
flags. We are no longer perform a number of checks, most notable of all the
VMA flags asserts becase:

1. We might be operating on a VMA that is not yet added to the tree.

2. We might be operating on a VMA that is now detached.

3. Really in all but core code, you should be using vma_desc_xxx().

4. Other VMA fields are manipulated with no such checks.

5. It'd be egregious to have to add variants of flag functions just to
   account for cases such as the above, especially when we don't do so for
   other VMA fields. Drivers are the problematic cases and why it was
   especially important (and also for debug as VMA locks were introduced),
   the mmap_prepare work is solving this generally.

Additionally, we can fairly safely assume by this point the soft dirty
flags are being set correctly, so it's reasonable to drop this also.

Finally, update the VMA tests to reflect this.

Link: https://lkml.kernel.org/r/51afbb2b8c3681003cc7926647e37335d793836e.1774034900.git.ljs@kernel.org
Signed-off-by: Lorenzo Stoakes (Oracle) 
Acked-by: Vlastimil Babka (SUSE) 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Al Viro 
Cc: Anton Ivanov 
Cc: "Borislav Petkov (AMD)" 
Cc: Catalin Marinas 
Cc: Chengming Zhou 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: David Hildenbrand 
Cc: Dinh Nguyen 
Cc: Heiko Carstens 
Cc: "H. Peter Anvin" 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Johannes Berg 
Cc: Kees Cook 
Cc: Liam Howlett 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: Ondrej Mosnacek 
Cc: Palmer Dabbelt 
Cc: Paul Moore 
Cc: Pedro Falcato 
Cc: Richard Weinberger 
Cc: Russell King 
Cc: Stephen Smalley 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: WANG Xuerui 
Cc: Will Deacon 
Cc: xu xin 
Signed-off-by: Andrew Morton

mm: userfaultfd: add pgtable_supports_uffd_wp()

2025-11-24T23:08:54+00:00

Some platforms can customize the PTE/PMD entry uffd-wp bit making it
unavailable even if the architecture provides the resource.  This patch
adds a macro API pgtable_supports_uffd_wp() that allows architectures to
define their specific implementations to check if the uffd-wp bit is
available on which device the kernel is running.

Also this patch is removing "ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP" and
"ifdef CONFIG_PTE_MARKER_UFFD_WP" in favor of pgtable_supports_uffd_wp()
and uffd_supports_wp_marker() checks respectively that default to
IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP) and
"IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP) &&
IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP)" if not overridden by the
architecture, no change in behavior is expected.

Link: https://lkml.kernel.org/r/20251113072806.795029-3-zhangchunyan@iscas.ac.cn
Signed-off-by: Chunyan Zhang 
Acked-by: David Hildenbrand 
Cc: Albert Ou 
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Al Viro 
Cc: Andrew Jones 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Christian Brauner 
Cc: Conor Dooley 
Cc: Conor Dooley 
Cc: Deepak Gupta 
Cc: Jan Kara 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Peter Xu 
Cc: Rob Herring 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Yuanchu Xie 
Signed-off-by: Andrew Morton

mm: avoid unnecessary uses of is_swap_pte()

2025-11-24T23:08:50+00:00

There's an established convention in the kernel that we treat PTEs as
containing swap entries (and the unfortunately named non-swap swap
entries) should they be neither empty (i.e.  pte_none() evaluating true)
nor present (i.e.  pte_present() evaluating true).

However, there is some inconsistency in how this is applied, as we also
have the is_swap_pte() helper which explicitly performs this check:

	/* check whether a pte points to a swap entry */
	static inline int is_swap_pte(pte_t pte)
	{
		return !pte_none(pte) && !pte_present(pte);
	}

As this represents a predicate, and it's logical to assume that in order
to establish that a PTE entry can correctly be manipulated as a
swap/non-swap entry, this predicate seems as if it must first be checked.

But we instead, we far more often utilise the established convention of
checking pte_none() / pte_present() before operating on entries as if they
were swap/non-swap.

This patch works towards correcting this inconsistency by removing all
uses of is_swap_pte() where we are already in a position where we perform
pte_none()/pte_present() checks anyway or otherwise it is clearly logical
to do so.

We also take advantage of the fact that pte_swp_uffd_wp() is only set on
swap entries.

Additionally, update comments referencing to is_swap_pte() and
non_swap_entry().

No functional change intended.

Link: https://lkml.kernel.org/r/17fd6d7f46a846517fd455fadd640af47fcd7c55.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Zi Yan 
Reviewed-by: Vlastimil Babka 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Signed-off-by: Andrew Morton