linux.git/mm/userfaultfd.c, branch v5.13

userfaultfd: hugetlbfs: fix new flag usage in error path

2021-05-23T01:09:07+00:00

In commit d6995da31122 ("hugetlb: use page.private for hugetlb specific
page flags") the use of PagePrivate to indicate a reservation count
should be restored at free time was changed to the hugetlb specific flag
HPageRestoreReserve.  Changes to a userfaultfd error path as well as a
VM_BUG_ON() in remove_inode_hugepages() were overlooked.

Users could see incorrect hugetlb reserve counts if they experience an
error with a UFFDIO_COPY operation.  Specifically, this would be the
result of an unlikely copy_huge_page_from_user error.  There is not an
increased chance of hitting the VM_BUG_ON.

Link: https://lkml.kernel.org/r/20210521233952.236434-1-mike.kravetz@oracle.com
Fixes: d6995da31122 ("hugetlb: use page.private for hugetlb specific page flags")
Signed-off-by: Mike Kravetz 
Reviewed-by: Mina Almasry 
Cc: Oscar Salvador 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: David Hildenbrand 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Mina Almasry 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

userfaultfd: add UFFDIO_CONTINUE ioctl

2021-05-05T18:27:22+00:00

This ioctl is how userspace ought to resolve "minor" userfaults.  The
idea is, userspace is notified that a minor fault has occurred.  It
might change the contents of the page using its second non-UFFD mapping,
or not.  Then, it calls UFFDIO_CONTINUE to tell the kernel "I have
ensured the page contents are correct, carry on setting up the mapping".

Note that it doesn't make much sense to use UFFDIO_{COPY,ZEROPAGE} for
MINOR registered VMAs.  ZEROPAGE maps the VMA to the zero page; but in
the minor fault case, we already have some pre-existing underlying page.
Likewise, UFFDIO_COPY isn't useful if we have a second non-UFFD mapping.
We'd just use memcpy() or similar instead.

It turns out hugetlb_mcopy_atomic_pte() already does very close to what
we want, if an existing page is provided via `struct page **pagep`.  We
already special-case the behavior a bit for the UFFDIO_ZEROPAGE case, so
just extend that design: add an enum for the three modes of operation,
and make the small adjustments needed for the MCOPY_ATOMIC_CONTINUE
case.  (Basically, look up the existing page, and avoid adding the
existing page to the page cache or calling set_page_huge_active() on
it.)

Link: https://lkml.kernel.org/r/20210301222728.176417-5-axelrasmussen@google.com
Signed-off-by: Axel Rasmussen 
Reviewed-by: Peter Xu 
Cc: Adam Ruprecht 
Cc: Alexander Viro 
Cc: Alexey Dobriyan 
Cc: Andrea Arcangeli 
Cc: Anshuman Khandual 
Cc: Cannon Matthews 
Cc: Catalin Marinas 
Cc: Chinwen Chang 
Cc: David Rientjes 
Cc: "Dr . David Alan Gilbert" 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Jerome Glisse 
Cc: Kirill A. Shutemov 
Cc: Lokesh Gidra 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: "Michal Koutn" 
Cc: Michel Lespinasse 
Cc: Mike Kravetz 
Cc: Mike Rapoport 
Cc: Mina Almasry 
Cc: Nicholas Piggin 
Cc: Oliver Upton 
Cc: Shaohua Li 
Cc: Shawn Anastasio 
Cc: Steven Price 
Cc: Steven Rostedt 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

hugetlb: pass vma into huge_pte_alloc() and huge_pmd_share()

2021-05-05T18:27:20+00:00

Patch series "hugetlb: Disable huge pmd unshare for uffd-wp", v4.

This series tries to disable huge pmd unshare of hugetlbfs backed memory
for uffd-wp.  Although uffd-wp of hugetlbfs is still during rfc stage,
the idea of this series may be needed for multiple tasks (Axel's uffd
minor fault series, and Mike's soft dirty series), so I picked it out
from the larger series.

This patch (of 4):

It is a preparation work to be able to behave differently in the per
architecture huge_pte_alloc() according to different VMA attributes.

Pass it deeper into huge_pmd_share() so that we can avoid the find_vma() call.

[peterx@redhat.com: build fix]
  Link: https://lkml.kernel.org/r/20210304164653.GB397383@xz-x1Link: https://lkml.kernel.org/r/20210218230633.15028-1-peterx@redhat.com

Link: https://lkml.kernel.org/r/20210218230633.15028-2-peterx@redhat.com
Signed-off-by: Peter Xu 
Suggested-by: Mike Kravetz 
Cc: Adam Ruprecht 
Cc: Alexander Viro 
Cc: Alexey Dobriyan 
Cc: Andrea Arcangeli 
Cc: Anshuman Khandual 
Cc: Axel Rasmussen 
Cc: Cannon Matthews 
Cc: Catalin Marinas 
Cc: Chinwen Chang 
Cc: David Rientjes 
Cc: "Dr . David Alan Gilbert" 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Jerome Glisse 
Cc: Kirill A. Shutemov 
Cc: Lokesh Gidra 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: "Michal Koutn" 
Cc: Michel Lespinasse 
Cc: Mike Rapoport 
Cc: Mina Almasry 
Cc: Nicholas Piggin 
Cc: Oliver Upton 
Cc: Shaohua Li 
Cc: Shawn Anastasio 
Cc: Steven Price 
Cc: Steven Rostedt 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/vmscan: protect the workingset on anonymous LRU

2020-08-12T17:57:55+00:00

In current implementation, newly created or swap-in anonymous page is
started on active list.  Growing active list results in rebalancing
active/inactive list so old pages on active list are demoted to inactive
list.  Hence, the page on active list isn't protected at all.

Following is an example of this situation.

Assume that 50 hot pages on active list.  Numbers denote the number of
pages on active/inactive list (active | inactive).

1. 50 hot pages on active list
50(h) | 0

2. workload: 50 newly created (used-once) pages
50(uo) | 50(h)

3. workload: another 50 newly created (used-once) pages
50(uo) | 50(uo), swap-out 50(h)

This patch tries to fix this issue.  Like as file LRU, newly created or
swap-in anonymous pages will be inserted to the inactive list.  They are
promoted to active list if enough reference happens.  This simple
modification changes the above example as following.

1. 50 hot pages on active list
50(h) | 0

2. workload: 50 newly created (used-once) pages
50(h) | 50(uo)

3. workload: another 50 newly created (used-once) pages
50(h) | 50(uo), swap-out 50(uo)

As you can see, hot pages on active list would be protected.

Note that, this implementation has a drawback that the page cannot be
promoted and will be swapped-out if re-access interval is greater than the
size of inactive list but less than the size of total(active+inactive).
To solve this potential issue, following patch will apply workingset
detection similar to the one that's already applied to file LRU.

Signed-off-by: Joonsoo Kim 
Signed-off-by: Andrew Morton 
Acked-by: Johannes Weiner 
Acked-by: Vlastimil Babka 
Cc: Hugh Dickins 
Cc: Matthew Wilcox 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Minchan Kim 
Link: http://lkml.kernel.org/r/1595490560-15117-3-git-send-email-iamjoonsoo.kim@lge.com
Signed-off-by: Linus Torvalds

mmap locking API: convert mmap_sem comments

2020-06-09T16:39:14+00:00

Convert comments that reference mmap_sem to reference mmap_lock instead.

[akpm@linux-foundation.org: fix up linux-next leftovers]
[akpm@linux-foundation.org: s/lockaphore/lock/, per Vlastimil]
[akpm@linux-foundation.org: more linux-next fixups, per Michel]

Signed-off-by: Michel Lespinasse 
Signed-off-by: Andrew Morton 
Reviewed-by: Vlastimil Babka 
Reviewed-by: Daniel Jordan 
Cc: Davidlohr Bueso 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: Jerome Glisse 
Cc: John Hubbard 
Cc: Laurent Dufour 
Cc: Liam Howlett 
Cc: Matthew Wilcox 
Cc: Peter Zijlstra 
Cc: Ying Han 
Link: http://lkml.kernel.org/r/20200520052908.204642-13-walken@google.com
Signed-off-by: Linus Torvalds

mmap locking API: use coccinelle to convert mmap_sem rwsem call sites

2020-06-09T16:39:14+00:00

This change converts the existing mmap_sem rwsem calls to use the new mmap
locking API instead.

The change is generated using coccinelle with the following rule:

// spatch --sp-file mmap_lock_api.cocci --in-place --include-headers --dir .

@@
expression mm;
@@
(
-init_rwsem
+mmap_init_lock
|
-down_write
+mmap_write_lock
|
-down_write_killable
+mmap_write_lock_killable
|
-down_write_trylock
+mmap_write_trylock
|
-up_write
+mmap_write_unlock
|
-downgrade_write
+mmap_write_downgrade
|
-down_read
+mmap_read_lock
|
-down_read_killable
+mmap_read_lock_killable
|
-down_read_trylock
+mmap_read_trylock
|
-up_read
+mmap_read_unlock
)
-(&mm->mmap_sem)
+(mm)

Signed-off-by: Michel Lespinasse 
Signed-off-by: Andrew Morton 
Reviewed-by: Daniel Jordan 
Reviewed-by: Laurent Dufour 
Reviewed-by: Vlastimil Babka 
Cc: Davidlohr Bueso 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: Jerome Glisse 
Cc: John Hubbard 
Cc: Liam Howlett 
Cc: Matthew Wilcox 
Cc: Peter Zijlstra 
Cc: Ying Han 
Link: http://lkml.kernel.org/r/20200520052908.204642-5-walken@google.com
Signed-off-by: Linus Torvalds

mm: memcontrol: delete unused lrucare handling

2020-06-04T03:09:48+00:00

Swapin faults were the last event to charge pages after they had already
been put on the LRU list.  Now that we charge directly on swapin, the
lrucare portion of the charge code is unused.

Signed-off-by: Johannes Weiner 
Signed-off-by: Andrew Morton 
Reviewed-by: Joonsoo Kim 
Cc: Alex Shi 
Cc: Hugh Dickins 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Balbir Singh 
Cc: Shakeel Butt 
Link: http://lkml.kernel.org/r/20200508183105.225460-19-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds

mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API

2020-06-04T03:09:48+00:00

With the page->mapping requirement gone from memcg, we can charge anon and
file-thp pages in one single step, right after they're allocated.

This removes two out of three API calls - especially the tricky commit
step that needed to happen at just the right time between when the page is
"set up" and when it's "published" - somewhat vague and fluid concepts
that varied by page type.  All we need is a freshly allocated page and a
memcg context to charge.

v2: prevent double charges on pre-allocated hugepages in khugepaged

[hannes@cmpxchg.org: Fix crash - *hpage could be ERR_PTR instead of NULL]
  Link: http://lkml.kernel.org/r/20200512215813.GA487759@cmpxchg.org
Signed-off-by: Johannes Weiner 
Signed-off-by: Andrew Morton 
Reviewed-by: Joonsoo Kim 
Cc: Alex Shi 
Cc: Hugh Dickins 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Shakeel Butt 
Cc: Balbir Singh 
Cc: Qian Cai 
Link: http://lkml.kernel.org/r/20200508183105.225460-13-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds

mm: memcontrol: switch to native NR_ANON_MAPPED counter

2020-06-04T03:09:47+00:00

Memcg maintains a private MEMCG_RSS counter.  This divergence from the
generic VM accounting means unnecessary code overhead, and creates a
dependency for memcg that page->mapping is set up at the time of charging,
so that page types can be told apart.

Convert the generic accounting sites to mod_lruvec_page_state and friends
to maintain the per-cgroup vmstat counter of NR_ANON_MAPPED.  We use
lock_page_memcg() to stabilize page->mem_cgroup during rmap changes, the
same way we do for NR_FILE_MAPPED.

With the previous patch removing MEMCG_CACHE and the private NR_SHMEM
counter, this patch finally eliminates the need to have page->mapping set
up at charge time.  However, we need to have page->mem_cgroup set up by
the time rmap runs and does the accounting, so switch the commit and the
rmap callbacks around.

v2: fix temporary accounting bug by switching rmap<->commit (Joonsoo)

Signed-off-by: Johannes Weiner 
Signed-off-by: Andrew Morton 
Cc: Alex Shi 
Cc: Hugh Dickins 
Cc: Joonsoo Kim 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Shakeel Butt 
Cc: Balbir Singh 
Link: http://lkml.kernel.org/r/20200508183105.225460-11-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds

mm: memcontrol: drop @compound parameter from memcg charging API

2020-06-04T03:09:47+00:00

The memcg charging API carries a boolean @compound parameter that tells
whether the page we're dealing with is a hugepage.
mem_cgroup_commit_charge() has another boolean @lrucare that indicates
whether the page needs LRU locking or not while charging.  The majority of
callsites know those parameters at compile time, which results in a lot of
naked "false, false" argument lists.  This makes for cryptic code and is a
breeding ground for subtle mistakes.

Thankfully, the huge page state can be inferred from the page itself and
doesn't need to be passed along.  This is safe because charging completes
before the page is published and somebody may split it.

Simplify the callsites by removing @compound, and let memcg infer the
state by using hpage_nr_pages() unconditionally.  That function does
PageTransHuge() to identify huge pages, which also helpfully asserts that
nobody passes in tail pages by accident.

The following patches will introduce a new charging API, best not to carry
over unnecessary weight.

Signed-off-by: Johannes Weiner 
Signed-off-by: Andrew Morton 
Reviewed-by: Alex Shi 
Reviewed-by: Joonsoo Kim 
Reviewed-by: Shakeel Butt 
Cc: Hugh Dickins 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Balbir Singh 
Link: http://lkml.kernel.org/r/20200508183105.225460-4-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds