linux-stable.git/mm/huge_memory.c, branch v5.7

userfaultfd: wp: support swap and page migration

2020-04-07T17:43:39+00:00

For either swap and page migration, we all use the bit 2 of the entry to
identify whether this entry is uffd write-protected.  It plays a similar
role as the existing soft dirty bit in swap entries but only for keeping
the uffd-wp tracking for a specific PTE/PMD.

Something special here is that when we want to recover the uffd-wp bit
from a swap/migration entry to the PTE bit we'll also need to take care of
the _PAGE_RW bit and make sure it's cleared, otherwise even with the
_PAGE_UFFD_WP bit we can't trap it at all.

In change_pte_range() we do nothing for uffd if the PTE is a swap entry.
That can lead to data mismatch if the page that we are going to write
protect is swapped out when sending the UFFDIO_WRITEPROTECT.  This patch
also applies/removes the uffd-wp bit even for the swap entries.

Signed-off-by: Peter Xu 
Signed-off-by: Andrew Morton 
Cc: Andrea Arcangeli 
Cc: Bobby Powers 
Cc: Brian Geffon 
Cc: David Hildenbrand 
Cc: Denis Plotnikov 
Cc: "Dr . David Alan Gilbert" 
Cc: Hugh Dickins 
Cc: Jerome Glisse 
Cc: Johannes Weiner 
Cc: "Kirill A . Shutemov" 
Cc: Martin Cracauer 
Cc: Marty McFadden 
Cc: Maya Gokhale 
Cc: Mel Gorman 
Cc: Mike Kravetz 
Cc: Mike Rapoport 
Cc: Pavel Emelyanov 
Cc: Rik van Riel 
Cc: Shaohua Li 
Link: http://lkml.kernel.org/r/20200220163112.11409-11-peterx@redhat.com
Signed-off-by: Linus Torvalds

userfaultfd: wp: drop _PAGE_UFFD_WP properly when fork

2020-04-07T17:43:39+00:00

UFFD_EVENT_FORK support for uffd-wp should be already there, except that
we should clean the uffd-wp bit if uffd fork event is not enabled.  Detect
that to avoid _PAGE_UFFD_WP being set even if the VMA is not being tracked
by VM_UFFD_WP.  Do this for both small PTEs and huge PMDs.

Signed-off-by: Peter Xu 
Signed-off-by: Andrew Morton 
Reviewed-by: Jerome Glisse 
Reviewed-by: Mike Rapoport 
Cc: Andrea Arcangeli 
Cc: Bobby Powers 
Cc: Brian Geffon 
Cc: David Hildenbrand 
Cc: Denis Plotnikov 
Cc: "Dr . David Alan Gilbert" 
Cc: Hugh Dickins 
Cc: Johannes Weiner 
Cc: "Kirill A . Shutemov" 
Cc: Martin Cracauer 
Cc: Marty McFadden 
Cc: Maya Gokhale 
Cc: Mel Gorman 
Cc: Mike Kravetz 
Cc: Pavel Emelyanov 
Cc: Rik van Riel 
Cc: Shaohua Li 
Link: http://lkml.kernel.org/r/20200220163112.11409-9-peterx@redhat.com
Signed-off-by: Linus Torvalds

userfaultfd: wp: apply _PAGE_UFFD_WP bit

2020-04-07T17:43:39+00:00

Firstly, introduce two new flags MM_CP_UFFD_WP[_RESOLVE] for
change_protection() when used with uffd-wp and make sure the two new flags
are exclusively used.  Then,

  - For MM_CP_UFFD_WP: apply the _PAGE_UFFD_WP bit and remove _PAGE_RW
    when a range of memory is write protected by uffd

  - For MM_CP_UFFD_WP_RESOLVE: remove the _PAGE_UFFD_WP bit and recover
    _PAGE_RW when write protection is resolved from userspace

And use this new interface in mwriteprotect_range() to replace the old
MM_CP_DIRTY_ACCT.

Do this change for both PTEs and huge PMDs.  Then we can start to identify
which PTE/PMD is write protected by general (e.g., COW or soft dirty
tracking), and which is for userfaultfd-wp.

Since we should keep the _PAGE_UFFD_WP when doing pte_modify(), add it
into _PAGE_CHG_MASK as well.  Meanwhile, since we have this new bit, we
can be even more strict when detecting uffd-wp page faults in either
do_wp_page() or wp_huge_pmd().

After we're with _PAGE_UFFD_WP, a special case is when a page is both
protected by the general COW logic and also userfault-wp.  Here the
userfault-wp will have higher priority and will be handled first.  Only
after the uffd-wp bit is cleared on the PTE/PMD will we continue to handle
the general COW.  These are the steps on what will happen with such a
page:

  1. CPU accesses write protected shared page (so both protected by
     general COW and uffd-wp), blocked by uffd-wp first because in
     do_wp_page we'll handle uffd-wp first, so it has higher priority
     than general COW.

  2. Uffd service thread receives the request, do UFFDIO_WRITEPROTECT
     to remove the uffd-wp bit upon the PTE/PMD.  However here we
     still keep the write bit cleared.  Notify the blocked CPU.

  3. The blocked CPU resumes the page fault process with a fault
     retry, during retry it'll notice it was not with the uffd-wp bit
     this time but it is still write protected by general COW, then
     it'll go though the COW path in the fault handler, copy the page,
     apply write bit where necessary, and retry again.

  4. The CPU will be able to access this page with write bit set.

Suggested-by: Andrea Arcangeli 
Signed-off-by: Peter Xu 
Signed-off-by: Andrew Morton 
Cc: Brian Geffon 
Cc: Pavel Emelyanov 
Cc: Mike Kravetz 
Cc: David Hildenbrand 
Cc: Martin Cracauer 
Cc: Mel Gorman 
Cc: Bobby Powers 
Cc: Mike Rapoport 
Cc: "Kirill A . Shutemov" 
Cc: Maya Gokhale 
Cc: Johannes Weiner 
Cc: Marty McFadden 
Cc: Denis Plotnikov 
Cc: Hugh Dickins 
Cc: "Dr . David Alan Gilbert" 
Cc: Jerome Glisse 
Cc: Rik van Riel 
Cc: Shaohua Li 
Link: http://lkml.kernel.org/r/20200220163112.11409-8-peterx@redhat.com
Signed-off-by: Linus Torvalds

mm: merge parameters for change_protection()

2020-04-07T17:43:39+00:00

change_protection() was used by either the NUMA or mprotect() code,
there's one parameter for each of the callers (dirty_accountable and
prot_numa).  Further, these parameters are passed along the calls:

  - change_protection_range()
  - change_p4d_range()
  - change_pud_range()
  - change_pmd_range()
  - ...

Now we introduce a flag for change_protect() and all these helpers to
replace these parameters.  Then we can avoid passing multiple parameters
multiple times along the way.

More importantly, it'll greatly simplify the work if we want to introduce
any new parameters to change_protection().  In the follow up patches, a
new parameter for userfaultfd write protection will be introduced.

No functional change at all.

Signed-off-by: Peter Xu 
Signed-off-by: Andrew Morton 
Reviewed-by: Jerome Glisse 
Cc: Andrea Arcangeli 
Cc: Bobby Powers 
Cc: Brian Geffon 
Cc: David Hildenbrand 
Cc: Denis Plotnikov 
Cc: "Dr . David Alan Gilbert" 
Cc: Hugh Dickins 
Cc: Johannes Weiner 
Cc: "Kirill A . Shutemov" 
Cc: Martin Cracauer 
Cc: Marty McFadden 
Cc: Maya Gokhale 
Cc: Mel Gorman 
Cc: Mike Kravetz 
Cc: Mike Rapoport 
Cc: Pavel Emelyanov 
Cc: Rik van Riel 
Cc: Shaohua Li 
Link: http://lkml.kernel.org/r/20200220163112.11409-7-peterx@redhat.com
Signed-off-by: Linus Torvalds

mm: remove CONFIG_TRANSPARENT_HUGE_PAGECACHE

2020-04-07T17:43:38+00:00

Commit e496cf3d7821 ("thp: introduce CONFIG_TRANSPARENT_HUGE_PAGECACHE")
notes that it should be reverted when the PowerPC problem was fixed.  The
commit fixing the PowerPC problem (953c66c2b22a) did not revert the
commit; instead setting CONFIG_TRANSPARENT_HUGE_PAGECACHE to the same as
CONFIG_TRANSPARENT_HUGEPAGE.  Checking with Kirill and Aneesh, this was an
oversight, so remove the Kconfig symbol and undo the work of commit
e496cf3d7821.

Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton 
Acked-by: Kirill A. Shutemov 
Cc: Aneesh Kumar K.V 
Cc: Christoph Hellwig 
Cc: Pankaj Gupta 
Link: http://lkml.kernel.org/r/20200318140253.6141-6-willy@infradead.org
Signed-off-by: Linus Torvalds

mm, thp: track fallbacks due to failed memcg charges separately

2020-04-07T17:43:38+00:00

The thp_fault_fallback and thp_file_fallback vmstats are incremented if
either the hugepage allocation fails through the page allocator or the
hugepage charge fails through mem cgroup.

This patch leaves this field untouched but adds two new fields,
thp_{fault,file}_fallback_charge, which is incremented only when the mem
cgroup charge fails.

This distinguishes between attempted hugepage allocations that fail due to
fragmentation (or low memory conditions) and those that fail due to mem
cgroup limits.  That can be used to determine the impact of fragmentation
on the system by excluding faults that failed due to memcg usage.

Signed-off-by: David Rientjes 
Signed-off-by: Andrew Morton 
Reviewed-by: Yang Shi 
Acked-by: Kirill A. Shutemov 
Cc: Mike Rapoport 
Cc: Jeremy Cline 
Cc: Andrea Arcangeli 
Cc: Mike Kravetz 
Cc: Michal Hocko 
Cc: Vlastimil Babka 
Link: http://lkml.kernel.org/r/alpine.DEB.2.21.2003061422070.7412@chino.kir.corp.google.com
Signed-off-by: Linus Torvalds

Merge tag 'drm-next-2020-04-03-1' of git://anongit.freedesktop.org/drm/drm

2020-04-04T18:58:55+00:00

Pull drm hugepage support from Dave Airlie:
 "This adds support for hugepages to TTM and has been tested with the
  vmwgfx drivers, though I expect other drivers to start using it"

* tag 'drm-next-2020-04-03-1' of git://anongit.freedesktop.org/drm/drm:
  drm/vmwgfx: Hook up the helpers to align buffer objects
  drm/vmwgfx: Introduce a huge page aligning TTM range manager
  drm: Add a drm_get_unmapped_area() helper
  drm/vmwgfx: Support huge page faults
  drm/ttm, drm/vmwgfx: Support huge TTM pagefaults
  mm: Add vmf_insert_pfn_xxx_prot() for huge page-table entries
  mm: Split huge pages on write-notify or COW
  mm: Introduce vma_is_special_huge
  fs: Constify vma argument to vma_is_dax

mm/gup: track FOLL_PIN pages

2020-04-02T16:35:27+00:00

Add tracking of pages that were pinned via FOLL_PIN.  This tracking is
implemented via overloading of page->_refcount: pins are added by adding
GUP_PIN_COUNTING_BIAS (1024) to the refcount.  This provides a fuzzy
indication of pinning, and it can have false positives (and that's OK).
Please see the pre-existing Documentation/core-api/pin_user_pages.rst for
details.

As mentioned in pin_user_pages.rst, callers who effectively set FOLL_PIN
(typically via pin_user_pages*()) are required to ultimately free such
pages via unpin_user_page().

Please also note the limitation, discussed in pin_user_pages.rst under the
"TODO: for 1GB and larger huge pages" section.  (That limitation will be
removed in a following patch.)

The effect of a FOLL_PIN flag is similar to that of FOLL_GET, and may be
thought of as "FOLL_GET for DIO and/or RDMA use".

Pages that have been pinned via FOLL_PIN are identifiable via a new
function call:

   bool page_maybe_dma_pinned(struct page *page);

What to do in response to encountering such a page, is left to later
patchsets. There is discussion about this in [1], [2], [3], and [4].

This also changes a BUG_ON(), to a WARN_ON(), in follow_page_mask().

[1] Some slow progress on get_user_pages() (Apr 2, 2019):
    https://lwn.net/Articles/784574/
[2] DMA and get_user_pages() (LPC: Dec 12, 2018):
    https://lwn.net/Articles/774411/
[3] The trouble with get_user_pages() (Apr 30, 2018):
    https://lwn.net/Articles/753027/
[4] LWN kernel index: get_user_pages():
    https://lwn.net/Kernel/Index/#Memory_management-get_user_pages

[jhubbard@nvidia.com: add kerneldoc]
  Link: http://lkml.kernel.org/r/20200307021157.235726-1-jhubbard@nvidia.com
[imbrenda@linux.ibm.com: if pin fails, we need to unpin, a simple put_page will not be enough]
  Link: http://lkml.kernel.org/r/20200306132537.783769-2-imbrenda@linux.ibm.com
[akpm@linux-foundation.org: fix put_compound_head defined but not used]
Suggested-by: Jan Kara 
Suggested-by: Jérôme Glisse 
Signed-off-by: John Hubbard 
Signed-off-by: Claudio Imbrenda 
Signed-off-by: Andrew Morton 
Reviewed-by: Jan Kara 
Acked-by: Kirill A. Shutemov 
Cc: Ira Weiny 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Al Viro 
Cc: Christoph Hellwig 
Cc: Dan Williams 
Cc: Dave Chinner 
Cc: Jason Gunthorpe 
Cc: Jonathan Corbet 
Cc: Michal Hocko 
Cc: Mike Kravetz 
Cc: Shuah Khan 
Cc: Vlastimil Babka 
Link: http://lkml.kernel.org/r/20200211001536.1027652-7-jhubbard@nvidia.com
Signed-off-by: Linus Torvalds

mm: Add vmf_insert_pfn_xxx_prot() for huge page-table entries

2020-03-24T17:48:09+00:00

For graphics drivers needing to modify the page-protection, add
huge page-table entries counterparts to vmf_insert_pfn_prot().

Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: "Matthew Wilcox (Oracle)" 
Cc: "Kirill A. Shutemov" 
Cc: Ralph Campbell 
Cc: "Jérôme Glisse" 
Cc: "Christian König" 
Cc: Dan Williams 
Signed-off-by: Thomas Hellstrom (VMware) 
Acked-by: Christian König 
Acked-by: Andrew Morton

mm: Introduce vma_is_special_huge

2020-03-24T17:47:17+00:00

For VM_PFNMAP and VM_MIXEDMAP vmas that want to support transhuge pages
and -page table entries, introduce vma_is_special_huge() that takes the
same codepaths as vma_is_dax().

The use of "special" follows the definition in memory.c, vm_normal_page():
"Special" mappings do not wish to be associated with a "struct page"
(either it doesn't exist, or it exists but they don't want to touch it)

For PAGE_SIZE pages, "special" is determined per page table entry to be
able to deal with COW pages. But since we don't have huge COW pages,
we can classify a vma as either "special huge" or "normal huge".

Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: "Matthew Wilcox (Oracle)" 
Cc: "Kirill A. Shutemov" 
Cc: Ralph Campbell 
Cc: "Jérôme Glisse" 
Cc: "Christian König" 
Cc: Dan Williams 
Signed-off-by: Thomas Hellstrom (VMware) 
Acked-by: Christian König 
Acked-by: Andrew Morton