linux.git/include/linux/hugetlb.h, branch v6.6-rc2

mm: convert free_huge_page() to free_huge_folio()

2023-08-21T21:28:43+00:00

Pass a folio instead of the head page to save a few instructions.  Update
the documentation, at least in English.

Link: https://lkml.kernel.org/r/20230816151201.3655946-4-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Sidhartha Kumar 
Cc: Yanteng Si 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Signed-off-by: Andrew Morton

mm: call free_huge_page() directly

2023-08-21T21:28:43+00:00

Indirect calls are expensive, thanks to Spectre.  Call free_huge_page()
directly if the folio belongs to hugetlb.

Link: https://lkml.kernel.org/r/20230816151201.3655946-3-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: David Hildenbrand 
Cc: Jens Axboe 
Cc: Sidhartha Kumar 
Cc: Yanteng Si 
Signed-off-by: Andrew Morton

mm/hugetlb: get rid of page_hstate()

2023-08-18T17:12:39+00:00

Convert the last page_hstate() user to use folio_hstate() so page_hstate()
can be safely removed.

Link: https://lkml.kernel.org/r/20230719184145.301911-1-sidhartha.kumar@oracle.com
Signed-off-by: Sidhartha Kumar 
Reviewed-by: Mike Kravetz 
Cc: Matthew Wilcox (Oracle) 
Cc: Muchun Song 
Signed-off-by: Andrew Morton

mm/hwpoison: check if a raw page in a hugetlb folio is raw HWPOISON

2023-08-18T17:12:26+00:00

Add the functionality, is_raw_hwpoison_page_in_hugepage, to tell if a raw
page in a hugetlb folio is HWPOISON.  This functionality relies on
RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list
becomes meaningless.

is_raw_hwpoison_page_in_hugepage holds mf_mutex in order to synchronize
with folio_set_hugetlb_hwpoison and folio_free_raw_hwp who iterate,
insert, or delete entry in raw_hwp_list.  llist itself doesn't ensure
insertion and removal are synchornized with the llist_for_each_entry used
by is_raw_hwpoison_page_in_hugepage (unless iterated entries are already
deleted from the list).  Caller can minimize the overhead of lock cycles
by first checking HWPOISON flag of the folio.

Exports this functionality to be immediately used in the read operation
for hugetlbfs.

Link: https://lkml.kernel.org/r/20230713001833.3778937-3-jiaqiyan@google.com
Signed-off-by: Jiaqi Yan 
Reviewed-by: Mike Kravetz 
Reviewed-by: Naoya Horiguchi 
Reviewed-by: Miaohe Lin 
Cc: James Houghton 
Cc: Muchun Song 
Cc: Yang Shi 
Signed-off-by: Andrew Morton

mm/gup: retire follow_hugetlb_page()

2023-08-18T17:12:04+00:00

Now __get_user_pages() should be well prepared to handle thp completely,
as long as hugetlb gup requests even without the hugetlb's special path.

Time to retire follow_hugetlb_page().

Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal.

Link: https://lkml.kernel.org/r/20230628215310.73782-7-peterx@redhat.com
Signed-off-by: Peter Xu 
Acked-by: David Hildenbrand 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Kirill A . Shutemov 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox 
Cc: Mike Kravetz 
Cc: Mike Rapoport (IBM) 
Cc: Vlastimil Babka 
Cc: Yang Shi 
Signed-off-by: Andrew Morton

mm/hugetlb: add page_mask for hugetlb_follow_page_mask()

2023-08-18T17:12:03+00:00

follow_page() doesn't need it, but we'll start to need it when unifying
gup for hugetlb.

Link: https://lkml.kernel.org/r/20230628215310.73782-4-peterx@redhat.com
Signed-off-by: Peter Xu 
Reviewed-by: David Hildenbrand 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: James Houghton 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Kirill A . Shutemov 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox 
Cc: Mike Kravetz 
Cc: Mike Rapoport (IBM) 
Cc: Vlastimil Babka 
Cc: Yang Shi 
Signed-off-by: Andrew Morton

mm/hugetlb: remove hugetlb_set_page_subpool()

2023-06-23T23:59:33+00:00

All users have been converted to hugetlb_set_folio_subpool() so we can
safely remove this function.

Link: https://lkml.kernel.org/r/20230623054948.280627-1-sidhartha.kumar@oracle.com
Signed-off-by: Sidhartha Kumar 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Tarun Sahu 
Signed-off-by: Andrew Morton

mm: ptep_get() conversion

2023-06-19T23:19:25+00:00

Convert all instances of direct pte_t* dereferencing to instead use
ptep_get() helper.  This means that by default, the accesses change from a
C dereference to a READ_ONCE().  This is technically the correct thing to
do since where pgtables are modified by HW (for access/dirty) they are
volatile and therefore we should always ensure READ_ONCE() semantics.

But more importantly, by always using the helper, it can be overridden by
the architecture to fully encapsulate the contents of the pte.  Arch code
is deliberately not converted, as the arch code knows best.  It is
intended that arch code (arm64) will override the default with its own
implementation that can (e.g.) hide certain bits from the core code, or
determine young/dirty status by mixing in state from another source.

Conversion was done using Coccinelle:

----

// $ make coccicheck \
//          COCCI=ptepget.cocci \
//          SPFLAGS="--include-headers" \
//          MODE=patch

virtual patch

@ depends on patch @
pte_t *v;
@@

- *v
+ ptep_get(v)

----

Then reviewed and hand-edited to avoid multiple unnecessary calls to
ptep_get(), instead opting to store the result of a single call in a
variable, where it is correct to do so.  This aims to negate any cost of
READ_ONCE() and will benefit arch-overrides that may be more complex.

Included is a fix for an issue in an earlier version of this patch that
was pointed out by kernel test robot.  The issue arose because config
MMU=n elides definition of the ptep helper functions, including
ptep_get().  HUGETLB_PAGE=n configs still define a simple
huge_ptep_clear_flush() for linking purposes, which dereferences the ptep.
So when both configs are disabled, this caused a build error because
ptep_get() is not defined.  Fix by continuing to do a direct dereference
when MMU=n.  This is safe because for this config the arch code cannot be
trying to virtualize the ptes because none of the ptep helpers are
defined.

Link: https://lkml.kernel.org/r/20230612151545.3317766-4-ryan.roberts@arm.com
Reported-by: kernel test robot 
Link: https://lore.kernel.org/oe-kbuild-all/202305120142.yXsNEo6H-lkp@intel.com/
Signed-off-by: Ryan Roberts 
Cc: Adrian Hunter 
Cc: Alexander Potapenko 
Cc: Alexander Shishkin 
Cc: Alex Williamson 
Cc: Al Viro 
Cc: Andrey Konovalov 
Cc: Andrey Ryabinin 
Cc: Christian Brauner 
Cc: Christoph Hellwig 
Cc: Daniel Vetter 
Cc: Dave Airlie 
Cc: Dimitri Sivanich 
Cc: Dmitry Vyukov 
Cc: Ian Rogers 
Cc: Jason Gunthorpe 
Cc: Jérôme Glisse 
Cc: Jiri Olsa 
Cc: Johannes Weiner 
Cc: Kirill A. Shutemov 
Cc: Lorenzo Stoakes 
Cc: Mark Rutland 
Cc: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Kravetz 
Cc: Mike Rapoport (IBM) 
Cc: Muchun Song 
Cc: Namhyung Kim 
Cc: Naoya Horiguchi 
Cc: Oleksandr Tyshchenko 
Cc: Pavel Tatashin 
Cc: Roman Gushchin 
Cc: SeongJae Park 
Cc: Shakeel Butt 
Cc: Uladzislau Rezki (Sony) 
Cc: Vincenzo Frascino 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/gup: remove vmas array from internal GUP functions

2023-06-09T23:25:26+00:00

Now we have eliminated all callers to GUP APIs which use the vmas
parameter, eliminate it altogether.

This eliminates a class of bugs where vmas might have been kept around
longer than the mmap_lock and thus we need not be concerned about locks
being dropped during this operation leaving behind dangling pointers.

This simplifies the GUP API and makes it considerably clearer as to its
purpose - follow flags are applied and if pinning, an array of pages is
returned.

Link: https://lkml.kernel.org/r/6811b4b2b4b3baf3dd07f422bb18853bb2cd09fb.1684350871.git.lstoakes@gmail.com
Signed-off-by: Lorenzo Stoakes 
Acked-by: David Hildenbrand 
Reviewed-by: Christoph Hellwig 
Cc: Catalin Marinas 
Cc: Christian König 
Cc: Dennis Dalessandro 
Cc: Greg Kroah-Hartman 
Cc: Janosch Frank 
Cc: Jarkko Sakkinen 
Cc: Jason Gunthorpe 
Cc: Jens Axboe 
Cc: Matthew Wilcox (Oracle) 
Cc: Sakari Ailus 
Cc: Sean Christopherson 
Signed-off-by: Andrew Morton

mm/hugetlb: remove hugetlb_page_subpool()

2023-06-09T23:25:25+00:00

All users of hugetlb_page_subpool() have been converted to use the folio
equivalent.  This function can be safely removed.

Link: https://lkml.kernel.org/r/20230516225205.1429196-1-sidhartha.kumar@oracle.com
Signed-off-by: Sidhartha Kumar 
Reviewed-by: David Hildenbrand 
Reviewed-by: Mike Kravetz 
Cc: Matthew Wilcox (Oracle) 
Cc: Muchun Song 
Signed-off-by: Andrew Morton