linux-stable.git/include/linux/memremap.h, branch v5.18.3

mm: Add three folio wrappers

2022-03-21T16:57:38+00:00

folio_is_zone_device() is equivalent to is_zone_device_page(),
folio_is_device_private() is equivalent to is_device_private_page(),
and folio_is_pinnable() is equivalent to is_pinnable_page().

All of these tests return the same result for every page in the folio,
so we can just pass the head page of the folio to the page variant of
the function.

Signed-off-by: Matthew Wilcox (Oracle)

mm: remove the extra ZONE_DEVICE struct page refcount

2022-03-03T17:47:33+00:00

ZONE_DEVICE struct pages have an extra reference count that complicates
the code for put_page() and several places in the kernel that need to
check the reference count to see that a page is not being used (gup,
compaction, migration, etc.). Clean up the code so the reference count
doesn't need to be treated specially for ZONE_DEVICE pages.

Note that this excludes the special idle page wakeup for fsdax pages,
which still happens at refcount 1.  This is a separate issue and will
be sorted out later.  Given that only fsdax pages require the
notifiacation when the refcount hits 1 now, the PAGEMAP_OPS Kconfig
symbol can go away and be replaced with a FS_DAX check for this hook
in the put_page fastpath.

Based on an earlier patch from Ralph Campbell .

Link: https://lkml.kernel.org/r/20220210072828.2930359-8-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Ralph Campbell 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Dan Williams 
Acked-by: Felix Kuehling 
Tested-by: "Sierra Guiza, Alejandro (Alex)" 

Cc: Alex Deucher 
Cc: Alistair Popple 
Cc: Ben Skeggs 
Cc: Chaitanya Kulkarni 
Cc: Christian Knig 
Cc: Karol Herbst 
Cc: Lyude Paul 
Cc: Miaohe Lin 
Cc: Muchun Song 
Cc: "Pan, Xinhui" 
Signed-off-by: Andrew Morton 
Signed-off-by: Matthew Wilcox (Oracle)

mm: don't include in

2022-03-03T17:47:33+00:00

Move the check for the actual pgmap types that need the free at refcount
one behavior into the out of line helper, and thus avoid the need to
pull memremap.h into mm.h.

Link: https://lkml.kernel.org/r/20220210072828.2930359-7-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Reviewed-by: Jason Gunthorpe 
Reviewed-by: Dan Williams 
Acked-by: Felix Kuehling 
Tested-by: "Sierra Guiza, Alejandro (Alex)" 

Cc: Alex Deucher 
Cc: Alistair Popple 
Cc: Ben Skeggs 
Cc: Chaitanya Kulkarni 
Cc: Karol Herbst 
Cc: Lyude Paul 
Cc: Miaohe Lin 
Cc: Muchun Song 
Cc: "Pan, Xinhui" 
Cc: Ralph Campbell 
Signed-off-by: Andrew Morton 
Signed-off-by: Matthew Wilcox (Oracle)

Merge branch 'akpm' (patches from Andrew)

2022-01-15T18:37:06+00:00

Merge misc updates from Andrew Morton:
 "146 patches.

  Subsystems affected by this patch series: kthread, ia64, scripts,
  ntfs, squashfs, ocfs2, vfs, and mm (slab-generic, slab, kmemleak,
  dax, kasan, debug, pagecache, gup, shmem, frontswap, memremap,
  memcg, selftests, pagemap, dma, vmalloc, memory-failure, hugetlb,
  userfaultfd, vmscan, mempolicy, oom-kill, hugetlbfs, migration, thp,
  ksm, page-poison, percpu, rmap, zswap, zram, cleanups, hmm, and
  damon)"

* emailed patches from Andrew Morton : (146 commits)
  mm/damon: hide kernel pointer from tracepoint event
  mm/damon/vaddr: hide kernel pointer from damon_va_three_regions() failure log
  mm/damon/vaddr: use pr_debug() for damon_va_three_regions() failure logging
  mm/damon/dbgfs: remove an unnecessary variable
  mm/damon: move the implementation of damon_insert_region to damon.h
  mm/damon: add access checking for hugetlb pages
  Docs/admin-guide/mm/damon/usage: update for schemes statistics
  mm/damon/dbgfs: support all DAMOS stats
  Docs/admin-guide/mm/damon/reclaim: document statistics parameters
  mm/damon/reclaim: provide reclamation statistics
  mm/damon/schemes: account how many times quota limit has exceeded
  mm/damon/schemes: account scheme actions that successfully applied
  mm/damon: remove a mistakenly added comment for a future feature
  Docs/admin-guide/mm/damon/usage: update for kdamond_pid and (mk|rm)_contexts
  Docs/admin-guide/mm/damon/usage: mention tracepoint at the beginning
  Docs/admin-guide/mm/damon/usage: remove redundant information
  Docs/admin-guide/mm/damon/usage: update for scheme quotas and watermarks
  mm/damon: convert macro functions to static inline functions
  mm/damon: modify damon_rand() macro to static inline function
  mm/damon: move damon_rand() definition into damon.h
  ...

mm/memremap: add ZONE_DEVICE support for compound pages

2022-01-15T14:30:25+00:00

Add a new @vmemmap_shift property for struct dev_pagemap which specifies
that a devmap is composed of a set of compound pages of order
@vmemmap_shift, instead of base pages.  When a compound page devmap is
requested, all but the first page are initialised as tail pages instead
of order-0 pages.

For certain ZONE_DEVICE users like device-dax which have a fixed page
size, this creates an opportunity to optimize GUP and GUP-fast walkers,
treating it the same way as THP or hugetlb pages.

Additionally, commit 7118fc2906e2 ("hugetlb: address ref count racing in
prep_compound_gigantic_page") removed set_page_count() because the
setting of page ref count to zero was redundant.  devmap pages don't
come from page allocator though and only head page refcount is used for
compound pages, hence initialize tail page count to zero.

Link: https://lkml.kernel.org/r/20211202204422.26777-5-joao.m.martins@oracle.com
Signed-off-by: Joao Martins 
Reviewed-by: Dan Williams 
Cc: Christoph Hellwig 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

memremap: remove support for external pgmap refcounts

2021-12-04T20:46:09+00:00

No driver is left using the external pgmap refcount, so remove the
code to support it.

Signed-off-by: Christoph Hellwig 
Acked-by: Bjorn Helgaas 
Link: https://lore.kernel.org/r/20211028151017.50234-1-hch@lst.de
Signed-off-by: Dan Williams

mm: fix spelling mistakes

2021-07-01T18:06:02+00:00

Fix some spelling mistakes in comments:
each having differents usage ==> each has a different usage
statments ==> statements
adresses ==> addresses
aggresive ==> aggressive
datas ==> data
posion ==> poison
higer ==> higher
precisly ==> precisely
wont ==> won't
We moves tha ==> We move the
endianess ==> endianness

Link: https://lkml.kernel.org/r/20210519065853.7723-2-thunder.leizhen@huawei.com
Signed-off-by: Zhen Lei 
Reviewed-by: Souptick Joarder 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm,memory_hotplug: allocate memmap from the added memory range

2021-05-05T18:27:26+00:00

Physical memory hotadd has to allocate a memmap (struct page array) for
the newly added memory section.  Currently, alloc_pages_node() is used
for those allocations.

This has some disadvantages:
 a) an existing memory is consumed for that purpose
    (eg: ~2MB per 128MB memory section on x86_64)
    This can even lead to extreme cases where system goes OOM because
    the physically hotplugged memory depletes the available memory before
    it is onlined.
 b) if the whole node is movable then we have off-node struct pages
    which has performance drawbacks.
 c) It might be there are no PMD_ALIGNED chunks so memmap array gets
    populated with base pages.

This can be improved when CONFIG_SPARSEMEM_VMEMMAP is enabled.

Vmemap page tables can map arbitrary memory.  That means that we can
reserve a part of the physically hotadded memory to back vmemmap page
tables.  This implementation uses the beginning of the hotplugged memory
for that purpose.

There are some non-obviously things to consider though.

Vmemmap pages are allocated/freed during the memory hotplug events
(add_memory_resource(), try_remove_memory()) when the memory is
added/removed.  This means that the reserved physical range is not
online although it is used.  The most obvious side effect is that
pfn_to_online_page() returns NULL for those pfns.  The current design
expects that this should be OK as the hotplugged memory is considered a
garbage until it is onlined.  For example hibernation wouldn't save the
content of those vmmemmaps into the image so it wouldn't be restored on
resume but this should be OK as there no real content to recover anyway
while metadata is reachable from other data structures (e.g.  vmemmap
page tables).

The reserved space is therefore (de)initialized during the {on,off}line
events (mhp_{de}init_memmap_on_memory).  That is done by extracting page
allocator independent initialization from the regular onlining path.
The primary reason to handle the reserved space outside of
{on,off}line_pages is to make each initialization specific to the
purpose rather than special case them in a single function.

As per above, the functions that are introduced are:

 - mhp_init_memmap_on_memory:
   Initializes vmemmap pages by calling move_pfn_range_to_zone(), calls
   kasan_add_zero_shadow(), and onlines as many sections as vmemmap pages
   fully span.

 - mhp_deinit_memmap_on_memory:
   Offlines as many sections as vmemmap pages fully span, removes the
   range from zhe zone by remove_pfn_range_from_zone(), and calls
   kasan_remove_zero_shadow() for the range.

The new function memory_block_online() calls mhp_init_memmap_on_memory()
before doing the actual online_pages().  Should online_pages() fail, we
clean up by calling mhp_deinit_memmap_on_memory().  Adjusting of
present_pages is done at the end once we know that online_pages()
succedeed.

On offline, memory_block_offline() needs to unaccount vmemmap pages from
present_pages() before calling offline_pages().  This is necessary because
offline_pages() tears down some structures based on the fact whether the
node or the zone become empty.  If offline_pages() fails, we account back
vmemmap pages.  If it succeeds, we call mhp_deinit_memmap_on_memory().

Hot-remove:

 We need to be careful when removing memory, as adding and
 removing memory needs to be done with the same granularity.
 To check that this assumption is not violated, we check the
 memory range we want to remove and if a) any memory block has
 vmemmap pages and b) the range spans more than a single memory
 block, we scream out loud and refuse to proceed.

 If all is good and the range was using memmap on memory (aka vmemmap pages),
 we construct an altmap structure so free_hugepage_table does the right
 thing and calls vmem_altmap_free instead of free_pagetable.

Link: https://lkml.kernel.org/r/20210421102701.25051-5-osalvador@suse.de
Signed-off-by: Oscar Salvador 
Reviewed-by: David Hildenbrand 
Acked-by: Michal Hocko 
Cc: Anshuman Khandual 
Cc: Pavel Tatashin 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: fix memory_failure() handling of dax-namespace metadata

2021-02-26T17:41:00+00:00

Given 'struct dev_pagemap' spans both data pages and metadata pages be
careful to consult the altmap if present to delineate metadata.  In fact
the pfn_first() helper already identifies the first valid data pfn, so
export that helper for other code paths via pgmap_pfn_valid().

Other usage of get_dev_pagemap() are not a concern because those are
operating on known data pfns having been looked up by get_user_pages().
I.e.  metadata pfns are never user mapped.

Link: https://lkml.kernel.org/r/161058501758.1840162.4239831989762604527.stgit@dwillia2-desk3.amr.corp.intel.com
Fixes: 6100e34b2526 ("mm, memory_failure: Teach memory_failure() about dev_pagemap pages")
Signed-off-by: Dan Williams 
Reported-by: David Hildenbrand 
Reviewed-by: David Hildenbrand 
Reviewed-by: Naoya Horiguchi 
Cc: Michal Hocko 
Cc: Oscar Salvador 
Cc: Qian Cai 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/memremap_pages: support multiple ranges per invocation

2020-10-14T01:38:28+00:00

In support of device-dax growing the ability to front physically
dis-contiguous ranges of memory, update devm_memremap_pages() to track
multiple ranges with a single reference counter and devm instance.

Convert all [devm_]memremap_pages() users to specify the number of ranges
they are mapping in their 'struct dev_pagemap' instance.

Signed-off-by: Dan Williams 
Signed-off-by: Andrew Morton 
Cc: Paul Mackerras 
Cc: Michael Ellerman 
Cc: Benjamin Herrenschmidt 
Cc: Vishal Verma 
Cc: Vivek Goyal 
Cc: Dave Jiang 
Cc: Ben Skeggs 
Cc: David Airlie 
Cc: Daniel Vetter 
Cc: Ira Weiny 
Cc: Bjorn Helgaas 
Cc: Boris Ostrovsky 
Cc: Juergen Gross 
Cc: Stefano Stabellini 
Cc: "Jérôme Glisse" 
Cc: Ard Biesheuvel 
Cc: Ard Biesheuvel 
Cc: Borislav Petkov 
Cc: Brice Goglin 
Cc: Catalin Marinas 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Greg Kroah-Hartman 
Cc: "H. Peter Anvin" 
Cc: Hulk Robot 
Cc: Ingo Molnar 
Cc: Jason Gunthorpe 
Cc: Jason Yan 
Cc: Jeff Moyer 
Cc: "Jérôme Glisse" 
Cc: Jia He 
Cc: Joao Martins 
Cc: Jonathan Cameron 
Cc: kernel test robot 
Cc: Mike Rapoport 
Cc: Pavel Tatashin 
Cc: Peter Zijlstra 
Cc: "Rafael J. Wysocki" 
Cc: Randy Dunlap 
Cc: Thomas Gleixner 
Cc: Tom Lendacky 
Cc: Wei Yang 
Cc: Will Deacon 
Link: https://lkml.kernel.org/r/159643103789.4062302.18426128170217903785.stgit@dwillia2-desk3.amr.corp.intel.com
Link: https://lkml.kernel.org/r/160106116293.30709.13350662794915396198.stgit@dwillia2-desk3.amr.corp.intel.com
Signed-off-by: Linus Torvalds