linux-stable.git/drivers/dax, branch linux-5.17.y

dax: make sure inodes are flushed before destroy cache

2022-04-08T11:58:07+00:00

[ Upstream commit a7e8de822e0b1979f08767c751f6c8a9c1d4ad86 ]

A bug can be triggered by following command

$ modprobe nd_pmem && modprobe -r nd_pmem

[   10.060014] BUG dax_cache (Not tainted): Objects remaining in dax_cache on __kmem_cache_shutdown()
[   10.060938] Slab 0x0000000085b729ac objects=9 used=1 fp=0x000000004f5ae469 flags=0x200000000010200(slab|head|node)
[   10.062433] Call Trace:
[   10.062673]  dump_stack_lvl+0x34/0x44
[   10.062865]  slab_err+0x90/0xd0
[   10.063619]  __kmem_cache_shutdown+0x13b/0x2f0
[   10.063848]  kmem_cache_destroy+0x4a/0x110
[   10.064058]  __x64_sys_delete_module+0x265/0x300

This is caused by dax_fs_exit() not flushing inodes before destroy cache.
To fix this issue, call rcu_barrier() before destroy cache.

Signed-off-by: Tong Zhang 
Reviewed-by: Ira Weiny 
Reviewed-by: Christoph Hellwig 
Link: https://lore.kernel.org/r/20220212071111.148575-1-ztong0001@gmail.com
Fixes: 7b6be8444e0f ("dax: refactor dax-fs into a generic provider of 'struct dax_device' instances")
Signed-off-by: Dan Williams 
Signed-off-by: Sasha Levin

Merge branch 'akpm' (patches from Andrew)

2022-01-15T18:37:06+00:00

Merge misc updates from Andrew Morton:
 "146 patches.

  Subsystems affected by this patch series: kthread, ia64, scripts,
  ntfs, squashfs, ocfs2, vfs, and mm (slab-generic, slab, kmemleak,
  dax, kasan, debug, pagecache, gup, shmem, frontswap, memremap,
  memcg, selftests, pagemap, dma, vmalloc, memory-failure, hugetlb,
  userfaultfd, vmscan, mempolicy, oom-kill, hugetlbfs, migration, thp,
  ksm, page-poison, percpu, rmap, zswap, zram, cleanups, hmm, and
  damon)"

* emailed patches from Andrew Morton : (146 commits)
  mm/damon: hide kernel pointer from tracepoint event
  mm/damon/vaddr: hide kernel pointer from damon_va_three_regions() failure log
  mm/damon/vaddr: use pr_debug() for damon_va_three_regions() failure logging
  mm/damon/dbgfs: remove an unnecessary variable
  mm/damon: move the implementation of damon_insert_region to damon.h
  mm/damon: add access checking for hugetlb pages
  Docs/admin-guide/mm/damon/usage: update for schemes statistics
  mm/damon/dbgfs: support all DAMOS stats
  Docs/admin-guide/mm/damon/reclaim: document statistics parameters
  mm/damon/reclaim: provide reclamation statistics
  mm/damon/schemes: account how many times quota limit has exceeded
  mm/damon/schemes: account scheme actions that successfully applied
  mm/damon: remove a mistakenly added comment for a future feature
  Docs/admin-guide/mm/damon/usage: update for kdamond_pid and (mk|rm)_contexts
  Docs/admin-guide/mm/damon/usage: mention tracepoint at the beginning
  Docs/admin-guide/mm/damon/usage: remove redundant information
  Docs/admin-guide/mm/damon/usage: update for scheme quotas and watermarks
  mm/damon: convert macro functions to static inline functions
  mm/damon: modify damon_rand() macro to static inline function
  mm/damon: move damon_rand() definition into damon.h
  ...

device-dax: compound devmap support

2022-01-15T14:30:26+00:00

Use the newly added compound devmap facility which maps the assigned dax
ranges as compound pages at a page size of @align.

dax devices are created with a fixed @align (huge page size) which is
enforced through as well at mmap() of the device.  Faults, consequently
happen too at the specified @align specified at the creation, and those
don't change throughout dax device lifetime.  MCEs unmap a whole dax
huge page, as well as splits occurring at the configured page size.

Performance measured by gup_test improves considerably for
unpin_user_pages() and altmap with NVDIMMs:

  $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S -a -n 512 -w
  (pin_user_pages_fast 2M pages) put:~71 ms -> put:~22 ms
  [altmap]
  (pin_user_pages_fast 2M pages) get:~524ms put:~525 ms -> get: ~127ms put:~71ms

   $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S -a -n 512 -w
  (pin_user_pages_fast 2M pages) put:~513 ms -> put:~188 ms
  [altmap with -m 127004]
  (pin_user_pages_fast 2M pages) get:~4.1 secs put:~4.12 secs -> get:~1sec put:~563ms

.. as well as unpin_user_page_range_dirty_lock() being just as effective
as THP/hugetlb[0] pages.

[0] https://lore.kernel.org/linux-mm/20210212130843.13865-5-joao.m.martins@oracle.com/

Link: https://lkml.kernel.org/r/20211202204422.26777-12-joao.m.martins@oracle.com
Signed-off-by: Joao Martins 
Reviewed-by: Dan Williams 
Cc: Christoph Hellwig 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: remove pfn from __dev_dax_{pte,pmd,pud}_fault()

2022-01-15T14:30:26+00:00

After moving the page mapping to be set prior to pte insertion, the pfn
in dev_dax_huge_fault() no longer is necessary.  Remove it, as well as
the @pfn argument passed to the internal fault handler helpers.

[akpm@linux-foundation.org: fix CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=n build]

Link: https://lkml.kernel.org/r/20211202204422.26777-11-joao.m.martins@oracle.com
Signed-off-by: Joao Martins 
Suggested-by: Christoph Hellwig 
Cc: Dan Williams 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: set mapping prior to vmf_insert_pfn{,_pmd,pud}()

2022-01-15T14:30:26+00:00

Normally, the @page mapping is set prior to inserting the page into a
page table entry.  Make device-dax adhere to the same ordering, rather
than setting mapping after the PTE is inserted.

The address_space never changes and it is always associated with the
same inode and underlying pages.  So, the page mapping is set once but
cleared when the struct pages are removed/freed (i.e.  after
{devm_}memunmap_pages()).

Link: https://lkml.kernel.org/r/20211202204422.26777-10-joao.m.martins@oracle.com
Suggested-by: Jason Gunthorpe 
Signed-off-by: Joao Martins 
Cc: Christoph Hellwig 
Cc: Dan Williams 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: factor out page mapping initialization

2022-01-15T14:30:25+00:00

Move initialization of page->mapping into a separate helper.

This is in preparation to move the mapping set to be prior to inserting
the page table entry and also for tidying up compound page handling into
one helper.

Link: https://lkml.kernel.org/r/20211202204422.26777-9-joao.m.martins@oracle.com
Signed-off-by: Joao Martins 
Cc: Christoph Hellwig 
Cc: Dan Williams 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: ensure dev_dax->pgmap is valid for dynamic devices

2022-01-15T14:30:25+00:00

Right now, only static dax regions have a valid @pgmap pointer in its
struct dev_dax.  Dynamic dax case however, do not.

In preparation for device-dax compound devmap support, make sure that
dev_dax pgmap field is set after it has been allocated and initialized.

dynamic dax device have the @pgmap is allocated at probe() and it's
managed by devm (contrast to static dax region which a pgmap is provided
and dax core kfrees it).  So in addition to ensure a valid @pgmap, clear
the pgmap when the dynamic dax device is released to avoid the same
pgmap ranges to be re-requested across multiple region device reconfigs.

Add a static_dev_dax() and use that helper in dev_dax_probe() to ensure
the initialization differences between dynamic and static regions are
more explicit.  While at it, consolidate the ranges initialization when
we allocate the @pgmap for the dynamic dax region case.  Also take the
opportunity to document the differences between static and dynamic da
regions.

Link: https://lkml.kernel.org/r/20211202204422.26777-8-joao.m.martins@oracle.com
Suggested-by: Dan Williams 
Signed-off-by: Joao Martins 
Cc: Christoph Hellwig 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: use struct_size()

2022-01-15T14:30:25+00:00

Use the struct_size() helper for the size of a struct with variable
array member at the end, rather than manually calculating it.

Link: https://lkml.kernel.org/r/20211202204422.26777-7-joao.m.martins@oracle.com
Suggested-by: Dan Williams 
Signed-off-by: Joao Martins 
Cc: Christoph Hellwig 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

device-dax: use ALIGN() for determining pgoff

2022-01-15T14:30:25+00:00

Rather than calculating @pgoff manually, switch to ALIGN() instead.

Link: https://lkml.kernel.org/r/20211202204422.26777-6-joao.m.martins@oracle.com
Suggested-by: Dan Williams 
Signed-off-by: Joao Martins 
Reviewed-by: Dan Williams 
Cc: Christoph Hellwig 
Cc: Dave Jiang 
Cc: Jane Chu 
Cc: Jason Gunthorpe 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Vishal Verma 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

dax: remove the copy_from_iter and copy_to_iter methods

2021-12-18T16:04:53+00:00

These methods indirect the actual DAX read/write path.  In the end pmem
uses magic flush and mc safe variants and fuse and dcssblk use plain ones
while device mapper picks redirects to the underlying device.

Add set_dax_nocache() and set_dax_nomc() APIs to control which copy
routines are used to remove indirect call from the read/write fast path
as well as a lot of boilerplate code.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Vivek Goyal  [virtiofs]
Link: https://lore.kernel.org/r/20211215084508.435401-5-hch@lst.de
Signed-off-by: Dan Williams