linux.git/mm/filemap.c, branch v6.7

mm/filemap: avoid buffered read/write race to read inconsistent data

2023-12-20T21:46:19+00:00

The following concurrency may cause the data read to be inconsistent with
the data on disk:

             cpu1                           cpu2
------------------------------|------------------------------
                               // Buffered write 2048 from 0
                               ext4_buffered_write_iter
                                generic_perform_write
                                 copy_page_from_iter_atomic
                                 ext4_da_write_end
                                  ext4_da_do_write_end
                                   block_write_end
                                    __block_commit_write
                                     folio_mark_uptodate
// Buffered read 4096 from 0          smp_wmb()
ext4_file_read_iter                   set_bit(PG_uptodate, folio_flags)
 generic_file_read_iter            i_size_write // 2048
  filemap_read                     unlock_page(page)
   filemap_get_pages
    filemap_get_read_batch
    folio_test_uptodate(folio)
     ret = test_bit(PG_uptodate, folio_flags)
     if (ret)
      smp_rmb();
      // Ensure that the data in page 0-2048 is up-to-date.

                               // New buffered write 2048 from 2048
                               ext4_buffered_write_iter
                                generic_perform_write
                                 copy_page_from_iter_atomic
                                 ext4_da_write_end
                                  ext4_da_do_write_end
                                   block_write_end
                                    __block_commit_write
                                     folio_mark_uptodate
                                      smp_wmb()
                                      set_bit(PG_uptodate, folio_flags)
                                   i_size_write // 4096
                                   unlock_page(page)

   isize = i_size_read(inode) // 4096
   // Read the latest isize 4096, but without smp_rmb(), there may be
   // Load-Load disorder resulting in the data in the 2048-4096 range
   // in the page is not up-to-date.
   copy_page_to_iter
   // copyout 4096

In the concurrency above, we read the updated i_size, but there is no read
barrier to ensure that the data in the page is the same as the i_size at
this point, so we may copy the unsynchronized page out.  Hence adding the
missing read memory barrier to fix this.

This is a Load-Load reordering issue, which only occurs on some weak
mem-ordering architectures (e.g.  ARM64, ALPHA), but not on strong
mem-ordering architectures (e.g.  X86).  And theoretically the problem
doesn't only happen on ext4, filesystems that call filemap_read() but
don't hold inode lock (e.g.  btrfs, f2fs, ubifs ...) will have this
problem, while filesystems with inode lock (e.g.  xfs, nfs) won't have
this problem.

Link: https://lkml.kernel.org/r/20231213062324.739009-1-libaokun1@huawei.com
Signed-off-by: Baokun Li 
Reviewed-by: Jan Kara 
Cc: Andreas Dilger 
Cc: Christoph Hellwig 
Cc: Dave Chinner 
Cc: Matthew Wilcox (Oracle) 
Cc: Ritesh Harjani (IBM) 
Cc: Theodore Ts'o 
Cc: yangerkun 
Cc: Yu Kuai 
Cc: Zhang Yi 
Cc: 
Signed-off-by: Andrew Morton

mm: fix oops when filemap_map_pmd() without prealloc_pte

2023-12-07T00:12:45+00:00

syzbot reports oops in lockdep's __lock_acquire(), called from
__pte_offset_map_lock() called from filemap_map_pages(); or when I run the
repro, the oops comes in pmd_install(), called from filemap_map_pmd()
called from filemap_map_pages(), just before the __pte_offset_map_lock().

The problem is that filemap_map_pmd() has been assuming that when it finds
pmd_none(), a page table has already been prepared in prealloc_pte; and
indeed do_fault_around() has been careful to preallocate one there, when
it finds pmd_none(): but what if *pmd became none in between?

My 6.6 mods in mm/khugepaged.c, avoiding mmap_lock for write, have made it
easy for *pmd to be cleared while servicing a page fault; but even before
those, a huge *pmd might be zapped while a fault is serviced.

The difference in symptomatic stack traces comes from the "memory model"
in use: pmd_install() uses pmd_populate() uses page_to_pfn(): in some
models that is strict, and will oops on the NULL prealloc_pte; in other
models, it will construct a bogus value to be populated into *pmd, then
__pte_offset_map_lock() oops when trying to access split ptlock pointer
(or some other symptom in normal case of ptlock embedded not pointer).

Link: https://lore.kernel.org/linux-mm/20231115065506.19780-1-jose.pekkarinen@foxhound.fi/
Link: https://lkml.kernel.org/r/6ed0c50c-78ef-0719-b3c5-60c0c010431c@google.com
Fixes: f9ce0be71d1f ("mm: Cleanup faultaround and finish_fault() codepaths")
Signed-off-by: Hugh Dickins 
Reported-and-tested-by: syzbot+89edd67979b52675ddec@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/linux-mm/0000000000005e44550608a0806c@google.com/
Reviewed-by: David Hildenbrand 
Cc: Jann Horn ,
Cc: José Pekkarinen 
Cc: Kirill A. Shutemov 
Cc: Matthew Wilcox (Oracle) 
Cc:     [5.12+]
Signed-off-by: Andrew Morton

mm: more ptep_get() conversion

2023-11-15T23:30:09+00:00

Commit c33c794828f2 ("mm: ptep_get() conversion") converted all (non-arch)
call sites to use ptep_get() instead of doing a direct dereference of the
pte.  Full rationale can be found in that commit's log.

Since then, three new call sites have snuck in, which directly dereference
the pte, so let's fix those up.

Unfortunately there is no reliable automated mechanism to catch these; I'm
relying on a combination of Coccinelle (which throws up a lot of false
positives) and some compiler magic to force a compiler error on
dereference (While this approach finds dereferences, it also yields a
non-booting kernel so can't be committed).

Link: https://lkml.kernel.org/r/20231114154945.490401-1-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts 
Cc: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton

mm: drop the assumption that VM_SHARED always implies writable

2023-10-18T21:34:19+00:00

Patch series "permit write-sealed memfd read-only shared mappings", v4.

The man page for fcntl() describing memfd file seals states the following
about F_SEAL_WRITE:-

    Furthermore, trying to create new shared, writable memory-mappings via
    mmap(2) will also fail with EPERM.

With emphasis on 'writable'.  In turns out in fact that currently the
kernel simply disallows all new shared memory mappings for a memfd with
F_SEAL_WRITE applied, rendering this documentation inaccurate.

This matters because users are therefore unable to obtain a shared mapping
to a memfd after write sealing altogether, which limits their usefulness. 
This was reported in the discussion thread [1] originating from a bug
report [2].

This is a product of both using the struct address_space->i_mmap_writable
atomic counter to determine whether writing may be permitted, and the
kernel adjusting this counter when any VM_SHARED mapping is performed and
more generally implicitly assuming VM_SHARED implies writable.

It seems sensible that we should only update this mapping if VM_MAYWRITE
is specified, i.e.  whether it is possible that this mapping could at any
point be written to.

If we do so then all we need to do to permit write seals to function as
documented is to clear VM_MAYWRITE when mapping read-only.  It turns out
this functionality already exists for F_SEAL_FUTURE_WRITE - we can
therefore simply adapt this logic to do the same for F_SEAL_WRITE.

We then hit a chicken and egg situation in mmap_region() where the check
for VM_MAYWRITE occurs before we are able to clear this flag.  To work
around this, perform this check after we invoke call_mmap(), with careful
consideration of error paths.

Thanks to Andy Lutomirski for the suggestion!

[1]:https://lore.kernel.org/all/20230324133646.16101dfa666f253c4715d965@linux-foundation.org/
[2]:https://bugzilla.kernel.org/show_bug.cgi?id=217238


This patch (of 3):

There is a general assumption that VMAs with the VM_SHARED flag set are
writable.  If the VM_MAYWRITE flag is not set, then this is simply not the
case.

Update those checks which affect the struct address_space->i_mmap_writable
field to explicitly test for this by introducing
[vma_]is_shared_maywrite() helper functions.

This remains entirely conservative, as the lack of VM_MAYWRITE guarantees
that the VMA cannot be written to.

Link: https://lkml.kernel.org/r/cover.1697116581.git.lstoakes@gmail.com
Link: https://lkml.kernel.org/r/d978aefefa83ec42d18dfa964ad180dbcde34795.1697116581.git.lstoakes@gmail.com
Signed-off-by: Lorenzo Stoakes 
Suggested-by: Andy Lutomirski 
Reviewed-by: Jan Kara 
Cc: Alexander Viro 
Cc: Christian Brauner 
Cc: Hugh Dickins 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Signed-off-by: Andrew Morton

filemap: remove use of wait bookmarks

2023-10-18T21:34:18+00:00

The original problem of the overly long list of waiters on a locked page
was solved properly by commit 9a1ea439b16b ("mm:
put_and_wait_on_page_locked() while page is migrated").  In the meantime,
using bookmarks for the writeback bit can cause livelocks, so we need to
stop using them.

Link: https://lkml.kernel.org/r/20231010035829.544242-1-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: Bin Lai 
Cc: Benjamin Segall 
Cc: Daniel Bristot de Oliveira 
Cc: Dietmar Eggemann 
Cc: Ingo Molnar 
Cc: Juri Lelli 
Cc: Matthew Wilcox (Oracle) 
Cc: Mel Gorman 
Cc: Peter Zijlstra 
Cc: Steven Rostedt (Google) 
Cc: Valentin Schneider 
Cc: Vincent Guittot 

Cc: Ingo Molnar 
Signed-off-by: Andrew Morton

memcontrol: only transfer the memcg data for migration

2023-10-18T21:34:17+00:00

For most migration use cases, only transfer the memcg data from the old
folio to the new folio, and clear the old folio's memcg data.  No charging
and uncharging will be done.

This shaves off some work on the migration path, and avoids the temporary
double charging of a folio during its migration.

The only exception is replace_page_cache_folio(), which will use the old
mem_cgroup_migrate() (now renamed to mem_cgroup_replace_folio).  In that
context, the isolation of the old page isn't quite as thorough as with
migration, so we cannot use our new implementation directly.

This patch is the result of the following discussion on the new hugetlb
memcg accounting behavior:

https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/

Link: https://lkml.kernel.org/r/20231006184629.155543-3-nphamcs@gmail.com
Signed-off-by: Nhat Pham 
Suggested-by: Johannes Weiner 
Acked-by: Johannes Weiner 
Cc: Frank van der Linden 
Cc: Michal Hocko 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Rik van Riel 
Cc: Roman Gushchin 
Cc: Shakeel Butt 
Cc: Shuah Khan 
Cc: Tejun heo 
Cc: Yosry Ahmed 
Cc: Zefan Li 
Signed-off-by: Andrew Morton

mm: use folio_xor_flags_has_waiters() in folio_end_writeback()

2023-10-18T21:34:17+00:00

Match how folio_unlock() works by combining the test for PG_waiters with
the clearing of PG_writeback.  This should have a small performance win,
and removes the last user of folio_wake().

Link: https://lkml.kernel.org/r/20231004165317.1061855-18-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Geert Uytterhoeven 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton

mm: make __end_folio_writeback() return void

2023-10-18T21:34:17+00:00

Rather than check the result of test-and-clear, just check that we have
the writeback bit set at the start.  This wouldn't catch every case, but
it's good enough (and enables the next patch).

Link: https://lkml.kernel.org/r/20231004165317.1061855-17-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Geert Uytterhoeven 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton

mm: add folio_xor_flags_has_waiters()

2023-10-18T21:34:17+00:00

Optimise folio_end_read() by setting the uptodate bit at the same time we
clear the unlock bit.  This saves at least one memory barrier and one
write-after-write hazard.

Link: https://lkml.kernel.org/r/20231004165317.1061855-16-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Geert Uytterhoeven 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton

mm: delete checks for xor_unlock_is_negative_byte()

2023-10-18T21:34:17+00:00

Architectures which don't define their own use the one in
asm-generic/bitops/lock.h.  Get rid of all the ifdefs around "maybe we
don't have it".

Link: https://lkml.kernel.org/r/20231004165317.1061855-15-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Geert Uytterhoeven 
Cc: Albert Ou 
Cc: Alexander Gordeev 
Cc: Andreas Dilger 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Heiko Carstens 
Cc: Ivan Kokshaysky 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Palmer Dabbelt 
Cc: Paul Walmsley 
Cc: Richard Henderson 
Cc: Sven Schnelle 
Cc: "Theodore Ts'o" 
Cc: Thomas Bogendoerfer 
Cc: Vasily Gorbik 
Signed-off-by: Andrew Morton