linux.git/mm/huge_memory.c, branch v6.7

mm: memcg: fix split queue list crash when large folio migration

2023-12-29T19:06:47+00:00

When running autonuma with enabling multi-size THP, I encountered the
following kernel crash issue:

[  134.290216] list_del corruption. prev->next should be fffff9ad42e1c490,
but was dead000000000100. (prev=fffff9ad42399890)
[  134.290877] kernel BUG at lib/list_debug.c:62!
[  134.291052] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[  134.291210] CPU: 56 PID: 8037 Comm: numa01 Kdump: loaded Tainted:
G            E      6.7.0-rc4+ #20
[  134.291649] RIP: 0010:__list_del_entry_valid_or_report+0x97/0xb0
......
[  134.294252] Call Trace:
[  134.294362]  
[  134.294440]  ? die+0x33/0x90
[  134.294561]  ? do_trap+0xe0/0x110
......
[  134.295681]  ? __list_del_entry_valid_or_report+0x97/0xb0
[  134.295842]  folio_undo_large_rmappable+0x99/0x100
[  134.296003]  destroy_large_folio+0x68/0x70
[  134.296172]  migrate_folio_move+0x12e/0x260
[  134.296264]  ? __pfx_remove_migration_pte+0x10/0x10
[  134.296389]  migrate_pages_batch+0x495/0x6b0
[  134.296523]  migrate_pages+0x1d0/0x500
[  134.296646]  ? __pfx_alloc_misplaced_dst_folio+0x10/0x10
[  134.296799]  migrate_misplaced_folio+0x12d/0x2b0
[  134.296953]  do_numa_page+0x1f4/0x570
[  134.297121]  __handle_mm_fault+0x2b0/0x6c0
[  134.297254]  handle_mm_fault+0x107/0x270
[  134.300897]  do_user_addr_fault+0x167/0x680
[  134.304561]  exc_page_fault+0x65/0x140
[  134.307919]  asm_exc_page_fault+0x22/0x30

The reason for the crash is that, the commit 85ce2c517ade ("memcontrol:
only transfer the memcg data for migration") removed the charging and
uncharging operations of the migration folios and cleared the memcg data
of the old folio.

During the subsequent release process of the old large folio in
destroy_large_folio(), if the large folio needs to be removed from the
split queue, an incorrect split queue can be obtained (which is
pgdat->deferred_split_queue) because the old folio's memcg is NULL now. 
This can lead to list operations being performed under the wrong split
queue lock protection, resulting in a list crash as above.

After the migration, the old folio is going to be freed, so we can remove
it from the split queue in mem_cgroup_migrate() a bit earlier before
clearing the memcg data to avoid getting incorrect split queue.

[akpm@linux-foundation.org: fix comment, per Zi Yan]
Link: https://lkml.kernel.org/r/61273e5e9b490682388377c20f52d19de4a80460.1703054559.git.baolin.wang@linux.alibaba.com
Fixes: 85ce2c517ade ("memcontrol: only transfer the memcg data for migration")
Signed-off-by: Baolin Wang 
Reviewed-by: Nhat Pham 
Reviewed-by: Yang Shi 
Reviewed-by: Zi Yan 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Roman Gushchin 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton

mm: fix for negative counter: nr_file_hugepages

2023-11-15T23:30:09+00:00

While qualifiying the 6.4 release, the following warning was detected in
messages:

vmstat_refresh: nr_file_hugepages -15664

The warning is caused by the incorrect updating of the NR_FILE_THPS
counter in the function split_huge_page_to_list.  The if case is checking
for folio_test_swapbacked, but the else case is missing the check for
folio_test_pmd_mappable.  The other functions that manipulate the counter
like __filemap_add_folio and filemap_unaccount_folio have the
corresponding check.

I have a test case, which reproduces the problem. It can be found here:
  https://github.com/sroeschus/testcase/blob/main/vmstat_refresh/madv.c

The test case reproduces on an XFS filesystem. Running the same test
case on a BTRFS filesystem does not reproduce the problem.

AFAIK version 6.1 until 6.6 are affected by this problem.

[akpm@linux-foundation.org: whitespace fix]
[shr@devkernel.io: test for folio_test_pmd_mappable()]
  Link: https://lkml.kernel.org/r/20231108171517.2436103-1-shr@devkernel.io
Link: https://lkml.kernel.org/r/20231106181918.1091043-1-shr@devkernel.io
Signed-off-by: Stefan Roesch 
Co-debugged-by: Johannes Weiner 
Acked-by: Johannes Weiner 
Reviewed-by: Matthew Wilcox (Oracle) 
Reviewed-by: David Hildenbrand 
Reviewed-by: Yang Shi 
Cc: Rik van Riel 
Cc: 
Signed-off-by: Andrew Morton

mm: huge_memory: use folio_xchg_last_cpupid() in __split_huge_page_tail()

2023-10-25T23:47:12+00:00

Convert to use folio_xchg_last_cpupid() in __split_huge_page_tail().

Link: https://lkml.kernel.org/r/20231018140806.2783514-16-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Cc: David Hildenbrand 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Juri Lelli 
Cc: Matthew Wilcox (Oracle) 
Cc: Peter Zijlstra 
Cc: Vincent Guittot 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: huge_memory: use a folio in change_huge_pmd()

2023-10-25T23:47:12+00:00

Use a folio in change_huge_pmd(), which helps to remove last
xchg_page_access_time() caller.

Link: https://lkml.kernel.org/r/20231018140806.2783514-11-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Cc: David Hildenbrand 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Juri Lelli 
Cc: Matthew Wilcox (Oracle) 
Cc: Peter Zijlstra 
Cc: Vincent Guittot 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: huge_memory: use folio_last_cpupid() in __split_huge_page_tail()

2023-10-25T23:47:12+00:00

Convert to use folio_last_cpupid() in __split_huge_page_tail().

Link: https://lkml.kernel.org/r/20231018140806.2783514-6-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Cc: David Hildenbrand 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Juri Lelli 
Cc: Matthew Wilcox (Oracle) 
Cc: Peter Zijlstra 
Cc: Vincent Guittot 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: huge_memory: use folio_last_cpupid() in do_huge_pmd_numa_page()

2023-10-25T23:47:11+00:00

Convert to use folio_last_cpupid() in do_huge_pmd_numa_page().

Link: https://lkml.kernel.org/r/20231018140806.2783514-5-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Cc: David Hildenbrand 
Cc: Huang Ying 
Cc: Ingo Molnar 
Cc: Juri Lelli 
Cc: Matthew Wilcox (Oracle) 
Cc: Peter Zijlstra 
Cc: Vincent Guittot 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/thp: fix "mm: thp: kill __transhuge_page_enabled()"

2023-10-18T21:34:18+00:00

The 6.0 commits:

commit 9fec51689ff6 ("mm: thp: kill transparent_hugepage_active()")
commit 7da4e2cb8b1f ("mm: thp: kill __transhuge_page_enabled()")

merged "can we have THPs in this VMA?" logic that was previously done
separately by fault-path, khugepaged, and smaps "THPeligible" checks.

During the process, the semantics of the fault path check changed in two
ways:

1) A VM_NO_KHUGEPAGED check was introduced (also added to smaps path).
2) We no longer checked if non-anonymous memory had a vm_ops->huge_fault
   handler that could satisfy the fault.  Previously, this check had been
   done in create_huge_pud() and create_huge_pmd() routines, but after
   the changes, we never reach those routines.

During the review of the above commits, it was determined that in-tree
users weren't affected by the change; most notably, since the only
relevant user (in terms of THP) of VM_MIXEDMAP or ->huge_fault is DAX,
which is explicitly approved early in approval logic.  However, this was a
bad assumption to make as it assumes the only reason to support
->huge_fault was for DAX (which is not true in general).

Remove the VM_NO_KHUGEPAGED check when not in collapse path and give any
->huge_fault handler a chance to handle the fault.  Note that we don't
validate the file mode or mapping alignment, which is consistent with the
behavior before the aforementioned commits.

Link: https://lkml.kernel.org/r/20230925200110.1979606-1-zokeefe@google.com
Fixes: 7da4e2cb8b1f ("mm: thp: kill __transhuge_page_enabled()")
Reported-by: Saurabh Singh Sengar 
Signed-off-by: Zach O'Keefe 
Cc: Yang Shi 
Cc: Matthew Wilcox 
Cc: David Hildenbrand 
Cc: Ryan Roberts 
Signed-off-by: Andrew Morton

mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()

2023-10-18T21:34:14+00:00

Let's convert it to consume a folio.

[akpm@linux-foundation.org: fix kerneldoc]
Link: https://lkml.kernel.org/r/20231002142949.235104-3-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Suren Baghdasaryan 
Reviewed-by: Vishal Moola (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton

mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap()

2023-10-18T21:34:14+00:00

Patch series "mm/rmap: convert page_move_anon_rmap() to
folio_move_anon_rmap()".

Convert page_move_anon_rmap() to folio_move_anon_rmap(), letting the
callers handle PageAnonExclusive.  I'm including cleanup patch #3 because
it fits into the picture and can be done cleaner by the conversion.


This patch (of 3):

Let's move it into the caller: there is a difference between whether an
anon folio can only be mapped by one process (e.g., into one VMA), and
whether it is truly exclusive (e.g., no references -- including GUP --
from other processes).

Further, for large folios the page might not actually be pointing at the
head page of the folio, so it better be handled in the caller.  This is a
preparation for converting page_move_anon_rmap() to consume a folio.

Link: https://lkml.kernel.org/r/20231002142949.235104-1-david@redhat.com
Link: https://lkml.kernel.org/r/20231002142949.235104-2-david@redhat.com
Signed-off-by: David Hildenbrand 
Reviewed-by: Suren Baghdasaryan 
Reviewed-by: Vishal Moola (Oracle) 
Cc: Mike Kravetz 
Cc: Muchun Song 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton

mm: memory: make numa_migrate_prep() to take a folio

2023-10-16T22:44:37+00:00

In preparation for large folio numa balancing, make numa_migrate_prep() to
take a folio, no functional change intended.

Link: https://lkml.kernel.org/r/20230921074417.24004-5-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Matthew Wilcox (Oracle) 
Cc: Mike Kravetz 
Cc: Zi Yan 
Signed-off-by: Andrew Morton