| Age | Commit message (Collapse) | Author |
|---|
|
mas_extend_spanning_null() had a clause if the end of the range being
written (mas->last) is the same as the end of the existing range it is
overwriting (wr_mas->r_max), action will be taken.
This code path is not possible because the only calling function
increments mas->last (unless it's ULONG_MAX) to walk to one beyond the
write and then resets the value back to the initial value.
In the case of mas->last == ULONG_MAX, then the second part of the
statement will always be false - mas->last cannot be less than the node
max.
This code never executed and is flawed anyways (the arguments are
incorrectly ordered), so removing it is the safest action. Since the code
never executes, it is not fixing any issue so Fixes tag is not given.
Link: https://lore.kernel.org/20260821192627.4085470-6-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Don't include the word flag in the quotes with the actual flag.
Link: https://lore.kernel.org/20260821192627.4085470-5-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Use the lockdep sequence numbers to ensure the write lock is not dropped
between write operations. The lockdep sequence is recorded on any walk
that starts from the top of the tree and re-checked prior to any operation
using an active node.
When lockdep detects an issue, it sets debug_locks to 0 disabling further
reports. __lock_sequnece() will return u32 ~0 when debug_locks is zero,
and the real sequnece count cannot return such a high value as it is less
than 32bits.
By always updating the sequence number, regardless of lock state and by
ignoring ~0 value in the sequence number will avoid ever printing a
WARN_ON when lockdep sets debug_locks to 0.
Link: https://lore.kernel.org/20260821192627.4085470-4-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Breno Leitao <leitao@debian.org>
Tested-by: Breno Leitao <leitao@debian.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add an 8 bit small sequence counter to the held_lock struct to detect if
the lock as been dropped and reacquired. This is useful when a data
structure depends on a constant locking context, but is not able to detect
locking and unlocking of the lock through its own API.
Since the __lock_unpin_lock() will no longer detect underflow by casting
the unsigned int to a signed int, update the casting code to use a temp
variable for calculations using a signed int.
Link: https://lore.kernel.org/20260821192627.4085470-3-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Will Deacon <will@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Waiman Long <longman@redhat.com>
Link: https://lore.kernel.org/all/h3tpnj5kzcrxms5picmimtkpg4aypcpip5wbd6bt2rpdj5k7eb@nhtzs3lefrkq/
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "maple_tree: lock checking and clean ups", v3.
In this series:
1. Try to detect lock issues
A number of syzbot reports are incorrectly pointing to the mm exit as
a source of the locking error. The first three patches attempt to
help users detect errors in their locking - but they still have to use
LOCKDEP. I guess it's still down to hope and prayers.
2. Documentation fixes
The documentation was lacking clarity, there are updates to try and
help the users, especially around the erase() cases.
3. Two benign issues
The cyclic allocator may have a race, although no in-kernel user can
hit it.
The erase functions may cause allocation issues if used with the
incorrect locking type, but none are present in-tree.
4. The erase gfp uses
mas_erase() and mtree_erase() do not take a gfp argument. To improve
reliability of the erase, the first attempt to allocate will be
GFP_NOWAIT, followed by a retry (if necessary of GFP_KERNEL |
GFP_NOFAIL. This will ensure the data is gone. I've updated the
documentation to make it more clear as well. mas_store() is not
addressed in the same way, but may need to be updated at a later date,
but that may require changing callers so it is out of scope here.
Beyond these goals there are some test fixes, some general speed-up
patches targeting extra work and cycles, and dropping dead code.
This patch (of 19):
When CONFIG_LOCKDEP and CONFIG_RCU_STRICT_GRACE_PERIOD is enabled, check
for rcu locking issues by recording the grace period in the maple state
and checking the rcu window is still valid whenever the maple state is
reused with a state that is not MA_START or MA_PAUSED.
Link: https://lore.kernel.org/20260821192627.4085470-1-liam@infradead.org
Link: https://lore.kernel.org/20260821192627.4085470-2-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
get_finfo() calls stat() to get metadata about the target directory, but
never checks the return value. On failure, stat() returns -1 and leaves
path_stat unmodified, so path_stat.st_mode may contain uninitialized stack
data.
The code then checks S_ISDIR(path_stat.st_mode) against this potentially
garbage value. This can produce a misleading "Not a directory" error when
the real problem is a nonexistent or inaccessible path, or, in the worst
case, the check could pass by chance on garbage data and let the function
continue using an invalid path_stat for the rest of its logic.
Check the return value and fail with a clear error message if stat()
fails, matching the error-handling style already used for statfs() and
read_file() later in the same function.
Link: https://lore.kernel.org/20260819121426.49500-1-anshumantewari123@gmail.com
Signed-off-by: Anshuman <anshumantewari123@gmail.com>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
A corrupt page table hands the same bogus entry to get_swap_device() on
every access to the mapping, and every rejection is logged. One machine
logged 6185620 copies of the same line in a few hours.
swap_dup_entry_direct() prints the same message from the fork path, once
per call: the WARN_ON_ONCE() guarding it warns once, the pr_err() inside
does not.
Rate limit all three prints.
Link: https://lore.kernel.org/20260818-swap_part_one-v1-1-a4fc58119fc0@debian.org
Fixes: 23b230ba8ac3 ("mm/swap: print bad swap offset entry in get_swap_device")
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Kairui Song <kasong@tencent.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
test_mprotect() calls ftruncate() to resize the backing file before
mmap()'ing it, but never checks the return value. If ftruncate() fails,
the file may remain shorter than the requested mapping size. The
subsequent mmap() with MAP_SHARED can still succeed in this case, but the
very next line writes directly into the mapped memory (*map = 1), which
can trigger SIGBUS if the mapping extends beyond the actual file size.
Check the return value and fail cleanly with ksft_exit_fail_msg() if
ftruncate() fails, matching the error-handling style already used for the
mmap() call immediately below it.
Link: https://lore.kernel.org/20260818133206.39503-1-anshumantewari123@gmail.com
Signed-off-by: Anshuman <anshumantewari123@gmail.com>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
consume_stock() can drive a stock slot's nr_pages to zero while its
cached[] pointer stays set, so the slot keeps pinning the css reference
that refill_stock() took. The offlining drain only flushes slots with
cached pages, so the reference is never released unless the slot happens
to be displaced by an unrelated charge or by CPU hotplug, and the memcg
lingers in the dying state - up to NR_MEMCG_STOCK (7) of them per CPU
under container churn.
Keeping the slot populated past the last page only saves a
css_get()/css_put() pair on the next charge of the same memcg, and costs
more than that: the offlining drain has to know about empty slots, and
refill_stock() cannot reuse them either, so a charge under a different
memcg evicts a live batch through the drain_idx rotation instead.
Drop the reference in consume_stock() when the slot empties. Empty slots
stop existing, so is_memcg_drain_needed() and the drain path stay as they
are, and refill_stock() reuses emptied slots directly. The cost is one
refcount pair per emptied slot, at most once per MEMCG_CHARGE_BATCH pages.
Link: https://lore.kernel.org/20260818130135.154315-1-husong@kylinos.cn
Fixes: d1a05b6973c7 ("memcg: do not try to drain per-cpu caches without pages")
Signed-off-by: Song Hu <husong@kylinos.cn>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Audra Mitchell <audra@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Nico Pache <npache@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
swapops.h uses MAX_SWAPFILES_SHIFT, SWP_MIGRATION_READ and SWP_PTE_MARKER,
all of which swap.h defines, but does not include swap.h. It compiles only
where the translation unit pulled swap.h in first. leafops.h includes
swapops.h on the line above swap.h, so a file whose include list reaches
leafops.h before swap.h gets:
In file included from include/linux/leafops.h:11:
include/linux/swapops.h:88:21: error: use of undeclared
identifier 'MAX_SWAPFILES_SHIFT'
A header that uses a definition has to include the header that provides it.
Link: https://lore.kernel.org/20260818115026.656406-1-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608181757.mza9RRj7-lkp@intel.com/
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
migrate_vma_insert_huge_pmd_page() calls flush_cache_page() before
replacing an existing huge zero PMD. However, the third argument to
flush_cache_page() is a PFN, while addr + HPAGE_PMD_SIZE is an end virtual
address.
More importantly, the mapping being invalidated is PMD-sized rather than
PAGE_SIZE-sized. Flush the whole PMD range with flush_cache_range(),
matching other huge PMD invalidation paths.
There is no userspace-visible effect today. The architectures that
currently enable ARCH_ENABLE_THP_MIGRATION use no-op implementations of
flush_cache_page()/flush_cache_range(). 32-bit ARM has non-trivial
implementations, but does not enable ARCH_ENABLE_THP_MIGRATION.
So this appears to be a latent API misuse rather than a currently
observable bug, and I don't think a stable backport is necessary.
Link: https://lore.kernel.org/20260817060845.377800-2-sh_def@163.com
Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Remove duplicate open() for local pagemap_fd in mprotect_tests() that
shadows the global pagemap_fd already opened in main(). The local fd is
never used in the function.
Link: https://lore.kernel.org/20260817080616.52946-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@arm.com>
Reviewed-by: SJ Park <sj@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct
vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This
moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu
pointer onto the same cacheline.
The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled,
generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a
176-CPU test system. Those writes bounce the line while memcg statistics
paths load vmstats_percpu.
Move cgwb_list into the existing alignment gap and cacheline-align
vmstats_percpu. This separates the pointer from the event counters
without increasing the size of struct mem_cgroup in the tested
configuration.
The blamed commit reduced median mremap throughput by 4.38% on the test
system with one socket. The patched kernel brings the performance to
within 0.5% of the parent which is within the observed boot-to-boot spread
(up to 1.2%).
Link: https://lore.kernel.org/20260817103835.2937733-1-usama.arif@linux.dev
Fixes: ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reported-by: kernel test robot <yi1.lai@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202608131743.c6a7dda4-lkp@intel.com
Tested-by: kernel test robot <yi1.lai@intel.com>
Link: http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Yi Lai <yi1.lai@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When building ARCH=riscv using clang with CONFIG_FORTIFY_SOURCE and
CONFIG_UBSAN_BOUNDS enabled, CONFIG_NR_CPUS > 64, and the default value of
2 for CONFIG_NODES_SHIFT, there is a compiletime warning from the fortify
routines.
In file included from mm/arch_numa.c:11:
In file included from include/linux/acpi.h:14:
In file included from include/linux/resource_ext.h:11:
In file included from include/linux/slab.h:17:
In file included from include/linux/gfp.h:7:
In file included from include/linux/mmzone.h:8:
In file included from include/linux/spinlock.h:60:
In file included from include/linux/interrupt_rc.h:17:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:11:
In file included from include/linux/bitmap.h:13:
In file included from include/linux/string.h:383:
include/linux/fortify-string.h:430:4: warning: call to '__write_overflow_field' declared with 'warning' attribute: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Wattribue-warning]
430 | __write_overflow_field(p_size_field, size);
| ^
include/linux/fortify-string.h:430:4: note: called by function 'fortify_memset_chk(unsigned long, unsigned long, unsigned long)'
include/linux/bitmap.h:248:3: note: inlined by function 'setup_node_to_cpumask_map'
248 | memset(dst, 0, len);
| ^
include/linux/fortify-string.h:462:25: note: expanded from macro 'memset'
462 | #define memset(p, c, s) __fortify_memset_chk(p, c, s, \
| ^
include/linux/fortify-string.h:453:2: note: expanded from macro '__fortify_memset_chk'
453 | fortify_memset_chk(__fortify_size, p_size, p_size_field), \
| ^
include/linux/fortify-string.h:430:4: note: use '-gline-directives-only' (implied by '-g1') or higher for more accurate inlining chain locations
430 | __write_overflow_field(p_size_field, size);
| ^
1 warning generated.
In this configuration, MAX_NUMNODES is 4. clang unrolls the for loop in
setup_node_to_cpumask_map() past this, which triggers the fortify check
when accessing node_to_cpumask_map on the theoretical fifth loop iteration
because it would be an out of bounds write.
Make it clear to clang that nr_node_ids is bounded by MAX_NUMNODES due to
the logic in setup_nr_node_ids() by early returning in
setup_node_to_cpumask_map() should that condition be violated.
Link: https://lore.kernel.org/20260813-arch_numa-avoid-fortify-warning-v2-1-093ad97a78df@kernel.org
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Closes: https://github.com/ClangBuiltLinux/linux/issues/2174
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Kees Cook <kees@kernel.org>
Cc: Bill Wendling <morbo@google.com>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Currently, in anon_vma_clone(), src vma's anon_vma is assigned to
active_anon_vma and is used when unlocking anon_vma after linking new
AVCs. However, the anon_vma is locked using src->anon_vma, instead of
active_anon_vma, making the lock and unlock target inconsistent.
Use active_anon_vma for both locking and unlocking.
Link: https://lore.kernel.org/OS7PR01MB139142FE16EC63B892559D40496DA2@OS7PR01MB13914.jpnprd01.prod.outlook.com
Signed-off-by: Eric Kim <seohyun.kim@outlook.kr>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mm/hmm.c:673 hmm_do_fault() error: we previously assumed 'hmm_vma_walk->locked' could be null (see line 654)
Stanislav says this can't happen. Waste a few cycles to make the warning
go away.
[akpm@linux-foundation.org: WARN_ON_ONCE() if the handler didn't set ->locked, per Stanislav]
Link: https://lore.kernel.org/anu1N-DOnQwxO1kF@skinsburskii
Fixes: 121170831228 ("mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/202608101053.PhnVUM4u-lkp@intel.com
Cc: Stanislav Kinsburskii <skinsburskii@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
For embedded devices, lacking support for NUMA, memory hotplug/hotremove,
CMA and huge pages is a quite common scenario. In this scenario, the
demand for contiguous physical memory allocation is very low. To reduce
the kernel image size, some devices disable the compaction. However,
their SoCs do support DDR ECC, meaning that memory-failure may be needed.
Migration is very useful for soft_offline_page() in memory-failure, which
may be triggered by correctable memory errors. Most anonymous and
file-mapped faulty pages can be migrated to other healthy pages.
Currently, MEMORY_FAILURE does not explicitly select MIGRATION. When
COMPACTION, MEMORY_HOTREMOVE, NUMA_MIGRATION and CMA are all disabled,
MEMORY_FAILURE can be enabled, but MIGRATION cannot be selected.
Make MEMORY_FAILURE select MIGRATION to handle this situation.
Link: https://lore.kernel.org/20260813134916.292733-1-xieyuanbin1@huawei.com
Signed-off-by: Xie Yuanbin <xieyuanbin1@huawei.com>
Suggested-by: Mike Rapoport <rppt@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: liaohua <liaohua4@huawei.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yuanbin Xie <xieyuanbin1@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Move pr_debug() inside the `if (dpage)` block to avoid printing garbage
pfn for NULL dpage, and correct the direction label from "sys to dev" to
"dev to sys".
Link: https://lore.kernel.org/20260812092856.55296-1-liuqiangneo@163.com
Signed-off-by: Qiang Liu <liuqiang@kylinos.cn>
Assisted-by: Qoder:Qwen-3.8-MAX-Preview
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
alpha requires percpu variables in modules to be defined as weak so that the
compiler generates GOT based external references for them. This puts two
extra restrictions on percpu variable definitions. The symbol must be
globally unique even when static and a static percpu variable can't be
defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic
code build coverage for these restrictions without building for alpha.
MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call
site and thus can't be built with weak percpu definitions, so it depends on
!DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU,
this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage.
allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build
coverage for restrictions which only matter to alpha module builds. Drop
DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on
alpha builds.
Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Gabriele Monaco <gmonaco@redhat.com> [include/rv/da_monitor.h]
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
sort_folio() has a shortcut for moving folios that are no longer evictable
but are still sitting on a generation list. However, this shortcut is
buggy. It does not follow the PG_lru usage convention, and it has a more
serious issue.
Unevictable folios are not threaded on lists[LRU_UNEVICTABLE], so that
folio->lru can be reused to hold folio->mlock_count (see the comment in
lruvec_init()). Hence lruvec_add_folio() skips the list_add() for them,
and every other place that turns a folio unevictable initialises
mlock_count explicitly: lru_add() sets it to 0, __mlock_folio() and
__mlock_new_folio() set it to !!folio_test_mlocked(folio). sort_folio()
sets nothing, and the lru_gen_del_folio() right above it may have already
poisoned folio->lru via list_del(), so mlock_count ends up aliasing
LIST_POISON2, which reads as 0x122, i.e. 290. The result is user
visible. On munlock, __munlock_folio() decrements that bogus count, finds
it still non-zero and bails out before clearing PG_mlocked, so the folio
remains unevictable and the Mlocked accounting stays inflated until the
folio is freed.
The shortcut also touches the LRU flags in the wrong order. It calls
lru_gen_del_folio() while PG_lru is still set, so a concurrent
folio_test_clear_lru() (e.g. compaction, folio_isolate_lru()) can succeed
on a folio that has already been taken off the generation list, which may
lead to unexpected behavior.
So fix it by isolating them as common folios and letting the generic
shrink path cull them. This matches the classical LRU behavior, and there
should be no visible effect on the generic eviction or isolation behavior.
There is no performance concern either, such a folio goes through this
once, and then it is off the generation lists for good.
Link: https://lore.kernel.org/20260812-mglru-mlock-fix-v2-1-a3fec5853c08@tencent.com
Fixes: ac35a4902374 ("mm: multi-gen LRU: minimal implementation")
Signed-off-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Brian Geffon <bgeffon@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jan Alexander Steffens (heftig) <heftig@archlinux.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Oleksandr Natalenko <oleksandr@natalenko.name>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Steven Barrett <steven@liquorix.net>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Steps 7 and 9 of the migration description still say "radix tree", unlike
steps 5 and 11 which already use "i_pages lock". The page cache moved to
the XArray at mapping->i_pages long ago. Use "page cache tree" for the
two remaining references.
Link: https://lore.kernel.org/20260812075739.325441-1-husong@kylinos.cn
Signed-off-by: Song Hu <husong@kylinos.cn>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The note about khugepaged counters references /proc/vmstat for the PMD
case, but never mentions where the mTHPs stats can be found (i.e.:
/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats/) Add a small
addition to this section for clarity.
Also fix a missing period while we are at it.
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-7-ddac39d61c4a@linux.dev
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Suggested-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We are currently dropping the anon_vma write lock before unmapping the
PTE. Although this is safe, due to us still holding the mmap_write_lock,
its safer and less confusing to switch the order of these two operations.
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-6-ddac39d61c4a@linux.dev
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix comment in collapse_scan_pmd() that still described the old
folio_mapcount() > folio_ref_count() check and a "512" false-positive
scenario. The code now uses folio_expected_ref_count() !=
folio_ref_count() which doesn't suffer from the same limitation.
Fix comment in collapse_huge_page() that referenced ptep_clear_flush, when
the code actually uses pmdp_collapse_flush.
Fix comment in __collapse_huge_page_swapin() that referenced the old
function name khugepaged_scan_pmd, now collapse_scan_pmd.
Also clean up some simple typos and stale terminology (mmap_sem ->
mmap_lock, PG_lock -> folio lock, page -> folio, grammar).
We also clarify a comment regarding where the max_ptes_none check is
deferred to in mthp_collapse() from the original collapse_scan_pmd check.
Update all comments that references a function to include parentheses.
[nico.pache@linux.dev: fix outdated comments]
Link: https://lore.kernel.org/1c96e2f3-802f-472b-81e6-4af17a721a3c@linux.dev
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-4-ddac39d61c4a@linux.dev
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Acked-by: Usama Arif <usama.arif@linux.dev>
Assisted-by: Cursor(claude-sonnet-4):4.6
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Provide a simple helper function to help reduce a often used, and
duplicate pattern across the khugepaged code.
When collapsing to a PMD we need to record a vm_event and the mTHP_stat
event. When doing mTHP collapse we only update the mTHP stat.
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-3-ddac39d61c4a@linux.dev
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This change deduplicates the "is this PTE/folio referenced enough to be
considered for a collapse" condition that was repeated in both
__collapse_huge_page_isolate() and collapse_scan_pmd(), extracting it into
a single inline helper function.
Also move the comment and use it as the function header. While we are at
it, updated the comment to clarify that a young pte is a recently accessed
one.
[nico.pache@linux.dev: drop the trivial helper kerneldoc and inline marker per review]
Link: https://lore.kernel.org/9038f552-926b-4c4c-b023-69271f45e3d5@linux.dev
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-2-ddac39d61c4a@linux.dev
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Acked-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
collapse_control_init_scan()
Patch series "mm/khugepaged: several cleanups", v4.
The following changes stem from a number of reviews during my khugepaged
mTHP support series [1]. Some of these are minor code cleanups, issues or
reviews that we decided to deferred to a followup series, or in the case
of the more major patch of the series, changes [2] Lance Yang attempted
while my series was in-flight and we decided to wait till later to try.
The first 3 patches introduce helper functions to increase code reuse and
readability. This includes a per-scan state clearing function, extracting
the young page check into a helper, and a count_collapse_event() function
to reduce a repetative pattern used across mTHP collapse.
The 4th patch was the byproduct of me throwing Claude at all the comments
in khugepaged verifying and looking for any outdated info.
The 5th patch is based on Lance Yang's commit series [2] trying to extract
the PTE state checking into a helper function. This required a bit of
rewriting due to differences after mTHP collapse was introduced. I also
took into account the changes requested during his patches review cycle.
The remaining 2 patches were review points during my mTHP series that we
agreed can be deferred to a later series.
Thank you to those whos reviews and work I leveraged to achieve these
cleanups.
This patch (of 6):
Extract the repeated clearing of node_load, alloc_nmask, and
mthp_present_ptes into a helper to reduce duplication in
collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not
current use the bitmap, they will in the future, and clearing it now is
harmless.
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-0-ddac39d61c4a@linux.dev
Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-1-ddac39d61c4a@linux.dev
Link: https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/ [1]
Link: https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/ [2]
Signed-off-by: Nico Pache (Red Hat) <nico.pache@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This file is a 'special' driver that implements /dev/zero and /dev/mem
among other things.
As such it makes sense for mm to be cc'd on mails and to have some say in
how things are changed there, so add it to the mm misc section.
Uniquely, it provides the 'old way' of obtaining an anonymous mapping -
MAP_PRIVATE of /dev/zero - so is directly tied to memory mapping,
therefore also add it to the memory mapping section.
scripts/get_maintainer.pl copes perfectly fine with files in multiple
sections so everything should work correctly.
Link: https://lore.kernel.org/20260812-add-drivers-mem-to-mm-maintainers-v1-1-6218b861f4c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Reviewed-by: Anshuman Khandual<anshuman.khandual@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
lru_gen_look_around() feeds its local 'young' counter into
suitable_to_scan(), which decides whether the current PMD is added to
the bloom filter and checked again on the next aging round.
The folio triggering the look-around is processed at function entry:
test_and_clear_young_ptes_notify() clears the accessed bits of the nr PTEs
it maps, and the function bails out if none of them is young. The loop
that follows therefore never recounts this folio, since its accessed bits
are already cleared. Every other young folio the loop finds is accounted
as a batch (young += nr), where nr is the number of consecutive PTEs it
maps. The triggering folio, however, still contributes a fixed young = 1
regardless of its size -- a leftover from before PTE batching. A large
triggering folio is thus accounted inconsistently with the rest of the
window.
Initialize young to nr so the triggering folio is accounted the same way
as any other young folio batch in the loop.
Note this is a deliberate overestimate, not a measured value. The
test-and-clear helper only reports whether any of the nr PTEs is young,
not how many were accessed, so the true number of accessed PTEs in a large
folio is unknown and can be smaller than nr. Counting the full batch is
intentional: the mm core tracks accessed/dirty state per folio, not per
page, so a per-page count is neither obtainable nor meaningful. The only
consumer is suitable_to_scan(), and the bloom filter it feeds tolerates
error. Overestimating is also the safe direction: at worst a PMD that saw
little access is rescanned, whereas underestimating could skip rescanning
a PMD whose folios are still hot and reclaim them incorrectly. (nr here
is the PTE batch size, not necessarily folio_nr_pages().)
Link: https://lore.kernel.org/20260813061019.49806-1-hui.zhu@linux.dev
Link: https://lore.kernel.org/20260812065933.103627-1-hui.zhu@linux.dev
Fixes: 56e5b60b2114 ("mm: support batched checking of the young flag for MGLRU")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
test_no_kmem_bypass() needs to read /sys/kernel/debug/zswap/stored_pages
via get_zswap_stored_pages() to verify that compressed pages are charged
to the memcg.
When running in an environment where debugfs is not mounted or
CONFIG_DEBUG_FS is disabled, get_zswap_stored_pages() fails, causing the
loop to terminate early and report a false negative (KSFT_FAIL).
Selftests should not fail if debugfs is unavailable, and it should print a
message when it is skipped.
While I'm here, also add a warning message if the test is being skipped
due to totalram size and make the check for totalram more readable.
[akpm@linux-foundation.org: clarify debugfs-unavailable error message]
Link: https://lore.kernel.org/20260812050848.848882-1-wfelipe@google.com
Link: https://lore.kernel.org/20260811051434.3805648-1-wfelipe@google.com
Signed-off-by: Wilson Felipe Pereira <wfelipe@google.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
quarantine_size tracks the total number of bytes stored in
global_quarantine[]. It is incremented when per-CPU quarantine objects
are moved into the global quarantine and decremented when a global batch
is evicted by kasan_quarantine_reduce().
kasan_quarantine_remove_cache() also removes objects from the global
quarantine. qlist_move_cache() rebuilds the source batch and updates its
.bytes field, but quarantine_size is not adjusted accordingly.
As a result, quarantine_size remains over-counted by the size of the
removed objects. The stale accounting accumulates across cache removals.
Once the inflated value exceeds quarantine_max_size,
kasan_quarantine_reduce() can evict a batch even though the actual number
of bytes in global_quarantine[] is still below quarantine_max_size,
shortening the quarantine window.
Fix the accounting by recording each batch's size before
qlist_move_cache() and subtracting the number of bytes actually removed
from quarantine_size while holding quarantine_lock.
A KUnit reproducer used during testing observed the over-count grow by
4698864 bytes after one kasan_quarantine_remove_cache() call with the fix
reverted. With this change applied, the over-count did not grow.
Link: https://lore.kernel.org/20260811073332.1351893-1-sh_def@163.com
Fixes: 64abdcb24351 ("kasan: eliminate long stalls during quarantine reduction")
Signed-off-by: Hui Su <sh_def@163.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260808031459.3032812-1-sh_def%40163.com
Reviewed-by: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Currently swap to and from file systems goes through two indirect calls
between the swap ops and the swap_rw method. Reduce this by directly
providing the swap_ops from the file system.
For this refactor swap_fs_submit into a swap_fs_prepare_rw helper that
initializes the iov_iter on the callers stack so that file systems can
call it directly, and use that to initialize file system specific ops in
the NFS and SMB clients, which then get passed to swap_fs_activate.
Link: https://lore.kernel.org/20260723054622.3460249-4-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Chris Li <chrisl@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steve French <sfrench@samba.org>
Cc: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add a new header to declare the swap_iocb, swap_ops and swap_ctx to allow
for swap_ops implementations outside of mm/page_io.c. This will be used
to remove the double indirection for file system-based swap. There is no
functional change, just a move of the declarations.
Note that there already is a swapops.h header, which is totally unrelated
to struct swap_ops. The close naming is a bit unfortunate, but I could
not think of a better name for this header.
Link: https://lore.kernel.org/20260723054622.3460249-3-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Chris Li <chrisl@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steve French <sfrench@samba.org>
Cc: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "swap_ops updates", v2.
This series is a follow on to the swap ops series now in mm-unstable.
The first patch reintroduces direct folio writes for synchronous
swap files, the other two remove the double indirect for file
system based swap.
This patch (of 3):
Kairui Song reported that zram benefits from submitting each folio
directly instead of batching up I/O because the classic LRU scanning
benefits from clearing the folio writeback bit in the scan loop.
Accommodate that by kicking off writes for synchronous devices for each
iteration.
Link: https://lore.kernel.org/20260723054622.3460249-1-hch@lst.de
Link: https://lore.kernel.org/20260723054622.3460249-2-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Chris Li <chrisl@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steve French <sfrench@samba.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
zs_lookup_class_index() lets zram recompression decide whether a newly
compressed object would use a smaller size class. It currently classifies
the payload size directly, while zs_malloc() adds ZS_HANDLE_SIZE before
selecting the class.
This makes lookup disagree with allocation near size-class boundaries.
With 4 KiB pages, CONFIG_ZSMALLOC_CHAIN_SIZE=8, and 64-bit handles, a
1025-to-1024-byte recompression appears to move from class 64 to class 62
although both allocations use class 64. Conversely, a 1049-to-1025-byte
recompression appears to stay in class 64 although the allocations move
from class 65 to class 64.
As a result, zram can accept replacements with no allocation benefit or
reject ones that would save memory, potentially marking the object
incompressible.
Factor size-class selection into lookup_size_class(), account for the
handle there, and use the helper for both lookup and allocation.
Link: https://lore.kernel.org/20260809115518.3791787-1-xialonglong2025@163.com
Fixes: 7c2af309abd2 ("zram: add size class equals check into recompression")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We have had bugs where set_page_dirty() was used on a page from GUP
without appropriate locking, leading to UAF, in:
- KVM, see
https://lore.kernel.org/r/20260810-x86-kvm-setpagedirty-v1-1-85f180892d4f@google.com
- i915, see commit 0d4bbe3d407f ("drm/i915/userptr: Try to acquire the
page lock around set_page_dirty()").
- VMCI, see commit 5a16c535409f ("VMCI: Use set_page_dirty_lock() when
unregistering guest memory")
- kpc2000 staging driver, see commit b6d13bd9f2c1 ("staging: kpc2000:
kpc_dma: Convert set_page_dirty() --> set_page_dirty_lock()")
I think set_page_dirty() and folio_mark_dirty() need more explicit
documentation on how they should be used with pages from GUP; so add a
comment on top of set_page_dirty() and make the comment above
folio_mark_dirty() more explicit.
Link: https://lore.kernel.org/20260810-set-page-dirty-warnings-v2-1-1bd40fadfacd@google.com
Signed-off-by: Jann Horn <jannh@google.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The slot lock is a bit operation on the whole __lock word, which flags and
ac_time alias as two u32s. On little-endian the lock bit lands in the
position ZRAM_ENTRY_LOCK reserves in flags, so the aliasing works out. On
64-bit big-endian it lands in ac_time instead: with
ZRAM_TRACK_ENTRY_ACTIME enabled, storing the access time from
mark_slot_accessed() or slot_free() wipes out the held lock bit, letting
another CPU take the same slot lock; an access time value with that bit
set makes the slot look locked forever.
Shift the lock bit into the flags half of the word on big-endian 64-bit.
Link: https://lore.kernel.org/20260810202241.2436603-1-devnexen@gmail.com
Fixes: 2e8ff2f51dde ("zram: use u32 for entry ac_time tracking")
Signed-off-by: David Carlier <devnexen@gmail.com>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix a spelling mistake n a code comment. This is a comment only change
with no functional impact.
Link: https://lore.kernel.org/20260813183847.474357-1-sercombe.joel.mark@gmail.com
Signed-off-by: Mark Sercombe <sercombe.joel.mark@gmail.com>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Andrew Ballance <andrewjballance@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The test writes min_unref_scans explicitly for every case, so its comments
describing 1 as the default are both unnecessary and, since the default is
now conditional, wrong. Refer to the threshold values directly.
No functional change.
Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-3-7b9689ac77cb@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
min_unref_scans now defaults to 2 when CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN and
CONFIG_DEBUG_KMEMLEAK_VERBOSE are both enabled, but the documentation
still states that the default is unconditionally 1.
Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-2-7b9689ac77cb@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: kmemleak: default min_unref_scans to 2 for verbose
kernels", v2.
When CONFIG_DEBUG_KMEMLEAK_VERBOSE is set, which means the host is in
auto scan mode, set min_unref_scans to 2, avoiding false positives.
CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN,
so a kernel built with it already runs the scan thread periodically and
the user has asked for detailed leak reports.
The confirming second scan comes for free there, so default
min_unref_scans to 2 in that case and keep it at 1 everywhere else.
CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so nothing changes for
kernels that do not opt in.
The other two patches bring the documentation and the selftest comments
in line with the new conditional default.
PS: A similar patch (v1 of this patchset) is applied to Meta's kernel,
in real production hosts.
This patch (of 3):
min_unref_scans defers reporting an object as leaked until it has stayed
unreferenced for that many consecutive scans, filtering out objects that
are only transiently unreferenced during a scan.
It defaults to 1, which reports on the first unreferenced scan.
CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN,
so a kernel built with it runs the scan thread periodically and the user
has opted into detailed leak reporting. A second confirming scan then
happens on its own.
Default min_unref_scans to 2 there to suppress transient false positives,
and keep it at 1 otherwise, where a manually triggered scan is expected to
report immediately. The value stays writable through the module
parameter.
CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so this does not change the
default for kernels that do not opt in.
Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-0-7b9689ac77cb@debian.org
Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-1-7b9689ac77cb@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add a functional test for the min_unref_scans kmemleak module parameter.
Using samples/kmemleak's helper module it checks that min_unref_scans=1
reports an orphan on the first scan, min_unref_scans=2 reports nothing on
the first scan but does on the second, and that the parameter reads back
what was written.
It counts only the helper module's own orphans (matched by their
[kmemleak_test] backtrace, with the module kept loaded so the symbols
resolve) so unrelated leaks already present on the system do not perturb
the result. The test skips when run as non-root, without
CONFIG_DEBUG_KMEMLEAK / CONFIG_SAMPLE_KMEMLEAK, on a kernel without the
parameter, or when the helper yields no detectable orphan.
Link: https://lore.kernel.org/20260713-catalin_pto-v1-4-5b93b1131089@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The reporting loop in kmemleak_scan() decided whether to tag an object as
a reported leak with a four-term compound condition whose last operand
also had a side effect (++object->unref_scans). Mixing the candidate
tests with the counter update made the check hard to read.
Move the state transition into confirm_leak(): it returns true when a
still-unreferenced suspect crosses min_unref_scans consecutive scans and
is newly flagged OBJECT_REPORTED, leaving only the reporting bookkeeping
in the caller. No functional change.
Link: https://lore.kernel.org/20260713-catalin_pto-v1-3-5b93b1131089@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
kmemleak reports an object the first scan it is found unreferenced. Its
mark phase runs without stopping the rest of the kernel and without a
write barrier, so a live object whose only reference is briefly invisible
during a concurrent RCU update -- e.g. a VMA moved between maple tree
nodes, or a page-cache xa_node -- can be seen as unreferenced for that one
scan. Because an object is flagged as reported only once, such a
transient race turns into a permanent false positive.
Track how many consecutive scans each object has been seen unreferenced
and only report it once that reaches min_unref_scans, a new module
parameter. It defaults to 1, leaving the behaviour unchanged; setting it
higher (e.g. 2) still reports a genuine leak, one scan later, while an
object referenced again before the threshold restarts its run and is never
reported.
min_unref_scans can be set at boot with kmemleak.min_unref_scans=<n> or at
run-time via /sys/module/kmemleak/parameters/min_unref_scans.
Link: https://lore.kernel.org/20260713-catalin_pto-v1-2-5b93b1131089@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: kmemleak: reduce transient false positives by confirming
leaks".
This series combines two kmemleak enhancements that were originally
submitted separately but both required rebasing after commit 79c37ae3733e9
("mm/kmemleak: fix checksum computation for per-cpu objects").
The first feature introduces a second scan to confirm suspected leaks:
https://lore.kernel.org/all/20260709173347.689607-1-catalin.marinas@arm.com/
The second feature adds a module parameter controlling the minimum number
of consecutive unreferenced scans before a leak is reported, as discussed
in:
https://lore.kernel.org/all/20260626-kmemleak_twice-v1-0-ab28f7cc0971@debian.org/
Changes from v1:
Now that commit 79c37ae3733e9 is upstream, the selftest includes an additional
priming phase scan as requested by Catalin.
Additionally, I've factored out the leak-detection conditional into a helper
function to be more digestible for the reader's eye.
This 4-patch series resolves all outstanding kmemleak issues I've been
tracking.
This patch (of 4):
The kmemleak marking phase is not atomic. While the object graph is
traversed, the kernel can modify pointers, free objects or allocate new
ones. If a reference to an object is moved from one location to another,
kmemleak scanning may miss it. We have explicit annotations like
kmemleak_transient_leak() but identifying and maintaining them is not
trivial.
Given that such transient leaks are short-lived, rather than just
reporting such objects as leaks, do another scan to confirm the suspected
objects. If no new leaks are found during the first scan, skip the
confirmation one.
Link: https://lore.kernel.org/20260713-catalin_pto-v1-0-5b93b1131089@debian.org
Link: https://lore.kernel.org/20260713-catalin_pto-v1-1-5b93b1131089@debian.org
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Breno Leitao <leitao@debian.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The pin longterm test currently stores its data globally, shared among
multiple concurrent users of the interface (multiple open file descriptors
-> multiple "struct file"'s). That makes the gup_test interface
problematic to use concurrently: two users, such as concurrent selftest
runs, can interfere with the same longterm pin state.
While this has not been observed as a problem so far in practice, let's
just handle it cleanly. There could be a way to trigger selftest failures
by e.g., running the cow.c and gup_longerm.c selftests concurrently, but
we usually run them sequentially. Let's add a "Fixes" tag to be safe, but
not need to CC stable.
Link: https://lore.kernel.org/20260810-gup_test_data-v1-1-fb1d41be5bb4@kernel.org
Fixes: c77369b437f9 ("mm/gup_test: start/stop/read functionality for PIN LONGTERM test")
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reported-by: yunhui cui <cuiyunhui@bytedance.com>
Closes: https://lore.kernel.org/r/20260608025043.88087-1-cuiyunhui@bytedance.com
Tested-by: Yunhui Cui <cuiyunhui@bytedance.com>
Tested-by: Lance Yang <lance.yang@linux.dev>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Yang Li <yang.lee@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
kasan_quarantine_remove_cache() first invokes per_cpu_remove_cache() on
all online CPUs. Each callback moves objects belonging to the cache from
cpu_quarantine to the CPU's shrink_qlist, where they can later be freed
from task context.
kmem_cache_destroy() invokes the quarantine removal path while holding
cpus_read_lock(), but kmem_cache_shrink() does not. The latter can
therefore race with CPU offlining as follows:
kmem_cache_shrink() CPU hotplug
------------------- -----------
on_each_cpu()
CPU1 moves objects to
CPU1's shrink_qlist
on_each_cpu() returns
CPU1 goes offline
kasan_cpu_offline()
drains cpu_quarantine
leaves shrink_qlist untouched
for_each_online_cpu()
skips CPU1
The objects left on CPU1's shrink_qlist are not returned to the slab
allocator. This may prevent kmem_cache_shrink() from releasing slabs that
would otherwise become empty. If CPU1 remains offline, a later
kmem_cache_destroy() also skips the list and can report that the cache
still contains objects.
An intermittent occurrence was observed with a virtio-9p filesystem. The
mount and umount commands both returned 0, but the kernel logged the
following during the userspace-triggered teardown:
[ 2994.380134][ T111] BUG 9p-fcall-cache-1 (Tainted: G B ): Objects remaining on __kmem_cache_shutdown()
[ 2994.381140][ T111] Object 0xff11000004361118 @offset=4376
[ 2994.381607][ T111] Allocated in p9_fcall_init+0x201/0x400 age=19564 cpu=1 pid=104
[ 2994.382591][ T111] p9_fcall_init+0x201/0x400
[ 2994.382810][ T111] p9_tag_alloc+0x12f/0x700
[ 2994.382982][ T111] p9_client_prepare_req+0x102/0x3e0
[ 2994.383165][ T111] p9_client_rpc+0x1ab/0xa50
[ 2994.383334][ T111] p9_client_getattr_dotl+0xb0/0x1a0
[ 2994.383515][ T111] v9fs_vfs_getattr_dotl+0x115/0x360
[ 2994.383719][ T111] vfs_getattr_nosec+0x22c/0x3a0
[ 2994.383910][ T111] vfs_statx+0xd7/0x170
[ 2994.384062][ T111] vfs_fstatat+0x45/0x80
[ 2994.384215][ T111] __do_sys_newfstatat+0x84/0xe0
[ 2994.384386][ T111] do_syscall_64+0x115/0x6a0
[ 2994.384566][ T111] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 2994.399720][ T111] WARNING: mm/slub.c:1244 at __kmem_cache_shutdown+0x363/0x500, CPU#0: busybox/111
[ 2994.405655][ T111] Call Trace:
[ 2994.406325][ T111] kmem_cache_destroy+0x73/0x1b0
[ 2994.406630][ T111] p9_client_destroy+0x271/0x3c0
[ 2994.407210][ T111] v9fs_session_close+0x3c/0x260
[ 2994.407409][ T111] v9fs_kill_super+0x48/0x90
[ 2994.407584][ T111] deactivate_locked_super+0xa3/0x160
[ 2994.407778][ T111] cleanup_mnt+0x1dd/0x3e0
Thus, a successful umount left objects in the 9p fcall cache and prevented
the cache from being destroyed cleanly.
Per-CPU shrink_qlist storage exists for every possible CPU, and each list
is protected by its own raw spinlock. Iterate over possible CPUs so that
a list populated before its CPU went offline is drained as well.
for_each_possible_cpu() can do more work than for_each_online_cpu(), but
this change only affects CONFIG_KASAN_GENERIC kernels. The extra work is
limited to cache shrink and cache destruction paths and does not affect
the normal allocation/free fast path. It adds one raw-spinlock-protected
scan of each possible CPU's shrink list. These lists are normally empty;
a non-empty list is traversed to remove objects belonging to the cache
being shrunk or destroyed.
Link: https://lore.kernel.org/20260808031459.3032812-1-sh_def@163.com
Fixes: 07d067e4f2ce ("kasan: fix sleeping function called from invalid context on RT kernel")
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: "Zhang, Qiang1" <qiang1.zhang@intel.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When no name is supplied, cma_new_area() generates names with format
"cma%d\n", introducing an unintended newline character ('\n') in the CMA
name.
Most CMA regions are created with explicit names, so this path is seldom
hit. The newline only creates cosmetic noise in debug logs, traces and
debugfs with no functional impact.
Link: https://lore.kernel.org/20260810093215.91419-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the
global cgroup rstat lock synchronously. On machines with many CPUs and
NUMA nodes, this creates severe lock contention in the kswapd reclaim
path:
- Multiple kswapd threads (one per NUMA node) run concurrently.
- do_shrink_slab() invokes zswap_shrinker_count() for each
memcg-aware shrinker pass.
- Each call flushes the full cgroup rstat hierarchy under the global
lock.
On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running
production workloads with zswap enabled, perf shows 2.88% of kernel cycles
in osq_lock contention from this path:
2.88% [k] osq_lock
--__mutex_lock.constprop.0
--__cgroup_rstat_lock
--cgroup_rstat_flush_locked
--cgroup_rstat_flush
--zswap_shrinker_count
do_shrink_slab
shrink_slab
shrink_node
balance_pgdat
kswapd
84% of kswapd kernel cycles are spent in
shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual
page reclaim (shrink_lruvec).
Controlled A/B on identical hardware and workload:
shrinker=Y: 2.88% osq_lock, memory PSI 1.58%
shrinker=N: 0.00% osq_lock, memory PSI 0.57%
eBPF-based rstat lock wait measurement across 8 production metals
confirms the contention splits cleanly along shrinker enablement:
shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s)
shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s)
zswap_shrinker_count() only produces a heuristic estimate, scaled by
compression ratio via mult_frac(). The actual writeback happens in
zswap_shrinker_scan(). Slightly stale stats are acceptable here.
Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if
the periodic 2-second flusher is one full cycle late. This matches the
approach already used in prepare_scan_control() (mm/vmscan.c) for the
same reclaim path.
After applying this patch, rstat flush latency and lock wait time on
shrinker=Y machines dropped to the same level as shrinker=N controls,
while the zswap shrinker continues to function (pool size remains
bounded under the max_pool_percent cap).
Previously discussed:
- Chengming Zhou (Dec 2023): rstat contention from
zswap_shrinker_count [1]
- Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync
flush [2]
- Yosry Ahmed (Aug 2024): suggested eliminating in-kernel
flushers [3]
- Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4]
Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com
Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1]
Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3]
Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4]
Suggested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Yunzhao Li <yunzhao@cloudflare.com>
Tested-by: Yunzhao Li <yunzhao@cloudflare.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Yosry Ahmed <yosry@kernel.org>
Cc: Yunzhao Li <yunzhao@cloudflare.com>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The !CONFIG_SWAP stub for swap_cluster_lock() has mismatched prototype: it
has an extra unused irq argument and uses pgoff_t instead of unsigned long
for offset. All callers are under CONFIG_SWAP so the extra parameter is
dead.
Delete the unused stub function entirely.
Link: https://lore.kernel.org/20260717071104.73467-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Acked-by: Kairui Song <kasong@tencent.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Hongfu Li <lihongfu@kylinos.cn>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
