| Age | Commit message (Collapse) | Author |
|---|
|
The "we just allocated them without exposing them to the swapcache" case
no longer exists, as Kairui has routed synchronous I/O through the
swapcache as well in his series "unify swapin use swap cache and cleanup
flags"[1]. As a result, folio_ref_count() should never be 1 in this path,
since at least two references are held (base ref plus swapcache). Remove
the folio_ref_count()==1 check and update the comment accordingly.
The ksm_might_need_to_copy() check may allocate a fresh folio with
folio_ref_count() == 1. Along that path, exclusive has already been
set to true, so the folio can still be reused correctly.
[akpm@linux-foundation.org: update comment, per David]
Link: https://lore.kernel.org/3a185a5d-2f2c-4e9d-9cd9-8bdb236dfc5c@kernel.org
Link: https://lore.kernel.org/20260701235955.36126-3-baohua@kernel.org
Link: https://lore.kernel.org/all/20251220-swap-table-p2-v5-0-8862a265a033@tencent.com/ [1]
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: drop redundant lru_add_drain in anon folio reuse paths",
v3.
We are doing a large number of redundant lru_add_drain() calls in both
wp_can_reuse_anon_folio() and do_swap_page(), leading to LRU lock
contention and unnecessary overhead.
In wp_can_reuse_anon_folio(), we can check the refcount against the
lru_cache before deciding to drain. In do_swap_page(), the drain is now
entirely redundant after Kairui's work to route SYNC I/O through the
swapcache in the same way as ASYNC I/O.
Build the kernel within a 1 GB memcg using 20 threads with zRAM swap. The
number of lru_add_drain() calls is reduced from 276,278 to 226,318, a
reduction of about 18%.
Build the kernel within an 800 MB memcg using 20 threads with zRAM swap.
The number of lru_add_drain() calls is reduced from 778,950 to 541,149, a
reduction of 30.5%.
This patch (of 4):
There is a case where `folio_ref_count(folio) == 3` and
`!folio_test_swapcache(folio)`. In that case, both
`folio_ref_count(folio) > 3` and
`folio_ref_count(folio) > 1 + folio_test_swapcache(folio)` evaluate
false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains.
For a kernel build with a minimal configuration, I observed more than
20,000 redundant drains.
Fix this by checking against: `1 + in_swapcache + in_lrucache` instead of
hardcoding `folio_ref_count(folio) > 3`.
[baohua@kernel.org: rename in_lru_cache to maybe_in_lru_cache]
Link: https://lore.kernel.org/20260708145718.82690-1-baohua@kernel.org
Link: https://lore.kernel.org/20260701235955.36126-1-baohua@kernel.org
Link: https://lore.kernel.org/20260701235955.36126-2-baohua@kernel.org
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Tail pages are expected to have (and optionally be checked) zeroed
->private when they are freed. It stays true during subsequent
reallocation, so replace the tail_page->private initialization with a
VM_WARN_ON_ONCE() in compound page preparation.
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-5-7e4fe155f5b9@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Any code using tail_page->private of a folio, a compound page or a
high-order page is supposed to reset it after use, otherwise ->private
data can leak to new page user and cause unexpected issues. Add a
bad_page() check at page free path for it.
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-4-7e4fe155f5b9@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
page->private should not be set in tail pages. Commit 4265d67e405a
("mm/migrate_device: add THP splitting during migration") removed a
DEBUG_VM check without a proper reason[1]. Add it back.
Link: https://lore.kernel.org/all/13f3fcda-7328-4aa5-afc6-75a294a82b2a@nvidia.com/ [1]
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-3-7e4fe155f5b9@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit 733aea0b3a7bb ("mm/compaction: add support for >0 order folio
memory compaction.") stores isolated free pages in an array indexed by
free page orders, it is no longer needed to store the order in each page's
->private field. And there is no code using the stored order. Stop doing
that.
It also prepares for an upcoming change that ensures subpage->private is
zero at page free time and the removal of set_page_private(0) from
prep_compound_tail().
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-2-7e4fe155f5b9@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "Keep tail page private zero at free and folio split", v3.
This patchset makes sure tail_page->private is zero before compound or
high-order pages are returned to the allocator. It also checks tail pages
that become new folio heads during large folio split, before their private
fields are used by new folios.
Note on ZONE_DEVICE and DAX page/folio
===
ZONE_DEVICE and DAX use prep_compound_tail() to reinitialize folios, so
tail_page->private was reset before this patchset. There was a concern
that after this patchset stale ->private can appear after ZONE_DEVICE/DAX
folio initialization. My reasoning is that no code sets ZONE_DEVICE/DAX
page->private, so their page->private stays zero all the time.
ZONE_DEVICE_PRIVATE page migration only supports anonymous memory without
swapcache, so after the migration ->private remains zero.
But let me know if my reasoning is wrong. It can be fixed by adding
->private zeroing code in ZONE_DEVICE/DAX folio initialization code.
Motivation
===
page->private is zeroed at page free time since commit ac1ea219590c0
("mm/page_alloc: clear page->private in free_pages_prepare()"), since we
concluded that it might be too much to ask every page user to free a page
with ->private zeroed. The holder of the last page reference might not
know whether ->private needs to be cleared.
For compound and high-order pages, tail_page->private can also leak to
later users if it is left uncleared. The page allocation path does not
zero every tail_page->private field, so they can be seen by new users and
cause unexpected issues[1].
Check tail_page->private at page free time, and check tail pages that
become new folio heads during large folio split. With those checks in
place, prep_compound_tail() no longer needs to clear tail_page->private
when preparing compound page metadata.
Overview
===
1. Patch 1 clears all pages ->private before percpu-km frees them.
2. Patch 2 removes setting page->private in compaction code when a free
page is taken out of the buddy allocator. cc->freepages is indexed by
page order, so storing the free page order in page->private is
redundant. In alloc_contig_frozen_range_noprof(),
isolate_freepages_range() is used to grab free pages from buddy
allocator and it leaves the aforementioned page->private set until
either split_free_frozen_pages() or prep_new_page() is called. That
stale value without resetting triggers the tail_page->private nonzero
check once set_page_private(0) is removed from prep_compound_tail().
3. Patch 3 adds back the page->private check for tail pages promoted to new
folio heads in __split_folio_to_order().
4. Patch 4 adds a tail_page->private check in the page free path.
5. Patch 5 removes tail_page->private zeroing from prep_compound_tail().
This patch (of 5):
page->private is cleared in free page path. In a subsequent commit,
tail_page->private will be checked and ensured to be zero. Clearing
percpu-km allocated pages' ->private to prevent triggering warnings later,
namely undo what we did in pcpu_create_chunk().
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-0-7e4fe155f5b9@nvidia.com
Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-1-7e4fe155f5b9@nvidia.com
Link: https://lore.kernel.org/all/20260206174017.128673-1-mikhail.v.gavrilov@gmail.com/ [1]
Signed-off-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_call() failure was causing the context to be stopped, but
asynchronously by the kdamond thread. To handle the race, the caller had
to stop the context. damon_call() is updated to do the stop on its own.
Remove the damon_stop() call from the damon_call() error handling.
Link: https://lore.kernel.org/20260706140628.87414-12-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_call() failure was causing the context to be stopped, but
asynchronously by the kdamond thread. To handle the race, the caller had
to stop the context. damon_call() is updated to do the stop on its own.
Remove the damon_stop() call from the damon_call() error handling.
Link: https://lore.kernel.org/20260706140628.87414-11-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_call() failure means the DAMON context started its termination. The
termination is asynchronously done in kdamond thread. The caller's error
handling should handle the race, too. It is complicated and easy to make
mistakes.
Update damon_call() to ensure the context is stopped in the case, by
waiting until the completion is confirmed.
Link: https://lore.kernel.org/20260706140628.87414-10-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() was theoretically able to return failure while keeping the
second context for mtier running. mtier stops its contexts one by one
with two damon_stop() call for the reason. damon_stop() is refactored to
always successfully stop all requested DAMON contexts. The two calls are
unnecessary. Use only single damon_stop() call for all contexts.
Link: https://lore.kernel.org/20260706140628.87414-9-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() always returns 0, and nobody cares. Change the return type
to void.
Link: https://lore.kernel.org/20260706140628.87414-8-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() return value is guaranteed to be 0. Ignore it.
Link: https://lore.kernel.org/20260706140628.87414-7-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() return value is guaranteed to be 0. Ignore it.
Link: https://lore.kernel.org/20260706140628.87414-6-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() return value is guaranteed to be 0. Ignore it.
Link: https://lore.kernel.org/20260706140628.87414-5-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_stop() has no reason to fail. It returns an error code only for
possible future changes that can make it fail. Such a change has not been
made yet, and this only makes the error handling complicated and
confusing. Ensure it returns no error.
Link: https://lore.kernel.org/20260706140628.87414-4-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_start() was able to fail while keeping the first context running.
mtier hence stops the first context in the case. damon_start() is
refactored to avoid that. The error handling is no longer necessary.
Remove it.
Link: https://lore.kernel.org/20260706140628.87414-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: refactor damon_{start,stop,commit}() for simple
error handling".
damon_start() and damon_stop() could leave a few of requested DAMON
contexts running when they return an error. damon_commit() failure stops
the DAMON context, but in an asynchronous way. Callers should stop the
left-over DAMON contexts. It is easy to make mistakes, and indeed a few
bugs from such mistakes were found and fixed.
Refactor the core API functions to guarantee contexts are completely
stopped under failures. Remove the caller side error handlers that are no
longer needed due to the refactoring.
Patches Sequence
================
Patch 1 refactors damon_start() to ensure all contexts are stopped for
failures. Patch 2 updates unnecessary damon_start() error handling from
mtier sample module.
Patch 3 refactors damon_stop() to always success. Patches 4-6 updates
callers (damon_{sysfs,reclaim lru_sort}) to ignore the return value.
Patch 7 update damon_stop() return value to void. Patch 8 simplifies
damon_stop() error handling in mtier sample module.
Patch 9 refactors damon_call() to return errors only after the context
is completely stopped. Patches 10 and 11 remove unnecessary error
handlings from callers (wsse and prcl sample modules).
This patch (of 11):
When multiple contexts are passed to damon_start(), the function starts
the contexts one by one. If any of the operations fails, it immediately
returns an error. Contexts that successfully started before the failure
keep running. The caller should catch this and stop the contexts. It is
complicated and easy to make mistakes. Stop all contexts in damon_start()
under the failures.
Link: https://lore.kernel.org/20260706140628.87414-1-sj@kernel.org
Link: https://lore.kernel.org/20260706140628.87414-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
In the ensure_file() function, the "not_exist" code path checks whether
$dir exists as a regular file. However, the intent is to verify that the
target file ($file) does not exist, not the $dir. Testing $dir makes the
existence check effectively useless -- it tests the wrong path and thus
never catches the case where the file is unexpectedly present.
Replace $dir with $file so the not_exist verification targets the correct
path.
Link: https://lore.kernel.org/20260706134305.5224-1-a929244872@163.com
Signed-off-by: wang wei <a929244872@163.com>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add the required space before the opening parenthesis in switch statements
to conform to kernel coding style.
Link: https://lore.kernel.org/20260623061730.2024-1-git@tazy.dev
Signed-off-by: Watson Wheeler <git@tazy.dev>
Reviewed-by: Liam Howlett <liam@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a macro that
only gets defined by the Makefiles in the kernel. This can be very
confusing when switching between userspace and kernelspace coding, or when
dealing with uapi headers that rather should use __ASSEMBLER__ instead.
So let's standardize now on the __ASSEMBLER__ macro that is provided by
the compilers.
This is a completely mechanical patch (done with a simple "sed -i"
statement.
Link: https://lore.kernel.org/20260619154704.41293-1-thuth@redhat.com
Signed-off-by: Thomas Huth <thuth@redhat.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add documentation for the new vm.panic_on_unrecoverable_memory_failure
sysctl, describing which failures trigger a panic (kernel-owned pages the
handler cannot recover) and which are intentionally left out (transient
allocator races and unclassified pages).
Link: https://lore.kernel.org/20260630-ecc_panic-v10-5-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add a sysctl panic_on_unrecoverable_memory_failure (disabled by default)
that triggers a kernel panic when memory_failure() encounters pages that
cannot be recovered. This provides a clean crash with useful debug
information rather than allowing silent data corruption or a delayed crash
at an unrelated code path.
Panic eligibility is intentionally narrow: only MF_MSG_KERNEL with result
== MF_IGNORED panics. After the previous patch, MF_MSG_KERNEL covers
PG_reserved pages and the kernel-owned pages promoted from
get_hwpoison_page() via -ENOTRECOVERABLE (slab, page tables,
large-kmalloc).
All other action types are excluded:
- MF_MSG_GET_HWPOISON and MF_MSG_KERNEL_HIGH_ORDER can be reached by
transient refcount races with the page allocator (an in-flight buddy
allocation has refcount 0 and is no longer on the buddy free list,
briefly), and panicking on them would risk killing the box for what
is actually a recoverable userspace page.
- MF_MSG_UNKNOWN means identify_page_state() could not classify the
page; that is precisely the wrong basis for a panic decision.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-4-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The previous patch teaches get_any_page() to return -ENOTRECOVERABLE for
stable unhandlable kernel pages (PG_reserved, slab, page tables,
large-kmalloc). memory_failure() still folds every negative return into
MF_MSG_GET_HWPOISON, so callers that want to react to the unrecoverable
cases (a panic option, smarter logging) cannot tell them apart from
transient page-allocator races.
Turn the post-call branch into a switch over the get_hwpoison_page()
return code: map -ENOTRECOVERABLE to MF_MSG_KERNEL and any other negative
return to MF_MSG_GET_HWPOISON. case 0 keeps the existing free-buddy /
kernel-high-order handling and case 1 falls through to the rest of
memory_failure() unchanged.
The MF_MSG_KERNEL label and tracepoint string are kept as "reserved kernel
page" to avoid breaking userspace tools that match on those literals; the
enum value still adequately tags the failure even though it now also
covers slab, page tables and large-kmalloc pages.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-3-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
get_any_page() collapses every HWPoisonHandlable() rejection into a
single -EIO via the __get_hwpoison_page() -> -EBUSY -> shake_page()
-> retry path. That is correct for the transient case (a userspace
folio briefly off LRU during migration or compaction, which a later
shake can drag back), but wrong for stable kernel-owned pages: slab,
page-table, large-kmalloc and PG_reserved pages will never become
HWPoisonHandlable(), so the retry loop is wasted work and the final
-EIO loses the "this is structurally unrecoverable" information.
memory_failure() then maps -EIO into MF_MSG_GET_HWPOISON, which the
panic-on-unrecoverable sysctl deliberately does not act on.
Introduce is_kernel_owned_page(), a small predicate that positively
identifies pages the hwpoison handler cannot recover from:
is_kernel_owned_page(p) :=
PageReserved(p) ||
PageSlab(head) || PageTable(head) || PageLargeKmalloc(head)
where head = compound_head(p).
PG_reserved is a per-page flag (PF_NO_COMPOUND) and is tested on the
page directly. The slab, page-table and large-kmalloc page-type bits
are only stored on the head page, so those tests resolve the compound
head first, then re-read compound_head(page) afterwards: a concurrent
split or compound free that moves head invalidates the just-read flags
and the loop retries. The lookup still takes no refcount, mirroring
the rest of get_any_page(); the recheck closes the common split race,
and a residual free->alloc->free in the same window can only mis-tag
a genuinely poisoned page, never reclassify a handlable one.
No MF_SOFT_OFFLINE / page_has_movable_ops() opt-out is needed: a
movable_ops page is always PageOffline or PageZsmalloc, whose page_type is
mutually exclusive with slab, page-table and large-kmalloc, and it never
carries PG_reserved, so it can never match any of the checks above.
The list is intentionally not exhaustive. vmalloc and kernel-stack pages,
for example, do not carry a page_type bit and would need a different
oracle; they keep going through the existing retry path unchanged. This
is the smallest set we can identify with certainty by page type.
Wire the helper into the top of get_any_page() to short-circuit those
pages before the retry loop runs. On a hit, drop the caller's
MF_COUNT_INCREASED reference (if any) and return -ENOTRECOVERABLE straight
away. Pages outside the helper's positive list still take the existing
retry path and return -EIO, leaving operator-visible behaviour for those
cases unchanged.
Extend the unhandlable-page pr_err() to fire for either errno and update
the get_hwpoison_page() kerneldoc to document the new return.
memory_failure() still folds every negative return into
MF_MSG_GET_HWPOISON via its existing "else if (res < 0)" branch, so this
patch on its own only changes the errno that soft_offline_page() can
propagate to its callers. A follow-up wires -ENOTRECOVERABLE through
memory_failure() and reports MF_MSG_KERNEL for the unrecoverable cases,
which is what the panic_on_unrecoverable_memory_failure sysctl observes.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-2-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Suggested-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/memory-failure: add panic option for unrecoverable
pages", v10.
A multi-bit ECC error on a kernel-owned page that the memory failure
handler cannot recover is currently swallowed: PG_hwpoison is set, the
event is logged, and the kernel keeps running. The corrupted memory
remains accessible to the kernel and either drives silent data corruption
or surfaces seconds-to-minutes later as an apparently unrelated crash. In
a large fleet that delayed, unattributable crash turns into significant
engineering effort to root-cause; in a kdump configuration, by the time
the crash happens the original error context (faulting PFN, MCE/GHES
record, page state) is long gone.
This series adds an opt-in sysctl,
vm.panic_on_unrecoverable_memory_failure, that converts an unrecoverable
kernel-page hwpoison event into an immediate panic with a clean
dmesg/vmcore that still contains the original failure context. The
default is disabled so existing workloads see no change.
There is a selftest that test different cases, and I tested it using
the following variants:
┌─────────┬──────────┬───────────────────────────────────────────────────────────┐
│ Variant │ PFN │ Result │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ rodata │ 0x2600 │ Panic with "Memory failure: 0x2600: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ slab │ 0x100032 │ Panic with "Memory failure: 0x100032: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ pgtable │ 0x100000 │ Panic with "Memory failure: 0x100000: unrecoverable page" │
└─────────┴──────────┴───────────────────────────────────────────────────────────┘
Each one shows the same call trace, exactly the path the series builds:
hard_offline_page_store
→ memory_failure
→ action_result
→ panic("Memory failure: %#lx: unrecoverable page")
This patch (of 5):
The first entry of error_states[],
{ reserved, reserved, MF_MSG_KERNEL, me_kernel },
is unreachable. identify_page_state() has two callers, and neither
one can dispatch a PG_reserved page to me_kernel():
* memory_failure() reaches identify_page_state() only after
get_hwpoison_page() returned 1. get_any_page() reaches that
return only via __get_hwpoison_page(), which only takes a
refcount when the page is HWPoisonHandlable().
HWPoisonHandlable() is an allowlist for LRU, free-buddy, and
(for soft-offline) movable_ops pages -- PG_reserved pages do
not satisfy any of these, so they fail with -EBUSY/-EIO long
before identify_page_state() runs.
* try_memory_failure_hugetlb() reaches identify_page_state() only
via the MF_HUGETLB_IN_USED branch, where the page is necessarily
a hugetlb folio. hugetlb folios don't carry PG_reserved at that
point: hugetlb_folio_init_vmemmap() calls __folio_clear_reserved()
during init, so the reserved entry would not match even if it
were still present.
me_kernel() never executes and the entry exists only to be matched
against by code that cannot see it.
Drop the entry, the me_kernel() helper, and the now-unused
"reserved" macro. Leave the MF_MSG_KERNEL enum value in place: it
remains part of the tracepoint and pr_err() string tables, and
follow-on work to classify unrecoverable kernel pages can reuse it
without churning the user-visible enum.
No functional change.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-0-c6ed5b62eea2@debian.org
Link: https://lore.kernel.org/20260630-ecc_panic-v10-1-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The legacy memory.limit_in_bytes and memory.memsw.limit_in_bytes writers
retry page_counter_set_max() by reclaiming synchronously in the writer
context. memory.force_empty similarly loops in synchronous reclaim until
the cgroup is empty or reclaim stops making progress.
These writes hold a kernfs active reference on the file. If cgroup
removal starts in parallel, the remover sets CSS_DYING and then waits in
kernfs_drain() under cgroup_mutex for the active reference to drain.
Continuing reclaim after the memcg is dying can therefore delay cgroup
removal and keep cgroup_mutex held for a long time.
Stop the v1 reclaim loops once the memcg is dying. For limit resizing,
keep the existing -EBUSY semantics when the new limit could not be
installed. For memory.force_empty, keep the existing best-effort success
semantics.
Link: https://lore.kernel.org/20260702120235.376752-5-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jiayuan Chen <jiayuan.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Proactive reclaim via memory.reclaim can run for a long time - swap I/O
or thrashing again dominating the latency - and delays cgroup removal in
the same way.
Mitigate this by stopping the reclaim once memcg_is_dying().
Link: https://lore.kernel.org/20260702120235.376752-4-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jiayuan Chen <jiayuan.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
memory.max has the same high-latency reclaim loop as memory.high, and may
additionally invoke the OOM killer on a cgroup that is already going away,
further delaying its removal.
Mitigate this by bailing out of the loop once memcg_is_dying().
Link: https://lore.kernel.org/20260702120235.376752-3-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jiayuan Chen <jiayuan.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "memcg: bail out reclaim when memcg is dying", v3.
This series mitigates a system-wide stall we hit when a cgroup is
removed while one of its memory control files is doing synchronous
reclaim.
Problem Description
===================
Writing to memory.high, memory.max or memory.reclaim runs reclaim
synchronously in the writer's context, looping until the usage drops below
the target (or, for memory.reclaim, until the requested amount has been
reclaimed). On a large cgroup this can take a long time. The latency is
especially bad when reclaim has to perform swap I/O, where it is bound by
the swap device write bandwidth, and under thrashing it is effectively
unbounded - each round reclaims a few pages that the workload immediately
faults back in, so the loop keeps making "progress" and never converges.
The legacy (v1) reclaim loops in memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty share the same pattern.
These writes go through cgroup_file_write(), which does not take
cgroup_mutex and does not pin the css. Instead, kernfs guarantees the
node (and thus the css) stays alive for the duration of the operation by
holding an active reference. So while the reclaim loop runs, the active
reference on the file is held.
If another task removes the same cgroup in parallel, cgroup_rmdir() takes
cgroup_mutex and then blocks in kernfs_drain() waiting for that active
reference to drain. Because cgroup_mutex is held throughout the wait,
every other task that needs it piles up behind the remover - in our case
the whole machine ground to a halt, with hung_task reports for the remover
and for unrelated tasks merely reading /proc/<pid>/cgroup:
INFO: task cgdelete:366634 blocked for more than 159 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
kernfs_drain+0xe6/0x150
__kernfs_remove.part.0+0xd0/0x200
kernfs_remove_by_name_ns+0x75/0xd0
cgroup_addrm_files+0x325/0x410
css_clear_dir+0x50/0xf0
cgroup_destroy_locked+0xdf/0x1e0
cgroup_rmdir+0x2d/0xd0
kernfs_iop_rmdir+0x53/0x90
vfs_rmdir+0x98/0x240
do_rmdir+0x172/0x1b0
__x64_sys_rmdir+0x42/0x70
x64_sys_call+0xeb0/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
INFO: task systemd-journal:2352 blocked for more than 182 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
schedule_preempt_disabled+0xe/0x20
__mutex_lock.constprop.0+0x3bb/0x640
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x3c/0x50
proc_cgroup_show+0x4d/0x380
proc_single_show+0x53/0xe0
seq_read_iter+0x12f/0x4b0
seq_read+0xcd/0x110
vfs_read+0xb1/0x360
? __seccomp_filter+0x368/0x590
ksys_read+0x73/0x100
__x64_sys_read+0x19/0x30
x64_sys_call+0x18d3/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
The system recovers only once the reclaim finally finishes and releases
the active reference. The reclaim itself is pointless here: the cgroup is
being torn down and its remaining pages will be reparented to the parent
anyway.
Even though we check signal_pending(current) in the reclaim loop, the
typical symptom is that cat /proc/<pid>/cgroup gets stuck. By the time
someone looks for which task is actually stuck in reclaim, the hung task
timeout has already been hit. This makes the problem particularly nasty
to debug from a hung-task report alone, because the blocked tasks shown
are often the victims, not the reclaim writer itself.
Our Mitigation
==============
cgroup destruction sets CSS_DYING in kill_css_sync() *before*
css_clear_dir() triggers the kernfs_drain() that blocks the remover. The
in-flight reclaim loop is therefore guaranteed to observe it before
starting another reclaim iteration. This series checks memcg_is_dying()
in the v2 reclaim loops (memory.high, memory.max and proactive reclaim)
and the v1 reclaim loops (memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty), and bails out early,
so the writer drops the active reference promptly and the remover can make
progress.
Unlike the no-progress guard (MAX_RECLAIM_RETRIES), which only fires when
reclaim makes zero progress, the dying check also covers the slow swap I/O
and thrashing cases, where reclaim keeps succeeding a little and the loop
would otherwise never converge.
For memory.reclaim, bailing out because the memcg is dying means the
requested reclaim amount was not satisfied, so the write returns -EAGAIN.
This is orthogonal to commit c8e6002bd611 ("memcg: introduce non-blocking
limit setting option"): O_NONBLOCK lets a caller avoid the synchronous
reclaim up front, while this series handles the case where reclaim is
already running when the cgroup starts being removed.
This patch (of 4):
memory.high reclaims synchronously in the writer's context, and the
latency can be very high - especially when reclaim performs swap I/O, or
under thrashing where the loop may not converge for a long time.
While this runs the kernfs active reference on the file is held, so a
concurrent removal of the same cgroup blocks in kernfs_drain() under
cgroup_mutex until it finishes. Reclaiming a dying cgroup is pointless,
as its pages are reparented to the parent anyway.
Mitigate this by bailing out of the reclaim loop once memcg_is_dying().
Link: https://lore.kernel.org/20260702120235.376752-1-jiayuan.chen@linux.dev
Link: https://lore.kernel.org/20260702120235.376752-2-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
vmscan.c changes which are required by "memcg: bail out proactive reclaim
when memcg is dying".
|
|
mlx5e_queue_start() deactivates and re-activates all channels but closes
only the queue being restarted. mlx5e_activate_txqsq() then
unconditionally calls netdev_tx_reset_queue(), zeroing the BQL counters
of channels that kept their in-flight TX WQEs. The next completion then
over-charges and trips the BUG_ON() in dql_completed():
kernel BUG at lib/dynamic_queue_limits.c:99!
RIP: 0010:dql_completed+0x23d/0x280
Call Trace:
<IRQ>
mlx5e_poll_tx_cq+0x668/0xa60
mlx5e_napi_poll+0x5b/0x7b0
net_rx_action+0x15a/0x580
Reset BQL only when the SQ has no bytes in flight (sq->cc == sq->pc).
In the case that reset is skipped, the outstanding WQEs will eventually
complete and rebalance the dql. The dql->limit is carried across the
reset.
Fixes: b2588ea40ec9 ("net/mlx5e: Implement queue mgmt ops and single channel swap")
Signed-off-by: Bobby Eshleman <bobbyeshleman@meta.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
Link: https://patch.msgid.link/20260803-mlx5-bql-v3-1-a30d4c66fe1d@meta.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
bonding_init() ignores register_netdevice_notifier() errors and still
returns success, which can leave the bonding module loaded without its
netdev notifier registered.
Check the error and unwind prior initialization on failure.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Acked-by: Jay Vosburgh <jv@jvosburgh.net>
Link: https://patch.msgid.link/20260803090012.142638-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
vrf_init_module() ignores register_netdevice_notifier() errors and
continues module initialization, which can leave VRF loaded without its
netdev notifier registered.
Check the error and fail module initialization early.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Reviewed-by: David Ahern <dsahern@kernel.org>
Link: https://patch.msgid.link/20260803090002.142453-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
macvlan_init_module() ignores register_netdevice_notifier() errors and
continues module initialization, which can leave macvlan loaded without
its netdev notifier registered.
Check the error and fail module initialization early.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Link: https://patch.msgid.link/20260803085950.142325-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
team_module_init() ignores register_netdevice_notifier() errors and
continues module initialization, which can leave the team module loaded
without its netdev notifier registered.
Check the error and fail module initialization early.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Link: https://patch.msgid.link/20260803085943.142261-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
lapbeth_init_driver()
lapbeth_init_driver() ignores register_netdevice_notifier() errors and
always returns success, which can leave the module loaded without its
netdev notifier registered.
Check the error and remove the packet type on failure.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Link: https://patch.msgid.link/20260803085936.142160-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
failover_init() ignores register_netdevice_notifier() errors and always
returns success, which can leave the failover module loaded without its
netdev notifier registered.
Return the notifier registration result directly so module initialization
fails when registration fails.
This is a future looking check, register_netdevice_notifier()
only fails on double registration or if the registered notifier
itself returns an error.
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Link: https://patch.msgid.link/20260731030338.82508-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
nexthop_init() ignores errors from register_pernet_subsys() and
register_netdevice_notifier(), so a partial initialization can appear
successful.
Check those steps and unwind prior registrations on failure.
Do not check rtnl_register_many(): for built-in code it panics on
failure, so the call cannot return an error to nexthop_init().
Cc: stable+noautosel@kernel.org # untested fix to unlikely error path
Signed-off-by: Minhong He <heminhong@kylinos.cn>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Link: https://patch.msgid.link/20260731025249.80026-1-heminhong@kylinos.cn
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
In __allocate_data_block(), when allocating a new data block
(dn->data_blkaddr == NULL_ADDR), inc_valid_block_count() is
called first to increment total_valid_block_count and i_blocks.
If the subsequent f2fs_allocate_data_block() fails, the function
returns the error directly without rolling back the
already-incremented block counts, causing a permanent leak.
Fix this by calling dec_valid_block_count() to undo the
increment before returning the error. The condition
old_blkaddr == NULL_ADDR precisely identifies the case where
inc_valid_block_count() was called.
Fixes: 7d009e048d7c ("f2fs: fix to handle segment allocation failure correctly")
Cc: <stable@vger.kernel.org>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Chen Changcheng <chenchangcheng@kylinos.cn>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
|
|
The sysfs store path already takes s_umount for GC thread control
entries, and ckpt_thread_ioprio is covered as well.
critical_task_priority also updates checkpoint or GC kthread scheduling
state, but it is not covered by that serialization. It can race with
remount or teardown paths that are stopping those threads.
Protect critical_task_priority sysfs writes with s_umount too.
Fixes: 52190933c37a ("f2fs: sysfs: introduce critical_task_priority")
Cc: stable@kernel.org
Signed-off-by: Wenjie Qi <qiwenjie@xiaomi.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
|
|
checkpoint_merge can be enabled even when no checkpoint merge thread is
running. A read-only mount is one case: f2fs does not start
f2fs_issue_ckpt there, but ckpt_thread_ioprio is still writable through
sysfs.
The ckpt_thread_ioprio store path updates the saved ioprio value and,
when checkpoint_merge is enabled, calls set_task_ioprio() for the
checkpoint thread. If cprc->f2fs_issue_ckpt is NULL, that dereferences a
NULL task pointer.
Protect ckpt_thread_ioprio sysfs writes with s_umount as well, so the
checkpoint thread cannot disappear under the store path while updating
its ioprio.
Fixes: e65920661708 ("f2fs: add ckpt_thread_ioprio sysfs node")
Cc: stable@kernel.org
Signed-off-by: Wenjie Qi <qiwenjie@xiaomi.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
|
|
ovs_flow_cmd_new() preallocates the optional reply skb before it takes
ovs_mutex and before it knows which existing flow will be updated.
That is normally fine because the skb is sized from the request flow
identifier. That identifier also becomes the inserted flow's identifier.
For updates, however, a request with a UFID may miss the UFID lookup and
then fall back to the flow key lookup. That lookup can legitimately find
an existing key-identified flow. UFIDs are optional and the flow key is
the primary identifier.
For echoed replies, ovs_flow_cmd_fill_info() writes the matched flow's
identifier, not the request identifier used for the preallocation. A short
request UFID can therefore leave too little room for the key identifier.
The fill can then fail with -EMSGSIZE and hit the BUG_ON(error < 0) in the
update path.
Once the update target has been resolved, reallocate the reply skb if the
matched flow needs a larger reply than the request identifier allowed. Do
this before replacing the actions so the request can still fail cleanly if
the rare extra allocation fails.
Fixes: 74ed7ab9264c ("openvswitch: Add support for unique flow IDs.")
Cc: stable@vger.kernel.org
Reported-by: Vega <vega@nebusec.ai>
Signed-off-by: Zhiling Zou <zhilinz@nebusec.ai>
Reviewed-by: Ilya Maximets <i.maximets@ovn.org>
Link: https://patch.msgid.link/f7bbd3c30ce81a39156e226b3872d73abed21d2f.1785644623.git.zhilinz@nebusec.ai
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
usbnet_probe() initializes max_mtu to ETH_MAX_MTU and only caps it
inside the if (info->bind) block. Drivers without a bind callback
never enter this block, so max_mtu stays at ETH_MAX_MTU.
QEMU's usb-net device (0x0525/0xa4a2) is claimed by the cdc_subset
driver which has no bind callback. The guest accepts any MTU from DHCP
(e.g. 65520 from passt), leading to TCP segments that exceed the
device's 2048-byte receive buffer and are silently dropped.
Initialize max_mtu to net->mtu at probe time and update it inside
the bind block.
Fixes: f77f0aee4da4 ("net: use core MTU range checking in USB NIC drivers")
Cc: jarod@redhat.com
Cc: stable@vger.kernel.org
Link: https://gitlab.com/qemu-project/qemu/-/issues/3268
Link: https://bugs.passt.top/show_bug.cgi?id=189
Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Link: https://patch.msgid.link/20260731092711.857684-1-lvivier@redhat.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
bnge_fix_rings_count() returns 0 on success or a negative errno on failure
However, bnge_adjust_rings() stores its return value in a u16 variable,
causing negative error codes such as -ENOMEM to be converted to a large
positive value.
Use an int for the return code variable so that error values are
preserved and propagated correctly.
Fixes: 627c67f038d2 ("bng_en: Add resource management support")
Signed-off-by: Alok Tiwari <alok.a.tiwari@oracle.com>
Reviewed-by: Bhargava Marreddy <bhargava.marreddy@broadcom.com>
Link: https://patch.msgid.link/20260801100923.1498570-1-alok.a.tiwari@oracle.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Yangyu Chen says:
====================
net: atlantic: fix two ring teardown leaks
These are the two fixes from the page_pool conversion series [1],
resent against net as requested in the review of that series. The
page_pool conversion itself stays in net-next and is not part of this
posting; it depends on these fixes, but they stand on their own.
Both patches are unchanged from [1] apart from the collected
Reviewed-by tags, and each carries a Fixes tag and a Cc: stable with
the affected range (patch 1: v4.11+, patch 2: v5.2+). They apply and
were build- and runtime-tested independently of each other and of the
conversion.
Patch 1: aq_vec_deinit() drains the TX rings with a single
aq_ring_tx_clean() call, which is capped at AQ_CFG_TX_CLEAN_BUDGET
descriptors and stops at hw_head, frozen once the hardware and NAPI
have been stopped. Everything beyond that keeps its skb or xdp_frame
when the interface goes down and is lost when the buffer ring is
freed.
Patch 2: aq_ring_rx_deinit() only walks [sw_head, sw_tail). Since the
page reuse strategy was added, a cleaned RX buffer keeps its page for
reuse and refill is batched, so consumed but not yet reposted slots
accumulate in the [sw_tail, sw_head) gap and their pages and DMA
mappings are never released.
Reproduction logs for both leaks (as page_pool stalled shutdowns,
which is how they become visible) are in the notes of the respective
patches.
[1] https://lore.kernel.org/lkml/tencent_1F173E0FC1606D2AC704DC9C98AF10984607@qq.com/
====================
Link: https://patch.msgid.link/tencent_29B860317921D68DE77C718242DA418EB608@qq.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
aq_ring_rx_deinit() only walks [sw_head, sw_tail), the region posted to
hardware. Since the page reuse strategy was added, a cleaned RX buffer
keeps its page (and its DMA mapping) in the ring for reuse, and refill
is batched: aq_ring_rx_fill() returns early until AQ_CFG_RX_REFILL_THRES
slots are free. Slots that were consumed but not yet reposted therefore
sit in the complementary [sw_tail, sw_head) gap with a live page, and
the deinit walk never visits them: up to a refill batch worth of pages
and DMA mappings leak on every interface down.
Walk the whole ring instead and release whatever is still there. Also
bail out if the buffer ring is already gone: a partial
aq_ptp_ring_alloc() failure frees the ring but leaves aq_nic set, so
aq_ptp_ring_deinit() still gets here on the unwind path.
Cc: stable@vger.kernel.org # v5.2+
Fixes: 46f4c29d9de6 ("net: aquantia: optimize rx performance by page reuse strategy")
Reviewed-by: Sukhdeep Singh <sukhdeeps@marvell.com>
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
Acked-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/tencent_607CBA8237DA438E36B844318B21538DE008@qq.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
aq_vec_deinit() drains the TX rings with a single aq_ring_tx_clean()
call, which frees at most AQ_CFG_TX_CLEAN_BUDGET (256) descriptors and
stops at hw_head, which no longer moves once aq_vec_stop() has stopped
the hardware and NAPI. Completed descriptors beyond the budget and
everything still posted in [hw_head, sw_tail) keep their skb or
xdp_frame when the interface goes down: aq_vec_ring_free() then frees
the buffer ring and the references are lost for good.
Today this is a silent memory leak on every interface down under
TX/XDP_TX load. With the conversion of the RX path to page_pool posted
for net-next it becomes much more visible: XDP_TX frames carry fragment
references on the RX ring's page_pool, so a single stranded frame keeps
the pool's inflight count above zero forever. page_pool_destroy() then
never completes, the pool is leaked together with its pages, and
"page_pool_release_retry() stalled pool shutdown" is warned every 60
seconds from that point on, on every ifdown, XDP detach or ring resize
under XDP_TX load.
Bring back aq_ring_tx_deinit() as it was before the removal and use it
for teardown again, with one extension: TX rings can hold xdp_frames
nowadays, so release those too. They are returned with
xdp_return_frame() since this runs in process context.
Fixes: eb36bedf28be ("net: aquantia: remove function aq_ring_tx_deinit")
Cc: stable@vger.kernel.org # v4.11+
Reviewed-by: Sukhdeep Singh <sukhdeeps@marvell.com>
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
Acked-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/tencent_EEDC35FAF2750A3A6A0B39BAE0E2C484860A@qq.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Since the referenced commit, changing the MTU on a running interface no
longer disconnects and reconnects the PHY; __stmmac_release() merely
stops phylink, which also suspends the PHY (BMCR power-down) when WoL
is not enabled. __stmmac_open() then performs the DMA software reset in
stmmac_hw_setup() before phylink_start() resumes the PHY again.
IEEE 802.3 22.2.4.1.5 allows a PHY to stop its receive clock while
powered down, and stmmac requires a running receive clock for the DMA
software reset to complete (the phylink config sets mac_requires_rxc).
On such setups, e.g. the RK3566-based Home Assistant Green with an
RTL8211F-VD PHY in RGMII mode, any runtime MTU change now times out and
leaves the interface dead:
rk_gmac-dwmac fe010000.ethernet end0: Failed to reset the dma
rk_gmac-dwmac fe010000.ethernet end0: stmmac_hw_setup: DMA engine initialization failed
rk_gmac-dwmac fe010000.ethernet end0: __stmmac_open: Hw setup failed
rk_gmac-dwmac fe010000.ethernet end0: failed reopening the interface after MTU change
In the field this is triggered by NetworkManager applying an MTU while
activating the connection, breaking networking entirely. The same
regression has also been reported on i.MX8MP and reproduced on SoCFPGA
based systems.
Resume the PHY in __stmmac_open() before the hardware setup, making it
the counterpart of the phylink_stop() in __stmmac_release(), like
stmmac_resume() already does for the same reason. phylink_start() also
resumes the PHY, but only after stmmac_hw_setup(), and it cannot be
moved before the hardware setup since it may bring the link up
immediately from a workqueue, racing with the initialization (see the
comment in stmmac_resume()). For the regular ndo_open path the PHY has
just been attached and is not suspended, in which case
phylink_prepare_resume() does nothing.
Fixes: db299a0c09e9 ("net: stmmac: move PHY handling out of __stmmac_open()/release()")
Link: https://github.com/home-assistant/operating-system/issues/4858
Tested-by: Alexander Stein <alexander.stein@ew.tq-group.com>
Signed-off-by: Stefan Agner <stefan@agner.ch>
Tested-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Reviewed-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Link: https://patch.msgid.link/20260803095156.132827-1-stefan@agner.ch
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The memory allocated for struct vcap_rule_internal, keyfields and
actionfields inside vcap_dup_rule() are not freed in some of the error
paths in vcap_decode_rule(). Fix that by calling vcap_free_rule().
Compile tested only. Issue found using a prototype static analysis tool
built on top of the LLVM compiler infrastructure.
Cc: stable+noautosel@kernel.org # untested fix to unlikely driver error path
Reviewed-by: Joe Damato <joe@dama.to>
Signed-off-by: Abdun Nihaal <nihaal@cse.iitm.ac.in>
Link: https://patch.msgid.link/20260801055507.47534-1-nihaal@cse.iitm.ac.in
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
