| Age | Commit message (Collapse) | Author |
|---|
|
Add an 8 bit small sequence counter to the held_lock struct to detect if
the lock as been dropped and reacquired. This is useful when a data
structure depends on a constant locking context, but is not able to detect
locking and unlocking of the lock through its own API.
Since the __lock_unpin_lock() will no longer detect underflow by casting
the unsigned int to a signed int, update the casting code to use a temp
variable for calculations using a signed int.
Link: https://lore.kernel.org/20260821192627.4085470-3-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Will Deacon <will@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Waiman Long <longman@redhat.com>
Link: https://lore.kernel.org/all/h3tpnj5kzcrxms5picmimtkpg4aypcpip5wbd6bt2rpdj5k7eb@nhtzs3lefrkq/
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "maple_tree: lock checking and clean ups", v3.
In this series:
1. Try to detect lock issues
A number of syzbot reports are incorrectly pointing to the mm exit as
a source of the locking error. The first three patches attempt to
help users detect errors in their locking - but they still have to use
LOCKDEP. I guess it's still down to hope and prayers.
2. Documentation fixes
The documentation was lacking clarity, there are updates to try and
help the users, especially around the erase() cases.
3. Two benign issues
The cyclic allocator may have a race, although no in-kernel user can
hit it.
The erase functions may cause allocation issues if used with the
incorrect locking type, but none are present in-tree.
4. The erase gfp uses
mas_erase() and mtree_erase() do not take a gfp argument. To improve
reliability of the erase, the first attempt to allocate will be
GFP_NOWAIT, followed by a retry (if necessary of GFP_KERNEL |
GFP_NOFAIL. This will ensure the data is gone. I've updated the
documentation to make it more clear as well. mas_store() is not
addressed in the same way, but may need to be updated at a later date,
but that may require changing callers so it is out of scope here.
Beyond these goals there are some test fixes, some general speed-up
patches targeting extra work and cycles, and dropping dead code.
This patch (of 19):
When CONFIG_LOCKDEP and CONFIG_RCU_STRICT_GRACE_PERIOD is enabled, check
for rcu locking issues by recording the grace period in the maple state
and checking the rcu window is still valid whenever the maple state is
reused with a state that is not MA_START or MA_PAUSED.
Link: https://lore.kernel.org/20260821192627.4085470-1-liam@infradead.org
Link: https://lore.kernel.org/20260821192627.4085470-2-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Breno Leitao <leitao@debian.org>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
swapops.h uses MAX_SWAPFILES_SHIFT, SWP_MIGRATION_READ and SWP_PTE_MARKER,
all of which swap.h defines, but does not include swap.h. It compiles only
where the translation unit pulled swap.h in first. leafops.h includes
swapops.h on the line above swap.h, so a file whose include list reaches
leafops.h before swap.h gets:
In file included from include/linux/leafops.h:11:
include/linux/swapops.h:88:21: error: use of undeclared
identifier 'MAX_SWAPFILES_SHIFT'
A header that uses a definition has to include the header that provides it.
Link: https://lore.kernel.org/20260818115026.656406-1-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608181757.mza9RRj7-lkp@intel.com/
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct
vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This
moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu
pointer onto the same cacheline.
The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled,
generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a
176-CPU test system. Those writes bounce the line while memcg statistics
paths load vmstats_percpu.
Move cgwb_list into the existing alignment gap and cacheline-align
vmstats_percpu. This separates the pointer from the event counters
without increasing the size of struct mem_cgroup in the tested
configuration.
The blamed commit reduced median mremap throughput by 4.38% on the test
system with one socket. The patched kernel brings the performance to
within 0.5% of the parent which is within the observed boot-to-boot spread
(up to 1.2%).
Link: https://lore.kernel.org/20260817103835.2937733-1-usama.arif@linux.dev
Fixes: ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reported-by: kernel test robot <yi1.lai@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202608131743.c6a7dda4-lkp@intel.com
Tested-by: kernel test robot <yi1.lai@intel.com>
Link: http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Yi Lai <yi1.lai@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
alpha requires percpu variables in modules to be defined as weak so that the
compiler generates GOT based external references for them. This puts two
extra restrictions on percpu variable definitions. The symbol must be
globally unique even when static and a static percpu variable can't be
defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic
code build coverage for these restrictions without building for alpha.
MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call
site and thus can't be built with weak percpu definitions, so it depends on
!DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU,
this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage.
allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build
coverage for restrictions which only matter to alpha module builds. Drop
DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on
alpha builds.
Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Gabriele Monaco <gmonaco@redhat.com> [include/rv/da_monitor.h]
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Currently swap to and from file systems goes through two indirect calls
between the swap ops and the swap_rw method. Reduce this by directly
providing the swap_ops from the file system.
For this refactor swap_fs_submit into a swap_fs_prepare_rw helper that
initializes the iov_iter on the callers stack so that file systems can
call it directly, and use that to initialize file system specific ops in
the NFS and SMB clients, which then get passed to swap_fs_activate.
Link: https://lore.kernel.org/20260723054622.3460249-4-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Chris Li <chrisl@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steve French <sfrench@samba.org>
Cc: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add a new header to declare the swap_iocb, swap_ops and swap_ctx to allow
for swap_ops implementations outside of mm/page_io.c. This will be used
to remove the double indirection for file system-based swap. There is no
functional change, just a move of the declarations.
Note that there already is a swapops.h header, which is totally unrelated
to struct swap_ops. The close naming is a bit unfortunate, but I could
not think of a better name for this header.
Link: https://lore.kernel.org/20260723054622.3460249-3-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Chris Li <chrisl@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steve French <sfrench@samba.org>
Cc: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Count how many swap I/Os we cause. Due to batching this can be different
than the current counter number of pages written/read, and tracking this
information is useful to see how efficient the batching is.
The counters are added at the end of enum vm_event_item and the
vmstat_text array under the assumption that the order of fields in
/proc/vmstat is an ABI. If that is not the case, they could be grouped
with the other swap counters.
Link: https://lore.kernel.org/20260713093350.2154226-8-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Provide a swap_fs_activate helper that directly sets up swap_fs_ops, and a
flag in struct swap_ops to indicate of NOFS swapping is allowed.
Link: https://lore.kernel.org/20260713093350.2154226-7-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This simplifies codes and makes logic clearer. And also makes later any
new swap device type being added easier to handle.
Currently there are two types of swap devices: fs and bdev.
[hch@lst.de: updated for the new submit and can_merge abstraction]
Link: https://lore.kernel.org/20260713093350.2154226-6-hch@lst.de
Signed-off-by: Baoquan He <baoquan.he@linux.dev>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Suggested-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "better block swap batching and a different take on swap_ops
v5".
This series makes use of the swap_iocb for block as well so that it
doesn't do inefficient single-bio I/O, and then rebases the swap_ops from
Baoquan on top of the now very different method structure.
When running doing kernels builds, which is a workload that doesn't really
do much THP anonymous memory it still gets 2x clustering for writeout and
1.2x for reading back swap in. The overall times do not actually change,
though.
This patch (of 7):
Provide a wrapper for the shmem abuses in drm to prepare for swap I/O
refactoring by keeping swap_iocb handling entirely contained in mm/.
Link: https://lore.kernel.org/20260713093350.2154226-1-hch@lst.de
Link: https://lore.kernel.org/20260713093350.2154226-2-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Extend the allocinfo filtering mechanism to allow users to filter tags
based on their accuracy.
[abhishekbapat@google.com: move `inaccurate` filtering criteria from `struct allocinfo_tag` to `struct allocinfo_filter`]
Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com
Link: https://lore.kernel.org/396a5e4bc3b2990223ab355f2cd3ceb6aa15499e.1783532853.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Extend the allocinfo filtering mechanism to allow users to filter tags
based on the total number of bytes allocated [min_size, max_size]. The
size range is inclusive.
Filtering by size involves retrieving allocinfo per-CPU counters, which is
an expensive operation. Hence, the performance of size-based filtering
will be worse than other filters.
Link: https://lore.kernel.org/0a7653b70ae0d64e967fbea0e933bc35f8ac656e.1783532853.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Extend the capability of the IOCTL mechanism to filter allocations based
on tag's module name, function name, file name and line number.
Link: https://lore.kernel.org/6a6100c0c58cb2911f39126b9fe177a8c17db16f.1783532853.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8.
Currently, memory allocation profiling data is primarily exposed through
/proc/allocinfo. While useful for manual inspection, this text-based
interface poses challenges for production monitoring and large-scale
analysis:
1. Userspace must parse large amounts of text to extract specific
fields.
2. To find specific tags, userspace must read the entire dataset,
requiring many context switches and high data copying.
3. The kernel currently aggregates per-CPU counters for every allocation
size, even those the user intends to filter out immediately.
This series introduces a new IOCTL-based binary interface for allocinfo
that supports kernel-side filtering. By allowing the user to specify a
filter mask, we significantly reduce the work performed in-kernel and the
amount of data transferred to userspace. The IOCTL mechanism was chosen
for allocinfo to address the per-CPU counter aggregation bottleneck. A
traditional read() operation must report the total allocation count and
sizes for every code tag in the system. Doing so requires iterating
across all CPUs to sum their per-CPU counters for thousands of tags, which
introduces substantial runtime overhead.
The IOCTL interface allows userspace to push selective filtering criteria
directly into the kernel before the per-CPU counter aggregation. The
kernel aggregates per-CPU counters only for a small subset of tags that
match the filter. This results in significant performance improvement.
Beyond fast filtered retrieval, the IOCTL foundation allows introducing a
context capture mechanism in the future to capture the context for
specific allocations.
Performance measurements were conducted on an Intel Xeon Platinum 8481C
(224 CPUs) with caches dropped before each run.
The IOCTL mechanism shows a ~20x performance improvement for
filtered queries. The kernel avoids the expensive per-CPU counter
aggregation (alloc_tag_read) for any tags that fail the initial string
or location filters.
Scenario 1: Specific File Filtering (arch/x86/events/rapl.c)
1. Traditional (cat /proc/allocinfo | grep): 22ms (sys)
2. IOCTL Interface: 1ms (sys)
Scenario 2: Compound Filtering (Filename + Size)
1. Traditional: (cat ... | grep | awk): 21ms (sys)
2. IOCTL Interface: 1ms (sys)
Scenario 3: Size-Based Filtering (min_size = 1MB)
1. Traditional: (cat ... | awk): 21ms (sys)
2. IOCTL Interface: 14ms (sys)
This patch (of 6):
Add the following ioctl commands for /proc/allocinfo file:
ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to
check whether the file content has changed specifically due to module
load/unload. Every time a module is loaded / unloaded, the returned value
will be different. By comparing the identifier value at the beginning and
at the end of the content retrieval operation, users can validate
retrieved information for consistency.
ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is
the position of a record in /proc/allocinfo.
ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one.
If no records were previously retrieved, returns the first record.
Note, function file and module names often have the same prefixes,
therefore when filtering for them, we compare the last 64 characters to
minimize the chances of name collisions.
[akpm@linux-foundation.org: include compat.h, per Suren]
Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/
Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com
Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix the following sparse warnings that appear while building f2fs:
./include/trace/events/f2fs.h:1469:1: warning: incorrect type in initializer (different base types)
./include/trace/events/f2fs.h:1469:1: expected unsigned long mask
./include/trace/events/f2fs.h:1469:1: got restricted vm_fault_t
Link: https://lore.kernel.org/e56c9e2aead04f79192c3110de80d846e41e3791.1786122711.git.bvanassche@acm.org
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.kahndual@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The kernel-doc for struct execmem_range incorrectly describes @fallback_end
as "start". Correct it to "end".
Link: https://lore.kernel.org/20260807091958.4735-1-henry.elderman.edu+linux@gmail.com
Signed-off-by: Henry Elderman <henry.elderman.edu+linux@gmail.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mem_section_usage_size() is only needed by sparsemem implementation code
after commit ae751d567baa ("mm/bootmem_info: stop marking
mem_section_usage as MIX_SECTION_INFO"), so keeping the declaration in
mmzone.h now exposes the helper to all mmzone.h users for no reason.
Move the helper to sparse.h so sparse.c and sparse-vmemmap.c can share it
through the internal header. While doing so, calculate the allocation
size with struct_size_t(), which ties the expression to the
pageblock_flags trailing array instead of open-coding the struct header
plus bitmap size.
Link: https://lore.kernel.org/20260805022536.1206575-1-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
pgdat_page_ext_init() sets pgdat->node_page_ext to NULL only on FLATMEM.
FLATMEM depends on !NUMA, so the pgdat is always the zero-initialized
contig_page_data and the store has no effect.
So remove the call site, the unused function and its declaration.
No functional change.
Link: https://lore.kernel.org/20260804151145.3419768-3-ekffu200098@gmail.com
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Acked-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte
counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the
type of its promoted left operand, so those are 32-bit arithmetic and wrap
at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast
being remembered; vmap() has one, two recent commits did not.
vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and
/proc/kcore returns it as zeros while reporting a successful read, which
drgn, crash or gdb cannot tell from real memory, and the vrealloc()
grow-in-place check declines a request that would have fit.
Widen the field so the class of bug goes away instead of one site at a
time. Everything feeding or consuming it widens too:
vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages
and old_nr_pages, the index range of vm_area_free_pages(), and three page
indexes that were plain int. Five casts go. Two prints needed fixing as
well, %u in vmalloc_dump_obj() and %d for the unsigned field in
vmalloc_info_show().
No bug report behind this, I found it reading the code. The 4 GiB wrap
needs only a machine with over 4 GiB of memory. Neither larger threshold
is a practical concern: 2^32 pages, where the field itself truncates, is
16 TiB and beyond what hardware can populate, and 2^31, where the plain
int indexes break, is 8 TiB and larger than anything in the tree asks for.
The int *nr cursor in the mapping path is unchanged and is separate work.
Users outside mm/vmalloc.c need no change either. Those handing the
count to a narrower parameter cannot drive it near 2^31, and
kho_preserve_vmalloc() stores it into a 32-bit ABI field that still
receives the same low bits; above 2^32 pages the truncation just moves out
of vm_struct into that store.
sizeof(struct vm_struct) on x86-64 stays 72 bytes with
CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled,
both inside the kmalloc-96 bucket it already comes from.
Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com
Fixes: 0bca23804632 ("mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas")
Fixes: d57ac904ffdc ("mm/vmalloc: use physical page count for vrealloc() grow-in-place check")
Signed-off-by: Artem Lytkin <iprintercanon@gmail.com>
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Assisted-by: Claude:claude-fable-5
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: <shivamkalra98@zohomail.in>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
KVM with guest_memfd wants to remove any folio references due to LRU
caches, as it really must only allow to convert folios from shared to
private when there are no unexpected folio references (e.g., from GUP
references).
So, to drive the refcount down, it needs a way to flush the LRU caches.
Let's factor out what we have in lru_cache_drain_for_folio(). Document
it, and also mention that concurrent folio (un)mapping might, in theory,
miss detecting LRU cache references. Keep obtaining the expected refcount
twice to minimize the possibility. For the current and future user that
should work, and we don't really have a better alternative: we could
detect if the mapcount changed, but it would still be racy and add more
complexity with questionable benefit.
Maybe there is a chance to avoid the draining entirely in the future, by
avoiding extra references from the LRU cache: Hugh thinks there might be a
way. But for the time being, this handling is unfortunately required.
Make folio_may_be_lru_cached() accept a const pointer so
lru_cache_drain_for_folio() can accept a const pointer as well.
Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The anon/file scan balance in get_scan_count() is driven by two scalars in
struct lruvec, anon_cost and file_cost, accumulated by every reclaim
producer under lruvec->lru_lock. The acquisition sites for cost work
specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely
to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned -
nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One
acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq()
purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost.
- lru_note_cost_unlock_irq() itself walks parent_lruvec and
re-acquires lru_lock on each ancestor to propagate the update,
adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on
memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru()
and folio_add_lru() takes it. The cost work itself is trivial (two scalar
bumps and one comparison), but it contends with and causes contention for
actual LRU manipulation. The parent_lruvec() walk also multiplies
cost-update overhead by memcg hierarchy depth.
The balance formula for anon and file, respectively, is this:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Instead of recording cost and running averaging logic directly when these
events occur, snapshot running vmstat counters once per reclaim cycle and
derive the balance from event deltas since the last run.
Use PGROTATE_* from the preceding patch for the rotation input.
WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event
counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can
be sampled per lruvec and aggregated through the memcg hierarchy. This is
overall cheaper and has fewer lock acquisition sites.
Moving accumulation and decay to the reclaim side also improves the cost
model across reclaim gaps. With producer-side decay, events that happen
while reclaim is idle still age each other before reclaim ever samples the
costs. If a workload refaults a large anon set and then a smaller file
set before reclaim runs again, the later file activity can age the earlier
anon activity out of the cost model. The new scheme observes the whole
between-reclaim delta and decays anon and file proportionally, so the
scan-balance history better represents what happened since the last
reclaim pass.
A dedicated per-lruvec spinlock, cost_lock, serialises the delta
extraction, the cost->count update and the halving loop against concurrent
reclaimers in the same memcg+node.
NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is
no longer needed and is removed.
memcg-v1's memory.stat anon_cost/file_cost is now sourced from
cost[].count instead of the removed lruvec anon_cost/file_cost fields.
The reported values only refresh when prepare_scan_control() runs and are
bounded at ~lrusize/4 by the halving loop; the scan-balance signal they
express is unchanged.
Under pure MGLRU the scan-balance signal itself is not consumed (both
prepare_scan_control() and get_scan_count() are short-circuited on the
MGLRU paths, and MGLRU's own type/tier selection comes from
read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted},
not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout
from either reclaim implementation. The preceding patch also bumps
PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work
is accounted consistently across both implementations.
Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Reclaim can spend substantial work on an LRU type without immediately
reclaiming or demoting a corresponding amount of memory. Record this work
in PGROTATE_ANON and PGROTATE_FILE.
For classic LRU reclaim:
- Inactive-list reclaim adds nr_scanned - nr_reclaimed to the
corresponding anon/file counter when isolation succeeds.
- Active-list reclaim adds referenced executable file folios that
are retained on the active list to PGROTATE_FILE. Active anon
reclaim does not contribute this component.
For MGLRU, add the number of initially isolated pages that remain
unreclaimed after both the initial and retry passes to the counter for the
selected anon/file type.
These counters are distinct from the existing pgrotated vm event.
pgrotated records an actual move to the inactive-list tail, primarily
after reclaim-marked writeback completes or failed invalidation leaves a
folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure
reclaim cost and do not imply that a folio moved to an LRU tail.
A subsequent patch will consume these counters for anon/file scan
balancing.
Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/vmscan: reduce lru_lock contention via vmstat-derived
scan-balance cost", v5.
The anon/file scan balance heuristic in get_scan_count() is fed by two
scalars in struct lruvec (anon_cost, file_cost) that every reclaim
producer updates under lruvec->lru_lock. The cost-recording work itself
is trivial, but it both contends for and contributes to contention on
lru_lock - which is often a contention point on memory-pressured
workloads. Specifically:
- shrink_inactive_list() re-acquires lru_lock at function exit just
to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to
record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under
lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and
re-acquires lru_lock on every ancestor, multiplying the cost
of every update by memcg-hierarchy depth.
This series removes those producer-side acquisitions entirely. The
rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters.
NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout();
charge it through lruvec_stat_mod_folio() so it is available per lruvec
and aggregated through the memcg hierarchy. Reclaim does not submit
filesystem folios for writeback from this path, so pageout contributes
only to anon cost. WORKINGSET_RESTORE_* already captures the refault
input.
PGROTATE_* are also useful independently of scan balancing. They are
cumulative base-page events, not unique-page counts. Classic inactive
reclaim records scan work that does not produce immediate reclaim or
demotion, while active reclaim records referenced executable file folios
retained on the active list. MGLRU records initially isolated pages that
remain unreclaimed after its retry passes. Read alongside pgscan_* and
pgsteal_*, their deltas identify which LRU type is consuming reclaim CPU
without producing immediate yield. Unlike the existing pgrotated event,
they do not imply a move to the inactive-list tail.
prepare_scan_control() reads the raw cost signals without lru_lock:
anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX
It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec
cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or
folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving
check. Hierarchy aggregation is implicit in rstat propagation, so the
parent_lruvec() walk and the lru_reparent_memcg() cost-splice both
disappear.
Moving accumulation and decay to the reclaim side also improves the cost
model across reclaim gaps. With producer-side decay, events that happen
while reclaim is idle still age each other before reclaim ever samples the
costs. If a workload refaults a large anon set and then a smaller file
set before reclaim runs again, the later file activity can age the earlier
anon activity out of the cost model. The new scheme observes the whole
between-reclaim delta and decays anon and file proportionally, so the
scan-balance history better represents what happened since the last
reclaim pass.
Trade-offs:
- Cost reads see rstat-aggregated values that can lag until periodic /
reader-triggered flushing.
- Per-lruvec footprint grows by 4 unsigned longs + a spinlock (a
struct lru_cost { count, last_rotated, last_io } per side), which
is a small cost.
- NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat,
adding memcg stat accounting to the reclaim writeout path while
preserving the existing node-level total.
== Numbers ==
Tested on a 176-core, 256 GB host. The benchmark drives sustained
swap-out/refault inside a tight memcg using vm-scalability/usemem:
usemem -n 16 --prealloc --prefault --random $((256*1024*1024))
run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon
working set has to fit in 512 MB -> continuous shrink_inactive_list +
workingset_refault). A 16 GB swap file is used. Measurement is a 30 s
`perf lock record -a` window over otherwise-idle hardware.
Workload rates are identical on both kernels (the bench drives the same
memory pressure):
baseline patched delta
pgscan_direct / s 172,662 171,817 ~0%
pgsteal_direct / s 67,162 66,306 ~0%
workingset_refault_anon / s
40,696 39,830 ~0%
perf lock contention (total wait per 30 s window):
Lock Name Before After % change
shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated)
(= lru_note_cost_unlock_irq)
workingset_refault+0x167 385.26 ms 0 -100% (eliminated)
(= lru_note_cost_refault)
shrink_node+0x4ad 689.43 ms 26.95 ms -96%
shrink_active_list 208.34 ms 15.97 ms -92%
lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53%
Total LRU lock wait ~4.23 s ~1.66 s -61%
The two specific contention sites the patch removes (shrink_lruvec+0x770 =
lru_note_cost_unlock_irq; workingset_refault+0x167 =
lru_note_cost_refault) are completely absent from the patched
perf-lock-contention output. Secondary reductions in shrink_node,
shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like
knock-on effects from removing the cost-recording overhead and the
parent_lruvec walk.
The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated
by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path
in shrink_lruvec.
The numbers above can be reproduced using the script in [1].
This patch (of 3):
lruvec_page_state(), node_page_state(), and global_node_page_state() all
clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU
delta skew presents as zero pages rather than as a garbage unsigned value.
This is the right behaviour for non-monotonic page-count readers.
It is however incorrect for callers that snapshot a monotonically-
incremented event counter and compute a delta from two samples. Once the
underlying signed long wraps past LONG_MAX, the clamped read drops to zero
while the previously-recorded snapshot still holds the pre-wrap value; the
unsigned subtraction then underflows into a ~2^31 spurious delta for
32-bit architecture and corrupts the caller's accumulator.
Add non-clamping siblings that return the underlying state value cast to
unsigned long:
global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()
With both samples read via the _monotonic variant, unsigned modular
subtraction stays correct across a signed-long wraparound as long as the
true growth between two samples fits in unsigned long (< 2^32 on 32-bit, <
2^64 on 64-bit); the 32-bit bound is the practically-reachable one that
motivates this helper.
The variants are only safe for monotonically-incremented counters.
Non-monotonic page-count readers must keep using the existing clamped
helpers so transient negative reads still present as zero.
This is a prerequisite for a later patch which replaces the producer-side
anon_cost/file_cost accumulators with a read-side accumulator in
prepare_scan_control() that samples monotonic per-LRU vmstat counters
(PGROTATE_*, NR_VMSCAN_WRITE, WORKINGSET_RESTORE_*) via
lruvec_page_state_monotonic() and folds their unsigned modular deltas into
lruvec->cost[].count.
Link: https://lore.kernel.org/20260727162550.2032-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260727162550.2032-2-usama.arif@linux.dev
Link: https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884 [1]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: vmscan: fix node reclaim ignoring swappiness parameter",
v4.
The per-node proactive reclaim interface
(/sys/devices/system/node/nodeX/reclaim) accepts a swappiness parameter,
but it is silently ignored when CONFIG_MEMCG is disabled. The root cause
is that sc_swappiness() has separate implementations for CONFIG_MEMCG and
!CONFIG_MEMCG, and the latter never checks proactive_swappiness.
Patch 1 moves mem_cgroup_swappiness() and vm_swappiness out of the public
include/linux/swap.h into the mm-private mm/swap.h, and makes the helper
handle both CONFIG_MEMCG and !CONFIG_MEMCG in a single inline function.
This is a prerequisite for unifying sc_swappiness().
Patch 2 consolidates sc_swappiness() into a single definition that works
regardless of CONFIG_MEMCG, fixing the node reclaim swappiness bug.
This patch (of 2):
The per-memcg swappiness knob is v1-only; v2 always uses global
vm_swappiness and ignores the per-cgroup field.
Both mem_cgroup_swappiness() and vm_swappiness are only used within mm/
(memcontrol.c, memcontrol-v1.c, vmscan.c), so move them out of the public
include/linux/swap.h into the mm-private mm/swap.h. This keeps unrelated
declarations out of include/linux/swap.h.
Guard memcg->swappiness with CONFIG_MEMCG_V1 as well, so v2-only
kernels drop the unused field.
No functional change for v1; v2-only kernels drop the unused field.
Link: https://lore.kernel.org/20260723032434.2016749-1-ridong.chen@linux.dev
Link: https://lore.kernel.org/20260723032434.2016749-2-ridong.chen@linux.dev
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Song Hu <husong@kylinos.cn>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "promote mapped executable folios after first usage for
MGLRU", v4.
Now MGLRU's protection of mapped executable file folios is less reliable.
Follow the classical LRU's logic, promoting mapped executable file folios
after their first usage to give executable code a better chance to stay in
memory and improve workload performance (See patch 2 for more details).
This patch (of 3):
Replace use of the legacy vm_flags_t flags with vma_flags_t values for
folio_referenced() and related logic. This is also a preparation for the
following changes.
No functional changes.
Link: https://lore.kernel.org/cover.1784509721.git.baolin.wang@linux.alibaba.com
Link: https://lore.kernel.org/2bd39e16ec19e3e3c4716aa9a1a25775c26cac57.1784509721.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Axel Rasmussen <axelrasmussen@google.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This function does not need to accept a page and requiring it to is
unnecessary and misleading.
make_[writable, readable]_device_private_entry() must be passed a
PMD-aligned PFN as they immediately used to obtain a softleaf PMD entry
and the same argument applies to folio_add_[anon, file]_rmap_pmd().
While we are here, update a VM_BUG_ON() to a VM_WARN_ON_ONCE().
No functional change intended.
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-12-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Gregory Price (Meta) <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Update the page_vma_mapped_walk structure to track whether the pgoff being
tracked is an anonymous pgoff or not and update the comments to reflect
this.
This is necessary in order to determine the correct VMA page offset in
vma_address_end() when pvmw->nr_pages > 1.
Also document that pvmw->pgoff is meaningless for pvmw->nr_pages == 1 and
for KSM.
Do not set this field where pgoff is not specified.
This is laying the groundwork for eventually using anonymous page offsets
as the index for all anonymous folios.
No functional change intended.
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-10-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Gregory Price (Meta) <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This function provides the anonymous equivalent of linear_page_index(),
instead offsetting based on the anonymous page offset of the VMA.
It is valid only for anonymous or MAP_PRIVATE file-backed mappings, in
other words CoW mappings.
For pure anon VMAs, this will be equal to linear_page_index().
Assert that both of these invariants are true in linear_anon_page_index()
and implement the algorithm in __linear_anon_page_index().
Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous()
but not fulfill this invariant, so when asserting this we check
vma->vm_file to account for this.
We do not update callsites yet, so no functional change intended.
Also const-ify vma_is_anonymous() to make it compatible with the
const-ified linear_anon_page_index().
While we're here, update linear_page_index() to be more succinct.
VMA userland tests are also updated accordingly.
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-3-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
All remaining callers of is_cow_mapping() are invoking it in the form of
is_cow_mapping(vma->vm_flags) or an indirected version of this.
Therefore, provide a helper - vma_is_cow_mapping() to directly test the
VMA.
Additionally provide a new helper vma_flags_is_cow_mapping() which
performs the check using the new vma_flags_t type, and share this logic
between vma_is_cow_mapping() and vma_desc_is_cow_mapping().
With these changes, no callers of is_cow_mapping() remain, so remove it.
Also update the userland VMA tests to reflect the change.
No functional change intended.
[akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo]
Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Gregory Price (Meta) <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous
pgoff", v5.
In memory management we've managed to manufacture a great deal of
confusion around the concept of anonymous memory. We have:
1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and
swap-backed (thus for reclaim purposes, treated as anonymous). These are
simple enough.
2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so
present in the page cache and mapped by an address_space object, but
whose folios are also swap-backed (thus treated as anonymous for reclaim
purposes).
3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have
vma->vm_file set, but which clears vma->vm_ops to satisfy
vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a
file, but are in every other sense anonymous, including the folios.
4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs
and have file-backed folios until CoW'd, at which point those CoW'd
folios are anonymous.
This series fixes issue 3.
In order for us to traverse VMAs using the reverse mapping, we require two
fields - folio->mapping and folio->index. The first tells the rmap code
where to look for VMAs, and the second tells it at which offset the folio
starts within the referenced object.
For anonymous folios, folio->mapping points at an anon_vma object. For
file-backed folios, it points at an address_space. And:
* For file-backed folios folio->index is simply the page offset of the start
of the folio within the file.
* For anonymous folios belonging to pure anon mappings, folio->index is
equal to the anonymous page offset of the folio.
* For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios
of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file
page offset.
This series establishes a new anonymous page offset property of VMAs to
allow us to map anonymous folios at their anonymous page offset,
consistent with pure anon.
The purpose of doing so is to lay the foundations for the scalable CoW
work. This is necessary because scalable CoW looks in the maple tree for
the VMA located at folio->index << PAGE_SHIFT, before falling back to
looking up tracked remaps if necessary.
The MAP_PRIVATE file-backed case means that folio indices will very often
conflict with one another and this remap tracking becomes substantially
more contended, and of course the fast path can never be used.
This also makes it possible, in future, to unshare anonymously mapped
folios with deep fork hierarchies on remap, eliminating the need for remap
tracking in the vast majority of cases.
Similar to page offset of pure anonymous VMAs, we update the anonymous
page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd
(i.e. vma->anon_vma is non-NULL).
Overall, there is little impact on mergeability, which remains exactly the
same for pure anonymous and shared file-backed mappings, with the only
impact being on MAP_PRIVATE-mapped file-backed mappings, which must now
match on anonymous page offset as well as file page offset to be merged.
To fail to merge like this would require CoW'ing the mapping, then finding
another VMA with identical file and compatible page offset to remap next
to.
This is therefore very much an edge case that should have very little
impact (and which scalable CoW may very well address in any case).
This patch (of 16):
Establish fields in vm_area_struct to store the anonymous page offset of
VMAs.
Initially, the anonymous page offset of a VMA is vma->vm_start >>
PAGE_SHIFT.
When a VMA is remapped to new_address its anonymous page offset is either
updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains
equal to the anonymous page offset it had when first faulted.
Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped
file-backed VMAs are tracked by their file offsets. By adding anonymous
offset as a property of VMAs, we can now track them by their anonymous
page offset instead.
By tracking this, we provide the means by which to eliminate this
inconsistency, and more importantly lay the foundations for future work
for the scalable CoW anonymous rmap rework.
This patch simply adds the fields and some simple helpers. Subsequent
patches will update mm code to make use of these fields correctly.
The fields chosen are packed in the VMA such that, for 64-bit kernel
builds, no additional space is taken up.
The first field is present on cacheline 0 containing key VMA fields, and
the second on cacheline 3, which contains file-backed reverse mapping
fields.
Given the relative time spent accessing reverse mapping fields as well as
updating them, there shouldn't be any performance impact here from false
sharing.
Update the VMA userland tests to account for this change.
No callsites are updated yet, so no functional change intended.
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org
Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Xu Xin <xu.xin16@zte.com.cn>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Deucher <alexander.deucher@amd.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Boris Brezillon <boris.brezillon@collabora.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christan König <christian.koenig@amd.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Huang Ray <Ray.Huang@amd.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Liviu Dudau <liviu.dudau@arm.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Namhyung kim <namhyung@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
<linux/maple_tree.h> consults the macro CONFIG_MAPLE_RCU_DISABLED to
determine whether to disable the mt_in_rcu() function (by making it always
return false). This macro is not reachable via Kconfig, despite its name,
and is not documented anywhere. Remove it to avoid polluting the CONFIG_*
namespace.
Discovered while searching for CONFIG_* symbols referenced in code but not
defined in any Kconfig file.
Link: https://lore.kernel.org/20260610220905.99860-1-enelsonmoore@gmail.com
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Acked-by: SeongJae Park <sj@kernel.org>
Reviewed-by: Liam Howlett <liam@infradead.org>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Cc: Andrew Ballance <andrewjballance@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
already-upstream changes to memcontrol.c, needed by "memcg: move
mem_cgroup_swappiness and vm_swappiness to mm/swap.h".
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth
Luiz Augusto von Dentz says:
====================
bluetooth pull request for net:
Core:
- hci_core: use skb_get() instead of skb_clone() for req_skb
- hci_conn: re-enable advertising only for peripheral role
- hci_event: clear HCI_LE_ADV only on a created connection
- hci_sync: Clear HCI_CMD_PENDING when dropping the last request
- hci_sync: add conditional locking annotations
- hci_sync: do not leak an hci_conn when a second LE connect is rejected
- eir: Fix OOB read in eir_get_service_data()
- mgmt: fix 'hdev->discovery.uuids' NULL dereference
- L2CAP: access chan->conn safely in get/setsockopt
- L2CAP: reject accept queue add unless BT_LISTEN
- L2CAP: fix race l2cap_sock_cleanup_listen() vs. put_chan
- RFCOMM: serialize security confirmation handling
- RFCOMM: serialize session teardown
- RFCOMM: Validate MTU in rfcomm_apply_pn() to prevent infinite loop
- ISO: fix use-after-free of listener socket in iso_conn_ready
Drivers:
- btnxpuart: Validate the FW dump header length
- btnxpuart: Check remote M.2 connector availability before pwrseq
- btmtksdio: Take exclusive ownership of the SKB before TX
- btmtksdio: Fix out-of-bounds DMA read in the TX path
- hci_uart: Fix false success return in hci_uart_setup()
- hci_bcm: fix usage_count leak when autosuspend_delay is negative
- hci_h5: fix usage_count leak when autosuspend_delay is negative
- hci_intel: fix usage_count leak when autosuspend_delay is negative
- btmtk: Do not report success when subsys reset fails
- btmtk: Do not discard the subsystem reset timeout
- btusb: limit RTL8761B BROKEN_EXT_SCAN quirk to 0bda:a728
- hci_bcm4377: Ignore reserved PHY in ext adv reports on BCM4378
* tag 'for-net-2026-08-24' of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth: (27 commits)
Bluetooth: RFCOMM: serialize session teardown
Bluetooth: do not leak an hci_conn when a second LE connect is rejected
Bluetooth: RFCOMM: serialize security confirmation handling
Bluetooth: btusb: limit RTL8761B BROKEN_EXT_SCAN quirk to 0bda:a728
Bluetooth: hci_uart: Fix false success return in hci_uart_setup()
Bluetooth: RFCOMM: Validate MTU in rfcomm_apply_pn() to prevent infinite loop
Bluetooth: ISO: fix use-after-free of listener socket in iso_conn_ready
Bluetooth: hci_core: use skb_get() instead of skb_clone() for req_skb
Bluetooth: hci_event: clear HCI_LE_ADV only on a created connection
Bluetooth: hci_conn: re-enable advertising only for peripheral role
Bluetooth: hci_bcm4377: Ignore reserved PHY in ext adv reports on BCM4378
Bluetooth: eir: Fix OOB read in eir_get_service_data()
Bluetooth: btnxpuart: Validate the FW dump header length
Bluetooth: hci_sync: add conditional locking annotations
Bluetooth: btnxpuart: Check remote M.2 connector availability before pwrseq
Bluetooth: btmtksdio: Fix out-of-bounds DMA read in the TX path
Bluetooth: btmtksdio: Take exclusive ownership of the SKB before TX
Bluetooth: btmtk: Do not discard the subsystem reset timeout
Bluetooth: btmtk: Do not report success when subsys reset fails
Bluetooth: L2CAP: fix race l2cap_sock_cleanup_listen() vs. put_chan
...
====================
Link: https://patch.msgid.link/20260824180639.3570348-1-luiz.dentz@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine
Pull dmaengine updates from Vinod Koul:
"Core:
- New API to combine configuration and preparation and users
New hardware support:
- Mediatek MT8189 SoC uart dma support
Updates:
- Designware dma driver flatten desc structures and simplify code,
interrupt-path groundwork changes, first part of PCI EP DMA support
- Updates to zynqmp_dma with runtime PM and device removal
improvments
- Xilinx dma optimizations for AXIDMA and MCDMA channel management"
* tag 'dmaengine-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine: (73 commits)
dmaengine: dw-edma: Mark emulated IRQ as level-triggered
dmaengine: idxd: assign all engines to group 0 in IAA defaults
dmaengine: qcom_hidma: remove conditional return with no effect
dmaengine: qcom-bam-dma: fix autosuspend cleanup during removal
dmaengine: fsl-edma: tracing: no ptr dereference during log output
dmaengine: dw-edma: Program endpoint function numbers
dmaengine: dw-edma-pcie: Add chip flags to match data
dmaengine: dw-edma-pcie: Handle optional data blocks
dmaengine: dw-edma-pcie: Factor out descriptor block address lookup
dmaengine: dw-edma-pcie: Add register offset match flag
dmaengine: dw-edma-pcie: Add platform ops to match data
dmaengine: dw-edma-pcie: Rename vsec_data to dma_data
dmaengine: dw-edma-pcie: Add capability match data
dmaengine: dw-edma-pcie: Track non-LL mode in DMA data
dmaengine: dw-edma: Add partial channel ownership mode
dmaengine: dw-edma: Initialize IRQ data before requesting IRQs
dmaengine: dw-edma: Add core quiesce operations
dmaengine: dw-edma: Add per-channel interrupt routing control
dmaengine: dw-edma: Factor out HDMA interrupt setup helper
dmaengine: dw-edma: Defer channel IRQ handling to workqueue
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/soundwire
Pull soundwire updates from Vinod Koul:
- Intel dmi quirks ghost list handling for Asus Zenbook Duo,
Asus ROG Zephyrus Duo and Asus Expertbook. Intel Peripheral
bra_block_alignment handling
- Cadence library BRA_NumBytes[8] support
- Qualcomm SCP address paging, bus mclk_freq support. Increase of
data ports to 17 and driver improvements
* tag 'soundwire-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/soundwire:
soundwire: dmi-quirks: Disable ghost Realtek on Asus ROG Zephyrus Duo
soundwire: stream: validate slave port properties
soundwire: honor clock_reg_supported in the clock scaling check
soundwire: qcom: set the bus mclk_freq property
soundwire: dmi-quirks: Disable ghost Realtek on Asus Zenbook Duo
soundwire: intel_ace2x: handle the max_data_per_frame property
soundwire: get mipi-sdw-bra-mode-max-data-per-frame property
soundwire: intel: handle Peripheral bra_block_alignment
soundwire: Add bra_block_alignment property support
soundwire: cadence_master: add BRA_NumBytes[8] support
soundwire: bus.h: repair kernel-doc comments
soundwire: intel_auxdevice: Add cs42l44 to wake_capable_list
soundwire: qcom: add SCP address paging support
soundwire: dmi-quirks: add a global ghost list
soundwire: dmi-quirks: Disable ghost Realtek on Asus Expertbook
soundwire: qcom: Allocate sruntime array dynamically
soundwire: qcom: Fix port exhaustion check in stream_alloc_ports
dt-bindings: soundwire: qcom: Increase max data ports to 17
|
|
of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux
Pull dma-mapping updates from Marek Szyprowski:
- swiotlb:
- new configuration option for the default pool size
(Jagadeesh Pagadala)
- reduce overhead for high watermark tracking (chenhuguanshen)
- minor code cleanups and improvements (Vova Sharaienko, Honglei Huang
and Marek Szyprowski)
- add proper tracking of the shared DMA state through direct, pool and
swiotlb paths (Aneesh Kumar K.V)
This is important for confidential-computing
* tag 'dma-mapping-7.3-2026-08-24' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux:
dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS
MAINTAINERS: update tree for DMA MAPPING HELPERS
dma/swiotlb: introduce Kconfig option for compile-time default pool size
dma-direct: Improve readability of the dma_direct_map_sg() for P2PDMA case
iommu/dma: simplify dma_iova_destroy() and drop the free_iova helper
dma-coherent: use KiB in DMA allocation logs
dma-coherent: fix spacing coding style issue
* tag 'dma-mapping-7.3-2026-08-24-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux: (23 commits)
swiotlb: remove unused SWIOTLB_FORCE flag
dma: swiotlb: handle set_memory_decrypted() failures
dma: swiotlb: free dynamic pools from process context
dma-direct: rename ret to cpu_addr in alloc helpers
dma-direct: select DMA address encoding from __DMA_ATTR_ALLOC_CC_SHARED
dma-direct: set decrypted flag for remapped DMA allocations
dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED
dma-direct: Move dma_direct_map_phys() to dma/direct.c
dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks
dma-mapping: make dma_pgprot() honor __DMA_ATTR_ALLOC_CC_SHARED
dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED
dma: swiotlb: pass mapping attributes by reference
dma-pool: track decrypted atomic pools and select them via attrs
dma-direct: use __DMA_ATTR_ALLOC_CC_SHARED in alloc/free paths
dma-mapping: Add internal shared allocation attribute
coco: arm64: s390: powerpc: Mark secure guests with CC_ATTR_GUEST_MEM_ENCRYPT
dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages
s390: Expose protected virtualization through cc_platform_has()
swiotlb: Preserve allocation virtual address for dynamic pools
dma: free atomic pool pages by physical address
...
|
|
The zero-copy path reads TX metadata whenever the UMEM has metadata space,
even if the descriptor does not set XDP_TX_METADATA. Pass descriptor
options through the metadata helpers and ignore metadata unless the option
is set.
This does not fix the existing per-WQE metadata handling for mlx5 MPWQEs.
Only the descriptor that starts a session passes through
xsk_tx_metadata_request() and configures offload state shared by the batch.
Metadata on descriptors joining an open session is therefore not validated
and does not configure its requested offloads. In addition, a non-NULL
metadata pointer from such a descriptor is treated as a timestamp
completion request even when XDP_TXMD_FLAGS_TIMESTAMP is not set, so its
metadata union can be overwritten with an unrequested timestamp. Fixing
mixed metadata states within one MPWQE requires a separate change.
Fixes: 48eb03dd2630 ("xsk: Add TX timestamp and TX checksum offload support")
Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Link: https://patch.msgid.link/20260819160535.1472459-3-sdf@fomichev.me
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Add explicit padding before launch_time so xsk_tx_metadata has the same
layout on 32-bit and 64-bit systems.
On several architectures (csky, i386, nios2, m65k, openrisc, sh), the old
native 32-bit layout put launch_time at offset 12 and had a natural size of
20 bytes. Using sizeof(struct xsk_tx_metadata) as tx_metadata_len was already
rejected because the length must be a multiple of eight, so the
straightforward use of the interface was broken on those ABIs. Userspace
could still register a padded length of 24 bytes, though; mixing the old and
new layouts then silently reads launch_time from the wrong offset and
misprograms packet launch times. This intentionally replaces that
incompatible layout because the affected architectures are unlikely to
have any notable users. (x86_64 and arm64 have the most users and are _not_
affected)
Fixes: ca4419f15abd ("xsk: Add launch time hardware offload support to XDP Tx metadata")
Reviewed-by: Simon Horman <horms@kernel.org>
Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20260819160535.1472459-2-sdf@fomichev.me
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
Pull slab updates from Vlastimil Babka:
- Add kfree_rcu_nolock() that can be used from contexts where spinning
on a lock might be unsafe, such as a BPF program attached to an
arbitrary function, or in NMI context. This complements the existing
kfree_nolock() support (Harry Yoo)
- Runtime instead of compile-time slabobj_ext sizing.
Avoid wasting memory when memory allocation profiling is compiled but
not enabled, with initial partial support to also avoid wasting
memory for objcg pointers when those are not needed, while profiling
is enabled (Vlastimil Babka)
- Various non-urgent fixes, cleanups and optimizations (Hao Li,
Hongling Zeng, Li RongQing, Li Xiasong, Seongjun Hong, Shengming Hu)
* tag 'slab-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: (31 commits)
mm/slab, kfence, memcg: completely remove obj_ext for kfence objects
mm/slab: stop allocating objcg pointers when unnecessary
mm/slab: add cache_ and slab_needs_objcg() helpers
mm/slab: stop exporting kvfree_rcu_barrier[_on_cache]()
slub_kunit: extend the test for kfree_rcu_nolock()
mm/slab: introduce kfree_rcu_nolock()
mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
mm/slab: reduce slabobj_ext memory with allocation profiling disabled
mm/slab: introduce slab_obj_ext_has_codetag()
mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
mm/slab: extend deferred free mechanism to handle rcu sheaves
mm/slab: use call_rcu() in unknown context if irqs are enabled
mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
mm/slab: change struct slabobj_ext to a union
mm/slab: replace slab.stride with obj_exts_in_object
mm/slab: abstract slabobj_ext.ref access
mm/slab: abstract slabobj_ext.objcg access
mm/slab: make slab_obj_ext() determine object index
mm: move struct slabobj_ext to mm/slab.h
mm/slab: remove objs_per_slab()
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86
Pull x86 platform driver updates from Ilpo Järvinen
"Highlights:
Major refactoring effort: stop setting acpi_device_name/class() and
pnp.device_class to facilitate their eventual removal
Many rollback/remove path fixes (presumably mostly found by AI)
Miscellaneous cleanups / refactoring / improvements
amd/halo:
- Add Halo RGB LED driver
amd/hsmp:
- Properly serialize probe, remove, and data paths
- Add support for protocol v7 used by Family 1AH Model 80H
- Fix error checking corner cases (largely from AI review)
- Reject negative power cap
amd/pmc:
- Improve behavior on platforms that do not support STB
- Add T14 Gen2 AMD (20XL) to s2idle quirk list
amd/pmf:
- Add ioctl interface to retrieve device metrics
- Add support for new metrics tables used by Family 1AH Model 80H
qcom-hamoa-ec (arm64):
- Reject short responses
asus-nb-wmi:
- Support ProArt key on ASUS ProArt PX13
asus-armoury:
- Gate PPT writes behind active fan curve
- Add power limits for more models
dell-wmi-base:
- Fix handling of ultra performance key
dell-wmi-sysman:
- Don't hex dump attribute security buffer
hp-bioscfg:
- Various fixes
- Improve reduced ACPI packages support (necessary for HP EliteBook 840 G2)
lg-laptop:
- Fix LED resource handling
- Add support for events used in newer models
- Fix keyboard backlight support on LG Gram 16T90SP
hp-wmi:
- Generalize thermal params to board params
- Manage CPU and GPU PWM independently
- Add GPU MUX switch support
- Add Victus 15-fb0xxx support
- Add OMEN MAX 16-ak0xxx, OMEN 16-n0xxx, OMEN 16-wd0xxx, OMEN
16-wf0xxx, and OMEN board ID 8D88 support
- Add OMEN Transcend 16-u0xxx support
huawei:
- Add support for Fn-lock ACPI interface found on newer Huawei
laptops such as MateBook 14 2024
ISST:
- Improve input validation (many fixes)
- Disallow SST-CP (core-power) feature if perf profile add fails
lenovo/yb9-kbdock:
- Add driver for Yoga Book 9 14IAH10
lenovo/ymc:
- Extend hinge switch query to support Yoga 9 2-in-1 14IPH11
- Prevent loading on Yoga Book 9 14IAH10 to avoid duplicated input
nodes
msi-ec:
- Add MSI Raider A18 HX A9WJG and MSI Katana GF76 11UEK support
msi-wmi:
- Add MSI Claw M-Center keys support
oxpec:
- Add support for OneXPlayer X2 Mini Pro
redmi-wmi:
- Report kbd backlight cycle, OEM preset power mode, and FnLock
toggle events to userspace
samsung-galaxybook:
- Add Samsung Galaxy Book6 Pro support
thinkpad_acpi:
- Add USB-C Security support
uniwill-laptop:
- Add keyboard backlight, AC auto boot, and USB powershare support
- Add MACHENIKE L16 Pro, AiStone X4SP4NAL, and Avell A60 MUV support
- Make lightbar max brightness configurable and add support for
LAPQC71A/B"
* tag 'platform-drivers-x86-v7.3-1' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86: (152 commits)
platform/x86: think-lmi: Fix current password length check
platform/x86: redmi-wmi: report EC state change events
MAINTAINERS: update Intel PMC Core maintainer contact
platform/x86: oxpec: Add support for OneXPlayer X2 Mini Pro
platform/x86: thinkpad_acpi: Fix fan speed reporting on Edge E330
platform/x86: msi-ec: Add MSI Katana GF76 11UEK EC firmware
platform/x86: think-lmi: Fix certificate thumbprint sysfs output
mlxbf-bootctl: fix the build error with FIELD_PREP()
platform/x86: think-lmi: Free system certificate signatures
platform/x86: ISST: Add a NULL check for sst_inst[]
platform/x86: ISST: Return error during profile addition
platform/x86: ISST: Just allow 2 bits for SST feature enable
platform/x86: ISST: Use PP level enable mask
platform/x86: ISST: Validate parameter for frequency and priority
platform/x86: ISST: Validate parameter for core power state
platform/x86: ISST: Validate max level for set feature
platform/x86: ISST: Validate logical CPU id and clos id
platform/x86: ISST: Validate level in perf mask ioctls
platform/x86: ISST: Validate socket ID in clos_assoc ioctl
platform/x86/amd/hsmp: Reject negative power cap writes in hwmon
...
|
|
For L2CAP sockets without owning sk->sk_socket, reading
l2cap_pi(sk)->chan may race against concurrent l2cap_sock_kill() ->
l2cap_sock_put_chan(). This excludes simultaneous proto_ops callbacks,
but access in l2cap_sock_cleanup_listen() has unsafe lockless read.
[Task 1] [Task 2 (hdev->workqueue)]
l2cap_sock_release(parent) l2cap_disconn_cfm
l2cap_sock_cleanup_listen l2cap_conn_del
bt_accept_dequeue l2cap_chan_del
lock_sock(sk) l2cap_sock_teardown_cb
bt_accept_unlink
bt_sk(sk)->parent = NULL
release_sock(sk) ----------------> lock_sock(sk)
parent = /* NULL */
lock_sock(sk) <--------------------- release_sock(sk)
sock_set_flag(sk, SOCK_ZAPPED)
l2cap_sock_close_cb
l2cap_sock_kill(sk)
l2cap_sock_put_chan
chan = READ l2cap_pi(sk)->chan l2cap_pi(sk)->chan = NULL
l2cap_chan_hold_unless_zero l2cap_put_chan(chan)
kref_get_unless_zero(&chan->ref)
Task 1 may observe NULL which causes null-ptr-deref.
Fix the race by taking lock_sock() in l2cap_sock_kill() to
synchronize with l2cap_sock_cleanup_listen(). hold_unless_zero() is not
needed here, l2cap_pi(sk)->chan owns reference if it is non-NULL.
Clarify code comments vs. locking.
Fixes: 6fef032af009 ("Bluetooth: L2CAP: Fix use-after-free in l2cap_sock_new_connection_cb()")
Reported-by: syzbot+e6382a2f53f5fc7453ac@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=e6382a2f53f5fc7453ac
Signed-off-by: Pauli Virtanen <pav@iki.fi>
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
|
|
'uuid_count' member of struct 'discovery_state' is assigned and read
without any locks, so there is a chance of situation when
uuid_count != 0, but uuids is NULL and there will be NULL pointer
dereference.
Possible race:
'hci_update_passive_scan_sync'
'hci_discovery_filter_clear'
hdev->discovery.uuid_count = 0;
<----------------------preempted----------------------------->
'start_service_discovery'
// Set uuid_count to value != 0
hdev->discovery.uuid_count = uuid_count;
hdev->discovery.uuids = kmemdup(...);
<----------------------preempted----------------------------->
spin_lock(&hdev->discovery.lock);
kfree(hdev->discovery.uuids);
hdev->discovery.uuids = NULL;
spin_unlock(&hdev->discovery.lock);
Now uuids == NULL and uuid_count != 0.
So 'mgmt_device_found' -> 'is_filter_match' -> 'eir_has_uuids' receives
non consistent discovery state, where NULL dereference of uuids happens.
To fix it let's add discovery.lock around every read/write of uuid_count,
uuids pair of struct members. It is also important to assign uuid_count
value only after success kmemdup() allocation in
start_service_discovery(), otherwise uuids is NULL, because kmemdup failed,
but uuid_count is already assigned to non zero value.
The following panic happens:
[ ] ------------[ cut here ]------------
[ ] Unable to handle kernel NULL pointer dereference at virtual
address 0000000000000000
[ ] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
[ ] CPU: 0 PID: 15056 Comm: kworker/u9:2
[ ] Workqueue: hci0 hci_rx_work
[ ] pstate: 10400009 (nzcV daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ ] pc : eir_has_uuids+0x2d8/0x590
[ ] lr : is_filter_match+0x258/0x320
...
[ ] Call trace:
[ ] eir_has_uuids+0x2d8/0x590
[ ] is_filter_match+0x258/0x320
[ ] mgmt_device_found+0x5b0/0xafc
[ ] process_adv_report.part.0+0x8c8/0xf14
[ ] hci_le_adv_report_evt+0x338/0x3f0
[ ] hci_le_meta_evt+0x1f0/0x4c8
[ ] hci_event_packet+0x440/0xc9c
[ ] hci_rx_work+0x44c/0xaf8
[ ] process_one_work+0x54c/0x103c
[ ] worker_thread+0x6c4/0x10c4
[ ] kthread+0x274/0x2ec
[ ] ret_from_fork+0x10/0x20
[ ] Code: 14000004 91004021 eb14003f 54000180 (f9400024)
[ ] ---[ end trace 0000000000000000 ]---
Fixes: 2935e556850e ("Bluetooth: hci_sync: fix double free in 'hci_discovery_filter_clear()'")
Signed-off-by: Pavel Shpakovskiy <pashpakovskii@salutedevices.com>
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 changes for 7.3
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
|
|
Merge series "mm/slab: introduce kfree_rcu_nolock() and improve
slub_kunit coverage" from Harry Yoo. From the cover letter [1]:
This series improves kmalloc_nolock() and kfree_nolock() coverage in
slub_kunit and introduces kfree_rcu_nolock() for unknown context as
suggested by Alexei Starovoitov.
Unknown context means the caller does not know whether spinning on a
lock is safe (e.g., a BPF program attached to an arbitrary kernel
function or in NMI context).
The slab allocator already supports unknown context via kmalloc_nolock()
and kfree_nolock(), but te slab allocator does not support freeing
objects by RCU in unknown context.
It is not ideal to have completely separate batching for unknown context
because the worst scenario where spinning on a lock would lead to
deadlock is very rare, and in most cases, it is safe to use the existing
mechanism (kfree_rcu_sheaf()).
Since most part of the slab allocator already supports unknown context
and sheaves support batching kvfree_rcu() calls for slab objects,
implement kfree_rcu_nolock() with minimal changes by teaching
kfree_rcu_sheaf() how to support unknown context and making it a little
bit harder to allocate an empty sheaf, instead of making intrusive
changes to the existing kvfree_rcu batching logic.
kfree_rcu_nolock() tries to free the object to the rcu sheaf if trylock
succeeds. Once the rcu sheaf becomes full, it is submitted to RCU via
call_rcu() if spinning is allowed or IRQs are enabled (to avoid calling
call_rcu() in the middle of call_rcu()). Otherwise, call_rcu() is
deferred via irq work.
When there is no sheaf available, kfree_rcu_sheaf() falls back to
defer_kfree_rcu(). It submits the object to kvfree_rcu batching via irq
work. To do this, patch 6 converts kvfree_rcu to use kvfree_rcu_head
without visible changes to the API for now.
Unlike kfree_rcu(), only the 2-argument variant is supported. This is
because the last resort of the 1-arg variant is synchronize_rcu(), which
cannot be used in an unknown context.
As suggested by Alexei Starovoitov, kfree_rcu_nolock() can be used with
struct kvfree_rcu_head (8 bytes), which is smaller than struct rcu_head
(16 bytes).
Link: https://lore.kernel.org/all/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org/ [1]
|
|
Revert commit 1b0866874833 ("locking: Switch to _irq_{disable,enable}()
variants in cleanup guards").
While the guards are properly nested, not all wrapped code is nice, as already
highlighted by that fair.c hunk.
Syzbot found another instance of this pattern in posix_timer_delete(), which
does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq).
Combined with this patch, that goes sideways most spectacular.
Undo this until we've developed stronger tools / debug for such issues.
Fixes: 1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824105523.GA4121620%40noisy.programming.kicks-ass.net
|
|
Fix IWYU issues:
/tmp/next/build/include/linux/io_uring_types.h:56:32: error: 'ITER_DEST'
undeclared here (not in a function)
56 | IO_BUF_DEST = 1 << ITER_DEST,
| ^~~~~~~~~
/tmp/next/build/include/linux/io_uring_types.h:57:32: error: 'ITER_SOURCE'
undeclared here (not in a function)
57 | IO_BUF_SOURCE = 1 << ITER_SOURCE,
| ^~~~~~~~~~~
Fixes: 95961b72c57b2 ("io_uring/rsrc: rename and export IO_IMU_DEST / IO_IMU_SOURCE")
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
https://gitlab.freedesktop.org/drm/msm into drm-next
Changes for v7.3
Bindings:
- Added Shikra support
- Document a840, a704, a722
Core:
- Use drm_client buffers for fbdev emulation
- teardown fixes
- ARM32 DMA fixup
- Remove objects from evict list when re-validated
- Bunch of corner case and error path fixes
DPU:
- Dropped dev_pm_opp_set_rate(0) preventing burnout
- Fixed SSPP offsets of Kaanapali
DP:
- Dropped dev_pm_opp_set_rate(0) preventing burnout
- Cleaned up core code in preparation for MST support
- Fixed prepare() to let Pipewire continue in case of the unplugged cable
GPU:
- Add support for a704
- Add support for a722
HDMI:
- Simplifed register access
Signed-off-by: Dave Airlie <airlied@redhat.com>
From: Rob Clark <rob.clark@oss.qualcomm.com>
Link: https://patch.msgid.link/CACSVV02Kp=J+w_RjEJbBbQnBYRb+SWdwMvVbCaAL70bq9EBagQ@mail.gmail.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux
Pull RCU updates from Paul McKenney:
"Make expedited grace periods expedite normal RCU callbacks
Miscellaneous fixes:
- Improve diagnostic output with character task states
- Mark accesses to inform KCSAN of concurrency design
- Move from kmalloc() to kmalloc_obj()
- Documentation updates
- Improve handling of RCU deferred quiescent states
- Clean up unused function arguments and structure fields
- Reduce show_rcu_gp_kthreads() stack space
Tasks RCU updates:
- Clean up after SRCU re-implementation of Tasks Trace RCU
- Mark accesses to inform KCSAN of concurrency design
- Add ->lazy_timer status to diagnostic output
- Remove an unnecessary memory barrier
- Fix a data race, courtesy of KCSAN
- Documentation updates
- Convert cond_resched_tasks_rcu_qs() from macro to static inline
function
SRCU updates:
- Add Rust helpers for SRCU
- Avoid losing queued work at cleanup_srcu_struct() time
Torture-test updates:
- Preparation work for immediate RCU priority deboosting
- Test RCU readers from real interrupt handlers (as opposed to
softirq)
- Simplify code through use of cpumask_next_wrap()
- Improve diagnostic output with character task states
- Add rcutorture.nwriters parameter to allow lightweight stall
testing, and rcutorture.stall_only to make doing so easier
- Test an RCU Tasks Trace grace period implying an RCU grace period
- Make RCU Tasks Trace torturing track reader batches
- Fix a data race, courtesy of KCSAN
- Plug a shuffle_tmp_mask memory leak on kthread spawn failure"
* tag 'rcu.2026.08.18a' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (59 commits)
rcu: Add closing parenthesis in comment in rcu_read_unlock_strict()
rcutorture: Make {,s}rcu_read_delay() better handle forward-progress testing
rcutorture: Announce declining to forward-progress test
torture: Don't leak shuffle_tmp_mask when shuffler kthread fails to start
rcutorture: Use this_cpu_inc() for rcu_torture_count[] and rcu_torture_batch[]
rcutorture: Make RCU Tasks Trace track Reader Batches
rcutorture: Test RCU Tasks Trace GP implying RCU GP
rcutorture: Add a stall_only module parameter
rcutorture: Add nwriters module parameter
rcutorture: Use task_state_to_char() for task-state reporting
rcutorture: Use cpumask_next_wrap() in rcu_torture_preempt()
rcutorture: Test RCU readers from hardware interrupt handlers
rcutorture: Check for immediate deboosting at reader end
srcu: Queue sdp->work when the delay timer is successfully deleted
rcu-tasks: Convert cond_resched_tasks_rcu_qs() to static inline
rcu-tasks: Fix some comments for call_rcu_tasks() and call_rcu_tasks_rude()
rcu-tasks: Rename tasks_rcu_exit_srcu_stall_timer to tasks_rcu_exit_stall_timer
rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s
rcu: Reduce stack usage in show_rcu_gp_kthreads()
rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs
...
|
|
In Hyper-V versions prior to WS2016/Win10, Hyper-V synthetic timers
interrupt the guest by delivering a message that is initially handled
by the Linux VMBus driver. Starting with WS2016/Win10, Hyper-V can
deliver stimer interrupts directly to an assigned interrupt vector
without involving the VMBus driver. This is called "Direct Mode".
With the overall removal of Linux support for running on Hyper-V
hosts earlier than WS2016 and Windows 10, it's no longer necessary
to support the legacy message-based delivery. Remove that delivery
mechanism and always use Direct Mode. If for some reason, the
Hyper-V host does not enumerate Direct Mode, output an error
message but continue to run using the LAPIC timer instead of an
stimer.
With these changes, the VMBus driver no longer calls the stimer
interrupt service routine. This removal has a broader benefit in
unblocking the disentangling of VMBus code and stimer code, as
they should be independent of each other. The final disentangling
will come as a follow-on patch set.
Signed-off-by: Michael Kelley <mhklinux@outlook.com>
Signed-off-by: Wei Liu <wei.liu@kernel.org>
|
