| Age | Commit message (Collapse) | Author |
|---|
|
Patch series "mm/damon: introduce data attributes only monitoring".
TL;DR: Introduce a way to get DAMON's best effort accuracy monitoring of
user-demanding non-access data attributes.
Background
==========
DAMON was initially designed for only access monitoring. It turned out
users want to get the information together with more data attributes. For
example, some users want to know how much of a hot memory region belongs
to huge pages or specific cgroups. Page level properties based monitoring
was introduced with commit 626ffabe67c2 ("mm/damon: clarify trying vs
applying on damos_stat kernel-doc comment") to fill the gap. Because it
works only at snapshot level and snapshot capturing in the mode can induce
high overhead, commit 45c49d9fd608 ("mm/damon/core: introduce struct
damon_probe") introduced data attributes monitoring.
Data attributes monitoring treats the attributes as only additional and
subordinate information. Data access monitoring is always turned on, and
regions are adjusted for best accuracy of the access information. In some
cases, users may be primarily interested in the attributes more than the
access. They might even not care about the access information at all.
Because DAMON treats data accesses as the only primary information, such
users cannot get high quality attributes information.
Design and Implementation
=========================
Introduce another way for treating data attributes as the primary
information. Add 'weight' property to each data attribute probe. When
any of the weights are set, the mode is enabled. Data access monitoring
is completely turned off in the mode. For region adjustment, the weighted
sum of probe hit counters is used instead of the nr_accesses.
Using the weights, users can specify to what attributes they are
interested in to what degree. DAMON will adjust the regions and provide
the best-effort quality monitoring that is optimized for the user demands.
Extend damon_operations for efficient use of probe hits. Update regions
merge and kdamond main logic to support the new mode. Add a new struct
field and a sysfs file for API callers and ABI users, respectively.
Test
====
On ~7 GiB memory idle system, run a simple AI-assisted program. The
program allocates and faults 2 GiB anonymous pages. Then, it does nothing
but wait until the user terminates it. Hence, the system ~2 GiB of
anonymous pages with no active accesses.
Monitor the distribution of the anonymous pages using DAMON attributes
monitoring mode, using DAMON user-space tool, damo [1].
$ sudo ./damo start --probe_filter allow anon
$ sudo ./damo report access --dont_merge_regions
heatmap: 00000000000000000000000000000000000000000000000399999995111111146666666666666666
# min/max temperatures: -2,470,000,000, -1,620,000,000, column size: 99.800 MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
# <start> <size> <freq> <age> <probe hits>
0 4.000 KiB 79.840 MiB 0 hz 24.700 s 2
1 79.844 MiB 718.562 MiB 0 hz 24.700 s 8
2 798.406 MiB 793.148 MiB 0 hz 24.700 s 7
3 1.554 GiB 797.828 MiB 0 hz 24.700 s 7
4 2.333 GiB 794.668 MiB 0 hz 24.600 s 8
5 3.109 GiB 791.117 MiB 0 hz 24.500 s 0
6 3.882 GiB 785.312 MiB 0 hz 24 s 2
7 4.649 GiB 787.867 MiB 0 hz 16.200 s 6
8 5.418 GiB 784.477 MiB 0 hz 23.300 s 6
9 6.184 GiB 783.820 MiB 0 hz 18.200 s 9
10 6.950 GiB 797.730 MiB 0 hz 18.900 s 7
11 7.729 GiB 69.625 MiB 0 hz 18.900 s 0
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
Note that the line after the line starting with "intervals:" is not
provided by the current version of 'damo'. I manually added the legends
line for easier understanding of these results.
Each of the 12 lines after the legend line shows the DAMON-found regions.
Each line shows 1) index of the region, 2) start address of the region, 3)
size of the region, 4) access frequency of the region, 5) age (how long
the access frequency on the region was kept) of the region, and finally 6)
the probe hit count.
Because data access is the primary information that adjusts region for,
and there is only nearly zero access on the system, regions are naively
adjusted with the same size. Still <probe hits> show different
distribution of the anonymous pages, but it is obviously very rough
information.
Switch to the attributes only mode and show how it changes the picture:
$ sudo ./damo tune --probe_filter allow anon --probe_weight 100
$ sudo ./damo report access --dont_merge_regions
heatmap: 88888888888888888889888999999889999999000004888888888888889999988888898888888888
# min/max temperatures: -4,430,000,000, 0, column size: 99.800 MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
# <start> <size> <freq> <age> <probe hits>
0 4.000 KiB 60.445 MiB 0 hz 700 ms 0
1 60.449 MiB 1.363 MiB 0 hz 600 ms 18
2 61.812 MiB 144.000 KiB 0 hz 0 ns 1
3 61.953 MiB 1.922 MiB 0 hz 2.400 s 19
4 63.875 MiB 12.133 MiB 0 hz 200 ms 0
[...]
500 5.132 GiB 8.000 KiB 0 hz 2 m 15.800 s 20
501 5.132 GiB 8.000 KiB 0 hz 2 m 16.200 s 0
502 5.132 GiB 16.000 KiB 0 hz 2 m 16.900 s 20
503 5.132 GiB 24.000 KiB 0 hz 2 m 14.200 s 0
504 5.132 GiB 8.000 KiB 0 hz 2 m 14.900 s 20
[...]
923 7.534 GiB 126.637 MiB 0 hz 0 ns 6
924 7.658 GiB 252.000 KiB 0 hz 54.800 s 20
925 7.658 GiB 142.242 MiB 0 hz 300 ms 0
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
As expected, regions are adjusted to provide the best accurate picture for
the anonymous pages distribution (<probe hits>). The region 0 (60.445 MiB
memory from the address 4.000 KiB) has nearly zero anonymous pages. The
region 1 (1.363 MiB memory from the address 60.449 MiB) is nearly full
with anonymous pages. Region 500 (8 KiB memory from the address 5.132
GiB) is certainly two anonymous pages.
Future Work
===========
Attributes only monitoring disables access monitoring. We will enable
that in future, by extending the supported attributes to include data
accesses. This patch series, and the future work are parts of the ongoing
project [2] for extending DAMON. The project aims to extend DAMON with
primitives other than page table accessed bits such as AMD IBS, Intel
PEBS, and Arm SPE, to provide more powerful and detailed information like
per-CPUs/threads/reads/writes monitoring.
Patches Sequence
================
Patch 1 introduces damon_probe->weight for specifying the weights of each
attribute. Patches 2-6 extends apply_probe() damon_ops callback to
efficiently support the new mode. Patch 7 fixes wrong use of abs() in the
regions merge code. Patch 8 extends regions merge function to work with
probe hits in the mode. Patch 8 also introduces the function for
detecting the mode enablement but always returns false, for safe and
incremental changes. Patches 9 and 10 adds user parameters validation to
prevent theoretical overflow of probe hits and the weighted sum. Patches
11-14 incrementally update kdamond_fn() to support the mode. Patch 15
completes the mode detection function implementation, so that the new mode
really works. Patch 16 introduces a new sysfs file for ABI users.
Finally, patches 17-19 respectively updates design, usage and ABI
documents for the new feature and interfaces.
[1] https://github.com/damonitor/damo
[2] https://lore.kernel.org/20260525225208.1179-1-sj@kernel.org/
This patch (of 19):
Add a new field, weight to damon_probe struct. The field is used to
specify the degree of the API caller's interest to the data attribute of
the probe.
Link: https://lore.kernel.org/20260710134651.18084-1-sj@kernel.org
Link: https://lore.kernel.org/20260710134651.18084-2-sj@kernel.org
Link: https://github.com/damonitor/damo [1]
Link: https://lore.kernel.org/20260525225208.1179-1-sj@kernel.org/ [2]
Signed-off-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Remove this entirely unused but exported function.
Link: https://lore.kernel.org/20260710051052.1839523-1-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Acked-by: SJ Park <sj@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since commit f002882ca369 ("mm: merge folio_is_secretmem() and
folio_fast_pin_allowed() into gup_fast_folio_allowed()"),
gup_fast_folio_allowed() falls back to the slow path for any order-0 folio
with a NULL mapping when CONFIG_SECRETMEM=y. This causes a performance
regression for drivers that allocate pages with alloc_page() and insert
them into VMAs via vm_insert_page(). These pages legitimately have a NULL
folio->mapping, but they cannot be secretmem pages.
Secretmem pages are always added to the secretmem inode's page cache via
filemap_add_folio(), which sets folio->mapping to the inode's i_mapping.
A folio with a NULL mapping can never be a secretmem folio. The
NULL-mapping check was intended to handle truncated file-backed pages (a
reject_file_backed concern), not secretmem detection.
When only check_secretmem is true (and reject_file_backed is false), a
NULL mapping is sufficient to prove the folio is not secretmem, so the
fast path can proceed.
Link: https://lore.kernel.org/20260708005745.164928-1-jhubbard@nvidia.com
Fixes: f002882ca369 ("mm: merge folio_is_secretmem() and folio_fast_pin_allowed() into gup_fast_folio_allowed()")
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Tested-by: Sourab Gupta <sougupta@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The KSM selftest uses conditional expressions to skip accesses to
merge_across_nodes on systems without NUMA support. However, the ternary
operator is combined with logical OR without parentheses:
a || numa_available() ? 0 : b || c
Due to operator precedence rules, this is parsed as:
(a || numa_available()) ? 0 : (b || c)
instead of the intended:
a || (numa_available() ? 0 : b) || c
Add parentheses around the conditional expressions to ensure the
correct evaluation order.
Link: https://lore.kernel.org/ce859430287ed2642848c933a90eb9a69da361f0.1783446924.git.sayalip@linux.ibm.com
Fixes: 9aa1af954db0 ("selftests: vm: check numa_available() before operating "merge_across_nodes" in ksm_tests")
Signed-off-by: Sayali Patil <sayalip@linux.ibm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Liam Howlett <liam@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The KSM NUMA merge test allocates identical pages on different NUMA nodes
and verifies KSM behavior with merge_across_nodes enabled and disabled.
On systems with memoryless NUMA nodes, for example:
#numactl -H
available: 2 nodes (0,4)
.....
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 14825 MB
node 0 free: 1382 MB
node 4 cpus:
node 4 size: 0 MB
node 4 free: 0 MB
the test may attempt to allocate memory on a node without memory, causing
numa_alloc_onnode() to fail and resulting in a spurious test failure.
The test currently checks numa_num_configured_nodes() to determine whether
sufficient NUMA nodes are available. However, configured nodes do not
necessarily have memory.
Reuse the existing get_first_mem_node() and get_next_mem_node() helpers to
locate NUMA nodes that actually contain memory, and skip the test when
fewer than two such nodes are available.
Before patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
mbind: Invalid argument
ok 1 KSM NUMA merging
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
mbind: Invalid argument
not ok 1 KSM NUMA merging
Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
[FAIL]
not ok 2 ksm_tests -N -m 0 # exit=1
After patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 2 ksm_tests -N -m 0
Link: https://lore.kernel.org/78a3b0e3fb94004c0710872c5bab6f7381b7d63c.1783446924.git.sayalip@linux.ibm.com
Fixes: e3820ab252dd ("selftest/vm: fix ksm selftest to run with different NUMA topologies")
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Sayali Patil <sayalip@linux.ibm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Liam Howlett <liam@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "selftests/mm: avoid false failures in hugetlb and KSM
tests", v3.
This series fixes issues in the hugetlb and KSM MM selftest categories
that can report failures when the prerequisites for the tests are not
satisfied.
Patch 1 updates the hugetlb selftest helpers to handle -EINVAL when
attempting to configure gigantic HugeTLB pages via nr_hugepages. PowerPC
hash MMU pSeries systems expose gigantic hugepage sizes but do not allow
runtime allocation of such pages, causing the sysfs write to fail. Handle
this case gracefully and continue running the test instead of aborting.
Patch 2 fixes the KSM NUMA merge test on systems with memoryless NUMA
nodes. The test currently relies on the number of configured NUMA nodes
and may attempt allocations on nodes that have no memory, resulting in
spurious failures. Use the existing helpers to identify NUMA nodes that
contain memory and skip the test when fewer than two such nodes are
available.
Patch 3 fixes a pre-existing operator precedence issue in ksm_tests, where
a ternary expression combined with logical OR operators could be evaluated
differently than intended. Added parentheses to ensure the correct
evaluation order.
These changes improve handling of unsupported test configurations and
unmet test prerequisites, avoiding spurious failures.
This patch (of 3):
Some MM selftests attempt to configure the amount of HugeTLB pages of
different sizes by writing to nr_hugepages.
PowerPC hash MMU pSeries systems advertise gigantic hugepage sizes but do
not support runtime allocation of such pages, writes to the corresponding
nr_hugepages file fail with -EINVAL. This causes the test to bail out
even though the failure is due to a platform limitation rather than the
functionality being tested.
Ignore -EINVAL when configuring nr_hugepages so that tests continue to run
on systems where gigantic hugepage allocation is unsupported.
Before patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
Bail out! /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages
write(0) failed: Invalid argument
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
[FAIL]
After patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
Link: https://lore.kernel.org/cover.1783446924.git.sayalip@linux.ibm.com
Link: https://lore.kernel.org/2e3b585cbb30b2fc495dcd49d75de6f6da61861c.1783446924.git.sayalip@linux.ibm.com
Fixes: 27477b28b74f ("selftests/mm: hugepage_settings: add APIs to get and set nr_hugepages")
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Sayali Patil <sayalip@linux.ibm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Liam Howlett <liam@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Several early return paths in run_migration_benchmark() skip
hmm_buffer_free(), leaking the buffer. Replace with a single cleanup
label.
Link: https://lore.kernel.org/20260709081843.1451202-1-lihongfu@kylinos.cn
Fixes: 271a7b2e3c13 ("selftests/mm/hmm-tests: new throughput tests including THP")
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Acked-by: Balbir Singh <balbirs@nvidia.com>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Hongfu Li <lihongfu@kylinos.cn>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since we now have linear_page_index available that we can use here,
allowing for optimizing the RMAP walk, we can also use it to locate more
precisely all related-processes when error hits the KSM page, which will
decrease a lot of invalid iterations.
Link: https://lore.kernel.org/20260709173312403qgj1Af6pRkFMDSsmc19sM@zte.com.cn
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "KSM: use linear_page_index in collect_procs_ksm()", v2.
In collect_procs_ksm() which is used to collect processes when the error
hit an ksm page, there is the same issue with rmap_walk_ksm (see the
previous discussion at [1]). So we apply the similar logic changes to the
collect_procs_ksm().
The patch [1/2] move the initializaion of addr from the position inside
loop to the position before the loop, since the variable will not change
in the loop.
The patch [2/2] optimize collect_procs_ksm by passing a suitable page
offset range to the anon_vma_interval_tree_foreach loop to reduce
ineffective checks.
This patch (of 2):
Similar to 318d87b8fa7 ("ksm: initialize the addr only once in
rmap_walk_ksm"), only initialize the addr once in rmap_walk_ksm because
the addr variable doesn't change across iterations.
Link: https://lore.kernel.org/20260709173212190rZdwynySRyLr9EtPuXBRU@zte.com.cn
Link: https://lore.kernel.org/all/20260703162253688u8Str9eFLR8TGCmo7nIOF@zte.com.cn/ [1]
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mm/internal.h becomes more and more bloated.
Move declarations related to vmalloc to a new mm/vmalloc.h header.
No functional changes.
Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Pratyush Yadav <pratyush@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Alexander Graf <graf@amazon.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mm/internal.h becomes more and more bloated.
Move declarations related to SPARSE and SPARSE_VMEMMAP memory models to
a new mm/sparse.h header.
No functional changes.
Link: https://lore.kernel.org/20260709-internal-h-v2-2-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Pratyush Yadav <pratyush@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Alexander Graf <graf@amazon.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: split a couple of headers from internal.h", v2.
mm/internal.h becomes more and more bloated.
Split declarations related to mm_init, memblock, vmalloc and sparse into
new headers.
This patch (of 3):
mm/internal.h becomes more and more bloated.
Move declarations for related to mm/mm_init.c and mm/memblock.c to a new
mm/mm_init.h header.
No functional changes.
[rppt@kernel.org: split stubfs from internal.h to mm_init.h]
Link: https://lore.kernel.org/alJd1BLypyK9Mpaw@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-0-695631425968@kernel.org
Link: https://lore.kernel.org/20260709-internal-h-v2-1-695631425968@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: Pratyush Yadav <pratyush@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Alexander Graf <graf@amazon.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The existing tools/testing/selftests/mm/rmap.c has already one testcase
for ksm_rmap_walk in TEST_F(migrate, ksm), which takes use of migration of
page from one NUMA node to another NUMA node. However, it just lacks the
scenario of mremapped VMAs.
We add the calling of mremap() and then trigger KSM to merge pages before
migrating, which is specifically to test an optimization which is
introduced by this patch ("ksm: Optimize rmap_walk_ksm by passing a
suitable address pgoff").
This test can reproduce the issue that Hugh points out at
https://lore.kernel.org/all/02e1b8df-d568-8cbb-b8f6-46d5476d9d75@google.com/
Link: https://lore.kernel.org/20260703162637070FU4ekl58Hw_Z7OSuJryZB@zte.com.cn
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
User impact / Why this matters to Linux users
=============================================
When a system runs with KSM enabled and memory becomes tight, KSM pages
may be swapped out or migrated. The kernel then performs a reverse map
walk by rmap_walk_ksm to locate all page table entries that reference
these pages. If A large number of unrelated VMAs can attach to a single
anon_vma related with this KSM page, then rmap_walk might be severe
performance bottleneck. In our embedded test environment, we observed
~20,000 VMAs sharing one anon_vma without any fork purely from VMA
splits
which cause 200~700ms duration of rmap_walk_ksm.
When one of those VMAs mapped a KSM page, then this KSM page's rmapping
will become bottleneck with hold its anon_vma lock for a long time. The
anon_vma lock is not only used by KSM; it is a core lock protecting the
VMA interval tree and is acquired by many critical memory operations:
' Page faults: do_anonymous_page(), do_wp_page() (during COW)
' Memory reclaim: try_to_unmap()
' Page migration & compaction: migrate_pages(), compact_zone()
' mlock / munlock: mlock_fixup()
' Process exit: exit_mmap() (tearing down VMAs)
' Cgroup memory accounting: mem_cgroup_move_charge()
If one thread holds the anon_vma lock for hundreds of milliseconds
because of an inefficient KSM rmap walk, any other thread that
tries to acquire the same lock (e.g., an application taking a page
fault, kswapd reclaiming pages, or a migration thread) will block.
This leads to stalled application threads, increased latency
spikes, and in extreme cases container timeouts or watchdog
triggers.
This patch reduces the worst-case anon_vma lock hold time during
ksm_rmap_walk from >500 ms to <1 ms, thereby almost eliminating
this source of lock contention and improving system responsiveness
under memory pressure.
Real-world examples:
====================
- JVM / Go runtime: These use mmap for heap regions and later call
mprotect(PROT_NONE) for garbage collection barriers or guard pages,
splitting the original VMA into thousands of small pieces over time.
- Database engines (MySQL, PostgreSQL): Large shared memory buffers
or anonymous mappings are managed with madvise(MADV_DONTNEED) to
release specific pages, which also splits VMAs.
Root Cause
==========
Through local debugging trace analysis, we found that most of the
latency of rmap_walk_ksm occurs within anon_vma_interval_tree_foreach,
leading to an excessively long hold time on the anon_vma lock (even
reaching 500ms or more), which in turn causes upper-layer applications
(waiting for the anon_vma lock) to be blocked for extended periods.
Further investigation revealed that 99.9% of iterations inside the
anon_vma_interval_tree_foreach loop are skipped due to the first check
"if (addr < vma->vm_start || addr >= vma->vm_end)), indicating that a
large number of loop iterations are ineffective. This inefficiency
arises because the start page index and the end page index parameters
passed to anon_vma_interval_tree_foreach span the entire address space
from 0 to ULONG_MAX, resulting in very poor loop efficiency.
Solution
========
We cannot rely solely on anon_vma to locate all PTEs mapping this page
but also need to have the original page's linear_page_index. Since the
implementation of anon_vma_interval_tree_foreach it essentially
iterates to find a suitable VMA such that the provided page index
falls within the candidate's vm_pgoff range.
vm_pgoff <= original linear page offset <= (vm_pgoff + vma_pages(v) - 1)
Fortunately, an earlier commit introduced the linear_page_index to struct
ksm_rmap_item, allowing for optimizing the RMAP walk.
Test results
============
A rmap testbench can be obtained with two Out-Of-Tree patches at [1][2].
After applying the OOT patches and building rmap_benchmark from:
tools/testing/rmap/rmap_benchmark.c, we can start the performance test.
The testing result in QEMU is shown as follows:
KSM rmapping Maximum duration Average duration
Before: 705.12 ms (705119858 ns) 532.04 ms (532041586 ns)
After: 1.67 ms (1665917 ns) 1.44 ms (1443784 ns)
The benchmark numbers are realistic, since we observed ~20,000 VMAs
sharing one anon_vma on a production system running a Java application
with KSM enabled. The lock hold time before the patch was measured at
228ms (max) during rmap walks triggered by memory compaction and page
migration. The benchmark reproduces that VMA count and lockhold
behavior in a controlled environment.
Link: https://lore.kernel.org/20260703162510242nxmjbcLy5ccp1dbZSK3EU@zte.com.cn
Link: https://lore.kernel.org/all/202605301703094695zmVgcSC27BNR0rH0N8_x@zte.com.cn [1]
Link: https://lore.kernel.org/all/20260530170404509QpJmBtpSjn3uQHeVKA2iA@zte.com.cn/ [2]
Co-developed-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "KSM: performance optimizations for rmap_walk_ksm", v11.
This series fixes a severe KSM reverse-mapping performance problem that
can freeze applications for hundreds of milliseconds under memory pressure
especially when a lot of unrelated VMAs sharing a single anon_vma.
Two key highlights:
1. Lock hold time drops from >500ms to <2ms
- In our benchmark (20,000 VMAs sharing an anon_vma), worst-case
anon_vma lock hold time during KSM rmap walk went from 705ms
down to 1.67ms (max) and 1.44ms (avg).
2. Real user impact
- The anon_vma lock is also acquired by page faults, reclaim,
migration, compaction, mlock, exit_mmap, and cgroup accounting.
- A long hold due to inefficient rmap walks stalls application
threads, causing latency spikes, reduced throughput, or even
container timeouts.
- The problem occurs even without fork() – VMA splitting (e.g.,
via mprotect or madvise over time) can create tens of thousands
of VMAs all attached to the same anon_vma.
Real-world examples:
- JVM / Go runtime: These use mmap for heap regions and later call
mprotect(PROT_NONE) for garbage collection barriers or guard pages,
splitting the original VMA into thousands of small pieces over time.
- Database engines (MySQL, PostgreSQL): Large shared memory buffers or
anonymous mappings are managed with madvise(MADV_DONTNEED) to release
specific pages, which also splits VMAs.
Why the benchmark numbers are realistic: We observed ~20,000 VMAs sharing
one anon_vma on a production system running a Java application with KSM
enabled. The lock hold time before the patch was measured at 228 ms
(max) during rmap walks triggered by memory compaction and page migration.
The benchmark reproduces that VMA count and lock‑hold behavior in a
controlled environment.
For systems that do not have thousands of VMAs per anon_vma, the patch
adds negligible overhead (a single pgoff comparison). For systems that do
suffer from this issue, the improvement is dramatic: 1) Worst‑case
anon_vma lock hold time drops from hundreds of milliseconds to under
2 ms.2)This directly reduces blocking of parallel operations that need
the same lock – page faults, reclaim, migration, compaction, mlock, and
exit_mmap.
End‑users will see lower tail latency (fewer application stalls), higher
throughput under memory pressure, and no more spurious lockup warnings or
container timeouts caused by excessive lock hold times.
In short: workloads that do not hit this pathological pattern are
unaffected; those that do will see a 100x to 500x reduction in lock hold
times, which translates directly into a more responsive system.
This patch (of 3):
As preparation for KSM rmap optimizations, let's track the original
linear_page_index() of a de-duplicated page in its ksm_rmap_item, so we
can efficiently search for the page in an address space, avoiding scanning
the entire address space. This was previously discussed in [1, 2].
To avoid growing ksm_rmap_item, let's squeeze it into the existing
structure by overlying some members (oldchecksum, age, remaining_skips)
that are only relevant while on the unstable tree. The new entry will
only be relevant for entries in the stable tree.
However, as the age information is read by should_skip_rmap_item() with
the smart-scanning approach even while we have an entry in the stable
tree, but the page changes (no longer a KSM page, for example due to COW),
we have to change the handling there a bit.
We'll calculate the linear page index in try_to_merge_with_ksm_page(),
when adding it to the stable tree, and reset the index (to reset overlayed
data) when removing an item from the stable tree -- in
remove_rmap_item_from_tree(), remove_node_from_stable_tree() and
break_cow().
To be specially clarified, the reason for resetting the stored index at
break_cow() is:
- When a page successfully becomes a KSM page (i.e., after
stable_tree_append() sets STABLE_FLAG), both anon_vma and the index are
stored and remain valid.
- However, during the merging process, there are several failure paths
where we already prepared an rmap item to be added to the stable tree,
but must revert that as some part of the merge process failed. Examples
include:
1 The second call to try_to_merge_with_ksm_page() fails in
try_to_merge_two_pages().
2 stable_tree_insert() fails in cmp_and_merge_page().
In such cases, break_cow() is invoked to break the COW mapping and
discard the KSM state.
Currently, break_cow() already contains a
put_anon_vma(rmap_item->anon_vma) to release the reference taken during
the aborted merge. Because the index is logically paired with anon_vma
(both are only meaningful when the rmap_item is in a stable state), it
must also be cleared (or reset) in break_cow() to avoid leaving stale
linear_page_index values that could confuse subsequent rmap walks or
scanning logic.
Link: https://lore.kernel.org/20260703162253688u8Str9eFLR8TGCmo7nIOF@zte.com.cn
Link: https://lore.kernel.org/20260703162357853iIa-RP7if9hRlAIuTh5La@zte.com.cn
Link: https://lore.kernel.org/all/adTPQSb-qSSHviJN@lucifer/ [1]
Link: https://lore.kernel.org/all/202604091806051535BJWZ_FTtdIm3Snk24ei_@zte.com.cn/ [2]
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
If damon_update_monitoring_result() is called at the end of the
aggregation interval, probe_hits is not reset. That's because the value
will be exposed to the user via damon_region_aggregated trace event.
Meanwhile, damon_probe_hits_mvsum() can be called in this state. Due to
its logic, it will return a value that is incorrectly high. This could
happen if the user requested DAMOS schemes applied regions sysfs files
update exactly in the time sequence.
The impact is minor, but better to avoid. Check the timing and simply
return the fully aggregated last_probe_hits, like
damon_nr_accesses_mvsum() also does. It is not 100% accurate since it is
the last interval's aggregation. But better than the value that is
completely reset.
Link: https://lore.kernel.org/20260708135359.122587-8-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Users can update DAMON parameters at runtime. If the samples and/or
aggregation intervals are updated in this way, monitoring results
depending on the intervals should also be updated for a more accurate
snapshot. The age and nr_accesses are properly updated, while probe_hits
are not updated in the way. Do the update.
Link: https://lore.kernel.org/20260708135359.122587-7-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_nr_accesses_for_new_attrs() can be used for not only nr_accesses but
also any positive sample count, like probe_hits. Rename to be able to be
used for such general uses without confusion.
Link: https://lore.kernel.org/20260708135359.122587-6-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_nr_accesses_to_accesses_bp() actually converts a positive sample
count to the ratio. Rename it to better describe what it really does and
not confusing for more general uses.
Link: https://lore.kernel.org/20260708135359.122587-5-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
accesses_bp_to_nr_accesses() actually converts a positive samples ratio to
the count. Rename it to better describe what it really does and not
confusing for more general uses.
Link: https://lore.kernel.org/20260708135359.122587-4-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_max_nr_accesses() actually returns the number of samples DAMON
checks for each region per each aggregation interval. Rename it to better
describe what it really does and not confusing for more general uses.
Link: https://lore.kernel.org/20260708135359.122587-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: update probe hits for runtime parameter commits".
DAMON users can update DAMON parameters such as sampling and aggregation
intervals at runtime. For such changes, monitoring results that depend on
the intervals should be properly updated for better accuracy. For
example, the access frequency counter (nr_accesses) is updated. The data
attribute monitoring counter (probe_hits) is not being updated, though.
Do the updates for new parameters.
Patch 1 removes obsolete comments and test code for a function that this
series will touch. Patches 2-5 rename functions that are being used for
nr_accesses update, to be able to be used for probe_hits without
confusion. Patch 6 does the probe_hits update. Patch 7 update
damon_probe_hits_mvsum() to cover a corner case from the update for better
accuracy.
This patch (of 7):
The comments on damon_nr_accesses_to_accesses_bp() and its unit test warn
it can trigger division-by-zero when the aggregation interval is zero.
Commit 35d4a3cf70a8 ("mm/damon/ops-common: handle extreme intervals in
damon_hot_score()") modified damon_max_nr_accesses() to always return
non-zero. Hence no division-by-zero of the note can happen. Remove the
obsolete comment on the function. The test code was written to test the
division-by-zero case, which cannot happen anymore. Having it makes no
sense. Entirely remove the test code and its comment.
Link: https://lore.kernel.org/20260708135359.122587-1-sj@kernel.org
Link: https://lore.kernel.org/20260708135359.122587-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The whole infrastructure is unused now. Let's remove the config option
along with mm/bootmem_info. + include/linux/bootmem_info.h.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-10-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
No longer required, so let's remove it.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-9-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We never set CONFIG_HAVE_BOOTMEM_INFO_NODE, so we can just switch to
free_reserved_page() and drop the register_page_bootmem_memmap() call.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-8-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
CONFIG_HAVE_BOOTMEM_INFO_NODE now essentially doesn't do anything.
So let's remove support for CONFIG_HAVE_BOOTMEM_INFO_NODE.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-7-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
There is no good reason to mark boot page tables as MIX_SECTION_INFO: we
only free boot page tables when they are completely empty, and memory
offlining/hotunplug doesn't benefit from it in any way.
So just stop marking page tables as MIX_SECTION_INFO. In
free_pagetable(), we can now simply free reserved pages directly.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-6-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We added the whole bootmem registration machinery in commit 04753278769f
("memory hotplug: register section/node id to free").
The main use case was to remember to which memory section memmap pages
belonged, so the memmap could be handled accordingly when freeing memory.
However, all that machinery is not required anymore: a memory section can
only get offlined if *all* pages can get offlined; and it can only get
unplugged once offline. If some of these pages are unmovable memmap
pages: bad luck, doesn't work. Offlining will fail.
Further, a lot of this machinery was required for pre-vmemmap support.
Now we only support the vmemmap with memory hotplug.
So the whole machinery is useless today. Let's start by removing the last
pieces by first stopping to mark vmemmap pages as SECTION_INFO. In
free_vmemmap_pages(), we can now always just free the reserved pages
directly.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-5-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
bootmem_type
As preparation for further changes, let's temporarily allow freeing pages
that were not previously registered.
This will allow freeing unregistered vmemmap pages allocated during boot
through free_bootmem_page() from hugetlb code, until we fully rip all of
that out.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-4-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Let's use our new generic helper.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-3-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Let's extend free_reserved_page() in page_alloc.c to
free_reserved_pages(), dropping the custom x86 variant. The common-code
variant will consume an order, so adjust the x86 callers accordingly.
Make free_reserved_pages() assume that we are freeing ordinary high-order
pages, just with the special "reserved" flavor. The target use case for
now is freeing vmemmap PMD pages.
Set the refcount directly to 0 (instead of 1) and call
__free_frozen_pages(). Set the page count to 0 before clearing
PG_reserved, so someone checking PG_reserved (and not finding it set) to
then try grabbing a ref would not suddenly have that ref be dropped. That
is arguably cleaner and safer than the old way of doing it.
Add some kerneldoc. Use a single adjust_managed_page_count() call.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-2-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)", v2.
Let's remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE,
performing some smaller cleanups around freeing of reserved vmemmap pages
on the way.
This patch (of 10):
All callers pass 0, so let's drop the parameter.
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-0-4afc76c73d61@kernel.org
Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-1-4afc76c73d61@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When vma_iter_prealloc() fails in do_mmap(), it jumps to error_just_free
as a error path of this function, but there are several possible issues.
1) It jumps to error_just_free without updating ret to -ENOMEM, meaning
do_mmap() will return 0 on failure.
2) The error path unconditionally frees the region struct. Since the
region was already added to the global nommu_region_tree via
add_nommu_region(), leaving it makes a potential dangling pointer in
the tree and may cause a use-after-free on the next tree walk.
3) If do_mmap() finds an existing overlapping shared region, it
increments its usage, sets region to this existing pregion, and jumps
to share:
region = pregion;
result = start;
goto share;
When vma_iter_prealloc() fails and jumps to error_just_free, the
error path unconditionally frees the region:
error:
...
if (region->vm_file)
fput(region->vm_file);
kmem_cache_free(vm_region_jar, region);
This potentially leaves a dangling pointer in nommu_region_tree and
causes RB-tree corruption.
4) When establishing a new private mapping, do_mmap_private() allocates
physical pages and assigns them to region->vm_start:
base = alloc_pages_exact(total << PAGE_SHIFT, GFP_KERNEL);
...
region->vm_start = (unsigned long) base;
If we later fail at vma_iter_prealloc() and jump to error_just_free,
the region struct is freed, but the backing physical memory isn't freed
via free_page_series().
5) In the error label of do_mmap(), the vm_area_struct allocated is
freed by vm_area_free(vma) but not called after vma_close(), leaving
potential memory leak which should be handled by a custom .close
handler of vm_ops.
This commit fixes those issues by introducing new jump label,
error_vma_iter_prealloc, to correctly handle the error case of
vma_iter_prealloc(), updating ret value (1), and move the region updates
after the place that the allocation is finished (2).
Additionally, the commit removes the existing goto label, error, and
consolidates to error_just_free as existing `goto error;` code blocks
always release nommu_region_sem.
Moreover, it only frees region allocated in this request to avoid
freeing the shared, existing region shared by other processes (3), and
free physical memory when do_mmap_private() allocates (4). It also add
vma_close() before vm_area_free() to fix the potential leak (5).
Those issues are discovered by Sashiko, linked below.
Link: https://lore.kernel.org/20260708083829.576036-1-thehajime@gmail.com
Link: https://sashiko.dev/#/patchset/20260702012830.667205-1-thehajime%40gmail.com
Link: https://sashiko.dev/#/patchset/c8513ee5aa8444ec9bf6c276043c9f833016a2fa.1783304131.git.thehajime%40gmail.com
Link: https://sashiko.dev/#/patchset/20260707235137.498738-1-thehajime%40gmail.com
Fixes: b5df09226450 ("mm: set up vma iterator for vma_iter_prealloc() calls")
Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add an admin-guide section covering UFFDIO_REGISTER_MODE_RWP:
- sync and async fault models;
- UFFDIO_RWPROTECT semantics;
- UFFD_FEATURE_RWP_ASYNC;
- UFFDIO_SET_MODE runtime mode flips.
It also covers typical VMM working-set-tracking workflow from detection
loop through sync-mode eviction and back to async.
Link: https://lore.kernel.org/20260708111417.173443-16-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: kernel test robot <lkp@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Coverage for UFFDIO_REGISTER_MODE_RWP and UFFDIO_RWPROTECT:
rwp-async async mode — touch pages, verify permissions are
auto-restored without a message
rwp-sync sync mode — access blocks, handler resolves via
UFFDIO_RWPROTECT
rwp-pagemap PAGEMAP_SCAN reports still-cold pages via
inverted PAGE_IS_ACCESSED
rwp-mprotect RWP survives mprotect(PROT_NONE) ->
mprotect(PROT_READ|PROT_WRITE) round-trip
rwp-gup GUP walks through a protnone RWP PTE (pipe
write/read drives the GUP path)
rwp-async-toggle UFFDIO_SET_MODE flips between sync and async
without re-registering
rwp-close closing the uffd restores page permissions
rwp-fork RWP survives fork() with EVENT_FORK; child's
PTEs keep the uffd bit
rwp-fork-pin RWP survives fork() on an RO-longterm-pinned
anon page (forces copy_present_page()); child
read auto-resolves and clears the bit, proving
PAGE_NONE was in place
rwp-wp-exclusive register with MODE_WP|MODE_RWP returns -EINVAL
All tests run against anon, shmem, shmem-private, hugetlb, and
hugetlb-private memory, except rwp-fork-pin which is anon-only —
copy_present_page() is the private-anon pinned-exclusive fork path.
Snapshot the RWP additions into tools/include/uapi/linux/userfaultfd.h so
the selftest builds without requiring "make headers" first, matching the
mechanism established by commit 580ea358af0a ("selftests/mm: fix
additional build errors for selftests").
Link: https://lore.kernel.org/ak-Z9KO2mP9HMOPW@thinkstation
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add an ioctl to toggle async mode at runtime without re-registering the
userfaultfd. This allows a VMM to switch between sync and async RWP modes
on-the-fly -- for example, starting in async mode for working set
scanning, then switching to sync mode to intercept faults during page
eviction.
UFFDIO_SET_MODE takes an enable/disable bitmask of UFFD_FEATURE_* flags.
Only UFFD_FEATURE_RWP_ASYNC is toggleable today; the ioctl rejects any
other bit with -EINVAL. Enabling RWP_ASYNC also requires RWP to have been
negotiated at UFFDIO_API time, mirroring the UFFDIO_API invariant.
Fault-path readers of ctx->features run under mmap_read_lock or a per-VMA
lock; the RMW takes mmap_write_lock and calls vma_start_write() on every
UFFD-armed VMA, so those readers are fully excluded.
userfaultfd_show_fdinfo(), however, reads ctx->features without any lock,
so the RMW is written as a single WRITE_ONCE and fdinfo reads it with
READ_ONCE. That keeps the lockless observer from seeing a mid-RMW
intermediate and removes the audit burden when new toggleable bits are
added later.
When switching to async, pending sync waiters are woken so they retry and
auto-resolve under the new mode.
Link: https://lore.kernel.org/20260708111417.173443-14-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Sync RWP delivers a message and blocks the faulting thread until the
handler resolves the fault. For working-set tracking the VMM does not
need the message: it just needs to know, at scan time, which pages were
touched. Async RWP serves that use case — the kernel restores access
in-place and the faulting thread continues without blocking.
The VMM reconstructs the access pattern after the fact via PAGEMAP_SCAN:
pages whose uffd bit is still set (inverted PAGE_IS_ACCESSED) were not
re-accessed since the last RWP cycle.
Worth calling out: async resolution upgrades writable private anon PTEs
via pte_mkwrite() when can_change_pte_writable() allows, mirroring
do_numa_page(). Without it, every re-access of an RWP'd writable page
would COW-fault a second time.
UFFD_FEATURE_RWP_ASYNC requires UFFD_FEATURE_RWP.
Link: https://lore.kernel.org/20260708111417.173443-13-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
PAGEMAP_SCAN already reports PAGE_IS_WRITTEN from the inverted uffd PTE
bit, targeting the UFFDIO_WRITEPROTECT workflow. UFFDIO_RWPROTECT reuses
the same PTE bit as a marker for read-write protection, but "has been
written" and "has been accessed" are distinct semantic signals — they
happen to share one PTE bit today only because the two implementations
share infrastructure.
Give RWP its own pagemap category so the UAPI does not conflate them:
PAGE_IS_WRITTEN reported on VM_UFFD_WP VMAs, !pte_uffd(pte)
PAGE_IS_ACCESSED reported on VM_UFFD_RWP VMAs, !pte_uffd(pte)
Both still read the same PTE bit today, but each is scoped to the VMA
whose registered mode makes the bit meaningful. If a future
implementation moves RWP to a separate PTE bit, only PAGE_IS_ACCESSED
switches over.
This is a UAPI narrowing. Outside VM_UFFD_WP VMAs the uffd bit is always
clear, so PAGEMAP_SCAN used to flag PAGE_IS_WRITTEN on every present PTE
there — a meaningless duplicate of PAGE_IS_PRESENT. Now PAGE_IS_WRITTEN
fires only inside VM_UFFD_WP VMAs.
pagemap_hugetlb_category() now takes the vma like its PTE/PMD peers.
Link: https://lore.kernel.org/20260708111417.173443-12-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Wire the fault side of read-write protection tracking and turn the
userspace interface on.
An RWP-protected PTE is PAGE_NONE with the uffd bit set. The PROT_NONE
triggers a fault on any access; the uffd bit distinguishes it from plain
mprotect(PROT_NONE) or NUMA hinting.
Fault dispatch, per level:
PTE handle_pte_fault() -> do_uffd_rwp()
PMD __handle_mm_fault() -> do_huge_pmd_uffd_rwp()
hugetlb hugetlb_fault() -> hugetlb_handle_userfault()
The RWP branches gate on userfaultfd_pte_rwp() /
userfaultfd_huge_pmd_rwp() (VM_UFFD_RWP plus the uffd bit) and fall
through to do_numa_page() / do_huge_pmd_numa_page() otherwise. Each
delivers a UFFD_PAGEFAULT_FLAG_RWP message through handle_userfault(); the
handler resolves it with UFFDIO_RWPROTECT clearing MODE_RWP.
userfaultfd_must_wait() and userfaultfd_huge_must_wait() add matching
protnone+uffd waiters so sync-mode fault handlers block correctly.
Expose the UAPI:
UFFDIO_REGISTER_MODE_RWP -> UFFD_API_REGISTER_MODES
UFFD_FEATURE_RWP -> UFFD_API_FEATURES
_UFFDIO_RWPROTECT -> UFFD_API_RANGE_IOCTLS
UFFD_API_RANGE_IOCTLS_BASIC
UFFD_FEATURE_RWP is masked out at UFFDIO_API time when PROT_NONE is not
available or VM_UFFD_RWP aliases VM_NONE (32-bit), so userspace never sees
an advertised-but-broken feature.
Works on anonymous, shmem, and hugetlb memory.
Link: https://lore.kernel.org/20260708111417.173443-11-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add the userspace interface for read-write protection tracking:
- UFFDIO_REGISTER_MODE_RWP register a range for RWP tracking
- UFFD_FEATURE_RWP capability bit
- UFFDIO_RWPROTECT install / remove RWP on a range
Introduce CONFIG_USERFAULTFD_RWP, auto-selected on 64-bit kernels with
ARCH_HAS_PTE_PROTNONE and HAVE_ARCH_USERFAULTFD_WP. The symbol gates
VM_UFFD_RWP (previously aliased to VM_NONE) and the smaps/trace-flag hooks
added in the preparatory patches; without it the UAPI bits added here have
nothing to drive and would be unreachable.
Registration sets VM_UFFD_RWP on the VMA. Combining MODE_WP with MODE_RWP
is rejected because both modes claim the uffd PTE bit.
UFFDIO_RWPROTECT is the bidirectional counterpart of
UFFDIO_WRITEPROTECT:
- MODE_RWP change_protection() with MM_CP_UFFD_RWP
installs PAGE_NONE and sets the uffd bit on
present PTEs
- !MODE_RWP change_protection() with MM_CP_UFFD_RWP_RESOLVE
restores vma->vm_page_prot and clears the bit
userfaultfd_clear_vma() runs the same resolve pass on unregister so RWP
state cannot outlive the uffd.
Re-registering a range must not drop a mode that installs per-PTE markers
(WP or RWP); doing so returns -EBUSY. This also closes a pre-existing
window where re-registering without MODE_WP would strand uffd-wp markers:
before, those caused extra write-faults but were otherwise benign; with
RWP preservation in place, a subsequent mprotect() on a VM_UFFD_RWP VMA
would silently promote the stale markers to RWP.
The feature is not yet advertised. UFFDIO_REGISTER_MODE_RWP,
UFFD_FEATURE_RWP, and _UFFDIO_RWPROTECT are intentionally absent from
UFFD_API_REGISTER_MODES, UFFD_API_FEATURES, and UFFD_API_RANGE_IOCTLS, so
UFFDIO_API masks them out and the register-mode validator rejects the bit.
The follow-up patch adds fault dispatch and exposes the UAPI.
Link: https://lore.kernel.org/20260708111417.173443-10-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Three mm paths outside the fault handler gate on the uffd PTE bit today:
khugepaged (skip collapse on ranges carrying markers), rmap (cap unmap
batching), and GUP (force a fault through gup_can_follow_protnone).
Extend each to treat VM_UFFD_RWP the same as VM_UFFD_WP; otherwise per-PTE
RWP state is silently destroyed or bypassed.
khugepaged: try_collapse_pte_mapped_thp() and
file_backed_vma_is_retractable() already refuse to collapse or retract
page tables on ranges carrying the uffd PTE bit. Broaden the VMA
predicate from userfaultfd_wp() to userfaultfd_protected() so VM_UFFD_RWP
ranges get the same protection. hpage_collapse_scan_pmd() needs no change
— its existing pte_uffd() check already catches an RWP PTE because it
carries the uffd bit.
rmap: folio_unmap_pte_batch() caps batching at 1 for VM_UFFD_RWP so the
restore path handles each PTE with its own marker.
GUP: gup_can_follow_protnone() forces a fault on VM_UFFD_RWP VMAs
regardless of FOLL_HONOR_NUMA_FAULT. RWP uses protnone as an
access-tracking marker, not for NUMA hinting, so any GUP — read or write
— must go through the userfaultfd fault path.
Link: https://lore.kernel.org/20260708111417.173443-9-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The uffd PTE bit must survive any kernel path that rewrites a PTE on a
VM_UFFD_RWP VMA, otherwise the marker that carries PAGE_NONE semantics is
silently dropped and the next access leaks past RWP tracking. Wire the
preservation through every path that rewrites a VM_UFFD_RWP PTE.
Swap and device-exclusive: do_swap_page(), restore_exclusive_pte(), and
unuse_pte() (swapoff()) re-apply PAGE_NONE when the swap PTE carries the
uffd bit and the VMA has VM_UFFD_RWP.
Migration: remove_migration_pte() and remove_migration_pmd() do the same
after the migration entry is replaced with a real PTE/PMD.
Fork: __copy_present_ptes(), copy_present_page(), copy_nonpresent_pte(),
copy_huge_pmd(), copy_huge_non_present_pmd(), and
copy_hugetlb_page_range() keep the uffd bit on the child when the
destination VMA has VM_UFFD_RWP, matching the existing VM_UFFD_WP
handling. Add VM_UFFD_RWP to VM_COPY_ON_FORK so the flag itself
propagates.
mprotect(): change_pte_range() and change_huge_pmd() restore PAGE_NONE
after pte_modify()/pmd_modify() have recomputed the base protection from a
(possibly user-changed) vm_page_prot. pte_modify() preserves _PAGE_UFFD,
so the bit stays; we just have to force PAGE_NONE back on top.
Link: https://lore.kernel.org/20260708111417.173443-8-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Preparatory patch. Add the change_protection() primitive that userfaultfd
RWP will use.
An RWP-protected PTE is PAGE_NONE with the uffd PTE bit set. The
PROT_NONE half makes the CPU fault on any access; the uffd bit
distinguishes an RWP fault from a plain mprotect(PROT_NONE) or NUMA
hinting fault. MM_CP_UFFD_WP and MM_CP_UFFD_RWP share the same PTE bit,
so the two cannot be used together on the same range.
Two new change_protection() flags:
MM_CP_UFFD_RWP install PAGE_NONE and set the uffd bit
MM_CP_UFFD_RWP_RESOLVE restore vma->vm_page_prot, clear the uffd bit
Both are wired through change_pte_range(), change_huge_pmd(), and
hugetlb_change_protection() so anon, shmem, THP, and hugetlb all share the
same semantics.
Link: https://lore.kernel.org/20260708111417.173443-7-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Preparatory patch for userfaultfd read-write protection (RWP). RWP
extends userfaultfd protection from plain write-protection (WP) to full
read-write protection: accesses to an RWP-protected range -- reads as well
as writes -- trap through userfaultfd.
Reserve VM_UFFD_RWP, add the userfaultfd_rwp() and userfaultfd_protected()
helpers, and wire up the smaps "ur" entry and the trace-flag table the
rest of the series will use. The flag is gated on CONFIG_USERFAULTFD_RWP,
which is introduced together with the UAPI in a later patch; until then
VM_UFFD_RWP aliases VM_NONE and every downstream check folds to dead code.
Nothing sets or queries the flag yet.
Link: https://lore.kernel.org/20260708111417.173443-6-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The uffd VMA-flag helpers read vma->vm_flags directly. Now that
config-gated per-mode masks exist, switch them to the vma_flags_t accessor
vma_test_any_mask(), which is the going-forward API and keeps a single
place (the VMA_UFFD_* masks) that knows which modes are available on the
current build.
No functional change: vma_flags_t is in union with vm_flags, so the same
bits are read, and the masks fold to the same code the open-coded vm_flags
tests produced -- verified identical on gcc and clang, 32- and 64-bit.
Link: https://lore.kernel.org/20260708111417.173443-5-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Suggested-by: Lorenzo Stoakes <ljs@kernel.org>
Assisted-by: Claude:claude-opus-4-8
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Userfaultfd RWP will reuse the uffd-wp PTE bit to mark access-tracking
PTEs, alongside the write-protected ones it already marks. The bit's
meaning now depends on the VMA flag (WP or RWP), not on its name.
Rename the kernel-internal names that describe the bit:
- pte/pmd/huge_pte accessors (and swap variants)
- pgtable_supports_uffd() capability query
- SCAN_PTE_UFFD khugepaged enum
The ftrace string emitted by mm_khugepaged_scan_pmd for this enum is
kept as "pte_uffd_wp" so existing trace-based tooling keeps matching.
Pure mechanical rename -- no behavior change.
Link: https://lore.kernel.org/20260708111417.173443-4-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The uffd-wp PTE bit is about to gain a second consumer: userfaultfd RWP
will use the same bit to mark access-tracking PTEs, distinct from
mprotect(PROT_NONE) or NUMA-hinting PTEs. WP vs RWP semantics come from
the VMA flag; the bit is just "uffd has claimed this entry." Drop the
"_wp" suffix from the arch-private bit macros so they reflect that.
x86: _PAGE_BIT_UFFD_WP -> _PAGE_BIT_UFFD
_PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
arm64: PTE_UFFD_WP -> PTE_UFFD
PTE_SWP_UFFD_WP -> PTE_SWP_UFFD
riscv: _PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
Pure mechanical rename -- no behavior change.
Link: https://lore.kernel.org/20260708111417.173443-3-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "userfaultfd: working set tracking for VM guest memory", v10.
This series adds userfaultfd support for tracking the working set of VM
guest memory, so a VMM can identify hot pages and reclaim cold ones to
tiered or remote storage.
This patch (of 15):
pte_protnone() and pmd_protnone() detect present-but-inaccessible page
table entries. This capability is useful beyond NUMA balancing -- for
example, userfaultfd working set tracking uses protnone PTEs to track page
access without unmapping pages.
Introduce CONFIG_ARCH_HAS_PTE_PROTNONE to decouple the protnone PTE
infrastructure from CONFIG_NUMA_BALANCING. The six architectures that
support protnone PTEs (x86_64, arm64, powerpc, s390, riscv, loongarch) now
select this option, and CONFIG_NUMA_BALANCING depends on it.
No functional change -- the same set of architectures continues to have
working protnone support, but the infrastructure is now available
independently of NUMA balancing.
Link: https://lore.kernel.org/20260708111417.173443-1-kirill@shutemov.name
Link: https://lore.kernel.org/20260708111417.173443-2-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Acked-by: SeongJae Park <sj@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
shmem_getattr() reads info->alloced, info->swapped and i_mapping->nrpages
without info->lock to decide whether shmem_recalc_inode() should be
called. shmem_recalc_inode() recomputes these under info->lock, so a
stale read here only affects whether shmem_recalc_inode() runs.
Annotate data_race() to silence KCSAN.
Link: https://lore.kernel.org/20260708111941.35460-1-jaeyeon.lee.dev@gmail.com
Signed-off-by: Jaeyeon Lee <jaeyeon.lee.dev@gmail.com>
Reported-by: syzbot+dfb578404df369f6599b@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=dfb578404df369f6599b
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Marco Elver <elver@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Keep include/linux/swap.h focused on swap-facing interfaces by moving
MM-internal reclaim and workingset declarations into mm/internal.h.
Leave the small set of LRU helper declarations that are used outside mm/
in swap.h so this cleanup does not need a new public header under
include/linux/.
Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-3-2bc72b4f8730@gmail.com
Signed-off-by: Jianyue Wu <wujianyue000@gmail.com>
Suggested-by: Barry Song <baohua@kernel.org>
Suggested-by: Baoquan He <bhe@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
