| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
Current implementation isn't working well when device tree nodes have a
phandle on a fixed-layout nvmem node. As the fixed layout is handled in
nvmem core, no driver is ever associated with the layout, and the device
consumer driver probe is deferred indefinitely.
Remove the specific handling of fixed-layout and add a layout driver.
This makes the fixed-layout similar to all other layouts, fixing the
whole issue.
Fixes: fc29fd821d9a ("nvmem: core: Rework layouts to become regular devices")
Cc: stable@vger.kernel.org
Reviewed-by: Miquel Raynal <miquel.raynal@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Srinivas Kandagatla <srini@kernel.org>
Link: https://patch.msgid.link/20260724223404.629248-3-srini@kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
|
|
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a
macro that only gets defined by the Makefiles in the kernel.
This can be very confusing when switching between userspace
and kernelspace coding, or when dealing with uapi headers that
rather should use __ASSEMBLER__ instead. So let's standardize now
on the __ASSEMBLER__ macro that is provided by the compilers.
This is a completely mechanical patch (done with a simple "sed -i"
statement).
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260619124936.208519-1-thuth@redhat.com
|
|
The only time that 'false' is passed as the 'excl' arg to the ->create
inode_operation is in lookup_open() when ->atomic_open is not provided
by the parent directory.
*all* directory inode_operations which do not have ->atomic_open
completely ignore the 'excl' arg.
Therefore we don't need the 'excl' arg. Those few ->create operations
which pay attention to the arg are only ever called with a value of
'true'.
We remove that arg and change all ->create operations to behave as those
thhe arg were 'true'.
Signed-off-by: NeilBrown <neil@brown.name>
Link: https://patch.msgid.link/178290671516.27465.15984496764174914338@noble.neil.brown.name
Reviewed-by: Jori Koolstra <jkoolstra@xs4all.nl>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The symbol mangle_path conflicts with a gcov symbol which
can break the build of ARCH=um with gcov, and it's also
not very specific and descriptive.
Rename mangle_path() to seq_mangle_path(), and also remove
the export since it's not needed or used by any modules.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Alex Hung <alex.hung@amd.com>
Link: https://patch.msgid.link/20260727195730.2306887-1-alex.hung@amd.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
1. put the dead val into a macro so that it can be used in other places
2. __lockref_is_dead():
- drop the __ suffix, this is not an internal routine
- drop the spurious cast, the value is already a signed int
- use READ_ONCE to prevent any compile shenanigans
3. provide lockref_is_dead_or_zero()
Signed-off-by: Mateusz Guzik <mjguzik@gmail.com>
Link: https://patch.msgid.link/20260724171422.429284-2-mjguzik@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Pipes keep two separate page caches:
a) The per-pipe, lock-protected tmp_page[2]
b) An on-stack anon_pipe_prealloc burst pool of up to eight pages
filled before the lock
Converge them into a single per-pipe pool (struct anon_pipe_prealloc
embedded in pipe_inode_info) with the same budget as before: up to
PIPE_PREALLOC_MAX (8) pages, trimmed back to PIPE_PREALLOC_KEEP (2)
after each operation. tmp_page[2] is removed.
Pages are still allocated and freed outside pipe->mutex; only the
assignment into the pool is done under it. The pool count is also read
locklessly in the prefill path, so it is annotated __data_racy.
anon_pipe_prefill_and_lock() tops the pool up to the write's page count
-- and returns with pipe->mutex held, so a write acquires the lock only
once.
anon_pipe_trim_and_unlock() trims the pool under that same lock before
dropping it, then frees the excess.
Signed-off-by: Breno Leitao <leitao@debian.org>
Link: https://patch.msgid.link/20260720-b4-pipe-unification-v5-1-9002a3fe5e6d@debian.org
Reviewed-by: Mateusz Guzik <mjguzik@gmail.com>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
syzbot reported a KCSAN race between fill_mg_cmtime() and
inode_set_ctime_to_ts() on inode->i_ctime_{sec,nsec}.
stat/getattr can sample inode timestamps while update paths store new
values concurrently, so KCSAN can report benign races on these fields.
Annotate the timestamp accessors with READ_ONCE()/WRITE_ONCE(), and use
the ctime accessor for the remaining ctime loads. This avoids the KCSAN
reports without changing timestamp semantics.
Fixes: 4e40eff0b573 ("fs: add infrastructure for multigrain timestamps")
Reported-by: syzbot+8b3bd9f8a06658479d4a@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8b3bd9f8a06658479d4a
Signed-off-by: Yu Peng <pengyu@kylinos.cn>
Link: https://patch.msgid.link/20260708080232.2564807-1-pengyu@kylinos.cn
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Add support for devices using SETAASA, such as SPD5118 and SPD5108
attached to DDR5 memory modules that do not support ENTDAA. Follow the
guidelines proposed by the MIPI Discovery and Configuration
Specification [1] for discovering such devices.
SETAASA (Set All Addresses to Static Address) differs from standard I3C
address assignment that uses ENTDAA or SETDASA to assign dynamic
addresses. Devices using SETAASA assign their pre-defined static addresses
as their dynamic addresses during DAA, and it is not mandatory for these
devices to implement standard CCC commands like GETPID, GETDCR, or GETBCR.
For such devices, it is generally recommended to issue SETHID (specified
by JEDEC JESD300) as a prerequisite for SETAASA to stop HID bit flipping.
[1] https://www.mipi.org/mipi-disco-for-i3c-download
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Link: https://www.mipi.org/mipi-disco-for-i3c-download
Link: https://patch.msgid.link/20260728065955.809445-5-akhilrajeev@nvidia.com
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Replace all OF-specific functions with unified device property functions
as a prerequisite to support both ACPI and device tree.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Akhil R <akhilrajeev@nvidia.com>
Link: https://patch.msgid.link/20260728065955.809445-3-akhilrajeev@nvidia.com
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Add optional_bytes to struct i3c_ccc_cmd_payload so callers describe
variable-length GET CCC responses. GETMRL and GETMXDS set optional_bytes
at the call site. Extend i3c_ccc_validate_payload_len() to honour it.
Signed-off-by: Adrian Ng Ho Yin <adrian.ho.yin.ng@altera.com>
Signed-off-by: Tze Yee Ng <tze.yee.ng@altera.com>
Reviewed-by: Alexandre Mergnat <amergnat@baylibre.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Tested-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com> # on RZ/G3S
Link: https://patch.msgid.link/2e07dc944eab1c4358be1da87fa5000e711ac8bd.1783493868.git.tze.yee.ng@altera.com
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Add retries to struct i3c_ccc_cmd. Validate GET payload length in
i3c_master_send_ccc_cmd_locked() after a successful transfer.
Retry failed Direct GET CCCs up to cmd->retries times when the driver
reports failure or an I3C error; validation failures are not retried.
SET CCCs are not retried by default.
Signed-off-by: Adrian Ng Ho Yin <adrian.ho.yin.ng@altera.com>
Signed-off-by: Tze Yee Ng <tze.yee.ng@altera.com>
Reviewed-by: Alexandre Mergnat <amergnat@baylibre.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Tested-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com> # on RZ/G3S
Link: https://patch.msgid.link/b467f01edfaaa0710f30e719ce7f2753b06c1a3f.1783493868.git.tze.yee.ng@altera.com
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Add actual_len to struct i3c_ccc_cmd_payload so drivers can report how
many bytes were received on a GET CCC without overwriting the requested
buffer length in len.
Signed-off-by: Adrian Ng Ho Yin <adrian.ho.yin.ng@altera.com>
Signed-off-by: Tze Yee Ng <tze.yee.ng@altera.com>
Reviewed-by: Alexandre Mergnat <amergnat@baylibre.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Tommaso Merciai <tommaso.merciai.xr@bp.renesas.com>
Tested-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com> # on RZ/G3S
Link: https://patch.msgid.link/e452777c3a9be734a97e20b9822d8a4264ceadba.1783493868.git.tze.yee.ng@altera.com
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
SWIOTLB_FORCE has no remaining in-tree users. Forced bouncing is now
controlled through the swiotlb=force command line option via
swiotlb_force_bounce.
Remove the unused flag and simplify the force_bounce initialization.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-24-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
swiotlb_dyn_free() is used after removing a dynamic swiotlb pool from
RCU-protected lists. It can call swiotlb_free_tlb(), which may need to
restore the encryption state of an unencrypted pool with
set_memory_encrypted() before freeing the pages.
RCU callbacks run in atomic context, but set_memory_encrypted() is not
guaranteed to be atomic-safe on all architectures. For example, page
attribute updates may allocate page tables or take sleeping locks.
Use queue_rcu_work() for dynamic pool freeing instead. This keeps the RCU
grace period before freeing a published pool, while running the actual pool
teardown from workqueue context. Use the same helper for the transient-pool
error path, since that path may also be reached from atomic DMA mapping
context.
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-22-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability
check can reject encrypted DMA addresses for devices that require
unencrypted/shared DMA.
Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected
SWIOTLB pool is decrypted so the capability check sees the correct DMA
address attribute.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-16-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Teach swiotlb to distinguish between encrypted and decrypted bounce
buffer pools, and make allocation and mapping paths select a pool whose
state matches the requested DMA attributes.
Add a cc_shared flag to io_tlb_mem, initialize it for the default and
restricted pools, and propagate __DMA_ATTR_ALLOC_CC_SHARED into swiotlb
pool allocation. Reject swiotlb alloc/map requests when the selected pool
does not match the required encrypted/decrypted state.
Also return DMA addresses with the matching phys_to_dma_{encrypted,
unencrypted} helper so the DMA address encoding stays consistent with the
chosen pool.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-14-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Change swiotlb_tbl_map_single() to take the DMA mapping attributes by
reference and update the direct callers accordingly.
This is a preparatory change for a follow-up patch which updates the
attributes based on the selected swiotlb pool. Keeping the signature change
separate makes the follow-up patch easier to review.
No functional change in this patch.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Petr Tesarik <ptesarik@suse.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-13-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Teach the atomic DMA pool code to distinguish between encrypted and
unencrypted pools, and make pool allocation select the matching pool based
on DMA attributes.
Introduce a dma_gen_pool wrapper that records whether a pool is
unencrypted, initialize that state when the atomic pools are created, and
use it when expanding and resizing the pools. Update dma_alloc_from_pool()
to take attrs and skip pools whose encrypted state does not match
__DMA_ATTR_ALLOC_CC_SHARED. Update dma_free_from_pool() accordingly.
Also pass __DMA_ATTR_ALLOC_CC_SHARED from the swiotlb atomic allocation
path so decrypted swiotlb allocations are taken from the correct atomic
pool.
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-12-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
DMA_ATTR_CC_SHARED describes an existing DMA mapping whose backing memory
is already shared, or decrypted, for confidential computing. It is a
mapping attribute: callers use it to request a shared DMA address encoding
for memory that has already been prepared for shared DMA.
Allocation paths need a related but different state. Once the DMA core
decides that an allocation must use shared backing pages, the lower-level
allocation helpers need to select shared pools, decrypt newly allocated
pages, derive the DMA address with the shared-memory translation and
restore encryption on free. That state is internal to the DMA-mapping
implementation and should not be passed by drivers to dma_alloc_attrs().
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-10-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in
dma_direct_alloc() / dma_direct_alloc_pages().
This is needed for follow-up changes that simplify the handling of
memory encryption/decryption based on the DMA attribute flags.
swiotlb backing pages are already mapped decrypted by
swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so
dma-direct should not call dma_set_decrypted() on allocation nor
dma_set_encrypted() on free for swiotlb-backed memory.
Update alloc/free paths to detect swiotlb-backed pages and skip
encrypt/decrypt transitions for those paths. Keep the existing highmem
rejection in dma_direct_alloc_pages() for swiotlb allocations.
Only for "restricted-dma-pool", we currently set `for_alloc = true`, while
rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is
typically used together with "shared-dma-pool", where the shared region is
accessed after remap/ioremap and the returned address is suitable for
decrypted memory access. So existing code paths remain valid.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Jiri Pirko <jiri@nvidia.com>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Link: https://lore.kernel.org/r/20260717180442.110954-8-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
dma_direct_alloc_pages() may satisfy atomic allocations from the coherent
atomic pools. The pool allocation is keyed by the virtual address stored in
the gen_pool, but the pages API returns only the backing struct page.
On architectures with CONFIG_DMA_DIRECT_REMAP, atomic pool chunks are added
to the gen_pool using their remapped virtual address.
dma_direct_free_pages() reconstructs a linear-map address with
page_address(page) and passes that to dma_free_from_pool(). That address
does not match the gen_pool virtual range, so the pool lookup can fail and
the code can fall through to freeing a pool-owned page through the normal
page allocator path.
Add a page-based pool free helper that looks up the owning pool chunk by
physical address, translates it back to the gen_pool virtual address, and
frees that address to the pool. Use it from dma_direct_free_pages() while
keeping the existing virtual-address helper for coherent allocation frees.
Tested-by: Michael Kelley <mhklinux@outlook.com>
Tested-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260717180442.110954-5-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Qualcomm remoteproc drivers such as qcom_q6v5_mss, which do not use the
Peripheral Authentication Service (PAS), always map the MBA region before
use and unmap it once the usage is complete. This behavior was introduced
to avoid issues seen in the past where speculative accesses from the
application processor to the MBA region after it was assigned to the remote
Q6 led to an XPU violation. The issue was mitigated by unmapping the region
before handing control to the remote Q6.
Currently, most Qualcomm SoCs using the PAS driver run either with a
standalone QHEE or the Gunyah hypervisor. In these environments, the
hypervisor unmaps the Q6 memory from HLOS Stage-2 and remaps it into the
Q6 Stage-2 page table. As a result, speculative accesses from HLOS cannot
reach the region even if it remains mapped in HLOS Stage-1; therefore, XPU
violations cannot occur.
However, when the same SoC runs Linux at EL2, Linux itself must perform the
unmapping to avoid such issues. It is still correct to apply this mapping/
unmapping sequence even for SoCs that run under Gunyah, so this behavior
should not be conditional.
Introduce qcom_pas_ctx_map() in qcom_pas.h to centralise the ioremap_wc
pattern used by both qcom_q6v5_pas and qcom_mdt_pas_load, and use it in
both places.
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Mukesh Ojha <mukesh.ojha@oss.qualcomm.com>
Link: https://lore.kernel.org/r/20260724182858.1868271-5-mukesh.ojha@oss.qualcomm.com
Signed-off-by: Bjorn Andersson <andersson@kernel.org>
|
|
The kernel doc comment of damon_region clearly specifies every region
should have positive size. But it is unclear who should verify it.
damon_set_regions() is the recommended DAMON core function for setting
regions from the callers, and has the verification. Update the comment to
clarify the callers should be ok to pass any values for region addresses,
as long as they use damon_set_regions().
Link: https://lore.kernel.org/20260705155600.96555-7-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "Fix incorrect access of hugetlb pte entries", v3.
There are various places which use ptep_get() to get the pte entry
corresponding to a hugetlb folio. Some arches (like s390) have special
handling to compute the pteval, so they provide huge_ptep_get(). Use this
helper consistently.
Additionally, some code paths may provide huge_ptep_get with an unaligned
address. This is a problem on arm64 (I checked other arches and it looks
fine for them), which is fixed in patch 1. The fix is made to be
backport-friendly: the cleaner fix would be to perhaps pass the hstate to
huge_ptep_get() - that is wider churn and we can do that later.
This patch (of 5):
try_to_unmap_one() handles hugetlb folios when memory failure needs to
replace a poisoned hugetlb mapping with a hwpoison entry. In that case
page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in
pvmw.pte, but the code reads it with ptep_get().
On arches which provide their own huge_ptep_get() to dereference a huge
pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present()
etc to misbehave.
It is not clear whether this has a trivially visible effect to userspace.
Just use huge_ptep_get() for dereferencing a huge pte pointer.
Link: https://lore.kernel.org/20260703114202.365553-1-dev.jain@arm.com
Link: https://lore.kernel.org/20260703114202.365553-3-dev.jain@arm.com
Fixes: c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
Signed-off-by: Dev Jain <dev.jain@arm.com>
Reported-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Jann Horn <jannh@google.com>
Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Jun'ichi "Nick" Nomura <j-nomura@ce.jp.nec.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
All users of the flag are converted to SLAB_ALLOC_NO_RECURSE or
ALLOC_NO_CODETAG (from __GFP_NO_CODETAG which reused the NO_OBJ_EXT bit).
Free up the flag bit.
[Rebased onto __GFP_NO_CODETAG removal]
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-16-c87b714e19d3@google.com
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Now that alloc_pages has an entrypoint that allows passing alloc_flags, we
can take advantage of this to start removing GFP flags that are only used
for mm-internal stuff.
This requires also plumbing the alloc_flags into some more of the
allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags
arg to go along with its callees, and we now need to pass those flags
deeper into the allocator so they can reach the alloc_tag code.
While moving the flag definition into page_alloc.h, also update the
comment per Hao's suggestion.
No functional change intended.
Link: https://lore.kernel.org/all/b4916118-3537-4e19-8bc8-1d103dd0d225@linux.dev/
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-15-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
It's no longer used outside of mm/.
Since this means __alloc_pages_noprof() is no longer visible from gfp.h,
this also means moving the definition of alloc_pages_node_noprof into
the .c file.
Also remove references to this API from the documentation tree -
referring to the specific function name was already questionable but
now the function is not even public it definitely seems wrong.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-14-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
There were only a few users, which have been removed. The only advantage
of this API over alloc_pages_node() is avoiding a single conditional
branch. The disadvantages are:
1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of __alloc_pages_node()
were using the result of cpu_to_node(); if the CPU gets hotplugged
out this will return NUMA_NO_NODE. If one of these paths fails to
protect against a concurrent hotplug then page_alloc.c will use
NUMA_NO_NODE as an index into NODE_DATA() and cause some horrible
memory corruption or other. With alloc_pages_node(), the code might
just work fine.
Ulterior motive: this frees up the __* variants of the allocator APIs to
serve specifically for use as mm-internal API.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-13-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Some of this stuff in the public header is only used internally so shrink
the scope to avoid silently growing new users.
drain_local_pages() is still used from kernel/power/snapshot.c so that
needs to stay behind.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-7-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
It's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of alloc_flags.
- gfp_to_alloc_flags() sounds generic but it only influences the
slowpath.
Rename some variables to highlight which alloc_flags are
fastpath-specific. Rename gfp_to_alloc_flags() to highlight that it's
slowpath-specific.
gfp_to_alloc_flags_cma() and gfp_to_alloc_flags_nonblocking() currently
have perfectly harmless names, but to keep the naming consistent also
rename those to the alloc_flags_*() pattern (which already exists for
alloc_flags_nofragment()).
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-2-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: JP Kobryn <jp.kobryn@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Implement a function for getting a reasonable best effort quality pseudo
moving sums of probe_hits on demands. It reuses the internal function for
the pseudo moving sum for data access frequency (nr_accesses).
Link: https://lore.kernel.org/20260703170605.94472-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: provide pseudo moving sum probe_hits".
Data attribute counters (probe_hits) of DAMON are managed in the classical
way. The counter value is accumulated every sampling interval, gets the
complete view at the end of the aggregation interval, and is reset when
the next aggregation interval starts. Hence, the complete view can be
retrieved only once per aggregation interval, which can be quite long.
With the suggested intervals autotuning setup, it becomes 2-4 seconds in
common real production systems. It can span up to 200 seconds in theory.
This will restrict online monitoring use case of DAMON.
Actually DAMON is already providing online monitoring of probe_hits.
DAMON sysfs interface exposes the values via schemes tried regions
directory files. However, due to the above mentioned limitation, it
usually shows only partially accumulated hit counters and therefore not
useful.
DAMOS is not using probe_hits at the moment. In the future, using it can
further strengthen DAMOS. However, a recommended setup of DAMOS is
utilizing sampling/aggregation intervals auto-tuning, and having its own
DAMOS apply_interval (1 second is mostly recommended). In the setup,
DAMOS will nearly always show incompletely accumulated probe_hits, which
will not really be useful.
Data frequency counter (nr_accesses) of DAMON solves this problem using
the pseudo moving sum value. The infrastructure is not limited to
nr_accesses but general sampling based counters. Maintain and provide the
pseudo moving sum of probe_hits similar to nr_accesses, using the
infrastructure.
Tests
=====
On an idle system, I ran DAMON with an attribute probe filter for
non-anonymous page, using DAMON user-space tool, damo [1], like below.
$ sudo ./damo start --probe_filter allow non anon
Because the system is idle, nearly all memory is not an anonymous page but
a free page, so the probe_hits are expected to be nearly always full. In
this setup, since the sampling interval is 5ms and the aggregation
interval is 100ms, the counter value is expected to always be near 20.
On kernels not having this series, if we retrieve the probe hits in an
arbitrary time that is likely not aligned to the aggregation interval, the
values are usually much lower than the expectation like below. This is
because the tool is showing the incompletely aggregated values.
$ sudo ./damo report access --format append region "probe_hits: <probe hits>"
heatmap: 00000000000000000000000000000000000000008999999711111111000000000000000000000000
# min/max temperatures: -1,630,000,000, 0, column size: 99.800 MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
0 addr 4.000 KiB size 3.898 GiB access 0 hz age 16.300 s probe_hits: 11
1 addr 3.898 GiB size 77.859 MiB access 0 hz age 1.500 s probe_hits: 11
2 addr 3.974 GiB size 700.770 MiB access 0 hz age 0 ns probe_hits: 11
3 addr 4.659 GiB size 791.078 MiB access 0 hz age 13.700 s probe_hits: 11
4 addr 5.431 GiB size 1.472 GiB access 0 hz age 15.800 s probe_hits: 11
5 addr 6.903 GiB size 915.059 MiB access 0 hz age 15.300 s probe_hits: 11
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
After applying this series, I was able to reliably show the expected
results like below.
$ sudo ./damo report access --format append region "probe_hits: <probe hits>"
heatmap: 00000000333333330000000166666665111111139999999855555555333333333333333444444444
intervals: sample 5 ms aggr 100 ms (max access hz 200)
0 addr 4.000 KiB size 790.496 MiB access 0 hz age 1 m 33.300 s probe_hits: 20
1 addr 790.500 MiB size 791.160 MiB access 0 hz age 1 m 15.400 s probe_hits: 19
2 addr 1.545 GiB size 792.316 MiB access 0 hz age 1 m 32.400 s probe_hits: 19
3 addr 2.318 GiB size 795.465 MiB access 0 hz age 1 m 2.600 s probe_hits: 19
4 addr 3.095 GiB size 797.102 MiB access 0 hz age 1 m 23.500 s probe_hits: 20
5 addr 3.874 GiB size 797.293 MiB access 0 hz age 47.900 s probe_hits: 20
6 addr 4.652 GiB size 787.516 MiB access 0 hz age 1 m 3.800 s probe_hits: 20
7 addr 5.421 GiB size 784.461 MiB access 0 hz age 1 m 14.400 s probe_hits: 19
8 addr 6.187 GiB size 795.621 MiB access 0 hz age 1 m 15.700 s probe_hits: 20
9 addr 6.964 GiB size 798.000 MiB access 0 hz age 1 m 10.200 s probe_hits: 20
10 addr 7.744 GiB size 54.566 MiB access 0 hz age 1 m 9.300 s probe_hits: 20
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
FYI, 'damo report access' output format has changed on v3.3.0. Above
outputs can be reproduced on <3.3.0 versions of damo.
Patches Sequence
================
Patch 1 adds probe_hits counters for values that fully accumulated in the
last aggregation interval. This is required for using the moving sum
infrastructure. Patch 2 introduces a function for getting the moving sum
values on demand, using the infrastructure. Finally, patch 3 updates the
DAMON sysfs interface to expose the moving sum values to the schemes tried
regions directory.
This patch (of 3):
Add new damon_region filed, last_probe_hits. Maintain fully accumulated
probe_hits values from the last aggregation interval in the field.
Link: https://lore.kernel.org/20260703170605.94472-1-sj@kernel.org
Link: https://lore.kernel.org/20260703170605.94472-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Refactor out hugetlb_alloc_folio() from alloc_hugetlb_folio(), which
handles allocation of a folio and memory and HugeTLB charging to cgroups.
This refactoring decouples the HugeTLB page allocation from VMAs,
specifically:
1. Reservations (as in resv_map) are stored in the vma
2. mpol is stored at vma->vm_policy
3. A vma must be used for allocation even if the pages are not meant to be
used by host process.
Without this coupling, VMAs are no longer a requirement for allocation.
This opens up the allocation routine for usage without VMAs, which will
allow guest_memfd to use HugeTLB as a more generic allocator of huge
pages, since guest_memfd memory may not have any associated VMAs by
design. In addition, direct allocations from HugeTLB could possibly be
refactored to avoid the use of a pseudo-VMA.
Also, this decouples HugeTLB page allocation from HugeTLBfs, where the
subpool is stored at the fs mount. This is also a requirement for
guest_memfd, where the plan is to have a subpool created per-fd and stored
on the inode.
Provide and use alloc_flags to allow more allocation knobs in future
without expanding the number of parameters in hugetlb_alloc_folio().
No functional change intended.
Link: https://lore.kernel.org/20260702-hugetlb-open-up-v4-6-d53cefcccf34@google.com
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: James Houghton <jthoughton@google.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiaqi Yan <jiaqiyan@google.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Michael Roth <michael.roth@amd.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Pratyush Yadav <pratyush@kernel.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Shivank Garg <shivankg@amd.com>
Cc: Vishal Annapurve <vannapurve@google.com>
Cc: Yan Zhao <yan.y.zhao@intel.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
No code touches damon_region->nr_accesses_bp field. Remove it.
Link: https://lore.kernel.org/20260630040812.149729-19-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_update_region_access_rate() is not using attrs parameter. Remove
it.
Link: https://lore.kernel.org/20260630040812.149729-15-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: optimize out nr_accesses_bp".
TLDR: Replace damon_region->nr_accesses_bp, which is easy to be wrong,
with a simpler on-demand moving sum function, damon_nr_accesses_mvsum().
Background
==========
DAMON's monitoring output (access pattern snapshot, or more technically
speaking, damon_region->nr_accesses) is completed once per aggregation
interval, which is 100 ms by default. Users can arbitrarily increase the
interval for demand. Under the suggested intervals auto-tuning setup, it
can span up to 200 seconds. If the aggregation interval is too long, the
snapshot users cannot use it in reasonable time. To mitigate this, we
introduced a new field of damon_region, namely nr_accesses_bp. It
contains a pseudo moving sum of nr_accesses in bp units and is updated for
each sampling interval.
It turned out keeping it correctly updated every sampling interval is not
that easy. From online parameter update feature development and more
experimental hacks, we found it is easy to be corrupted. Once it is
corrupted, DAMON's monitoring outputs become quite insane. Hence we added
a few validation checks. It is easy to be corrupted because it requires
every update per sampling interval to be correct.
Solution
========
There is no real reason to keep it updated every sampling interval. Due
to the simple pseudo-moving sum mechanism and existing helper field
(last_nr_accesses), we can also calculate the pseudo moving sum on demand
in a much simpler way.
Implement a function for getting the pseudo moving sum on demand, and
replace nr_accessses_bp uses with the new function. Also remove no more
needed tests for nr_accesses_bp and the per-sampling interval update
functions. Finally, remove the nr_accesses_bp. The new function is quite
simple.
Discussion
==========
Depending on the use case, multiple nr_accesses readers could be executed
in the same kdamond_fn() main loop iteration, which is executed once per
sampling interval. Such readers include DAMON region exporting
tracepoints (damon_[region_]aggregated and damos_before_apply), DAMOS, and
DAMON sysfs interface logic for update_schemes_tried_regions command. In
this case, the new function will be called multiple times and this could
be overhead compared to the old logic, which simply reads the field
without any additional work. Nonetheless, the new function is quite
simple. And the new approach does nothing while there is no need to read.
The old approach had to execute its update function for each region for
every sampling interval. Hence the new approach is believed to be even
more lightweight in common case, and the overhead is anyway negligible.
One more advantage of this change is that one field from the damon_region
struct is removed. On setups that uses a high number of DAMON regions,
this could be a potential memory space benefit.
Patches Sequence
================
Patch 1 introduces the new function for getting the pseudo moving sum of
nr_accesses on demands. Patch 2 implements a unit test for the new
function's internal logic. Patch 3 and 4 update monitoring logic and the
new function to ready for safe use on the existing logic. Patches 5-7
replace uses of nr_accesses_bp in DAMOS, tracepoints and DAMON sysfs
interface with the new function, respectively. Patches 8-10 removes
nr_accesses_bp validation functions in DAMON core, one by one. Patches 11
and 12 further remove tests and test helper for nr_accesses_bp,
respectively. Patches 13 removes the setups and updates or nr_accesses_bp
field. Patches 14-16 cleans up function parameters that are no more being
used due to the previous patch. Patch 17 removes the function that was
used for updating nr_accesses_bp field with its unit test, which is the
single remaining caller of the function. Finally, patch 18 removes
damon_region->nr_accesses_bp field.
This patch (of 18):
Introduce a new DAMON core function, damon_nr_accesses_mvsum(). It
returns a pseudo moving sum value of a given region's nr_accesses for the
last aggregation interval. The internal logic is the same to
nr_accesses_bp. The difference is that nr_accesses_bp is updated for each
sampling interval, while the new function needs to be executed only when
requested. Hence the function's return value is the same as the value of
nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-1-sj@kernel.org
Link: https://lore.kernel.org/20260630040812.149729-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Gregory points out that these descriptions are cursed and confusing,
considering what these flags actually do. This is mostly due to historic
implementation choices and cgroup1 baggage. Improve the description of
their actual effects.
Link: https://lore.kernel.org/20260701182102.1586784-1-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Acked-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We miss a failed allocation check for pgdat->per_cpu_nodestats, which
results in a NULL deref when we offset into the per-cpu area.
Propagate -ENOMEM up the stack and leave per_cpu_nodestats pointing at
boot_nodestats so a later online can retry the allocation.
hotadd_init_pgdat() returns NULL on failure, which __try_online_node()
already maps to -ENOMEM.
On failure nothing needs to be unwound:
- the node is never marked online
- per_cpu_nodestats is left pointing at boot_nodestats
- __add_memory_resource() cleans up pending memblock resources
- later online attempts retry the per_cpu_nodestats allocation
Link: https://lore.kernel.org/20260701221613.2818148-1-gourry@gourry.net
Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
Signed-off-by: Gregory Price <gourry@gourry.net>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260627202243.758289-1-gourry%40gourry.net
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
KCSAN reports a data race between page_to_nid()/folio_pgdat() reading
page->flags and folio_trylock()/folio_lock() concurrently doing
test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:
BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp
The race is benign: nid/zone bits are set once at page init and never
overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside
memdesc_nid/zonenum() was checking a by-value copy of the flags word, not
the live page->flags, so it failed to annotate the real access.
Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and
memdesc_is_zone_device() to take a const memdesc_flags_t * and update all
callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS()
operates on the actual shared word.
Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under
ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone
field is absent. memdesc_section() needs no such guard, since
SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined.
Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead
of reading page->flags, since NODES_MASK is 0 and the check can never
fire; page_to_nid()/folio_nid() now just call memdesc_nid()
unconditionally and rely on that stub, instead of duplicating the
CONFIG_NUMA split at each call site.
[zhuhui@kylinos.cn: v11]
Link: https://lore.kernel.org/20260708083308.747930-1-hui.zhu@linux.dev
[ziy@nvidia.com: build fix]
Link: https://lore.kernel.org/DJSGLP9VFLRI.355AAWR2VHIEL@nvidia.com
Link: https://lore.kernel.org/20260630070810.470763-1-hui.zhu@linux.dev
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Clean up mm/vmpressure.c by separating the cgroup v1 userspace eventfd
interface from the shared and v2 in-kernel code.
Currently, almost half of mm/vmpressure.c exists to serve tree=true:
struct vmpressure_event, the events list and its mutex, the work_struct
and vmpressure_work_fn that drains tree_scanned/tree_reclaimed, the parent
walk, vmpressure_event(), vmpressure_register_event(),
vmpressure_unregister_event(), and vmpressure_prio() (which always calls
vmpressure() with tree=true).
Move it all into mm/memcontrol-v1.c (built only when CONFIG_MEMCG_V1=y) as
a single contiguous block, following the per-component layout already used
by that file. Keeping the v1 vmpressure code with the rest of the
deprecated cgroup v1 memory controller makes the full footprint of the
CONFIG_MEMCG_V1 option easy to see in one place, which matters more than
component-level file separation for code that has no active development.
vmpressure.c keeps the shared bits (constants, vmpressure_calc_level, the
runtime hierarchy check, the tree=false body, init/cleanup plumbing) and
calls into three small v1 hooks for the tree=true accumulator and the v1
portions of init/cleanup. The hooks have static-inline no-op stubs in
include/linux/vmpressure.h for the !MEMCG_V1 case, so callers don't need
ifdefs. vmpressure_prio() gets the same treatment, which means vmscan.c's
call site disappears at compile time on v2-only kernels.
The only #ifdef CONFIG_MEMCG_V1 in source remains around the v1-only
fields inside struct vmpressure itself.
Memory savings on CONFIG_MEMCG_V1=n (measured with pahole):
struct vmpressure : 112B -> 24B
struct mem_cgroup : 1664B -> 1536B
This split is the first step toward eventually making vmpressure
CONFIG_MEMCG_V1 only. The v2 in-kernel socket pressure path (tree=false)
cannot be removed today immediately: PSI is not an exact replacement for
vmpressure, and switching networking socket-buffer back-off to PSI may
regress networking performance or increase memory pressure in workloads
that today rely on vmpressure's hysteresis. The medium-term plan is to
introduce a PSI-based socket-pressure path, keep vmpressure available for
v2 behind a defconfig as an opt-out for several releases, and only then
drop the tree=false path entirely, at which point everything that remains
of the vmpressure block in mm/memcontrol-v1.c is the whole subsystem.
Link: https://lore.kernel.org/20260630112617.1198623-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
CONFIG_ARCH_ENABLE_THP_MIGRATION gates PMD-level migration entries.
PMD-level device-private entries use the same migration mechanism and
therefore require the same architecture support.
Upcoming PMD-level swap entries can use the same PMD softleaf helpers
without depending on page migration, so rename the architecture gate to
CONFIG_ARCH_HAS_PMD_SOFTLEAVES. This describes the PMD entry capability
rather than one current user of it.
This is a pure rename: the set of selecting architectures (x86, arm64,
s390, riscv, loongarch, and powerpc on PPC_BOOK3S_64) and the gating
semantics are unchanged.
No functional change intended.
Link: https://lore.kernel.org/20260706114320.1643046-7-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: preparatory patches for PMD level swap entries", v2.
This is the preparatory part of the PMD page table swapin work. The full
PMD swap entry series has been split into two parts:
1. this preparatory series, which contains the first 6 patches. Zi [1]
and Lance [2] suggested to separate this out from the core series.
2. the PMD swap entry core series, which depends on this one. I will
send this once the preparatory series is merged in mm-new as v3
as the combined is currently at v2 [1].
I have not marked this prep series as v3, as its not really adding
support for PMD swap entries.
This series does not introduce PMD swap entries and does not install any
new page-table entry type. It only cleans up existing PMD softleaf
helpers and call sites so the follow-up PMD swap entry series can be
smaller and easier to review.
It should be safe to merge independently. The patches are either helper
additions, refactors of existing open-coded logic, defensive checks that
preserve current migration/device-private behavior, or a mechanical rename
of the PMD softleaf Kconfig gate. The follow-up series depends on these
helpers, but this series does not depend on the follow-up series.
Patch breakdown:
1. mm: add softleaf_to_pmd() and convert existing callers
Add the PMD counterpart to softleaf_to_pte() and convert existing
swp_entry_to_pmd() users that are constructing PMD softleaf
entries.
2. mm: extract mm_prepare_for_swap_entries() helper
Hoist the "register mm with swapoff" double-checked-locking
pattern out of try_to_unmap_one() and copy_nonpresent_pte() so
future PMD-level users do not need another open-coded copy.
3. fs/proc: use softleaf_has_pfn() in pagemap PMD walker
Avoid assuming every non-present PMD softleaf entry encodes a PFN.
Existing migration/device-private behavior is preserved.
4. mm/huge_memory: move softleaf_to_folio() inside migration branch
Keep the folio lookup in change_non_present_huge_pmd() scoped to
the migration-entry branch that actually needs it.
5. mm/migrate_device: move softleaf_to_folio() inside device-private
branch
Apply the same ordering cleanup to migrate_vma_collect_pmd(): only
derive a folio after confirming the PMD entry is device-private.
6. mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_HAS_PMD_SOFTLEAVES
Rename the architecture gate to describe what it actually enables:
PMD softleaf entries. Migration remains the only current user in
this series; the follow-up series adds PMD swap entries.
This patch (of 6):
Add softleaf_to_pmd() as the PMD counterpart to softleaf_to_pte(),
completing the symmetry of the softleaf abstraction for page table leaf
entries.
The upcoming PMD swap entry support needs to construct PMD entries from
swap entries. Converting existing swp_entry_to_pmd() callers to
softleaf_to_pmd() in a prep patch keeps the feature patches focused on new
functionality rather than mixing refactoring with new code.
Link: https://lore.kernel.org/20260706114320.1643046-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260706114320.1643046-2-usama.arif@linux.dev
Link: https://lore.kernel.org/all/6E99CC4E-A026-4DE3-8A5A-34216771F521@nvidia.com/ [1]
Link: https://lore.kernel.org/all/b08cafbb-a4b7-4609-84ae-dbb2cfcfc8be@linux.dev/#t [2]
Link: https://lore.kernel.org/all/20260602142537.198755-1-usama.arif@linux.dev/ [3]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add support for packet spanning as defined in DSP0283 v1.1.
With the existing v1.0 implementation of multi-packet transfers, all we
need here is to adjust the buffer sizes to suit v1.1.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-9-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Using the existing prepare/complete API, we can persist the rx skb
across receives to implement v1.1 packet spanning.
Alter the packet-extraction loop to allow truncated packets, returning
early with the skb persisted for the next IN urb completion. When we see
we have a complete packet, netif_rx() that. If the packet boundary
aligns with the urb completion, we can netif_rx() the whole thing.
Those intermediate packets are cloned from the original
(large-transfer-data) skb. Unlike existing behaviour, if the clone
fails, we drop just that clone, instead of the existing transfer skb.
This allows us to process the rest of the skb data, and any continuation
of the span into the next transfer.
One subtle change: the mctp_usblib_rx() helper now handles skbs with the
full transport header, so we shift the skb_pull() for the header data to
the helper, before doing the rx_bytes stats update.
We still need to handle non-spanning mode, so error out on
truncated-packet cases there.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-8-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
In the v1.1 update to DSP0283, we have a larger header field, of 13 bits
rather than 8.
In order to accommodate this, in preparation for proper v1.1 support,
expand our struct mctp_usb_hdr's len field to a u16, and endian-convert
when necessary. Because we don't yet support spanning mode, we will
never receive or transmit with the top 5 bits set, so we always mask
out anyway.
This allows for a future change where we allow spanning mode with
>512-byte transfers.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-7-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The MCTP over USB spec allows us to pack multiple packets in one
transfer. Given the packet max length is 255, and the transfer max
length is 512, we can typically include two full-size packets per
urb submission.
To do this, we allow a struct mctp_usb_tx to persist a tx_ctx,
representing the ongoing context for a transmit. If possible, a TX skb
will be queued to the context and the send deferred until the context is
full, or the device queue reports no more packets.
This typically requires a linear buffer for the 512-byte TX, which we
allocate along with the TX context.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-6-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
With the RX processing in mctp-usblib, add TX processing alongside.
To accommodate packed transfers in DSP0283, where a transfer may contain
multiple MCTP packets, we move to a split process for the transmit API:
* push: create a new transmit context, and add a skb to it.
* send: callback to the driver implementation to send the (possibly
multi-packet) USB transfer
* complete: update skb accounting and release the tx context
The actual multi-packet transfer implementation will be added in the
next change; no tx context persists beyond the single send at present.
However, we use an anchor in the host driver implementation to track the
submitted TX urb when necessary.
While we're here, fix an inconsistency between tx and rx stats: both
should not include the transport header.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-5-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The processing of USB receive transfers is common to both sides of a
MCTP over USB transport. In order to support a future gadget driver,
move the current host-side driver into a new common file, mctp-usblib.
This currently handles the submit-complete-packetise process of the
receive path of the USB transport. We'll add transmit handling in an
upcoming change.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-3-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The max packet size is smaller than the max transfer size, as we only
have a u8 length field in the transport header.
Add a define for the maximum representable length, and use that for our
check. Use this for the MTU maximum calculation too.
Signed-off-by: Jeremy Kerr <jk@codeconstruct.com.au>
Link: https://patch.msgid.link/20260724-dev-mctp-usb-1-1-v5-2-e66bbba0dbdc@codeconstruct.com.au
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
