linux-stable.git/include/linux/pgalloc_tag.h, branch v7.0

mm: introduce memdesc_flags_t

2025-09-13T23:55:07+00:00

Patch series "Add and use memdesc_flags_t".

At some point struct page will be separated from struct slab and struct
folio.  This is a step towards that by introducing a type for the 'flags'
word of all three structures.  This gives us a certain amount of type
safety by establishing that some of these unsigned longs are different
from other unsigned longs in that they contain things like node ID,
section number and zone number in the upper bits.  That lets us have
functions that can be easily called by anyone who has a slab, folio or
page (but not easily by anyone else) to get the node or zone.

There's going to be some unusual merge problems with this as some odd bits
of the kernel decide they want to print out the flags value or something
similar by writing page->flags and now they'll need to write page->flags.f
instead.  That's most of the churn here.  Maybe we should be removing
these things from the debug output?


This patch (of 11):

Wrap the unsigned long flags in a typedef.  In upcoming patches, this will
provide a strong hint that you can't just pass a random unsigned long to
functions which take this as an argument.

[willy@infradead.org: s/flags/flags.f/ in several architectures]
  Link: https://lkml.kernel.org/r/aKMgPRLD-WnkPxYm@casper.infradead.org
[nicola.vetrini@gmail.com: mips: fix compilation error]
  Link: https://lore.kernel.org/lkml/CA+G9fYvkpmqGr6wjBNHY=dRp71PLCoi2341JxOudi60yqaeUdg@mail.gmail.com/
  Link: https://lkml.kernel.org/r/20250825214245.1838158-1-nicola.vetrini@gmail.com
Link: https://lkml.kernel.org/r/20250805172307.1302730-1-willy@infradead.org
Link: https://lkml.kernel.org/r/20250805172307.1302730-2-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Zi Yan 
Cc: Shakeel Butt 
Signed-off-by: Andrew Morton

mm/codetag: move tag retrieval back upfront in __free_pages()

2025-05-12T00:26:06+00:00

Commit 51ff4d7486f0 ("mm: avoid extra mem_alloc_profiling_enabled()
 checks") introduces a possible use-after-free scenario, when page
is non-compound, page[0] could be released by other thread right
after put_page_testzero failed in current thread, pgalloc_tag_sub_pages
afterwards would manipulate an invalid page for accounting remaining
pages:

[timeline]   [thread1]                     [thread2]
  |          alloc_page non-compound
  V
  |                                        get_page, rf counter inc
  V
  |          in ___free_pages
  |          put_page_testzero fails
  V
  |                                        put_page, page released
  V
  |          in ___free_pages,
  |          pgalloc_tag_sub_pages
  |          manipulate an invalid page
  V

Restore __free_pages() to its state before, retrieve alloc tag
beforehand.

Link: https://lkml.kernel.org/r/20250505193034.91682-1-00107082@163.com
Fixes: 51ff4d7486f0 ("mm: avoid extra mem_alloc_profiling_enabled() checks")
Signed-off-by: David Wang <00107082@163.com>
Acked-by: Suren Baghdasaryan 
Reviewed-by: Vlastimil Babka 
Cc: Brendan Jackman 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Shakeel Butt 
Cc: Vlastimil Babka 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

alloc_tag: uninline code gated by mem_alloc_profiling_key in page allocator

2025-03-17T05:06:03+00:00

When a sizable code section is protected by a disabled static key, that
code gets into the instruction cache even though it's not executed and
consumes the cache, increasing cache misses. This can be remedied by
moving such code into a separate uninlined function.
On a Pixel6 phone, page allocation profiling overhead measured with
CONFIG_MEM_ALLOC_PROFILING=y and profiling disabled is:

             baseline             modified
Big core     4.93%                1.53%
Medium core  4.39%                1.41%
Little core  1.02%                0.36%

This improvement comes at the expense of the configuration when profiling
gets enabled, since there is now an additional function call. The overhead
from this additional call on Pixel6 is:

Big core     0.24%
Middle core  0.63%
Little core  1.1%

However this is negligible when compared with the overall overhead of the
memory allocation profiling when it is enabled.
On x86 this patch does not make noticeable difference because the overhead
with mem_alloc_profiling_key disabled is much lower (under 1%) to start
with, so any improvement is less visible and hard to distinguish from the
noise. The overhead from additional call when profiling is enabled is also
within noise levels.

Link: https://lkml.kernel.org/r/20250201231803.2661189-3-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Shakeel Butt 
Cc: David Wang <00107082@163.com>
Cc: Kent Overstreet 
Cc: Minchan Kim 
Cc: Pasha Tatashin 
Cc: Peter Zijlstra (Intel) 
Cc: Sourav Panda 
Cc: Steven Rostedt 
Cc: Vlastimil Babka 
Cc: Yu Zhao 
Cc: Zhenhua Huang 
Signed-off-by: Andrew Morton

mm: avoid extra mem_alloc_profiling_enabled() checks

2025-03-17T05:06:03+00:00

Refactor code to avoid extra mem_alloc_profiling_enabled() checks inside
pgalloc_tag_get() function which is often called after that check was
already done.

Link: https://lkml.kernel.org/r/20250201231803.2661189-1-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Shakeel Butt 
Cc: David Wang <00107082@163.com>
Cc: Steven Rostedt 
Cc: Kent Overstreet 
Cc: Minchan Kim 
Cc: Pasha Tatashin 
Cc: Peter Zijlstra (Intel) 
Cc: Sourav Panda 
Cc: Vlastimil Babka 
Cc: Yu Zhao 
Cc: Zhenhua Huang 
Signed-off-by: Andrew Morton

mm/codetag: swap tags when migrate pages

2024-12-06T03:54:46+00:00

Current solution to adjust codetag references during page migration is
done in 3 steps:

1. sets the codetag reference of the old page as empty (not pointing
   to any codetag);

2. subtracts counters of the new page to compensate for its own
   allocation;

3. sets codetag reference of the new page to point to the codetag of
   the old page.

This does not work if CONFIG_MEM_ALLOC_PROFILING_DEBUG=n because
set_codetag_empty() becomes NOOP.  Instead, let's simply swap codetag
references so that the new page is referencing the old codetag and the old
page is referencing the new codetag.  This way accounting stays valid and
the logic makes more sense.

Link: https://lkml.kernel.org/r/20241129025213.34836-1-00107082@163.com
Fixes: e0a955bf7f61 ("mm/codetag: add pgalloc_tag_copy()")
Signed-off-by: David Wang <00107082@163.com>
Closes: https://lore.kernel.org/lkml/20241124074318.399027-1-00107082@163.com/
Acked-by: Suren Baghdasaryan 
Suggested-by: Suren Baghdasaryan 
Acked-by: Yu Zhao 
Cc: Kent Overstreet 
Signed-off-by: Andrew Morton

mm/codetag: uninline and move pgalloc_tag_copy and pgalloc_tag_split

2024-11-07T22:25:17+00:00

pgalloc_tag_copy() and pgalloc_tag_split() are sizable and outside of any
performance-critical paths, so it should be fine to uninline them.  Also
move their declarations into pgalloc_tag.h which seems like a more
appropriate place for them.  No functional changes other than uninlining.

Link: https://lkml.kernel.org/r/20241024162318.1640781-1-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Suggested-by: Andrew Morton 
Acked-by: Yu Zhao 
Cc: Kent Overstreet 
Cc: Pasha Tatashin 
Cc: Sourav Panda 
Signed-off-by: Andrew Morton

alloc_tag: support for page allocation tag compression

2024-11-07T22:25:16+00:00

Implement support for storing page allocation tag references directly in
the page flags instead of page extensions.  sysctl.vm.mem_profiling boot
parameter it extended to provide a way for a user to request this mode. 
Enabling compression eliminates memory overhead caused by page_ext and
results in better performance for page allocations.  However this mode
will not work if the number of available page flag bits is insufficient to
address all kernel allocations.  Such condition can happen during boot or
when loading a module.  If this condition is detected, memory allocation
profiling gets disabled with an appropriate warning.  By default
compression mode is disabled.

Link: https://lkml.kernel.org/r/20241023170759.999909-7-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Pasha Tatashin 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Petkov (AMD) 
Cc: Christoph Hellwig 
Cc: Daniel Gomez 
Cc: David Hildenbrand 
Cc: Davidlohr Bueso 
Cc: David Rientjes 
Cc: Dennis Zhou 
Cc: Johannes Weiner 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Joonsoo Kim 
Cc: Kalesh Singh 
Cc: Kees Cook 
Cc: Kent Overstreet 
Cc: Liam R. Howlett 
Cc: Luis Chamberlain 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Mike Rapoport (Microsoft) 
Cc: Minchan Kim 
Cc: Paul E. McKenney 
Cc: Petr Pavlu 
Cc: Roman Gushchin 
Cc: Sami Tolvanen 
Cc: Sourav Panda 
Cc: Steven Rostedt (Google) 
Cc: Thomas Gleixner 
Cc: Thomas Huth 
Cc: Uladzislau Rezki (Sony) 
Cc: Vlastimil Babka 
Cc: Xiongwei Song 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

alloc_tag: introduce pgtag_ref_handle to abstract page tag references

2024-11-07T22:25:16+00:00

To simplify later changes to page tag references, introduce new
pgtag_ref_handle type.  This allows easy replacement of page_ext as a
storage of page allocation tags.

Link: https://lkml.kernel.org/r/20241023170759.999909-6-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Pasha Tatashin 
Cc: Ard Biesheuvel 
Cc: Arnd Bergmann 
Cc: Borislav Petkov (AMD) 
Cc: Christoph Hellwig 
Cc: Daniel Gomez 
Cc: David Hildenbrand 
Cc: Davidlohr Bueso 
Cc: David Rientjes 
Cc: Dennis Zhou 
Cc: Johannes Weiner 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Joonsoo Kim 
Cc: Kalesh Singh 
Cc: Kees Cook 
Cc: Kent Overstreet 
Cc: Liam R. Howlett 
Cc: Luis Chamberlain 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Mike Rapoport (Microsoft) 
Cc: Minchan Kim 
Cc: Paul E. McKenney 
Cc: Petr Pavlu 
Cc: Roman Gushchin 
Cc: Sami Tolvanen 
Cc: Sourav Panda 
Cc: Steven Rostedt (Google) 
Cc: Thomas Gleixner 
Cc: Thomas Huth 
Cc: Uladzislau Rezki (Sony) 
Cc: Vlastimil Babka 
Cc: Xiongwei Song 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm/codetag: fix pgalloc_tag_split()

2024-09-09T23:39:18+00:00

The current assumption is that a large folio can only be split into
order-0 folios.  That is not the case for hugeTLB demotion, nor for THP
split: see commit c010d47f107f ("mm: thp: split huge page to any lower
order pages").

When a large folio is split into ones of a lower non-zero order, only the
new head pages should be tagged.  Tagging tail pages can cause imbalanced
"calls" counters, since only head pages are untagged by pgalloc_tag_sub()
and the "calls" counts on tail pages are leaked, e.g.,

  # echo 2048kB >/sys/kernel/mm/hugepages/hugepages-1048576kB/demote_size
  # echo 700 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
  # time echo 700 >/sys/kernel/mm/hugepages/hugepages-1048576kB/demote
  # echo 0 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  # grep alloc_gigantic_folio /proc/allocinfo

Before this patch:
  0  549427200  mm/hugetlb.c:1549 func:alloc_gigantic_folio

  real  0m2.057s
  user  0m0.000s
  sys   0m2.051s

After this patch:
  0          0  mm/hugetlb.c:1549 func:alloc_gigantic_folio

  real  0m1.711s
  user  0m0.000s
  sys   0m1.704s

Not tagging tail pages also improves the splitting time, e.g., by about
15% when demoting 1GB hugeTLB folios to 2MB ones, as shown above.

Link: https://lkml.kernel.org/r/20240906042108.1150526-2-yuzhao@google.com
Fixes: be25d1d4e822 ("mm: create new codetag references during page splitting")
Signed-off-by: Yu Zhao 
Acked-by: Suren Baghdasaryan 
Cc: Kent Overstreet 
Cc: Muchun Song 
Cc: 
Signed-off-by: Andrew Morton

alloc_tag: introduce clear_page_tag_ref() helper function

2024-08-16T05:16:16+00:00

In several cases we are freeing pages which were not allocated using
common page allocators.  For such cases, in order to keep allocation
accounting correct, we should clear the page tag to indicate that the page
being freed is expected to not have a valid allocation tag.  Introduce
clear_page_tag_ref() helper function to be used for this.

Link: https://lkml.kernel.org/r/20240813150758.855881-1-surenb@google.com
Fixes: d224eb0287fb ("codetag: debug: mark codetags for reserved pages as empty")
Signed-off-by: Suren Baghdasaryan 
Suggested-by: David Hildenbrand 
Acked-by: David Hildenbrand 
Reviewed-by: Pasha Tatashin 
Cc: Kees Cook 
Cc: Kent Overstreet 
Cc: Sourav Panda 
Cc: Vlastimil Babka 
Cc: 	[6.10]
Signed-off-by: Andrew Morton