linux.git/mm/memcontrol.c, branch v6.9

memcg: remove mem_cgroup_uncharge_list()

2024-03-05T01:01:25+00:00

All users have been converted to mem_cgroup_uncharge_folios() so we can
remove this API.

Link: https://lkml.kernel.org/r/20240227174254.710559-14-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Ryan Roberts 
Cc: David Hildenbrand 
Cc: Mel Gorman 
Signed-off-by: Andrew Morton

memcg: add mem_cgroup_uncharge_folios()

2024-03-05T01:01:23+00:00

Almost identical to mem_cgroup_uncharge_list(), except it takes a
folio_batch instead of a list_head.

Link: https://lkml.kernel.org/r/20240227174254.710559-6-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Ryan Roberts 
Cc: David Hildenbrand 
Cc: Mel Gorman 
Signed-off-by: Andrew Morton

mm: memcg: make memcg huge page split support any order split

2024-03-05T01:01:20+00:00

It sets memcg information for the pages after the split.  A new parameter
new_order is added to tell the order of subpages in the new page, always 0
for now.  It prepares for upcoming changes to support split huge page to
any lower order.

Link: https://lkml.kernel.org/r/20240226205534.1603748-6-zi.yan@sent.com
Signed-off-by: Zi Yan 
Acked-by: David Hildenbrand 
Cc: Hugh Dickins 
Cc: Kirill A. Shutemov 
Cc: Luis Chamberlain 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michal Koutny 
Cc: Roman Gushchin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Zach O'Keefe 
Signed-off-by: Andrew Morton

mm/memcg: use order instead of nr in split_page_memcg()

2024-03-05T01:01:19+00:00

We do not have non power of two pages, using nr is error prone if nr is
not power-of-two.  Use page order instead.

Link: https://lkml.kernel.org/r/20240226205534.1603748-4-zi.yan@sent.com
Signed-off-by: Zi Yan 
Acked-by: David Hildenbrand 
Cc: Hugh Dickins 
Cc: Kirill A. Shutemov 
Cc: Luis Chamberlain 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michal Koutny 
Cc: Roman Gushchin 
Cc: Ryan Roberts 
Cc: Yang Shi 
Cc: Yu Zhao 
Cc: Zach O'Keefe 
Signed-off-by: Andrew Morton

mm: memcg: use larger batches for proactive reclaim

2024-02-22T18:24:52+00:00

Before 388536ac291 ("mm:vmscan: fix inaccurate reclaim during proactive
reclaim") we passed the number of pages for the reclaim request directly
to try_to_free_mem_cgroup_pages, which could lead to significant
overreclaim.  After 0388536ac291 the number of pages was limited to a
maximum 32 (SWAP_CLUSTER_MAX) to reduce the amount of overreclaim. 
However such a small batch size caused a regression in reclaim performance
due to many more reclaim start/stop cycles inside memory_reclaim.  The
restart cost is amortized over more pages with larger batch sizes, and
becomes a significant component of the runtime if the batch size is too
small.

Reclaim tries to balance nr_to_reclaim fidelity with fairness across nodes
and cgroups over which the pages are spread.  As such, the bigger the
request, the bigger the absolute overreclaim error.  Historic in-kernel
users of reclaim have used fixed, small sized requests to approach an
appropriate reclaim rate over time.  When we reclaim a user request of
arbitrary size, use decaying batch sizes to manage error while maintaining
reasonable throughput.

MGLRU enabled - memcg LRU used
root - full reclaim       pages/sec   time (sec)
pre-0388536ac291      :    68047        10.46
post-0388536ac291     :    13742        inf
(reclaim-reclaimed)/4 :    67352        10.51

MGLRU enabled - memcg LRU not used
/uid_0 - 1G reclaim       pages/sec   time (sec)  overreclaim (MiB)
pre-0388536ac291      :    258822       1.12            107.8
post-0388536ac291     :    105174       2.49            3.5
(reclaim-reclaimed)/4 :    233396       1.12            -7.4

MGLRU enabled - memcg LRU not used
/uid_0 - full reclaim     pages/sec   time (sec)
pre-0388536ac291      :    72334        7.09
post-0388536ac291     :    38105        14.45
(reclaim-reclaimed)/4 :    72914        6.96

[tjmercier@google.com: v4]
  Link: https://lkml.kernel.org/r/20240206175251.3364296-1-tjmercier@google.com
Link: https://lkml.kernel.org/r/20240202233855.1236422-1-tjmercier@google.com
Fixes: 0388536ac291 ("mm:vmscan: fix inaccurate reclaim during proactive reclaim")
Signed-off-by: T.J. Mercier 
Reviewed-by: Yosry Ahmed 
Acked-by: Johannes Weiner 
Reviewed-by: Michal Koutny 
Acked-by: Shakeel Butt 
Acked-by: Michal Hocko 
Cc: Roman Gushchin 
Cc: Shakeel Butt 
Cc: Muchun Song 
Cc: Efly Young 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton

mm: memcg: don't periodically flush stats when memcg is disabled

2024-02-22T18:24:41+00:00

The root memcg is onlined even when memcg is disabled.  When it's onlined
a 2 second periodic stat flush is started, but no stat flushing is
required when memcg is disabled because there can be no child memcgs. 
Most calls to flush memcg stats are avoided when memcg is disabled as a
result of the mem_cgroup_disabled check added in 7d7ef0a4686a ("mm: memcg:
restore subtree stats flushing"), but the periodic flushing started in
mem_cgroup_css_online is not.  Skip it.

Link: https://lkml.kernel.org/r/20240126211927.1171338-1-tjmercier@google.com
Fixes: aa48e47e3906 ("memcg: infrastructure to flush memcg stats")
Signed-off-by: T.J. Mercier 
Acked-by: Shakeel Butt 
Acked-by: Johannes Weiner 
Acked-by: Chris Li 
Reported-by: Minchan Kim 
Reviewed-by: Yosry Ahmed 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Roman Gushchin 
Cc: Michal Koutn 
Signed-off-by: Andrew Morton

mm: writeback: ratelimit stat flush from mem_cgroup_wb_stats

2024-02-22T18:24:38+00:00

One of our workloads (Postgres 14) has regressed when migrated from 5.10
to 6.1 upstream kernel.  The regression can be reproduced by sysbench's
oltp_write_only benchmark.  It seems like the always on rstat flush in
mem_cgroup_wb_stats() is causing the regression.  So, rate limit that
specific rstat flush.  One potential consequence would be the dirty
throttling might be decided on stale memcg stats.  However from our
benchmarks and production traffic we have not observed any change in the
dirty throttling behavior of the application.

Link: https://lkml.kernel.org/r/20240118184235.618164-1-shakeelb@google.com
Fixes: 2d146aa3aa84 ("mm: memcontrol: switch to rstat")
Signed-off-by: Shakeel Butt 
Acked-by: Johannes Weiner 
Acked-by: Roman Gushchin 
Cc: Jan Kara 
Cc: Jens Axboe 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Tejun Heo 
Signed-off-by: Andrew Morton

memcg: use a folio in get_mctgt_type_thp

2024-02-22T00:00:03+00:00

Replace five calls to compound_head() with one.

Link: https://lkml.kernel.org/r/20240111181219.3462852-5-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Johannes Weiner 
Reviewed-by: Roman Gushchin 
Reviewed-by: Muchun Song 
Acked-by: Shakeel Butt 
Acked-by: Michal Hocko 
Signed-off-by: Andrew Morton

memcg: use a folio in get_mctgt_type

2024-02-22T00:00:03+00:00

Replace seven calls to compound_head() with one.  We still use the page as
page_mapped() is different from folio_mapped().

Link: https://lkml.kernel.org/r/20240111181219.3462852-4-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Johannes Weiner 
Reviewed-by: Roman Gushchin 
Reviewed-by: Muchun Song 
Acked-by: Shakeel Butt 
Acked-by: Michal Hocko 
Signed-off-by: Andrew Morton

memcg: return the folio in union mc_target

2024-02-22T00:00:03+00:00

All users of target.page convert it to the folio, so we can just return
the folio directly and save a few calls to compound_head().

Link: https://lkml.kernel.org/r/20240111181219.3462852-3-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Acked-by: Johannes Weiner 
Reviewed-by: Roman Gushchin 
Reviewed-by: Muchun Song 
Acked-by: Shakeel Butt 
Acked-by: Michal Hocko 
Signed-off-by: Andrew Morton