linux-stable.git/include/asm-generic/io.h, branch v6.14

New implementation for IO memcpy and IO memset

2024-10-28T21:44:29+00:00

The IO memcpy and IO memset functions in asm-generic/io.h simply call
memcpy and memset. This can lead to alignment problems or faults on
architectures that do not define their own version and fall back to
these defaults.
This patch introduces new implementations for IO memcpy and IO memset,
that use read{l,q} accessor functions, align accesses to machine word
size, and resort to byte accesses when the target memory is not aligned.
For new architectures and existing ones that were using the old
fallbacks these functions are save to use, because IO memory constraints
are taken into account. Moreover, architectures with similar
implementations can now use these new versions, not needing to implement
their own.

Reviewed-by: Yann Sionneau 
Signed-off-by: Julian Vetter 
Signed-off-by: Arnd Bergmann

asm-generic/io.h: Remove I/O port accessors for HAS_IOPORT=n

2024-10-28T21:44:28+00:00

With all subsystems and drivers either declaring their dependence on
HAS_IOPORT or fencing I/O port specific code sections we can finally
make inb()/outb() and friends compile-time dependent on HAS_IOPORT as
suggested by Linus in the linked mail. The main benefit of this is that
on platforms such as s390 which have no meaningful way of implementing
inb()/outb() their use without the proper HAS_IOPORT dependency will
result in easy to catch and fix compile-time errors instead of compiling
code that can never work.

Link: https://lore.kernel.org/lkml/CAHk-=wg80je=K7madF4e7WrRNp37e3qh6y10Svhdc7O8SZ_-8g@mail.gmail.com/
Co-developed-by: Arnd Bergmann 
Signed-off-by: Arnd Bergmann 
Signed-off-by: Niklas Schnelle 
Reviewed-by: Takashi Iwai 
Reviewed-by: Ville Syrjälä 
Reviewed-by: Jarkko Sakkinen 
Acked-by: Lucas De Marchi 
Acked-by: Maciej W. Rozycki 
Acked-by: Damien Le Moal 
Acked-by: Jaroslav Kysela 
Acked-by: Greg Kroah-Hartman 
Acked-by: Rafael J. Wysocki 
Acked-by: Johannes Berg  # for ARCH=um
Acked-by: Geert Uytterhoeven 
Acked-by: Kalle Valo 
Acked-by: Jakub Kicinski 
Acked-by: Corey Minyard 
Acked-by: Guenter Roeck 
Acked-by: Damien Le Moal 
Acked-by: Sebastian Reichel 
Signed-off-by: Arnd Bergmann

asm-generic/io.h: kill vmalloc.h dependency

2024-04-26T03:55:50+00:00

Needed to avoid a new circular dependency with the memory allocation
profiling series.

Naturally, a whole bunch of files needed to include vmalloc.h that were
previously getting it implicitly.

Link: https://lkml.kernel.org/r/20240321163705.3067592-3-surenb@google.com
Signed-off-by: Kent Overstreet 
Signed-off-by: Suren Baghdasaryan 
Reviewed-by: Pasha Tatashin 
Tested-by: Kees Cook 
Cc: Alexander Viro 
Cc: Alex Gaynor 
Cc: Alice Ryhl 
Cc: Andreas Hindborg 
Cc: Benno Lossin 
Cc: "Björn Roy Baron" 
Cc: Boqun Feng 
Cc: Christoph Lameter 
Cc: Dennis Zhou 
Cc: Gary Guo 
Cc: Miguel Ojeda 
Cc: Peter Zijlstra 
Cc: Tejun Heo 
Cc: Vlastimil Babka 
Cc: Wedson Almeida Filho 
Signed-off-by: Andrew Morton

fix missing vmalloc.h includes

2024-04-26T03:55:49+00:00

Patch series "Memory allocation profiling", v6.

Overview:
Low overhead [1] per-callsite memory allocation profiling. Not just for
debug kernels, overhead low enough to be deployed in production.

Example output:
  root@moria-kvm:~# sort -rn /proc/allocinfo
   127664128    31168 mm/page_ext.c:270 func:alloc_page_ext
    56373248     4737 mm/slub.c:2259 func:alloc_slab_page
    14880768     3633 mm/readahead.c:247 func:page_cache_ra_unbounded
    14417920     3520 mm/mm_init.c:2530 func:alloc_large_system_hash
    13377536      234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs
    11718656     2861 mm/filemap.c:1919 func:__filemap_get_folio
     9192960     2800 kernel/fork.c:307 func:alloc_thread_stack_node
     4206592        4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable
     4136960     1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start
     3940352      962 mm/memory.c:4214 func:alloc_anon_folio
     2894464    22613 fs/kernfs/dir.c:615 func:__kernfs_new_node
     ...

Usage:
kconfig options:
 - CONFIG_MEM_ALLOC_PROFILING
 - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT
 - CONFIG_MEM_ALLOC_PROFILING_DEBUG
   adds warnings for allocations that weren't accounted because of a
   missing annotation

sysctl:
  /proc/sys/vm/mem_profiling

Runtime info:
  /proc/allocinfo

Notes:

[1]: Overhead
To measure the overhead we are comparing the following configurations:
(1) Baseline with CONFIG_MEMCG_KMEM=n
(2) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
    CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n)
(3) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
    CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y)
(4) Enabled at runtime (CONFIG_MEM_ALLOC_PROFILING=y &&
    CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n && /proc/sys/vm/mem_profiling=1)
(5) Baseline with CONFIG_MEMCG_KMEM=y && allocating with __GFP_ACCOUNT
(6) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
    CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n)  && CONFIG_MEMCG_KMEM=y
(7) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
    CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y) && CONFIG_MEMCG_KMEM=y

Performance overhead:
To evaluate performance we implemented an in-kernel test executing
multiple get_free_page/free_page and kmalloc/kfree calls with allocation
sizes growing from 8 to 240 bytes with CPU frequency set to max and CPU
affinity set to a specific CPU to minimize the noise. Below are results
from running the test on Ubuntu 22.04.2 LTS with 6.8.0-rc1 kernel on
56 core Intel Xeon:

                        kmalloc                 pgalloc
(1 baseline)            6.764s                  16.902s
(2 default disabled)    6.793s  (+0.43%)        17.007s (+0.62%)
(3 default enabled)     7.197s  (+6.40%)        23.666s (+40.02%)
(4 runtime enabled)     7.405s  (+9.48%)        23.901s (+41.41%)
(5 memcg)               13.388s (+97.94%)       48.460s (+186.71%)
(6 def disabled+memcg)  13.332s (+97.10%)       48.105s (+184.61%)
(7 def enabled+memcg)   13.446s (+98.78%)       54.963s (+225.18%)

Memory overhead:
Kernel size:

   text           data        bss         dec         diff
(1) 26515311	      18890222    17018880    62424413
(2) 26524728	      19423818    16740352    62688898    264485
(3) 26524724	      19423818    16740352    62688894    264481
(4) 26524728	      19423818    16740352    62688898    264485
(5) 26541782	      18964374    16957440    62463596    39183

Memory consumption on a 56 core Intel CPU with 125GB of memory:
Code tags:           192 kB
PageExts:         262144 kB (256MB)
SlabExts:           9876 kB (9.6MB)
PcpuExts:            512 kB (0.5MB)

Total overhead is 0.2% of total memory.

Benchmarks:

Hackbench tests run 100 times:
hackbench -s 512 -l 200 -g 15 -f 25 -P
      baseline       disabled profiling           enabled profiling
avg   0.3543         0.3559 (+0.0016)             0.3566 (+0.0023)
stdev 0.0137         0.0188                       0.0077


hackbench -l 10000
      baseline       disabled profiling           enabled profiling
avg   6.4218         6.4306 (+0.0088)             6.5077 (+0.0859)
stdev 0.0933         0.0286                       0.0489

stress-ng tests:
stress-ng --class memory --seq 4 -t 60
stress-ng --class cpu --seq 4 -t 60
Results posted at: https://evilpiepirate.org/~kent/memalloc_prof_v4_stress-ng/

[2] https://lore.kernel.org/all/20240306182440.2003814-1-surenb@google.com/


This patch (of 37):

The next patch drops vmalloc.h from a system header in order to fix a
circular dependency; this adds it to all the files that were pulling it in
implicitly.

[kent.overstreet@linux.dev: fix arch/alpha/lib/memcpy.c]
  Link: https://lkml.kernel.org/r/20240327002152.3339937-1-kent.overstreet@linux.dev
[surenb@google.com: fix arch/x86/mm/numa_32.c]
  Link: https://lkml.kernel.org/r/20240402180933.1663992-1-surenb@google.com
[kent.overstreet@linux.dev: a few places were depending on sizes.h]
  Link: https://lkml.kernel.org/r/20240404034744.1664840-1-kent.overstreet@linux.dev
[arnd@arndb.de: fix mm/kasan/hw_tags.c]
  Link: https://lkml.kernel.org/r/20240404124435.3121534-1-arnd@kernel.org
[surenb@google.com: fix arc build]
  Link: https://lkml.kernel.org/r/20240405225115.431056-1-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-1-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-2-surenb@google.com
Signed-off-by: Kent Overstreet 
Signed-off-by: Suren Baghdasaryan 
Signed-off-by: Arnd Bergmann 
Reviewed-by: Pasha Tatashin 
Tested-by: Kees Cook 
Cc: Alexander Viro 
Cc: Alex Gaynor 
Cc: Alice Ryhl 
Cc: Andreas Hindborg 
Cc: Benno Lossin 
Cc: "Björn Roy Baron" 
Cc: Boqun Feng 
Cc: Christoph Lameter 
Cc: Dennis Zhou 
Cc: Gary Guo 
Cc: Miguel Ojeda 
Cc: Peter Zijlstra 
Cc: Tejun Heo 
Cc: Vlastimil Babka 
Cc: Wedson Almeida Filho 
Signed-off-by: Andrew Morton

mm: ioremap: remove unneeded ioremap_allowed and iounmap_allowed

2023-08-18T17:12:36+00:00

Now there are no users of ioremap_allowed and iounmap_allowed, clean
them up.

Link: https://lkml.kernel.org/r/20230706154520.11257-20-bhe@redhat.com
Signed-off-by: Baoquan He 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Kefeng Wang 
Reviewed-by: Mike Rapoport (IBM) 
Cc: Alexander Gordeev 
Cc: Arnd Bergmann 
Cc: Brian Cain 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Chris Zankel 
Cc: David Laight 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: "James E.J. Bottomley" 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Nathan Chancellor 
Cc: Nicholas Piggin 
Cc: Niklas Schnelle 
Cc: Rich Felker 
Cc: Stafford Horne 
Cc: Stefan Kristiansson 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

mm: ioremap: allow ARCH to have its own ioremap method definition

2023-08-18T17:12:33+00:00

Architectures can be converted to GENERIC_IOREMAP, to take standard
ioremap_xxx() and iounmap() way.  But some ARCH-es could have specific
handling for ioremap_prot(), ioremap() and iounmap(), than standard
methods.

In oder to convert these ARCH-es to take GENERIC_IOREMAP method, allow
these architecutres to have their own ioremap_prot(), ioremap() and
iounmap() definitions.

Link: https://lkml.kernel.org/r/20230706154520.11257-6-bhe@redhat.com
Signed-off-by: Baoquan He 
Acked-by: Arnd Bergmann 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Kefeng Wang 
Reviewed-by: Mike Rapoport (IBM) 
Cc: Alexander Gordeev 
Cc: Brian Cain 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Chris Zankel 
Cc: David Laight 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: "James E.J. Bottomley" 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Nathan Chancellor 
Cc: Nicholas Piggin 
Cc: Niklas Schnelle 
Cc: Rich Felker 
Cc: Stafford Horne 
Cc: Stefan Kristiansson 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

mm/ioremap: define generic_ioremap_prot() and generic_iounmap()

2023-08-18T17:12:32+00:00

Define a generic version of ioremap_prot() and iounmap() that
architectures can call after they have performed the necessary alteration
to parameters and/or necessary verifications.

Link: https://lkml.kernel.org/r/20230706154520.11257-5-bhe@redhat.com
Signed-off-by: Christophe Leroy 
Signed-off-by: Baoquan He 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Kefeng Wang 
Reviewed-by: Mike Rapoport (IBM) 
Cc: Alexander Gordeev 
Cc: Arnd Bergmann 
Cc: Brian Cain 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: Chris Zankel 
Cc: David Laight 
Cc: Geert Uytterhoeven 
Cc: Gerald Schaefer 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: "James E.J. Bottomley" 
Cc: John Paul Adrian Glaubitz 
Cc: Jonas Bonn 
Cc: Matthew Wilcox 
Cc: Max Filippov 
Cc: Michael Ellerman 
Cc: Nathan Chancellor 
Cc: Nicholas Piggin 
Cc: Niklas Schnelle 
Cc: Rich Felker 
Cc: Stafford Horne 
Cc: Stefan Kristiansson 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vineet Gupta 
Cc: Will Deacon 
Cc: Yoshinori Sato 
Signed-off-by: Andrew Morton

asm-generic/io.h: suppress endianness warnings for relaxed accessors

2023-04-04T15:58:11+00:00

Copy the forced type casts from the normal MMIO accessors to suppress
the sparse warnings that point out __raw_readl() returns a native endian
word (just like readl()).

Signed-off-by: Vladimir Oltean 
Signed-off-by: Arnd Bergmann

asm-generic/io.h: suppress endianness warnings for readq() and writeq()

2023-04-04T15:58:11+00:00

Commit c1d55d50139b ("asm-generic/io.h: Fix sparse warnings on
big-endian architectures") missed fixing the 64-bit accessors.

Arnd explains in the attached link why the casts are necessary, even if
__raw_readq() and __raw_writeq() do not take endian-specific types.

Link: https://lore.kernel.org/lkml/9105d6fc-880b-4734-857d-e3d30b87ccf6@app.fastmail.com/
Suggested-by: Arnd Bergmann 
Signed-off-by: Vladimir Oltean 
Reviewed-by: Jonathan Cameron 
Signed-off-by: Arnd Bergmann

asm-generic/io: Add _RET_IP_ to MMIO trace for more accurate debug info

2022-11-21T21:02:10+00:00

Due to compiler optimizations like inlining, there are cases where
MMIO traces using _THIS_IP_ for caller information might not be
sufficient to provide accurate debug traces.

1) With optimizations (Seen with GCC):

In this case, _THIS_IP_ works fine and prints the caller information
since it will be inlined into the caller and we get the debug traces
on who made the MMIO access, for ex:

rwmmio_read: qcom_smmu_tlb_sync+0xe0/0x1b0 width=32 addr=0xffff8000087447f4
rwmmio_post_read: qcom_smmu_tlb_sync+0xe0/0x1b0 width=32 val=0x0 addr=0xffff8000087447f4

2) Without optimizations (Seen with Clang):

_THIS_IP_ will not be sufficient in this case as it will print only
the MMIO accessors itself which is of not much use since it is not
inlined as below for example:

rwmmio_read: readl+0x4/0x80 width=32 addr=0xffff8000087447f4
rwmmio_post_read: readl+0x48/0x80 width=32 val=0x4 addr=0xffff8000087447f4

So in order to handle this second case as well irrespective of the compiler
optimizations, add _RET_IP_ to MMIO trace to make it provide more accurate
debug information in all these scenarios.

Before:

rwmmio_read: readl+0x4/0x80 width=32 addr=0xffff8000087447f4
rwmmio_post_read: readl+0x48/0x80 width=32 val=0x4 addr=0xffff8000087447f4

After:

rwmmio_read: qcom_smmu_tlb_sync+0xe0/0x1b0 -> readl+0x4/0x80 width=32 addr=0xffff8000087447f4
rwmmio_post_read: qcom_smmu_tlb_sync+0xe0/0x1b0 -> readl+0x4/0x80 width=32 val=0x0 addr=0xffff8000087447f4

Fixes: 210031971cdd ("asm-generic/io: Add logging support for MMIO accessors")
Signed-off-by: Sai Prakash Ranjan 
Signed-off-by: Arnd Bergmann