linux.git/kernel/locking/percpu-rwsem.c, branch v7.2-rc1

locking: Add contended_release tracepoint to sleepable locks

2026-06-11T11:41:25+00:00

Add the contended_release trace event. This tracepoint fires on the
holder side when a contended lock is released, complementing the
existing contention_begin/contention_end tracepoints which fire on the
waiter side.

This enables correlating lock hold time under contention with waiter
events by lock address.

Add trace_contended_release()/trace_call__contended_release() calls to
the slowpath unlock paths of sleepable locks: mutex, rtmutex, semaphore,
rwsem, percpu-rwsem, and RT-specific rwbase locks.

Where possible, trace_contended_release() fires before the lock is
released and before the waiter is woken. For some lock types, the
tracepoint fires after the release but before the wake. Making the
placement consistent across all lock types is not worth the added
complexity.

For reader/writer locks, the tracepoint fires for every reader releasing
while a writer is waiting, not only for the last reader.

Signed-off-by: Dmitry Ilvokhin 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Paul E. McKenney 
Acked-by: Usama Arif 
Link: https://patch.msgid.link/02f4f6c5ce6761e7f6587cf0ff2289d962ecddd4.1780506267.git.d@ilvokhin.com

locking/percpu-rwsem: Extract __percpu_up_read()

2026-06-11T11:41:25+00:00

Move the percpu_up_read() slowpath out of the inline function into a new
__percpu_up_read() to avoid binary size increase from adding a
tracepoint to an inlined function.

Signed-off-by: Dmitry Ilvokhin 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Usama Arif 
Link: https://patch.msgid.link/3dd2a1b9ab4f469e1892766cb63f41d6b0f53d29.1780506267.git.d@ilvokhin.com

locking/percpu-rwsem: add freezable alternative to down_read

2025-04-07T07:37:16+00:00

Percpu-rwsems are used for superblock locking.  However, we know the
read percpu-rwsem we take for sb_start_write() on a frozen filesystem
needs not to inhibit system from suspending or hibernating.  That
means it needs to wait with TASK_UNINTERRUPTIBLE | TASK_FREEZABLE.

Introduce a new percpu_down_read_freezable() that allows us to control
whether TASK_FREEZABLE is added to the wait flags.

Signed-off-by: James Bottomley 
Link: https://lore.kernel.org/r/20250327140613.25178-2-James.Bottomley@HansenPartnership.com
Signed-off-by: Christian Brauner

percpu: use TYPEOF_UNQUAL() in variable declarations

2025-03-17T05:05:53+00:00

Use TYPEOF_UNQUAL() to declare variables as a corresponding type without
named address space qualifier to avoid "`__seg_gs' specified for auto
variable `var'" errors.

Link: https://lkml.kernel.org/r/20250127160709.80604-4-ubizjak@gmail.com
Signed-off-by: Uros Bizjak 
Acked-by: Nadav Amit 
Acked-by: Christoph Lameter 
Cc: Dennis Zhou 
Cc: Tejun Heo 
Cc: Andy Lutomirski 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Borislav Petkov 
Cc: Dave Hansen 
Cc: "H. Peter Anvin" 
Cc: Kent Overstreet 
Cc: Arnd Bergmann 
Cc: "David S. Miller" 
Cc: Eric Dumazet 
Cc: Jakub Kicinski 
Cc: Paolo Abeni 
Cc: Peter Zijlstra 
Cc: Will Deacon 
Cc: Waiman Long 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Brian Gerst 
Cc: Denys Vlasenko 
Signed-off-by: Andrew Morton

locking/percpu-rwsem: Trigger contention tracepoints only if contended

2024-02-28T12:10:29+00:00

We mistakenly always fire lock contention tracepoints in the writer path,
while it should be conditional on the trylock result.

Signed-off-by: Namhyung Kim 
Signed-off-by: Ingo Molnar 
Reviewed-by: Waiman Long 
Link: https://lore.kernel.org/r/20231108215322.2845536-1-namhyung@kernel.org

locking/percpu-rwsem: Add percpu_is_write_locked() and percpu_is_read_locked()

2022-08-30T08:56:23+00:00

Implement simple accessors to probe percpu-rwsem's locked state:
percpu_is_write_locked(), percpu_is_read_locked().

Signed-off-by: Marco Elver 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Dmitry Vyukov 
Acked-by: Ian Rogers 
Link: https://lore.kernel.org/r/20220829124719.675715-11-elver@google.com

locking: Apply contention tracepoints in the slow path

2022-04-05T08:24:35+00:00

Adding the lock contention tracepoints in various lock function slow
paths.  Note that each arch can define spinlock differently, I only
added it only to the generic qspinlock for now.

Signed-off-by: Namhyung Kim 
Signed-off-by: Peter Zijlstra (Intel) 
Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Link: https://lkml.kernel.org/r/20220322185709.141236-3-namhyung@kernel.org

locking: Add missing __sched attributes

2022-02-11T11:13:55+00:00

This patch adds __sched attributes to a few missing places
to show blocked function rather than locking function
in get_wchan.

Signed-off-by: Minchan Kim 
Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20220115231657.84828-1-minchan@kernel.org

locking/percpu-rwsem: Use this_cpu_{inc,dec}() for read_count

2020-09-16T14:26:56+00:00

The __this_cpu*() accessors are (in general) IRQ-unsafe which, given
that percpu-rwsem is a blocking primitive, should be just fine.

However, file_end_write() is used from IRQ context and will cause
load-store issues on architectures where the per-cpu accessors are not
natively irq-safe.

Fix it by using the IRQ-safe this_cpu_*() for operations on
read_count. This will generate more expensive code on a number of
platforms, which might cause a performance regression for some of the
other percpu-rwsem users.

If any such is reported, we can consider alternative solutions.

Fixes: 70fe2f48152e ("aio: fix freeze protection of aio writes")
Signed-off-by: Hou Tao 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Acked-by: Oleg Nesterov 
Link: https://lkml.kernel.org/r/20200915140750.137881-1-houtao1@huawei.com

locking/percpu-rwsem: Fix a task_struct refcount

2020-04-08T10:05:06+00:00

The following commit:

  7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")

introduced task_struct memory leaks due to messing up the task_struct
refcount.

At the beginning of percpu_rwsem_wake_function(), it calls get_task_struct(),
but if the trylock failed, it will remain in the waitqueue. However, it
will run percpu_rwsem_wake_function() again with get_task_struct() to
increase the refcount but then only call put_task_struct() once the trylock
succeeded.

Fix it by adjusting percpu_rwsem_wake_function() a bit to guard against
when percpu_rwsem_wait() observing !private, terminating the wait and
doing a quick exit() while percpu_rwsem_wake_function() then doing
wake_up_process(p) as a use-after-free.

Fixes: 7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")
Suggested-by: Peter Zijlstra 
Signed-off-by: Qian Cai 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Link: https://lkml.kernel.org/r/20200330213002.2374-1-cai@lca.pw