linux.git/kernel/unwind/deferred.c, branch v7.0

Convert more 'alloc_obj' cases to default GFP_KERNEL arguments

2026-02-22T04:03:00+00:00

This converts some of the visually simpler cases that have been split
over multiple lines.  I only did the ones that are easy to verify the
resulting diff by having just that final GFP_KERNEL argument on the next
line.

Somebody should probably do a proper coccinelle script for this, but for
me the trivial script actually resulted in an assertion failure in the
middle of the script.  I probably had made it a bit _too_ trivial.

So after fighting that far a while I decided to just do some of the
syntactically simpler cases with variations of the previous 'sed'
scripts.

The more syntactically complex multi-line cases would mostly really want
whitespace cleanup anyway.

Signed-off-by: Linus Torvalds

treewide: Replace kmalloc with kmalloc_obj for non-scalar types

2026-02-21T09:02:28+00:00

This is the result of running the Coccinelle script from
scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to
avoid scalar types (which need careful case-by-case checking), and
instead replace kmalloc-family calls that allocate struct or union
object instances:

Single allocations:	kmalloc(sizeof(TYPE), ...)
are replaced with:	kmalloc_obj(TYPE, ...)

Array allocations:	kmalloc_array(COUNT, sizeof(TYPE), ...)
are replaced with:	kmalloc_objs(TYPE, COUNT, ...)

Flex array allocations:	kmalloc(struct_size(PTR, FAM, COUNT), ...)
are replaced with:	kmalloc_flex(*PTR, FAM, COUNT, ...)

(where TYPE may also be *VAR)

The resulting allocations no longer return "void *", instead returning
"TYPE *".

Signed-off-by: Kees Cook

unwind: Make unwind_task_info::unwind_mask consistent

2025-10-29T09:29:57+00:00

The unwind_task_info::unwind_mask was manipulated using a mixture of:

  regular store
  WRITE_ONCE()
  try_cmpxchg()
  set_bit()
  atomic_long_*()

Clean up and make it consistently atomic_long_t.

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://patch.msgid.link/20250924080119.384384486@infradead.org

unwind: Simplify unwind_user_faultable()

2025-10-29T09:29:56+00:00

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://patch.msgid.link/20250924080119.271671514@infradead.org

unwind: Clarify calling context

2025-10-29T09:29:56+00:00

The get_cookie() function hard relies on IRQs being disabled, but this
isn't immediately obvious when reading the function.

Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Steven Rostedt (Google) 
Link: https://patch.msgid.link/20250924080119.122507632@infradead.org

unwind: Fix unwind_deferred_request() vs NMI

2025-10-29T09:29:56+00:00

task_work_add(RWA_RESUME) isn't NMI-safe, use TWA_NMI_CURRENT when
used from NMI context.

Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Steven Rostedt (Google) 
Link: https://patch.msgid.link/20250924080119.005422353@infradead.org

unwind: Finish up unwind when a task exits

2025-07-31T14:20:11+00:00

On do_exit() when a task is exiting, if a unwind is requested and the
deferred user stacktrace is deferred via the task_work, the task_work
callback is called after exit_mm() is called in do_exit(). This means that
the user stack trace will not be retrieved and an empty stack is created.

Instead, add a function unwind_deferred_task_exit() and call it just
before exit_mm() so that the unwinder can call the requested callbacks
with the user space stack.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.504259474@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind deferred: Use SRCU unwind_deferred_task_work()

2025-07-31T14:20:11+00:00

Instead of using the callback_mutex to protect the link list of callbacks
in unwind_deferred_task_work(), use SRCU instead. This gets called every
time a task exits that has to record a stack trace that was requested.
This can happen for many tasks on several CPUs at the same time. A mutex
is a bottleneck and can cause a bit of contention and slow down performance.

As the callbacks themselves are allowed to sleep, regular RCU cannot be
used to protect the list. Instead use SRCU, as that still allows the
callbacks to sleep and the list can be read without needing to hold the
callback_mutex.

Link: https://lore.kernel.org/all/ca9bd83a-6c80-4ee0-a83c-224b9d60b755@efficios.com/

Cc: "Paul E. McKenney" 
Cc: Masami Hiramatsu 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.331548065@kernel.org
Suggested-by: Mathieu Desnoyers 
Signed-off-by: Steven Rostedt (Google)

unwind: Add USED bit to only have one conditional on way back to user space

2025-07-31T14:20:11+00:00

On the way back to user space, the function unwind_reset_info() is called
unconditionally (but always inlined). It currently has two conditionals.
One that checks the unwind_mask which is set whenever a deferred trace is
called and is used to know that the mask needs to be cleared. The other
checks if the cache has been allocated, and if so, it resets the
nr_entries so that the unwinder knows it needs to do the work to get a new
user space stack trace again (it only does it once per entering the
kernel).

Use one of the bits in the unwind mask as a "USED" bit that gets set
whenever a trace is created. This will make it possible to only check the
unwind_mask in the unwind_reset_info() to know if it needs to do work or
not and eliminates a conditional that happens every time the task goes
back to user space.

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182406.155422551@kernel.org
Signed-off-by: Steven Rostedt (Google)

unwind deferred: Add unwind_completed mask to stop spurious callbacks

2025-07-31T14:20:10+00:00

If there's more than one registered tracer to the unwind deferred
infrastructure, it is currently possible that one tracer could cause extra
callbacks to happen for another tracer if the former requests a deferred
stacktrace after the latter's callback was executed and before the task
went back to user space.

Here's an example of how this could occur:

  [Task enters kernel]
    tracer 1 request -> add cookie to its buffer
    tracer 1 request -> add cookie to its buffer
    <..>
    [ task work executes ]
    tracer 1 callback -> add trace + cookie to its buffer

    [tracer 2 requests and triggers the task work again]
    [ task work executes again ]
    tracer 1 callback -> add trace + cookie to its buffer
    tracer 2 callback -> add trace + cookie to its buffer
 [Task exits back to user space]

This is because the bit for tracer 1 gets set in the task's unwind_mask
when it did its request and does not get cleared until the task returns
back to user space. But if another tracer were to request another deferred
stacktrace, then the next task work will executed all tracer's callbacks
that have their bits set in the task's unwind_mask.

To fix this issue, add another mask called unwind_completed and place it
into the task's info->cache structure. The cache structure is allocated
on the first occurrence of a deferred stacktrace and this unwind_completed
mask is not needed until then. It's better to have it in the cache than to
permanently waste space in the task_struct.

After a tracer's callback is executed, it's bit gets set in this
unwind_completed mask. When the task_work enters, it will AND the task's
unwind_mask with the inverse of the unwind_completed which will eliminate
any work that already had its callback executed since the task entered the
kernel.

When the task leaves the kernel, it will reset this unwind_completed mask
just like it resets the other values as it enters user space.

Link: https://lore.kernel.org/all/20250716142609.47f0e4a5@batman.local.home/

Cc: Masami Hiramatsu 
Cc: Mathieu Desnoyers 
Cc: Josh Poimboeuf 
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Namhyung Kim 
Cc: Thomas Gleixner 
Cc: Andrii Nakryiko 
Cc: Indu Bhagat 
Cc: "Jose E. Marchesi" 
Cc: Beau Belgrave 
Cc: Jens Remus 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Jens Axboe 
Cc: Florian Weimer 
Cc: Sam James 
Link: https://lore.kernel.org/20250729182405.989222722@kernel.org
Signed-off-by: Steven Rostedt (Google)