linux.git/arch/powerpc/kernel/interrupt_64.S, branch v6.8

powerpc/64: Set task pt_regs->link to the LR value on scv entry

2024-02-05T11:43:19+00:00

Nysal reported that userspace backtraces are missing in offcputime bcc
tool. As an example:
    $ sudo ./bcc/tools/offcputime.py -uU
    Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.

    ^C
	write
	-                python (9107)
	    8

	write
	-                sudo (9105)
	    9

	mmap
	-                python (9107)
	    16

	clock_nanosleep
	-                multipathd (697)
	    3001604

The offcputime bcc tool attaches a bpf program to a kprobe on
finish_task_switch(), which is usually hit on a syscall from userspace.
With the switch to system call vectored, we started setting
pt_regs->link to zero. This is because system call vectored behaves like
a function call with LR pointing to the system call return address, and
with no modification to SRR0/SRR1. The LR value does indicate our next
instruction, so it is being saved as pt_regs->nip, and pt_regs->link is
being set to zero. This is not a problem by itself, but BPF uses perf
callchain infrastructure for capturing stack traces, and that stores LR
as the second entry in the stack trace. perf has code to cope with the
second entry being zero, and skips over it. However, generic userspace
unwinders assume that a zero entry indicates end of the stack trace,
resulting in a truncated userspace stack trace.

Rather than fixing all userspace unwinders to ignore/skip past the
second entry, store the real LR value in pt_regs->link so that there
continues to be a valid, though duplicate entry in the stack trace.

With this change:
    $ sudo ./bcc/tools/offcputime.py -uU
    Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.

    ^C
	write
	write
	[unknown]
	[unknown]
	[unknown]
	[unknown]
	[unknown]
	PyObject_VectorcallMethod
	[unknown]
	[unknown]
	PyObject_CallOneArg
	PyFile_WriteObject
	PyFile_WriteString
	[unknown]
	[unknown]
	PyObject_Vectorcall
	_PyEval_EvalFrameDefault
	PyEval_EvalCode
	[unknown]
	[unknown]
	[unknown]
	_PyRun_SimpleFileObject
	_PyRun_AnyFileObject
	Py_RunMain
	[unknown]
	Py_BytesMain
	[unknown]
	__libc_start_main
	-                python (1293)
	    7

	write
	write
	[unknown]
	sudo_ev_loop_v1
	sudo_ev_dispatch_v1
	[unknown]
	[unknown]
	[unknown]
	[unknown]
	__libc_start_main
	-                sudo (1291)
	    7

	syscall
	syscall
	bpf_open_perf_buffer_opts
	[unknown]
	[unknown]
	[unknown]
	[unknown]
	_PyObject_MakeTpCall
	PyObject_Vectorcall
	_PyEval_EvalFrameDefault
	PyEval_EvalCode
	[unknown]
	[unknown]
	[unknown]
	_PyRun_SimpleFileObject
	_PyRun_AnyFileObject
	Py_RunMain
	[unknown]
	Py_BytesMain
	[unknown]
	__libc_start_main
	-                python (1293)
	    11

	clock_nanosleep
	clock_nanosleep
	nanosleep
	sleep
	[unknown]
	[unknown]
	__clone
	-                multipathd (698)
	    3001661

Fixes: 7fa95f9adaee ("powerpc/64s: system call support for scv/rfscv instructions")
Cc: stable@vger.kernel.org
Reported-by: "Nysal Jan K.A" 
Signed-off-by: Naveen N Rao 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20240202154316.395276-1-naveen@kernel.org

powerpc: Fix merge conflict between pcrel and copy_thread changes

2023-04-26T06:20:16+00:00

Fix a conflict between commit 4e991e3c16a35 ("powerpc: add CFUNC
assembly label annotation") and commit b504b6aade040 ("powerpc:
differentiate kthread from user kernel thread start").

Fixes: 4e991e3c16a35 ("powerpc: add CFUNC assembly label annotation")
Fixes: b504b6aade040 ("powerpc: differentiate kthread from user kernel thread start")
Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230426055848.402993-2-npiggin@gmail.com

powerpc: add CFUNC assembly label annotation

2023-04-20T02:54:24+00:00

This macro is to be used in assembly where C functions are called.
pcrel addressing mode requires branches to functions with a
localentry value of 1 to have either a trailing nop or @notoc.
This macro permits the latter without changing callers.

Signed-off-by: Nicholas Piggin 
[mpe: Add dummy definitions to fix selftests build]
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230408021752.862660-5-npiggin@gmail.com

powerpc: copy_thread don't set _TIF_RESTOREALL

2023-04-11T13:13:33+00:00

In the kernel user thread path, don't set _TIF_RESTOREALL because
the thread is required to call kernel_execve() before it returns,
which will set _TIF_RESTOREALL if necessary via start_thread().

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230325122904.2375060-8-npiggin@gmail.com

powerpc: differentiate kthread from user kernel thread start

2023-04-11T13:13:33+00:00

Kernel created user threads start similarly to kernel threads in that
they call a kernel function after first returning from _switch, so
they share ret_from_kernel_thread for this. Kernel threads never return
from that function though, whereas user threads often do (although some
don't, e.g., IO threads).

Split these startup functions in two, and catch kernel threads that
improperly return from their function. This is intended to make the
complicated code a little bit easier to understand.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230325122904.2375060-7-npiggin@gmail.com

powerpc/64: ret_from_fork avoid restoring regs twice

2023-04-11T13:13:32+00:00

If the system call return path always restores NVGPRs then there is no
need for ret_from_fork to do it. The HANDLER_RESTORE_NVGPRS does the
right thing for this.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230325122904.2375060-5-npiggin@gmail.com

powerpc: use switch frame for ret_from_kernel_thread parameters

2023-04-11T13:13:32+00:00

The kernel thread path in copy_thread creates a user interrupt frame on
stack and stores the function and arg parameters there, and
ret_from_kernel_thread loads them. This is a slightly confusing way to
overload that frame. Non-volatile registers are loaded from the switch
frame, so the parameters can be stored there. The user interrupt frame
is now only used by user threads when they return to user.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://msgid.link/20230325122904.2375060-4-npiggin@gmail.com

powerpc/64s: Zeroise gprs on interrupt routine entry on Book3S

2022-12-02T09:46:05+00:00

Zeroise user state in gprs (assign to zero) to reduce the influence of user
registers on speculation within kernel syscall handlers. Clears occur
at the very beginning of the sc and scv 0 interrupt handlers, with
restores occurring following the execution of the syscall handler.

Zeroise GPRS r0, r2-r11, r14-r31, on entry into the kernel for all
other interrupt sources. The remaining gprs are overwritten by
entry macros to interrupt handlers, irrespective of whether or not a
given handler consumes these register values. If an interrupt does not
select the IMSR_R12 IOption, zeroise r12.

Prior to this commit, r14-r31 are restored on a per-interrupt basis at
exit, but now they are always restored on 64bit Book3S. Remove explicit
REST_NVGPRS invocations on 64-bit Book3S. 32-bit systems do not clear
user registers on interrupt, and continue to depend on the return value
of interrupt_exit_user_prepare to determine whether or not to restore
non-volatiles.

The mmap_bench benchmark in selftests should rapidly invoke pagefaults.
See ~0.8% performance regression with this mitigation, but this
indicates the worst-case performance due to heavier-weight interrupt
handlers. This mitigation is able to be enabled/disabled through
CONFIG_INTERRUPT_SANITIZE_REGISTERS.

Reviewed-by: Nicholas Piggin 
Signed-off-by: Rohan McLure 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20221201071019.1953023-5-rmclure@linux.ibm.com

powerpc/64: Sanitise common exit code for interrupts

2022-12-02T09:46:01+00:00

Interrupt code is shared between Book3E/S 64-bit systems for interrupt
handlers. Ensure that exit code correctly restores non-volatile gprs on
each system when CONFIG_INTERRUPT_SANITIZE_REGISTERS is enabled.

Also introduce macros for clearing/restoring registers on interrupt
entry for when this configuration option is either disabled or enabled.

Reviewed-by: Nicholas Piggin 
Signed-off-by: Rohan McLure 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20221201071019.1953023-3-rmclure@linux.ibm.com

powerpc: add a definition for the marker offset within the interrupt frame

2022-12-02T06:54:08+00:00

Define a constant rather than open-code the offset for the
"regs" marker.

Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20221127124942.1665522-9-npiggin@gmail.com