linux-stable.git/arch/x86/kernel/relocate_kernel_64.S, branch linux-rolling-stable

x86/kexec: Disable LASS during relocate kernel

2025-11-18T18:38:26+00:00

The relocate kernel mechanism uses an identity mapping to copy the new
kernel, which leads to a LASS violation when executing from a low
address.

LASS must be disabled after the original CR4 value is saved because
kexec paths that preserve context need to restore CR4.LASS. But,
disabling it along with CET during identity_mapped() is too late. So,
disable LASS immediately after saving CR4, along with PGE, and before
jumping to the identity-mapped page.

Signed-off-by: Sohil Mehta 
Signed-off-by: Dave Hansen 
Reviewed-by: Dave Hansen 
Link: https://patch.msgid.link/20251118182911.2983253-6-sohil.mehta%40intel.com

x86/sme: Use percpu boolean to control WBINVD during kexec

2025-09-05T17:40:40+00:00

TL;DR:

Prepare to unify how TDX and SME do cache flushing during kexec by
making a percpu boolean control whether to do the WBINVD.

-- Background --

On SME platforms, dirty cacheline aliases with and without encryption
bit can coexist, and the CPU can flush them back to memory in random
order. During kexec, the caches must be flushed before jumping to the
new kernel otherwise the dirty cachelines could silently corrupt the
memory used by the new kernel due to different encryption property.

TDX also needs a cache flush during kexec for the same reason. It would
be good to have a generic way to flush the cache instead of scattering
checks for each feature all around.

When SME is enabled, the kernel basically encrypts all memory including
the kernel itself and a simple memory write from the kernel could dirty
cachelines. Currently, the kernel uses WBINVD to flush the cache for
SME during kexec in two places:

1) the one in stop_this_cpu() for all remote CPUs when the kexec-ing CPU
stops them;
2) the one in the relocate_kernel() where the kexec-ing CPU jumps to the
new kernel.

-- Solution --

Unlike SME, TDX can only dirty cachelines when it is used (i.e., when
SEAMCALLs are performed). Since there are no more SEAMCALLs after the
aforementioned WBINVDs, leverage this for TDX.

To unify the approach for SME and TDX, use a percpu boolean to indicate
the cache may be in an incoherent state and needs flushing during kexec,
and set the boolean for SME. TDX can then leverage it.

While SME could use a global flag (since it's enabled at early boot and
enabled on all CPUs), the percpu flag fits TDX better:

The percpu flag can be set when a CPU makes a SEAMCALL, and cleared when
another WBINVD on the CPU obviates the need for a kexec-time WBINVD.
Saving kexec-time WBINVD is valuable, because there is an existing
race[*] where kexec could proceed while another CPU is active. WBINVD
could make this race worse, so it's worth skipping it when possible.

-- Side effect to SME --

Today the first WBINVD in the stop_this_cpu() is performed when SME is
*supported* by the platform, and the second WBINVD is done in
relocate_kernel() when SME is *activated* by the kernel. Make things
simple by changing to do the second WBINVD when the platform supports
SME. This allows the kernel to simply turn on this percpu boolean when
bringing up a CPU by checking whether the platform supports SME.

No other functional change intended.

[*] The aforementioned race:

During kexec native_stop_other_cpus() is called to stop all remote CPUs
before jumping to the new kernel. native_stop_other_cpus() firstly
sends normal REBOOT vector IPIs to stop remote CPUs and waits them to
stop. If that times out, it sends NMI to stop the CPUs that are still
alive. The race happens when native_stop_other_cpus() has to send NMIs
and could potentially result in the system hang (for more information
please see [1]).

Signed-off-by: Kai Huang
Signed-off-by: Paolo Bonzini
Signed-off-by: Dave Hansen
Reviewed-by: Tom Lendacky
Reviewed-by: Borislav Petkov (AMD)
Tested-by: Tom Lendacky
Link: https://lore.kernel.org/kvm/b963fcd60abe26c7ec5dc20b42f1a2ebbcc72397.1750934177.git.kai.huang@intel.com/ [1]
Link: https://lore.kernel.org/all/20250901160930.1785244-3-pbonzini%40redhat.com

x86/kexec: Consolidate relocate_kernel() function parameters

2025-09-05T17:40:40+00:00

During kexec, the kernel jumps to the new kernel in relocate_kernel(),
which is implemented in assembly and both 32-bit and 64-bit have their
own version.

Currently, for both 32-bit and 64-bit, the last two parameters of the
relocate_kernel() are both 'unsigned int' but actually they only convey
a boolean, i.e., one bit information.  The 'unsigned int' has enough
space to carry two bits information therefore there's no need to pass
the two booleans in two separate 'unsigned int'.

Consolidate the last two function parameters of relocate_kernel() into a
single 'unsigned int' and pass flags instead.

Only consolidate the 64-bit version albeit the similar optimization can
be done for the 32-bit version too.  Don't bother changing the 32-bit
version while it is working (since assembly code change is required).

Signed-off-by: Kai Huang 
Signed-off-by: Paolo Bonzini 
Signed-off-by: Dave Hansen 
Reviewed-by: Tom Lendacky 
Reviewed-by: Borislav Petkov (AMD) 
Reviewed-by: David Woodhouse 
Link: https://lore.kernel.org/all/20250901160930.1785244-2-pbonzini%40redhat.com

x86/asm: Replace "REP; NOP" with PAUSE mnemonic

2025-04-18T08:19:25+00:00

Current minimum required version of binutils is 2.25,
which supports PAUSE instruction mnemonic.

Replace "REP; NOP" with this proper mnemonic.

No functional change intended.

Reviewed-by: Nikolay Borisov 
Signed-off-by: Uros Bizjak 
Signed-off-by: Ingo Molnar 
Cc: H. Peter Anvin 
Link: https://lore.kernel.org/r/20250418080805.83679-2-ubizjak@gmail.com

x86/asm: Remove semicolon from "rep" prefixes

2025-04-18T07:33:33+00:00

Minimum version of binutils required to compile the kernel is 2.25.
This version correctly handles the "rep" prefixes, so it is possible
to remove the semicolon, which was used to support ancient versions
of GNU as.

Due to the semicolon, the compiler considers "rep; insn" (or its
alternate "rep\n\tinsn" form) as two separate instructions. Removing
the semicolon makes asm length calculations more accurate, consequently
making scheduling and inlining decisions of the compiler more accurate.

Removing the semicolon also enables assembler checks involving "rep"
prefixes. Trying to assemble e.g. "rep addl %eax, %ebx" results in:

  Error: invalid instruction `add' after `rep'

Signed-off-by: Uros Bizjak 
Signed-off-by: Ingo Molnar 
Cc: Andy Lutomirski 
Cc: Boris Ostrovsky 
Cc: Brian Gerst 
Cc: H. Peter Anvin 
Cc: Josh Poimboeuf 
Cc: Juergen Gross 
Cc: Kees Cook 
Cc: Linus Torvalds 
Cc: Pavel Machek 
Cc: Rafael J. Wysocki 
Link: https://lore.kernel.org/r/20250418071437.4144391-2-ubizjak@gmail.com

x86/kexec: Invalidate GDT/IDT from relocate_kernel() instead of earlier

2025-04-10T10:17:14+00:00

Reduce the window during which exceptions are unhandled, by leaving the
GDT/IDT in place all the way into the relocate_kernel() function, until
the moment that %cr3 gets replaced.

Signed-off-by: David Woodhouse 
Signed-off-by: Ingo Molnar 
Cc: Brian Gerst 
Cc: Juergen Gross 
Cc: H. Peter Anvin 
Cc: Linus Torvalds 
Cc: Josh Poimboeuf 
Cc: Kees Cook 
Cc: Ard Biesheuvel 
Link: https://lore.kernel.org/r/20250326142404.256980-4-dwmw2@infradead.org

x86/kexec: Add 8250 MMIO serial port output

2025-04-10T10:17:14+00:00

This supports the same 32-bit MMIO-mapped 8250 as the early_printk code.

It's not clear why the early_printk code supports this form and only this
form; the actual runtime 8250_pci doesn't seem to support it. But having
hacked up QEMU to expose such a device, early_printk does work with it,
and now so does the kexec debug code.

Signed-off-by: David Woodhouse 
Signed-off-by: Ingo Molnar 
Cc: Andy Shevchenko 
Cc: Brian Gerst 
Cc: Juergen Gross 
Cc: H. Peter Anvin 
Cc: Linus Torvalds 
Cc: Josh Poimboeuf 
Cc: Kees Cook 
Cc: Ard Biesheuvel 
Link: https://lore.kernel.org/r/20250326142404.256980-3-dwmw2@infradead.org

x86/kexec: Add 8250 serial port output

2025-04-10T10:17:13+00:00

If a serial port was configured for early_printk, use it for debug output
from the relocate_kernel exception handler too.

Signed-off-by: David Woodhouse 
Signed-off-by: Ingo Molnar 
Cc: Andy Shevchenko 
Cc: Brian Gerst 
Cc: Juergen Gross 
Cc: H. Peter Anvin 
Cc: Linus Torvalds 
Cc: Josh Poimboeuf 
Cc: Kees Cook 
Cc: Ard Biesheuvel 
Link: https://lore.kernel.org/r/20250326142404.256980-2-dwmw2@infradead.org

x86/kexec: Debugging support: Dump registers on exception

2025-03-25T11:49:05+00:00

The actual serial output function is a no-op for now.

Signed-off-by: David Woodhouse 
Signed-off-by: Ingo Molnar 
Cc: Brian Gerst 
Cc: Juergen Gross 
Cc: H. Peter Anvin 
Cc: Linus Torvalds 
Cc: Josh Poimboeuf 
Cc: Kees Cook 
Link: https://lore.kernel.org/r/20250314173226.3062535-3-dwmw2@infradead.org

x86/kexec: Debugging support: Load an IDT and basic exception entry points

2025-03-25T11:49:05+00:00

[ mingo: Minor readability edits ]

Signed-off-by: David Woodhouse 
Signed-off-by: Ingo Molnar 
Cc: Brian Gerst 
Cc: Juergen Gross 
Cc: H. Peter Anvin 
Cc: Linus Torvalds 
Cc: Josh Poimboeuf 
Cc: Kees Cook 
Link: https://lore.kernel.org/r/20250314173226.3062535-2-dwmw2@infradead.org