linux.git/arch/arm64/include/asm/processor.h, branch v6.8

arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD

2023-12-12T14:31:55+00:00

Now that kernel mode FPSIMD state is context switched along with other
task state, we can enable the existing logic that keeps track of which
task's FPSIMD state the CPU is holding in its registers. If it is the
context of the task that we are switching to, we can elide the reload of
the FPSIMD state from memory.

Note that we also need to check whether the FPSIMD state on this CPU is
the most recent: if a task gets migrated away and back again, the state
in memory may be more recent than the state in the CPU. So add another
CPU id field to task_struct to keep track of this. (We could reuse the
existing CPU id field used for user mode context, but that might result
in user state to be discarded unnecessarily, given that two distinct
CPUs could be holding the most recent user mode state and the most
recent kernel mode state)

Signed-off-by: Ard Biesheuvel 
Reviewed-by: Mark Brown 
Acked-by: Mark Rutland 
Link: https://lore.kernel.org/r/20231208113218.3001940-9-ardb@google.com
Signed-off-by: Will Deacon

arm64: fpsimd: Preserve/restore kernel mode NEON at context switch

2023-12-12T14:31:54+00:00

Currently, the FPSIMD register file is not preserved and restored along
with the general registers on exception entry/exit or context switch.
For this reason, we disable preemption when enabling FPSIMD for kernel
mode use in task context, and suspend the processing of softirqs so that
there are no concurrent uses in the kernel. (Kernel mode FPSIMD may not
be used at all in other contexts).

Disabling preemption while doing CPU intensive work on inputs of
potentially unbounded size is bad for real-time performance, which is
why we try and ensure that SIMD crypto code does not operate on more
than ~4k at a time, which is an arbitrary limit and requires assembler
code to implement efficiently.

We can avoid the need for disabling preemption if we can ensure that any
in-kernel users of the NEON will not lose the FPSIMD register state
across a context switch. And given that disabling softirqs implicitly
disables preemption as well, we will also have to ensure that a softirq
that runs code using FPSIMD can safely interrupt an in-kernel user.

So introduce a thread_info flag TIF_KERNEL_FPSTATE, and modify the
context switch hook for FPSIMD to preserve and restore the kernel mode
FPSIMD to/from struct thread_struct when it is set. This avoids any
scheduling blackouts due to prolonged use of FPSIMD in kernel mode,
without the need for manual yielding.

In order to support softirq processing while FPSIMD is being used in
kernel task context, use the same flag to decide whether the kernel mode
FPSIMD state needs to be preserved and restored before allowing FPSIMD
to be used in softirq context.

Signed-off-by: Ard Biesheuvel 
Reviewed-by: Mark Brown 
Reviewed-by: Mark Rutland 
Link: https://lore.kernel.org/r/20231208113218.3001940-8-ardb@google.com
Signed-off-by: Will Deacon

locking: remove spin_lock_prefetch

2023-08-12T16:18:47+00:00

The only remaining consumer is new_inode, where it showed up in 2001 as
commit c37fa164f793 ("v2.4.9.9 -> v2.4.9.10") in a historical repo [1]
with a changelog which does not mention it.

Since then the line got only touched up to keep compiling.

While it may have been of benefit back in the day, it is guaranteed to
at best not get in the way in the multicore setting -- as the code
performs *a lot* of work between the prefetch and actual lock acquire,
any contention means the cacheline is already invalid by the time the
routine calls spin_lock().  It adds spurious traffic, for short.

On top of it prefetch is notoriously tricky to use for single-threaded
purposes, making it questionable from the get go.

As such, remove it.

I admit upfront I did not see value in benchmarking this change, but I
can do it if that is deemed appropriate.

Removal from new_inode and of the entire thing are in the same patch as
requested by Linus, so whatever weird looks can be directed at that guy.

Link: https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/fs/inode.c?id=c37fa164f793735b32aa3f53154ff1a7659e6442 [1]
Signed-off-by: Mateusz Guzik 
Signed-off-by: Linus Torvalds

arm64/sme: Rename za_state to sme_state

2023-01-20T12:23:05+00:00

In preparation for adding support for storage for ZT0 to the thread_struct
rename za_state to sme_state. Since ZT0 is accessible when PSTATE.ZA is
set just like ZA itself we will extend the allocation done for ZA to
cover it, avoiding the need to further expand task_struct for non-SME
tasks.

No functional changes.

Signed-off-by: Mark Brown 
Link: https://lore.kernel.org/r/20221208-arm64-sme2-v4-1-f2fa0aef982f@kernel.org
Signed-off-by: Catalin Marinas

Merge branch 'for-next/trivial' into for-next/core

2022-12-06T11:33:29+00:00

* for-next/trivial:
  arm64: alternatives: add __init/__initconst to some functions/variables
  arm64/asm: Remove unused assembler DAIF save/restore macros
  arm64/kpti: Move DAIF masking to C code
  Revert "arm64/mm: Drop redundant BUG_ON(!pgtable_alloc)"
  arm64/mm: Drop unused restore_ttbr1
  arm64: alternatives: make apply_alternatives_vdso() static
  arm64/mm: Drop idmap_pg_end[] declaration
  arm64/mm: Drop redundant BUG_ON(!pgtable_alloc)
  arm64: make is_ttbrX_addr() noinstr-safe
  arm64/signal: Document our convention for choosing magic numbers
  arm64: atomics: lse: remove stale dependency on JUMP_LABEL
  arm64: paravirt: remove conduit check in has_pv_steal_clock
  arm64: entry: Fix typo
  arm64/booting: Add missing colon to FA64 entry
  arm64/mm: Drop ARM64_KERNEL_USES_PMD_MAPS
  arm64/asm: Remove unused enable_da macro

Merge branch 'for-next/sve-state' into for-next/core

2022-12-06T11:27:28+00:00

* for-next/sve-state:
  arm64/fp: Use a struct to pass data to fpsimd_bind_state_to_cpu()
  arm64/sve: Leave SVE enabled on syscall if we don't context switch
  arm64/fpsimd: SME no longer requires SVE register state
  arm64/fpsimd: Load FP state based on recorded data type
  arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM
  arm64/fpsimd: Have KVM explicitly say which FP registers to save
  arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE
  KVM: arm64: Discard any SVE state when entering KVM guests

arm64/fpsimd: Have KVM explicitly say which FP registers to save

2022-11-29T15:01:56+00:00

In order to avoid needlessly saving and restoring the guest registers KVM
relies on the host FPSMID code to save the guest registers when we context
switch away from the guest. This is done by binding the KVM guest state to
the CPU on top of the task state that was originally there, then carefully
managing the TIF_SVE flag for the task to cause the host to save the full
SVE state when needed regardless of the needs of the host task. This works
well enough but isn't terribly direct about what is going on and makes it
much more complicated to try to optimise what we're doing with the SVE
register state.

Let's instead have KVM pass in the register state it wants saving when it
binds to the CPU. We introduce a new FP_STATE_CURRENT for use
during normal task binding to indicate that we should base our
decisions on the current task. This should not be used when
actually saving. Ideally we might want to use a separate enum for
the type to save but this enum and the enum values would then
need to be named which has problems with clarity and ambiguity.

In order to ease any future debugging that might be required this patch
does not actually update any of the decision making about what to save,
it merely starts tracking the new information and warns if the requested
state is not what we would otherwise have decided to save.

Signed-off-by: Mark Brown 
Reviewed-by: Catalin Marinas 
Reviewed-by: Marc Zyngier 
Link: https://lore.kernel.org/r/20221115094640.112848-4-broonie@kernel.org
Signed-off-by: Will Deacon

arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE

2022-11-29T15:01:56+00:00

When we save the state for the floating point registers this can be done
in the form visible through either the FPSIMD V registers or the SVE Z and
P registers. At present we track which format is currently used based on
TIF_SVE and the SME streaming mode state but particularly in the SVE case
this limits our options for optimising things, especially around syscalls.
Introduce a new enum which we place together with saved floating point
state in both thread_struct and the KVM guest state which explicitly
states which format is active and keep it up to date when we change it.

At present we do not use this state except to verify that it has the
expected value when loading the state, future patches will introduce
functional changes.

Signed-off-by: Mark Brown 
Reviewed-by: Catalin Marinas 
Reviewed-by: Marc Zyngier 
Link: https://lore.kernel.org/r/20221115094640.112848-3-broonie@kernel.org
Signed-off-by: Will Deacon

arm64: move on_thread_stack() to

2022-11-18T14:36:47+00:00

Currently on_thread_stack() is defined in , depending
upon definitiong from  despite this header not being
included. This ends up being fragile, and any user of on_thread_stack()
must include both  and .

We organised things this way due to header dependencies back in commit:

  0b3e336601b82c6a ("arm64: Add support for STACKLEAK gcc plugin")

... but now that we no longer use current_top_of_stack(), and given that
stackleak includes  via , we no
longer need the definition to live in .

Move on_thread_stack() to , where all its dependencies
are guaranteed to be defined. This requires having arm64's irq.c
explicitly include , and I've taken the opportunity to
sort the includes, which were slightly out of order.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Cc: Catalin Marinas 
Cc: Kees Cook 
Cc: Will Deacon 
Reviewed-by: Kees Cook 
Reviewed-by: Mark Brown 
Link: https://lore.kernel.org/r/20221117120902.3974163-3-mark.rutland@arm.com
Signed-off-by: Will Deacon

arm64: remove current_top_of_stack()

2022-11-18T14:36:47+00:00

We no longer use current_top_of_stack() on arm64, so it can be removed.

We introduced current_top_of_stack() for STACKLEAK in commit:

  0b3e336601b82c6a ("arm64: Add support for STACKLEAK gcc plugin")

... then we figured out the intended semantics were unclear, and
reworked it in commit:

  e85094c31ddb794a ("arm64: stackleak: fix current_top_of_stack()")

... then we removed the only user in commit:

  0cfa2ccd285d98ad ("stackleak: rework stack high bound handling")

Given that it's no longer used, and it's very easy to misuse, this patch
removes current_top_of_stack(). For the moment, on_thread_stack() is
left where it is as moving it will change some header dependencies.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Cc: Catalin Marinas 
Cc: Kees Cook 
Cc: Will Deacon 
Reviewed-by: Kees Cook 
Reviewed-by: Mark Brown 
Link: https://lore.kernel.org/r/20221117120902.3974163-2-mark.rutland@arm.com
Signed-off-by: Will Deacon