linux-stable.git/arch/x86/lib, branch linux-6.3.y Linux kernel stable tree x86: fix clear_user_rep_good() exception handling annotation 2023-05-17T12:02:08+00:00 Linus Torvalds torvalds@linux-foundation.org 2023-05-14T22:46:19+00:00 b805d212c394f291f116b12c53401e7ba0c4d408 This code no longer exists in mainline, because it was removed in commit d2c95f9d6802 ("x86: don't use REP_GOOD or ERMS for user memory clearing") upstream. However, rather than backport the full range of x86 memory clearing and copying cleanups, fix the exception table annotation placement for the final 'rep movsb' in clear_user_rep_good(): rather than pointing at the actual instruction that did the user space access, it pointed to the register move just before it. That made sense from a code flow standpoint, but not from an actual usage standpoint: it means that if user access takes an exception, the exception handler won't actually find the instruction in the exception tables. As a result, rather than fixing it up and returning -EFAULT, it would then turn it into a kernel oops report instead, something like: BUG: unable to handle page fault for address: 0000000020081000 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page ... RIP: 0010:clear_user_rep_good+0x1c/0x30 arch/x86/lib/clear_page_64.S:147 ... Call Trace: __clear_user arch/x86/include/asm/uaccess_64.h:103 [inline] clear_user arch/x86/include/asm/uaccess_64.h:124 [inline] iov_iter_zero+0x709/0x1290 lib/iov_iter.c:800 iomap_dio_hole_iter fs/iomap/direct-io.c:389 [inline] iomap_dio_iter fs/iomap/direct-io.c:440 [inline] __iomap_dio_rw+0xe3d/0x1cd0 fs/iomap/direct-io.c:601 iomap_dio_rw+0x40/0xa0 fs/iomap/direct-io.c:689 ext4_dio_read_iter fs/ext4/file.c:94 [inline] ext4_file_read_iter+0x4be/0x690 fs/ext4/file.c:145 call_read_iter include/linux/fs.h:2183 [inline] do_iter_readv_writev+0x2e0/0x3b0 fs/read_write.c:733 do_iter_read+0x2f2/0x750 fs/read_write.c:796 vfs_readv+0xe5/0x150 fs/read_write.c:916 do_preadv+0x1b6/0x270 fs/read_write.c:1008 __do_sys_preadv2 fs/read_write.c:1070 [inline] __se_sys_preadv2 fs/read_write.c:1061 [inline] __x64_sys_preadv2+0xef/0x150 fs/read_write.c:1061 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd which then looks like a filesystem bug rather than the incorrect exception annotation that it is. [ The alternative to this one-liner fix is to take the upstream series that cleans this all up: 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small memory copies") 20f3337d350c ("x86: don't use REP_GOOD or ERMS for small memory clearing") adfcf4231b8c ("x86: don't use REP_GOOD or ERMS for user memory copies") * d2c95f9d6802 ("x86: don't use REP_GOOD or ERMS for user memory clearing") 3639a535587d ("x86: move stac/clac from user copy routines into callers") 577e6a7fd50d ("x86: inline the 'rep movs' in user copies for the FSRM case") 8c9b6a88b7e2 ("x86: improve on the non-rep 'clear_user' function") 427fda2c8a49 ("x86: improve on the non-rep 'copy_user' function") * e046fe5a36a9 ("x86: set FSRS automatically on AMD CPUs that have FSRM") e1f2750edc4a ("x86: remove 'zerorest' argument from __copy_user_nocache()") 034ff37d3407 ("x86: rewrite '__copy_user_nocache' function") with either the whole series or at a minimum the two marked commits being needed to fix this issue ] Reported-by: syzbot <syzbot+401145a9a237779feb26@syzkaller.appspotmail.com> Link: https://syzkaller.appspot.com/bug?extid=401145a9a237779feb26 Fixes: 0db7058e8e23 ("x86/clear_user: Make it faster") Cc: Borislav Petkov <bp@alien8.de> Cc: stable@kernel.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
This code no longer exists in mainline, because it was removed in
commit d2c95f9d6802 ("x86: don't use REP_GOOD or ERMS for user memory
clearing") upstream.

However, rather than backport the full range of x86 memory clearing and
copying cleanups, fix the exception table annotation placement for the
final 'rep movsb' in clear_user_rep_good(): rather than pointing at the
actual instruction that did the user space access, it pointed to the
register move just before it.

That made sense from a code flow standpoint, but not from an actual
usage standpoint: it means that if user access takes an exception, the
exception handler won't actually find the instruction in the exception
tables.

As a result, rather than fixing it up and returning -EFAULT, it would
then turn it into a kernel oops report instead, something like:

    BUG: unable to handle page fault for address: 0000000020081000
    #PF: supervisor write access in kernel mode
    #PF: error_code(0x0002) - not-present page
    ...
    RIP: 0010:clear_user_rep_good+0x1c/0x30 arch/x86/lib/clear_page_64.S:147
    ...
    Call Trace:
      __clear_user arch/x86/include/asm/uaccess_64.h:103 [inline]
      clear_user arch/x86/include/asm/uaccess_64.h:124 [inline]
      iov_iter_zero+0x709/0x1290 lib/iov_iter.c:800
      iomap_dio_hole_iter fs/iomap/direct-io.c:389 [inline]
      iomap_dio_iter fs/iomap/direct-io.c:440 [inline]
      __iomap_dio_rw+0xe3d/0x1cd0 fs/iomap/direct-io.c:601
      iomap_dio_rw+0x40/0xa0 fs/iomap/direct-io.c:689
      ext4_dio_read_iter fs/ext4/file.c:94 [inline]
      ext4_file_read_iter+0x4be/0x690 fs/ext4/file.c:145
      call_read_iter include/linux/fs.h:2183 [inline]
      do_iter_readv_writev+0x2e0/0x3b0 fs/read_write.c:733
      do_iter_read+0x2f2/0x750 fs/read_write.c:796
      vfs_readv+0xe5/0x150 fs/read_write.c:916
      do_preadv+0x1b6/0x270 fs/read_write.c:1008
      __do_sys_preadv2 fs/read_write.c:1070 [inline]
      __se_sys_preadv2 fs/read_write.c:1061 [inline]
      __x64_sys_preadv2+0xef/0x150 fs/read_write.c:1061
      do_syscall_x64 arch/x86/entry/common.c:50 [inline]
      do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
      entry_SYSCALL_64_after_hwframe+0x63/0xcd

which then looks like a filesystem bug rather than the incorrect
exception annotation that it is.

[ The alternative to this one-liner fix is to take the upstream series
  that cleans this all up:

    68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small memory copies")
    20f3337d350c ("x86: don't use REP_GOOD or ERMS for small memory clearing")
    adfcf4231b8c ("x86: don't use REP_GOOD or ERMS for user memory copies")
  * d2c95f9d6802 ("x86: don't use REP_GOOD or ERMS for user memory clearing")
    3639a535587d ("x86: move stac/clac from user copy routines into callers")
    577e6a7fd50d ("x86: inline the 'rep movs' in user copies for the FSRM case")
    8c9b6a88b7e2 ("x86: improve on the non-rep 'clear_user' function")
    427fda2c8a49 ("x86: improve on the non-rep 'copy_user' function")
  * e046fe5a36a9 ("x86: set FSRS automatically on AMD CPUs that have FSRM")
    e1f2750edc4a ("x86: remove 'zerorest' argument from __copy_user_nocache()")
    034ff37d3407 ("x86: rewrite '__copy_user_nocache' function")

  with either the whole series or at a minimum the two marked commits
  being needed to fix this issue ]

Reported-by: syzbot <syzbot+401145a9a237779feb26@syzkaller.appspotmail.com>
Link: https://syzkaller.appspot.com/bug?extid=401145a9a237779feb26
Fixes: 0db7058e8e23 ("x86/clear_user: Make it faster")
Cc: Borislav Petkov <bp@alien8.de>
Cc: stable@kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
x86/retbleed: Fix return thunk alignment 2023-05-17T12:01:51+00:00 Borislav Petkov (AMD) bp@alien8.de 2023-05-12T21:12:26+00:00 e16be8fb7291568f5c1bbfc6367ba7e389eb0e79 commit 9a48d604672220545d209e9996c2a1edbb5637f6 upstream. SYM_FUNC_START_LOCAL_NOALIGN() adds an endbr leading to this layout (leaving only the last 2 bytes of the address): 3bff <zen_untrain_ret>: 3bff: f3 0f 1e fa endbr64 3c03: f6 test $0xcc,%bl 3c04 <__x86_return_thunk>: 3c04: c3 ret 3c05: cc int3 3c06: 0f ae e8 lfence However, "the RET at __x86_return_thunk must be on a 64 byte boundary, for alignment within the BTB." Use SYM_START instead. Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de> Reviewed-by: Thomas Gleixner <tglx@linutronix.de> Cc: <stable@kernel.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
commit 9a48d604672220545d209e9996c2a1edbb5637f6 upstream.

SYM_FUNC_START_LOCAL_NOALIGN() adds an endbr leading to this layout
(leaving only the last 2 bytes of the address):

  3bff <zen_untrain_ret>:
  3bff:       f3 0f 1e fa             endbr64
  3c03:       f6                      test   $0xcc,%bl

  3c04 <__x86_return_thunk>:
  3c04:       c3                      ret
  3c05:       cc                      int3
  3c06:       0f ae e8                lfence

However, "the RET at __x86_return_thunk must be on a 64 byte boundary,
for alignment within the BTB."

Use SYM_START instead.

Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Cc: <stable@kernel.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Merge tag 'x86_cpu_for_v6.3_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2023-02-21T22:51:40+00:00 Linus Torvalds torvalds@linux-foundation.org 2023-02-21T22:51:40+00:00 877934769e5b91798d304d4641647900ee614ce8 Pull x86 cpuid updates from Borislav Petkov: - Cache the AMD debug registers in per-CPU variables to avoid MSR writes where possible, when supporting a debug registers swap feature for SEV-ES guests - Add support for AMD's version of eIBRS called Automatic IBRS which is a set-and-forget control of indirect branch restriction speculation resources on privilege change - Add support for a new x86 instruction - LKGS - Load kernel GS which is part of the FRED infrastructure - Reset SPEC_CTRL upon init to accomodate use cases like kexec which rediscover - Other smaller fixes and cleanups * tag 'x86_cpu_for_v6.3_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: x86/amd: Cache debug register values in percpu variables KVM: x86: Propagate the AMD Automatic IBRS feature to the guest x86/cpu: Support AMD Automatic IBRS x86/cpu, kvm: Add the SMM_CTL MSR not present feature x86/cpu, kvm: Add the Null Selector Clears Base feature x86/cpu, kvm: Move X86_FEATURE_LFENCE_RDTSC to its native leaf x86/cpu, kvm: Add the NO_NESTED_DATA_BP feature KVM: x86: Move open-coded CPUID leaf 0x80000021 EAX bit propagation code x86/cpu, kvm: Add support for CPUID_80000021_EAX x86/gsseg: Add the new <asm/gsseg.h> header to <asm/asm-prototypes.h> x86/gsseg: Use the LKGS instruction if available for load_gs_index() x86/gsseg: Move load_gs_index() to its own new header file x86/gsseg: Make asm_load_gs_index() take an u16 x86/opcode: Add the LKGS instruction to x86-opcode-map x86/cpufeature: Add the CPU feature bit for LKGS x86/bugs: Reset speculation control settings on init x86/cpu: Remove redundant extern x86_read_arch_cap_msr() 
Pull x86 cpuid updates from Borislav Petkov:

 - Cache the AMD debug registers in per-CPU variables to avoid MSR
   writes where possible, when supporting a debug registers swap feature
   for SEV-ES guests

 - Add support for AMD's version of eIBRS called Automatic IBRS which is
   a set-and-forget control of indirect branch restriction speculation
   resources on privilege change

 - Add support for a new x86 instruction - LKGS - Load kernel GS which
   is part of the FRED infrastructure

 - Reset SPEC_CTRL upon init to accomodate use cases like kexec which
   rediscover

 - Other smaller fixes and cleanups

* tag 'x86_cpu_for_v6.3_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/amd: Cache debug register values in percpu variables
  KVM: x86: Propagate the AMD Automatic IBRS feature to the guest
  x86/cpu: Support AMD Automatic IBRS
  x86/cpu, kvm: Add the SMM_CTL MSR not present feature
  x86/cpu, kvm: Add the Null Selector Clears Base feature
  x86/cpu, kvm: Move X86_FEATURE_LFENCE_RDTSC to its native leaf
  x86/cpu, kvm: Add the NO_NESTED_DATA_BP feature
  KVM: x86: Move open-coded CPUID leaf 0x80000021 EAX bit propagation code
  x86/cpu, kvm: Add support for CPUID_80000021_EAX
  x86/gsseg: Add the new <asm/gsseg.h> header to <asm/asm-prototypes.h>
  x86/gsseg: Use the LKGS instruction if available for load_gs_index()
  x86/gsseg: Move load_gs_index() to its own new header file
  x86/gsseg: Make asm_load_gs_index() take an u16
  x86/opcode: Add the LKGS instruction to x86-opcode-map
  x86/cpufeature: Add the CPU feature bit for LKGS
  x86/bugs: Reset speculation control settings on init
  x86/cpu: Remove redundant extern x86_read_arch_cap_msr()
Merge tag 'x86-asm-2023-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2023-02-21T02:25:30+00:00 Linus Torvalds torvalds@linux-foundation.org 2023-02-21T02:25:30+00:00 6be3dafc7c1d323debb5910d697c5cd2cb19825f Pull x86 asm updates from Ingo Molnar: "Header fixes and a DocBook fix" * tag 'x86-asm-2023-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: x86/lib: Fix compiler and kernel-doc warnings x86/lib: Include <asm/misc.h> to fix a missing prototypes warning at build time 
Pull x86 asm updates from Ingo Molnar:
 "Header fixes and a DocBook fix"

* tag 'x86-asm-2023-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/lib: Fix compiler and kernel-doc warnings
  x86/lib: Include <asm/misc.h> to fix a missing prototypes warning at build time
Merge tag 'v6.2-rc6' into sched/core, to pick up fixes 2023-01-31T14:01:20+00:00 Ingo Molnar mingo@kernel.org 2023-01-31T14:01:20+00:00 57a30218fa25c469ed507964bbf028b7a064309a Pick up fixes before merging another batch of cpuidle updates. Signed-off-by: Ingo Molnar <mingo@kernel.org> 
Pick up fixes before merging another batch of cpuidle updates.

Signed-off-by: Ingo Molnar <mingo@kernel.org>
entry, kasan, x86: Disallow overriding mem*() functions 2023-01-13T10:48:17+00:00 Peter Zijlstra peterz@infradead.org 2023-01-12T19:43:58+00:00 69d4c0d3218692ffa56b0e1b9c76c50c699d7044 KASAN cannot just hijack the mem*() functions, it needs to emit __asan_mem*() variants if it wants instrumentation (other sanitizers already do this). vmlinux.o: warning: objtool: sync_regs+0x24: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: vc_switch_off_ist+0xbe: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: fixup_bad_iret+0x36: call to memset() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_get_ghcb+0xa0: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_put_ghcb+0x35: call to memcpy() leaves .noinstr.text section Remove the weak aliases to ensure nobody hijacks these functions and add them to the noinstr section. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Tested-by: Tony Lindgren <tony@atomide.com> Tested-by: Ulf Hansson <ulf.hansson@linaro.org> Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Acked-by: Frederic Weisbecker <frederic@kernel.org> Link: https://lore.kernel.org/r/20230112195542.028523143@infradead.org 
KASAN cannot just hijack the mem*() functions, it needs to emit
__asan_mem*() variants if it wants instrumentation (other sanitizers
already do this).

  vmlinux.o: warning: objtool: sync_regs+0x24: call to memcpy() leaves .noinstr.text section
  vmlinux.o: warning: objtool: vc_switch_off_ist+0xbe: call to memcpy() leaves .noinstr.text section
  vmlinux.o: warning: objtool: fixup_bad_iret+0x36: call to memset() leaves .noinstr.text section
  vmlinux.o: warning: objtool: __sev_get_ghcb+0xa0: call to memcpy() leaves .noinstr.text section
  vmlinux.o: warning: objtool: __sev_put_ghcb+0x35: call to memcpy() leaves .noinstr.text section

Remove the weak aliases to ensure nobody hijacks these functions and
add them to the noinstr section.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Tested-by: Tony Lindgren <tony@atomide.com>
Tested-by: Ulf Hansson <ulf.hansson@linaro.org>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://lore.kernel.org/r/20230112195542.028523143@infradead.org
x86/opcode: Add the LKGS instruction to x86-opcode-map 2023-01-12T12:06:36+00:00 H. Peter Anvin (Intel) hpa@zytor.com 2023-01-12T07:20:29+00:00 5a91f12660fe7249e37b11372bf599e02b6a319c Add the instruction opcode used by LKGS to x86-opcode-map. Opcode number is per public FRED draft spec v3.0. Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com> Signed-off-by: Xin Li <xin3.li@intel.com> Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: https://lore.kernel.org/r/20230112072032.35626-3-xin3.li@intel.com 
Add the instruction opcode used by LKGS to x86-opcode-map.

Opcode number is per public FRED draft spec v3.0.

Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Signed-off-by: Xin Li <xin3.li@intel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20230112072032.35626-3-xin3.li@intel.com
x86/lib: Fix compiler and kernel-doc warnings 2023-01-03T17:46:21+00:00 Anuradha Weeraman anuradha@debian.org 2023-01-03T11:47:24+00:00 65c24d7b0f9142c6acc9bd6dabeba22767077681 Fix the following W=1 warnings: arch/x86/lib/cmdline.c: - Include <asm/cmdline.h> to fix missing-prototypes warnings. - Update comment for __cmdline_find_option_bool to fix a kernel-doc warning. Signed-off-by: Anuradha Weeraman <anuradha@debian.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: https://lore.kernel.org/r/20230103114725.108431-1-anuradha@debian.org 
Fix the following W=1 warnings:

arch/x86/lib/cmdline.c:

  - Include <asm/cmdline.h> to fix missing-prototypes warnings.

  - Update comment for __cmdline_find_option_bool to fix a kernel-doc warning.

Signed-off-by: Anuradha Weeraman <anuradha@debian.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20230103114725.108431-1-anuradha@debian.org
x86/insn: Avoid namespace clash by separating instruction decoder MMIO type from MMIO trace type 2023-01-03T17:46:06+00:00 Jason A. Donenfeld Jason@zx2c4.com 2023-01-01T16:29:04+00:00 72bb8f8cc088730c4d84117a6906f458c2fc64bb Both <linux/mmiotrace.h> and <asm/insn-eval.h> define various MMIO_ enum constants, whose namespace overlaps. Rename the <asm/insn-eval.h> ones to have a INSN_ prefix, so that the headers can be used from the same source file. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: https://lore.kernel.org/r/20230101162910.710293-2-Jason@zx2c4.com 
Both <linux/mmiotrace.h> and <asm/insn-eval.h> define various MMIO_ enum constants,
whose namespace overlaps.

Rename the <asm/insn-eval.h> ones to have a INSN_ prefix, so that the headers can be
used from the same source file.

Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20230101162910.710293-2-Jason@zx2c4.com
x86/asm: Fix an assembler warning with current binutils 2023-01-03T16:55:11+00:00 Mikulas Patocka mpatocka@redhat.com 2023-01-03T15:24:11+00:00 55d235361fccef573990dfa5724ab453866e7816 Fix a warning: "found `movsd'; assuming `movsl' was meant" Signed-off-by: Mikulas Patocka <mpatocka@redhat.com> Signed-off-by: Ingo Molnar <mingo@kernel.org> Cc: linux-kernel@vger.kernel.org 
Fix a warning: "found `movsd'; assuming `movsl' was meant"

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: linux-kernel@vger.kernel.org