tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'x86_urgent_for_v6.5_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 fixes from Borislav Petkov:
"Extraordinary embargoed times call for extraordinary measures. That's
why this week's x86/urgent branch is larger than usual, containing all
the known fallout fixes after the SRSO mitigation got merged.

I know, it is a bit late in the game but everyone who has reported a
bug stemming from the SRSO pile, has tested that branch and has
confirmed that it fixes their bug.

Also, I've run it on every possible hardware I have and it is looking
good. It is running on this very machine while I'm typing, for 2 days
now without an issue. Famous last words...

- Use LEA ...%rsp instead of ADD %rsp in the Zen1/2 SRSO return
sequence as latter clobbers flags which interferes with fastop
emulation in KVM, leading to guests freezing during boot

- A fix for the DIV(0) quotient data leak on Zen1 to clear the
divider buffers at the right time

- Disable the SRSO mitigation on unaffected configurations as it got
enabled there unnecessarily

- Change .text section name to fix CONFIG_LTO_CLANG builds

- Improve the optprobe indirect jmp check so that certain
configurations can still be able to use optprobes at all

- A serious and good scrubbing of the untraining routines by PeterZ:
- Add proper speculation stopping traps so that objtool is happy
- Adjust objtool to handle the new thunks
- Make the thunk pointer assignable to the different untraining
sequences at runtime, thus avoiding the alternative at the
return thunk. It simplifies the code a bit too.
- Add a entry_untrain_ret() main entry point which selects the
respective untraining sequence
- Rename things so that they're more clear
- Fix stack validation with FRAME_POINTER=y builds

- Fix static call patching to handle when a JMP to the return thunk
is the last insn on the very last module memory page

- Add more documentation about what each untraining routine does and
why"

* tag 'x86_urgent_for_v6.5_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/srso: Correct the mitigation status when SMT is disabled
x86/static_call: Fix __static_call_fixup()
objtool/x86: Fixup frame-pointer vs rethunk
x86/srso: Explain the untraining sequences a bit more
x86/cpu/kvm: Provide UNTRAIN_RET_VM
x86/cpu: Cleanup the untrain mess
x86/cpu: Rename srso_(.*)_alias to srso_alias_\1
x86/cpu: Rename original retbleed methods
x86/cpu: Clean up SRSO return thunk mess
x86/alternative: Make custom return thunk unconditional
objtool/x86: Fix SRSO mess
x86/cpu: Fix up srso_safe_ret() and __x86_return_thunk()
x86/cpu: Fix __x86_return_thunk symbol type
x86/retpoline,kprobes: Skip optprobe check for indirect jumps with retpolines and IBT
x86/retpoline,kprobes: Fix position of thunk sections with CONFIG_LTO_CLANG
x86/srso: Disable the mitigation on unaffected configurations
x86/CPU/AMD: Fix the DIV(0) initial fix attempt
x86/retpoline: Don't clobber RFLAGS during srso_safe_ret()

Linus Torvalds bf98bae3 4e7ffde6

+238 -135
+2 -2
Documentation/admin-guide/hw-vuln/srso.rst
··· 141 141 To ensure the safety of this mitigation, the kernel must ensure that the 142 142 safe return sequence is itself free from attacker interference. In Zen3 143 143 and Zen4, this is accomplished by creating a BTB alias between the 144 - untraining function srso_untrain_ret_alias() and the safe return 145 - function srso_safe_ret_alias() which results in evicting a potentially 144 + untraining function srso_alias_untrain_ret() and the safe return 145 + function srso_alias_safe_ret() which results in evicting a potentially 146 146 poisoned BTB entry and using that safe one for all function returns. 147 147 148 148 In older Zen1 and Zen2, this is accomplished using a reinterpretation
+1
arch/x86/include/asm/entry-common.h
··· 92 92 static __always_inline void arch_exit_to_user_mode(void) 93 93 { 94 94 mds_user_clear_cpu_buffers(); 95 + amd_clear_divider(); 95 96 } 96 97 #define arch_exit_to_user_mode arch_exit_to_user_mode 97 98
+27 -22
arch/x86/include/asm/nospec-branch.h
··· 272 272 .endm 273 273 274 274 #ifdef CONFIG_CPU_UNRET_ENTRY 275 - #define CALL_ZEN_UNTRAIN_RET "call zen_untrain_ret" 275 + #define CALL_UNTRAIN_RET "call entry_untrain_ret" 276 276 #else 277 - #define CALL_ZEN_UNTRAIN_RET "" 277 + #define CALL_UNTRAIN_RET "" 278 278 #endif 279 279 280 280 /* ··· 282 282 * return thunk isn't mapped into the userspace tables (then again, AMD 283 283 * typically has NO_MELTDOWN). 284 284 * 285 - * While zen_untrain_ret() doesn't clobber anything but requires stack, 285 + * While retbleed_untrain_ret() doesn't clobber anything but requires stack, 286 286 * entry_ibpb() will clobber AX, CX, DX. 287 287 * 288 288 * As such, this must be placed after every *SWITCH_TO_KERNEL_CR3 at a point ··· 293 293 defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) 294 294 VALIDATE_UNRET_END 295 295 ALTERNATIVE_3 "", \ 296 - CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ 296 + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ 297 297 "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ 298 298 __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH 299 299 #endif 300 + .endm 300 301 301 - #ifdef CONFIG_CPU_SRSO 302 - ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ 303 - "call srso_untrain_ret_alias", X86_FEATURE_SRSO_ALIAS 302 + .macro UNTRAIN_RET_VM 303 + #if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ 304 + defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) 305 + VALIDATE_UNRET_END 306 + ALTERNATIVE_3 "", \ 307 + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ 308 + "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT, \ 309 + __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH 304 310 #endif 305 311 .endm 306 312 ··· 315 309 defined(CONFIG_CALL_DEPTH_TRACKING) 316 310 VALIDATE_UNRET_END 317 311 ALTERNATIVE_3 "", \ 318 - CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ 312 + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ 319 313 "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ 320 314 __stringify(RESET_CALL_DEPTH_FROM_CALL), X86_FEATURE_CALL_DEPTH 321 - #endif 322 - 323 - #ifdef CONFIG_CPU_SRSO 324 - ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ 325 - "call srso_untrain_ret_alias", X86_FEATURE_SRSO_ALIAS 326 315 #endif 327 316 .endm 328 317 ··· 342 341 extern retpoline_thunk_t __x86_indirect_call_thunk_array[]; 343 342 extern retpoline_thunk_t __x86_indirect_jump_thunk_array[]; 344 343 344 + #ifdef CONFIG_RETHUNK 345 345 extern void __x86_return_thunk(void); 346 - extern void zen_untrain_ret(void); 346 + #else 347 + static inline void __x86_return_thunk(void) {} 348 + #endif 349 + 350 + extern void retbleed_return_thunk(void); 351 + extern void srso_return_thunk(void); 352 + extern void srso_alias_return_thunk(void); 353 + 354 + extern void retbleed_untrain_ret(void); 347 355 extern void srso_untrain_ret(void); 348 - extern void srso_untrain_ret_alias(void); 356 + extern void srso_alias_untrain_ret(void); 357 + 358 + extern void entry_untrain_ret(void); 349 359 extern void entry_ibpb(void); 350 360 351 - #ifdef CONFIG_CALL_THUNKS 352 361 extern void (*x86_return_thunk)(void); 353 - #else 354 - #define x86_return_thunk (&__x86_return_thunk) 355 - #endif 356 362 357 363 #ifdef CONFIG_CALL_DEPTH_TRACKING 358 364 extern void __x86_return_skl(void); ··· 485 477 SPEC_STORE_BYPASS_PRCTL, 486 478 SPEC_STORE_BYPASS_SECCOMP, 487 479 }; 488 - 489 - extern char __indirect_thunk_start[]; 490 - extern char __indirect_thunk_end[]; 491 480 492 481 static __always_inline 493 482 void alternative_msr_write(unsigned int msr, u64 val, unsigned int feature)
-4
arch/x86/kernel/alternative.c
··· 687 687 688 688 #ifdef CONFIG_RETHUNK 689 689 690 - #ifdef CONFIG_CALL_THUNKS 691 - void (*x86_return_thunk)(void) __ro_after_init = &__x86_return_thunk; 692 - #endif 693 - 694 690 /* 695 691 * Rewrite the compiler generated return thunk tail-calls. 696 692 *
+1
arch/x86/kernel/cpu/amd.c
··· 1329 1329 asm volatile(ALTERNATIVE("", "div %2\n\t", X86_BUG_DIV0) 1330 1330 :: "a" (0), "d" (0), "r" (1)); 1331 1331 } 1332 + EXPORT_SYMBOL_GPL(amd_clear_divider);
+23 -4
arch/x86/kernel/cpu/bugs.c
··· 63 63 64 64 static DEFINE_MUTEX(spec_ctrl_mutex); 65 65 66 + void (*x86_return_thunk)(void) __ro_after_init = &__x86_return_thunk; 67 + 66 68 /* Update SPEC_CTRL MSR and its cached copy unconditionally */ 67 69 static void update_spec_ctrl(u64 val) 68 70 { ··· 167 165 md_clear_select_mitigation(); 168 166 srbds_select_mitigation(); 169 167 l1d_flush_select_mitigation(); 168 + 169 + /* 170 + * srso_select_mitigation() depends and must run after 171 + * retbleed_select_mitigation(). 172 + */ 170 173 srso_select_mitigation(); 171 174 gds_select_mitigation(); 172 175 } ··· 1042 1035 setup_force_cpu_cap(X86_FEATURE_RETHUNK); 1043 1036 setup_force_cpu_cap(X86_FEATURE_UNRET); 1044 1037 1038 + if (IS_ENABLED(CONFIG_RETHUNK)) 1039 + x86_return_thunk = retbleed_return_thunk; 1040 + 1045 1041 if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD && 1046 1042 boot_cpu_data.x86_vendor != X86_VENDOR_HYGON) 1047 1043 pr_err(RETBLEED_UNTRAIN_MSG); ··· 1054 1044 1055 1045 case RETBLEED_MITIGATION_IBPB: 1056 1046 setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); 1047 + setup_force_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT); 1057 1048 mitigate_smt = true; 1058 1049 break; 1059 1050 ··· 2428 2417 * Zen1/2 with SMT off aren't vulnerable after the right 2429 2418 * IBPB microcode has been applied. 2430 2419 */ 2431 - if ((boot_cpu_data.x86 < 0x19) && 2432 - (!cpu_smt_possible() || (cpu_smt_control == CPU_SMT_DISABLED))) 2420 + if (boot_cpu_data.x86 < 0x19 && !cpu_smt_possible()) { 2433 2421 setup_force_cpu_cap(X86_FEATURE_SRSO_NO); 2422 + return; 2423 + } 2434 2424 } 2435 2425 2436 2426 if (retbleed_mitigation == RETBLEED_MITIGATION_IBPB) { ··· 2460 2448 * like ftrace, static_call, etc. 2461 2449 */ 2462 2450 setup_force_cpu_cap(X86_FEATURE_RETHUNK); 2451 + setup_force_cpu_cap(X86_FEATURE_UNRET); 2463 2452 2464 - if (boot_cpu_data.x86 == 0x19) 2453 + if (boot_cpu_data.x86 == 0x19) { 2465 2454 setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); 2466 - else 2455 + x86_return_thunk = srso_alias_return_thunk; 2456 + } else { 2467 2457 setup_force_cpu_cap(X86_FEATURE_SRSO); 2458 + x86_return_thunk = srso_return_thunk; 2459 + } 2468 2460 srso_mitigation = SRSO_MITIGATION_SAFE_RET; 2469 2461 } else { 2470 2462 pr_err("WARNING: kernel not compiled with CPU_SRSO.\n"); ··· 2712 2696 2713 2697 static ssize_t srso_show_state(char *buf) 2714 2698 { 2699 + if (boot_cpu_has(X86_FEATURE_SRSO_NO)) 2700 + return sysfs_emit(buf, "Mitigation: SMT disabled\n"); 2701 + 2715 2702 return sysfs_emit(buf, "%s%s\n", 2716 2703 srso_strings[srso_mitigation], 2717 2704 (cpu_has_ibpb_brtype_microcode() ? "" : ", no microcode"));
+16 -24
arch/x86/kernel/kprobes/opt.c
··· 226 226 } 227 227 228 228 /* Check whether insn is indirect jump */ 229 - static int __insn_is_indirect_jump(struct insn *insn) 229 + static int insn_is_indirect_jump(struct insn *insn) 230 230 { 231 231 return ((insn->opcode.bytes[0] == 0xff && 232 232 (X86_MODRM_REG(insn->modrm.value) & 6) == 4) || /* Jump */ ··· 258 258 target = (unsigned long)insn->next_byte + insn->immediate.value; 259 259 260 260 return (start <= target && target <= start + len); 261 - } 262 - 263 - static int insn_is_indirect_jump(struct insn *insn) 264 - { 265 - int ret = __insn_is_indirect_jump(insn); 266 - 267 - #ifdef CONFIG_RETPOLINE 268 - /* 269 - * Jump to x86_indirect_thunk_* is treated as an indirect jump. 270 - * Note that even with CONFIG_RETPOLINE=y, the kernel compiled with 271 - * older gcc may use indirect jump. So we add this check instead of 272 - * replace indirect-jump check. 273 - */ 274 - if (!ret) 275 - ret = insn_jump_into_range(insn, 276 - (unsigned long)__indirect_thunk_start, 277 - (unsigned long)__indirect_thunk_end - 278 - (unsigned long)__indirect_thunk_start); 279 - #endif 280 - return ret; 281 261 } 282 262 283 263 /* Decode whole function to ensure any instructions don't jump into target */ ··· 314 334 /* Recover address */ 315 335 insn.kaddr = (void *)addr; 316 336 insn.next_byte = (void *)(addr + insn.length); 317 - /* Check any instructions don't jump into target */ 318 - if (insn_is_indirect_jump(&insn) || 319 - insn_jump_into_range(&insn, paddr + INT3_INSN_SIZE, 337 + /* 338 + * Check any instructions don't jump into target, indirectly or 339 + * directly. 340 + * 341 + * The indirect case is present to handle a code with jump 342 + * tables. When the kernel uses retpolines, the check should in 343 + * theory additionally look for jumps to indirect thunks. 344 + * However, the kernel built with retpolines or IBT has jump 345 + * tables disabled so the check can be skipped altogether. 346 + */ 347 + if (!IS_ENABLED(CONFIG_RETPOLINE) && 348 + !IS_ENABLED(CONFIG_X86_KERNEL_IBT) && 349 + insn_is_indirect_jump(&insn)) 350 + return 0; 351 + if (insn_jump_into_range(&insn, paddr + INT3_INSN_SIZE, 320 352 DISP32_SIZE)) 321 353 return 0; 322 354 addr += insn.length;
+13
arch/x86/kernel/static_call.c
··· 186 186 */ 187 187 bool __static_call_fixup(void *tramp, u8 op, void *dest) 188 188 { 189 + unsigned long addr = (unsigned long)tramp; 190 + /* 191 + * Not all .return_sites are a static_call trampoline (most are not). 192 + * Check if the 3 bytes after the return are still kernel text, if not, 193 + * then this definitely is not a trampoline and we need not worry 194 + * further. 195 + * 196 + * This avoids the memcmp() below tripping over pagefaults etc.. 197 + */ 198 + if (((addr >> PAGE_SHIFT) != ((addr + 7) >> PAGE_SHIFT)) && 199 + !kernel_text_address(addr + 7)) 200 + return false; 201 + 189 202 if (memcmp(tramp+5, tramp_ud, 3)) { 190 203 /* Not a trampoline site, not our problem. */ 191 204 return false;
-2
arch/x86/kernel/traps.c
··· 206 206 { 207 207 do_error_trap(regs, 0, "divide error", X86_TRAP_DE, SIGFPE, 208 208 FPE_INTDIV, error_get_trap_addr(regs)); 209 - 210 - amd_clear_divider(); 211 209 } 212 210 213 211 DEFINE_IDTENTRY(exc_overflow)
+9 -11
arch/x86/kernel/vmlinux.lds.S
··· 133 133 KPROBES_TEXT 134 134 SOFTIRQENTRY_TEXT 135 135 #ifdef CONFIG_RETPOLINE 136 - __indirect_thunk_start = .; 137 - *(.text.__x86.indirect_thunk) 138 - *(.text.__x86.return_thunk) 139 - __indirect_thunk_end = .; 136 + *(.text..__x86.indirect_thunk) 137 + *(.text..__x86.return_thunk) 140 138 #endif 141 139 STATIC_CALL_TEXT 142 140 143 141 ALIGN_ENTRY_TEXT_BEGIN 144 142 #ifdef CONFIG_CPU_SRSO 145 - *(.text.__x86.rethunk_untrain) 143 + *(.text..__x86.rethunk_untrain) 146 144 #endif 147 145 148 146 ENTRY_TEXT 149 147 150 148 #ifdef CONFIG_CPU_SRSO 151 149 /* 152 - * See the comment above srso_untrain_ret_alias()'s 150 + * See the comment above srso_alias_untrain_ret()'s 153 151 * definition. 154 152 */ 155 - . = srso_untrain_ret_alias | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20); 156 - *(.text.__x86.rethunk_safe) 153 + . = srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20); 154 + *(.text..__x86.rethunk_safe) 157 155 #endif 158 156 ALIGN_ENTRY_TEXT_END 159 157 *(.gnu.warning) ··· 521 523 #endif 522 524 523 525 #ifdef CONFIG_RETHUNK 524 - . = ASSERT((__ret & 0x3f) == 0, "__ret not cacheline-aligned"); 526 + . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); 525 527 . = ASSERT((srso_safe_ret & 0x3f) == 0, "srso_safe_ret not cacheline-aligned"); 526 528 #endif 527 529 ··· 536 538 * Instead do: (A | B) - (A & B) in order to compute the XOR 537 539 * of the two function addresses: 538 540 */ 539 - . = ASSERT(((ABSOLUTE(srso_untrain_ret_alias) | srso_safe_ret_alias) - 540 - (ABSOLUTE(srso_untrain_ret_alias) & srso_safe_ret_alias)) == ((1 << 2) | (1 << 8) | (1 << 14) | (1 << 20)), 541 + . = ASSERT(((ABSOLUTE(srso_alias_untrain_ret) | srso_alias_safe_ret) - 542 + (ABSOLUTE(srso_alias_untrain_ret) & srso_alias_safe_ret)) == ((1 << 2) | (1 << 8) | (1 << 14) | (1 << 20)), 541 543 "SRSO function pair won't alias"); 542 544 #endif 543 545
+2
arch/x86/kvm/svm/svm.c
··· 4006 4006 4007 4007 guest_state_enter_irqoff(); 4008 4008 4009 + amd_clear_divider(); 4010 + 4009 4011 if (sev_es_guest(vcpu->kvm)) 4010 4012 __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted); 4011 4013 else
+2 -5
arch/x86/kvm/svm/vmenter.S
··· 222 222 * because interrupt handlers won't sanitize 'ret' if the return is 223 223 * from the kernel. 224 224 */ 225 - UNTRAIN_RET 226 - 227 - /* SRSO */ 228 - ALTERNATIVE "", "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT 225 + UNTRAIN_RET_VM 229 226 230 227 /* 231 228 * Clear all general purpose registers except RSP and RAX to prevent ··· 359 362 * because interrupt handlers won't sanitize RET if the return is 360 363 * from the kernel. 361 364 */ 362 - UNTRAIN_RET 365 + UNTRAIN_RET_VM 363 366 364 367 /* "Pop" @spec_ctrl_intercepted. */ 365 368 pop %_ASM_BX
+97 -44
arch/x86/lib/retpoline.S
··· 13 13 #include <asm/frame.h> 14 14 #include <asm/nops.h> 15 15 16 - .section .text.__x86.indirect_thunk 16 + .section .text..__x86.indirect_thunk 17 17 18 18 19 19 .macro POLINE reg ··· 133 133 #ifdef CONFIG_RETHUNK 134 134 135 135 /* 136 - * srso_untrain_ret_alias() and srso_safe_ret_alias() are placed at 136 + * srso_alias_untrain_ret() and srso_alias_safe_ret() are placed at 137 137 * special addresses: 138 138 * 139 - * - srso_untrain_ret_alias() is 2M aligned 140 - * - srso_safe_ret_alias() is also in the same 2M page but bits 2, 8, 14 139 + * - srso_alias_untrain_ret() is 2M aligned 140 + * - srso_alias_safe_ret() is also in the same 2M page but bits 2, 8, 14 141 141 * and 20 in its virtual address are set (while those bits in the 142 - * srso_untrain_ret_alias() function are cleared). 142 + * srso_alias_untrain_ret() function are cleared). 143 143 * 144 144 * This guarantees that those two addresses will alias in the branch 145 145 * target buffer of Zen3/4 generations, leading to any potential 146 146 * poisoned entries at that BTB slot to get evicted. 147 147 * 148 - * As a result, srso_safe_ret_alias() becomes a safe return. 148 + * As a result, srso_alias_safe_ret() becomes a safe return. 149 149 */ 150 150 #ifdef CONFIG_CPU_SRSO 151 - .section .text.__x86.rethunk_untrain 151 + .section .text..__x86.rethunk_untrain 152 152 153 - SYM_START(srso_untrain_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) 153 + SYM_START(srso_alias_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) 154 + UNWIND_HINT_FUNC 154 155 ANNOTATE_NOENDBR 155 156 ASM_NOP2 156 157 lfence 157 - jmp __x86_return_thunk 158 - SYM_FUNC_END(srso_untrain_ret_alias) 159 - __EXPORT_THUNK(srso_untrain_ret_alias) 158 + jmp srso_alias_return_thunk 159 + SYM_FUNC_END(srso_alias_untrain_ret) 160 + __EXPORT_THUNK(srso_alias_untrain_ret) 160 161 161 - .section .text.__x86.rethunk_safe 162 - #endif 163 - 164 - /* Needs a definition for the __x86_return_thunk alternative below. */ 165 - SYM_START(srso_safe_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) 166 - #ifdef CONFIG_CPU_SRSO 167 - add $8, %_ASM_SP 168 - UNWIND_HINT_FUNC 169 - #endif 162 + .section .text..__x86.rethunk_safe 163 + #else 164 + /* dummy definition for alternatives */ 165 + SYM_START(srso_alias_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) 170 166 ANNOTATE_UNRET_SAFE 171 167 ret 172 168 int3 173 - SYM_FUNC_END(srso_safe_ret_alias) 169 + SYM_FUNC_END(srso_alias_untrain_ret) 170 + #endif 174 171 175 - .section .text.__x86.return_thunk 172 + SYM_START(srso_alias_safe_ret, SYM_L_GLOBAL, SYM_A_NONE) 173 + lea 8(%_ASM_SP), %_ASM_SP 174 + UNWIND_HINT_FUNC 175 + ANNOTATE_UNRET_SAFE 176 + ret 177 + int3 178 + SYM_FUNC_END(srso_alias_safe_ret) 179 + 180 + .section .text..__x86.return_thunk 181 + 182 + SYM_CODE_START(srso_alias_return_thunk) 183 + UNWIND_HINT_FUNC 184 + ANNOTATE_NOENDBR 185 + call srso_alias_safe_ret 186 + ud2 187 + SYM_CODE_END(srso_alias_return_thunk) 188 + 189 + /* 190 + * Some generic notes on the untraining sequences: 191 + * 192 + * They are interchangeable when it comes to flushing potentially wrong 193 + * RET predictions from the BTB. 194 + * 195 + * The SRSO Zen1/2 (MOVABS) untraining sequence is longer than the 196 + * Retbleed sequence because the return sequence done there 197 + * (srso_safe_ret()) is longer and the return sequence must fully nest 198 + * (end before) the untraining sequence. Therefore, the untraining 199 + * sequence must fully overlap the return sequence. 200 + * 201 + * Regarding alignment - the instructions which need to be untrained, 202 + * must all start at a cacheline boundary for Zen1/2 generations. That 203 + * is, instruction sequences starting at srso_safe_ret() and 204 + * the respective instruction sequences at retbleed_return_thunk() 205 + * must start at a cacheline boundary. 206 + */ 176 207 177 208 /* 178 209 * Safety details here pertain to the AMD Zen{1,2} microarchitecture: 179 - * 1) The RET at __x86_return_thunk must be on a 64 byte boundary, for 210 + * 1) The RET at retbleed_return_thunk must be on a 64 byte boundary, for 180 211 * alignment within the BTB. 181 - * 2) The instruction at zen_untrain_ret must contain, and not 212 + * 2) The instruction at retbleed_untrain_ret must contain, and not 182 213 * end with, the 0xc3 byte of the RET. 183 214 * 3) STIBP must be enabled, or SMT disabled, to prevent the sibling thread 184 215 * from re-poisioning the BTB prediction. 185 216 */ 186 217 .align 64 187 - .skip 64 - (__ret - zen_untrain_ret), 0xcc 188 - SYM_START(zen_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) 218 + .skip 64 - (retbleed_return_thunk - retbleed_untrain_ret), 0xcc 219 + SYM_START(retbleed_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) 189 220 ANNOTATE_NOENDBR 190 221 /* 191 - * As executed from zen_untrain_ret, this is: 222 + * As executed from retbleed_untrain_ret, this is: 192 223 * 193 224 * TEST $0xcc, %bl 194 225 * LFENCE 195 - * JMP __x86_return_thunk 226 + * JMP retbleed_return_thunk 196 227 * 197 228 * Executing the TEST instruction has a side effect of evicting any BTB 198 229 * prediction (potentially attacker controlled) attached to the RET, as 199 - * __x86_return_thunk + 1 isn't an instruction boundary at the moment. 230 + * retbleed_return_thunk + 1 isn't an instruction boundary at the moment. 200 231 */ 201 232 .byte 0xf6 202 233 203 234 /* 204 - * As executed from __x86_return_thunk, this is a plain RET. 235 + * As executed from retbleed_return_thunk, this is a plain RET. 205 236 * 206 237 * As part of the TEST above, RET is the ModRM byte, and INT3 the imm8. 207 238 * ··· 244 213 * With SMT enabled and STIBP active, a sibling thread cannot poison 245 214 * RET's prediction to a type of its choice, but can evict the 246 215 * prediction due to competitive sharing. If the prediction is 247 - * evicted, __x86_return_thunk will suffer Straight Line Speculation 216 + * evicted, retbleed_return_thunk will suffer Straight Line Speculation 248 217 * which will be contained safely by the INT3. 249 218 */ 250 - SYM_INNER_LABEL(__ret, SYM_L_GLOBAL) 219 + SYM_INNER_LABEL(retbleed_return_thunk, SYM_L_GLOBAL) 251 220 ret 252 221 int3 253 - SYM_CODE_END(__ret) 222 + SYM_CODE_END(retbleed_return_thunk) 254 223 255 224 /* 256 225 * Ensure the TEST decoding / BTB invalidation is complete. ··· 261 230 * Jump back and execute the RET in the middle of the TEST instruction. 262 231 * INT3 is for SLS protection. 263 232 */ 264 - jmp __ret 233 + jmp retbleed_return_thunk 265 234 int3 266 - SYM_FUNC_END(zen_untrain_ret) 267 - __EXPORT_THUNK(zen_untrain_ret) 235 + SYM_FUNC_END(retbleed_untrain_ret) 236 + __EXPORT_THUNK(retbleed_untrain_ret) 268 237 269 238 /* 270 - * SRSO untraining sequence for Zen1/2, similar to zen_untrain_ret() 239 + * SRSO untraining sequence for Zen1/2, similar to retbleed_untrain_ret() 271 240 * above. On kernel entry, srso_untrain_ret() is executed which is a 272 241 * 273 - * movabs $0xccccccc308c48348,%rax 242 + * movabs $0xccccc30824648d48,%rax 274 243 * 275 244 * and when the return thunk executes the inner label srso_safe_ret() 276 245 * later, it is a stack manipulation and a RET which is mispredicted and ··· 282 251 ANNOTATE_NOENDBR 283 252 .byte 0x48, 0xb8 284 253 254 + /* 255 + * This forces the function return instruction to speculate into a trap 256 + * (UD2 in srso_return_thunk() below). This RET will then mispredict 257 + * and execution will continue at the return site read from the top of 258 + * the stack. 259 + */ 285 260 SYM_INNER_LABEL(srso_safe_ret, SYM_L_GLOBAL) 286 - add $8, %_ASM_SP 261 + lea 8(%_ASM_SP), %_ASM_SP 287 262 ret 288 263 int3 289 264 int3 290 - int3 265 + /* end of movabs */ 291 266 lfence 292 267 call srso_safe_ret 293 - int3 268 + ud2 294 269 SYM_CODE_END(srso_safe_ret) 295 270 SYM_FUNC_END(srso_untrain_ret) 296 271 __EXPORT_THUNK(srso_untrain_ret) 297 272 298 - SYM_FUNC_START(__x86_return_thunk) 299 - ALTERNATIVE_2 "jmp __ret", "call srso_safe_ret", X86_FEATURE_SRSO, \ 300 - "call srso_safe_ret_alias", X86_FEATURE_SRSO_ALIAS 273 + SYM_CODE_START(srso_return_thunk) 274 + UNWIND_HINT_FUNC 275 + ANNOTATE_NOENDBR 276 + call srso_safe_ret 277 + ud2 278 + SYM_CODE_END(srso_return_thunk) 279 + 280 + SYM_FUNC_START(entry_untrain_ret) 281 + ALTERNATIVE_2 "jmp retbleed_untrain_ret", \ 282 + "jmp srso_untrain_ret", X86_FEATURE_SRSO, \ 283 + "jmp srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS 284 + SYM_FUNC_END(entry_untrain_ret) 285 + __EXPORT_THUNK(entry_untrain_ret) 286 + 287 + SYM_CODE_START(__x86_return_thunk) 288 + UNWIND_HINT_FUNC 289 + ANNOTATE_NOENDBR 290 + ANNOTATE_UNRET_SAFE 291 + ret 301 292 int3 302 293 SYM_CODE_END(__x86_return_thunk) 303 294 EXPORT_SYMBOL(__x86_return_thunk)
+7 -4
tools/objtool/arch/x86/decode.c
··· 824 824 825 825 bool arch_is_rethunk(struct symbol *sym) 826 826 { 827 - return !strcmp(sym->name, "__x86_return_thunk") || 828 - !strcmp(sym->name, "srso_untrain_ret") || 829 - !strcmp(sym->name, "srso_safe_ret") || 830 - !strcmp(sym->name, "__ret"); 827 + return !strcmp(sym->name, "__x86_return_thunk"); 828 + } 829 + 830 + bool arch_is_embedded_insn(struct symbol *sym) 831 + { 832 + return !strcmp(sym->name, "retbleed_return_thunk") || 833 + !strcmp(sym->name, "srso_safe_ret"); 831 834 }
+35 -10
tools/objtool/check.c
··· 389 389 if (!strcmp(sec->name, ".noinstr.text") || 390 390 !strcmp(sec->name, ".entry.text") || 391 391 !strcmp(sec->name, ".cpuidle.text") || 392 - !strncmp(sec->name, ".text.__x86.", 12)) 392 + !strncmp(sec->name, ".text..__x86.", 13)) 393 393 sec->noinstr = true; 394 394 395 395 /* ··· 455 455 return -1; 456 456 } 457 457 458 - if (func->return_thunk || func->alias != func) 458 + if (func->embedded_insn || func->alias != func) 459 459 continue; 460 460 461 461 if (!find_insn(file, sec, func->offset)) { ··· 1288 1288 return 0; 1289 1289 } 1290 1290 1291 + /* 1292 + * Symbols that replace INSN_CALL_DYNAMIC, every (tail) call to such a symbol 1293 + * will be added to the .retpoline_sites section. 1294 + */ 1291 1295 __weak bool arch_is_retpoline(struct symbol *sym) 1292 1296 { 1293 1297 return false; 1294 1298 } 1295 1299 1300 + /* 1301 + * Symbols that replace INSN_RETURN, every (tail) call to such a symbol 1302 + * will be added to the .return_sites section. 1303 + */ 1296 1304 __weak bool arch_is_rethunk(struct symbol *sym) 1305 + { 1306 + return false; 1307 + } 1308 + 1309 + /* 1310 + * Symbols that are embedded inside other instructions, because sometimes crazy 1311 + * code exists. These are mostly ignored for validation purposes. 1312 + */ 1313 + __weak bool arch_is_embedded_insn(struct symbol *sym) 1297 1314 { 1298 1315 return false; 1299 1316 } ··· 1593 1576 struct symbol *sym = find_symbol_by_offset(dest_sec, dest_off); 1594 1577 1595 1578 /* 1596 - * This is a special case for zen_untrain_ret(). 1579 + * This is a special case for retbleed_untrain_ret(). 1597 1580 * It jumps to __x86_return_thunk(), but objtool 1598 1581 * can't find the thunk's starting RET 1599 1582 * instruction, because the RET is also in the 1600 1583 * middle of another instruction. Objtool only 1601 1584 * knows about the outer instruction. 1602 1585 */ 1603 - if (sym && sym->return_thunk) { 1586 + if (sym && sym->embedded_insn) { 1604 1587 add_return_call(file, insn, false); 1605 1588 continue; 1606 1589 } ··· 2519 2502 if (arch_is_rethunk(func)) 2520 2503 func->return_thunk = true; 2521 2504 2505 + if (arch_is_embedded_insn(func)) 2506 + func->embedded_insn = true; 2507 + 2522 2508 if (arch_ftrace_match(func->name)) 2523 2509 func->fentry = true; 2524 2510 ··· 2650 2630 return 0; 2651 2631 } 2652 2632 2653 - static bool is_fentry_call(struct instruction *insn) 2633 + static bool is_special_call(struct instruction *insn) 2654 2634 { 2655 - if (insn->type == INSN_CALL && 2656 - insn_call_dest(insn) && 2657 - insn_call_dest(insn)->fentry) 2658 - return true; 2635 + if (insn->type == INSN_CALL) { 2636 + struct symbol *dest = insn_call_dest(insn); 2637 + 2638 + if (!dest) 2639 + return false; 2640 + 2641 + if (dest->fentry || dest->embedded_insn) 2642 + return true; 2643 + } 2659 2644 2660 2645 return false; 2661 2646 } ··· 3661 3636 if (ret) 3662 3637 return ret; 3663 3638 3664 - if (opts.stackval && func && !is_fentry_call(insn) && 3639 + if (opts.stackval && func && !is_special_call(insn) && 3665 3640 !has_valid_stack_frame(&state)) { 3666 3641 WARN_INSN(insn, "call without frame pointer save/setup"); 3667 3642 return 1;
+1
tools/objtool/include/objtool/arch.h
··· 90 90 91 91 bool arch_is_retpoline(struct symbol *sym); 92 92 bool arch_is_rethunk(struct symbol *sym); 93 + bool arch_is_embedded_insn(struct symbol *sym); 93 94 94 95 int arch_rewrite_retpolines(struct objtool_file *file); 95 96
+1
tools/objtool/include/objtool/elf.h
··· 66 66 u8 fentry : 1; 67 67 u8 profiling_func : 1; 68 68 u8 warned : 1; 69 + u8 embedded_insn : 1; 69 70 struct list_head pv_target; 70 71 struct reloc *relocs; 71 72 };
+1 -3
tools/perf/util/thread-stack.c
··· 1038 1038 1039 1039 static bool is_x86_retpoline(const char *name) 1040 1040 { 1041 - const char *p = strstr(name, "__x86_indirect_thunk_"); 1042 - 1043 - return p == name || !strcmp(name, "__indirect_thunk_start"); 1041 + return strstr(name, "__x86_indirect_thunk_") == name; 1044 1042 } 1045 1043 1046 1044 /*