x86/decompressor: Only call the trampoline when changing paging levels

Since the current and desired number of paging levels are known when the
trampoline is being prepared, avoid calling the trampoline at all if it
is clear that calling it is not going to result in a change to the
number of paging levels.

Given that the CPU is already running in long mode, the PAE and LA57
settings are necessarily consistent with the currently active page
tables, and other fields in CR4 will be initialized by the startup code
in the kernel proper. So limit the manipulation of CR4 to toggling the
LA57 bit, which is the only thing that really needs doing at this point
in the boot. This also means that there is no need to pass the value of
l5_required to toggle_la57(), as it will not be called unless CR4.LA57
needs to toggle.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Link: https://lore.kernel.org/r/20230807162720.545787-14-ardb@kernel.org

authored by

Ard Biesheuvel and committed by

Borislav Petkov (AMD) 2 years ago f97b67a7 64ef578b

+13 -54

2 changed files

expand all

arch

x86

boot

compressed

head_64.S

pgtable_64.c

+4 -41

arch/x86/boot/compressed/head_64.S

··· 387 387 * For the trampoline, we need the top page table to reside in lower 388 388 * memory as we don't have a way to load 64-bit values into CR3 in 389 389 * 32-bit mode. 390 - * 391 - * We go though the trampoline even if we don't have to: if we're 392 - * already in a desired paging mode. This way the trampoline code gets 393 - * tested on every boot. 394 390 */ 395 391 396 392 /* Make sure we have GDT with 32-bit code segment */ ··· 522 526 * 523 527 * Return address is at the top of the stack (might be above 4G). 524 528 * The first argument (EDI) contains the 32-bit addressable base of the 525 - * trampoline memory. A non-zero second argument (ESI) means that the 526 - * trampoline needs to enable 5-level paging. 529 + * trampoline memory. 527 530 */ 528 531 .section ".rodata", "a", @progbits 529 532 SYM_CODE_START(trampoline_32bit_src) ··· 574 579 btrl $X86_CR0_PG_BIT, %eax 575 580 movl %eax, %cr0 576 581 577 - /* Check what paging mode we want to be in after the trampoline */ 578 - testl %esi, %esi 579 - jz 1f 580 - 581 - /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ 582 - movl %cr4, %eax 583 - testl $X86_CR4_LA57, %eax 584 - jnz 3f 585 - jmp 2f 586 - 1: 587 - /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ 588 - movl %cr4, %eax 589 - testl $X86_CR4_LA57, %eax 590 - jz 3f 591 - 2: 592 582 /* Point CR3 to the trampoline's new top level page table */ 593 583 leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%edi), %eax 594 584 movl %eax, %cr3 595 - 3: 585 + 596 586 /* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */ 597 587 movl $MSR_EFER, %ecx 598 588 rdmsr ··· 586 606 jc 1f 587 607 wrmsr 588 608 1: 589 - #ifdef CONFIG_X86_MCE 590 - /* 591 - * Preserve CR4.MCE if the kernel will enable #MC support. 592 - * Clearing MCE may fault in some environments (that also force #MC 593 - * support). Any machine check that occurs before #MC support is fully 594 - * configured will crash the system regardless of the CR4.MCE value set 595 - * here. 596 - */ 609 + /* Toggle CR4.LA57 */ 597 610 movl %cr4, %eax 598 - andl $X86_CR4_MCE, %eax 599 - #else 600 - movl $0, %eax 601 - #endif 602 - 603 - /* Enable PAE and LA57 (if required) paging modes */ 604 - orl $X86_CR4_PAE, %eax 605 - testl %esi, %esi 606 - jz 1f 607 - orl $X86_CR4_LA57, %eax 608 - 1: 611 + btcl $X86_CR4_LA57_BIT, %eax 609 612 movl %eax, %cr4 610 613 611 614 /* Enable paging again. */

+9 -13

arch/x86/boot/compressed/pgtable_64.c

··· 103 103 104 104 asmlinkage void configure_5level_paging(struct boot_params *bp) 105 105 { 106 - void (*toggle_la57)(void *trampoline, bool enable_5lvl); 106 + void (*toggle_la57)(void *trampoline); 107 107 bool l5_required = false; 108 108 109 109 /* Initialize boot_params. Required for cmdline_find_option_bool(). */ ··· 133 133 ptrs_per_p4d = 512; 134 134 } 135 135 136 + /* 137 + * The trampoline will not be used if the paging mode is already set to 138 + * the desired one. 139 + */ 140 + if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) 141 + return; 142 + 136 143 trampoline_32bit = (unsigned long *)find_trampoline_placement(); 137 144 138 145 /* Preserve trampoline memory */ ··· 167 160 * 168 161 * The new page table will be used by trampoline code for switching 169 162 * from 4- to 5-level paging or vice versa. 170 - * 171 - * If switching is not required, the page table is unused: trampoline 172 - * code wouldn't touch CR3. 173 163 */ 174 - 175 - /* 176 - * We are not going to use the page table in trampoline memory if we 177 - * are already in the desired paging mode. 178 - */ 179 - if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) 180 - goto out; 181 164 182 165 if (l5_required) { 183 166 /* ··· 191 194 (void *)src, PAGE_SIZE); 192 195 } 193 196 194 - out: 195 - toggle_la57(trampoline_32bit, l5_required); 197 + toggle_la57(trampoline_32bit); 196 198 } 197 199 198 200 void cleanup_trampoline(void *pgtable)

Configure Feed

Configure Feed