ARM: fix branch predictor hardening

__do_user_fault() may be called with indeterminent interrupt enable
state, which means we may be preemptive at this point. This causes
problems when calling harden_branch_predictor(). For example, when
called from a data abort, do_alignment_fault()->do_bad_area().

Move harden_branch_predictor() out of __do_user_fault() and into the
calling contexts.

Moving it into do_kernel_address_page_fault(), we can be sure that
interrupts will be disabled here.

Converting do_translation_fault() to use do_kernel_address_page_fault()
rather than do_bad_area() means that we keep branch predictor handling
for translation faults. Interrupts will also be disabled at this call
site.

do_sect_fault() needs special handling, so detect user mode accesses
to kernel-addresses, and add an explicit call to branch predictor
hardening.

Finally, add branch predictor hardening to do_alignment() for the
faulting case (user mode accessing kernel addresses) before interrupts
are enabled.

This should cover all cases where harden_branch_predictor() is called,
ensuring that it is always has interrupts disabled, also ensuring that
it is called early in each call path.

Reviewed-by: Xie Yuanbin <xieyuanbin1@huawei.com>
Tested-by: Xie Yuanbin <xieyuanbin1@huawei.com>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>

Russell King (Oracle) 6 months ago fd2dee1c 7733bc7d

+31 -14

2 changed files

expand all

arch

arm

alignment.c

fault.c

+5 -1

arch/arm/mm/alignment.c

··· 19 19 #include <linux/init.h> 20 20 #include <linux/sched/signal.h> 21 21 #include <linux/uaccess.h> 22 + #include <linux/unaligned.h> 22 23 23 24 #include <asm/cp15.h> 24 25 #include <asm/system_info.h> 25 - #include <linux/unaligned.h> 26 + #include <asm/system_misc.h> 26 27 #include <asm/opcodes.h> 27 28 28 29 #include "fault.h" ··· 809 808 int isize = 4; 810 809 int thumb2_32b = 0; 811 810 int fault; 811 + 812 + if (addr >= TASK_SIZE && user_mode(regs)) 813 + harden_branch_predictor(); 812 814 813 815 if (interrupts_enabled(regs)) 814 816 local_irq_enable();

+26 -13

arch/arm/mm/fault.c

··· 198 198 { 199 199 struct task_struct *tsk = current; 200 200 201 - if (addr > TASK_SIZE) 202 - harden_branch_predictor(); 203 - 204 201 #ifdef CONFIG_DEBUG_USER 205 202 if (((user_debug & UDBG_SEGV) && (sig == SIGSEGV)) || 206 203 ((user_debug & UDBG_BUS) && (sig == SIGBUS))) { ··· 266 269 /* 267 270 * Fault from user mode for a kernel space address. User mode 268 271 * should not be faulting in kernel space, which includes the 269 - * vector/khelper page. Send a SIGSEGV. 272 + * vector/khelper page. Handle the branch predictor hardening 273 + * while interrupts are still disabled, then send a SIGSEGV. 270 274 */ 275 + harden_branch_predictor(); 271 276 __do_user_fault(addr, fsr, SIGSEGV, SEGV_MAPERR, regs); 272 277 } else { 273 278 /* ··· 484 485 * We enter here because the first level page table doesn't contain 485 486 * a valid entry for the address. 486 487 * 487 - * If the address is in kernel space (>= TASK_SIZE), then we are 488 - * probably faulting in the vmalloc() area. 488 + * If this is a user address (addr < TASK_SIZE), we handle this as a 489 + * normal page fault. This leaves the remainder of the function to handle 490 + * kernel address translation faults. 489 491 * 490 - * If the init_task's first level page tables contains the relevant 491 - * entry, we copy the it to this task. If not, we send the process 492 - * a signal, fixup the exception, or oops the kernel. 492 + * Since user mode is not permitted to access kernel addresses, pass these 493 + * directly to do_kernel_address_page_fault() to handle. 493 494 * 494 - * NOTE! We MUST NOT take any locks for this case. We may be in an 495 - * interrupt or a critical region, and should only copy the information 496 - * from the master page table, nothing more. 495 + * Otherwise, we're probably faulting in the vmalloc() area, so try to fix 496 + * that up. Note that we must not take any locks or enable interrupts in 497 + * this case. 498 + * 499 + * If vmalloc() fixup fails, that means the non-leaf page tables did not 500 + * contain an entry for this address, so handle this via 501 + * do_kernel_address_page_fault(). 497 502 */ 498 503 #ifdef CONFIG_MMU 499 504 static int __kprobes ··· 563 560 return 0; 564 561 565 562 bad_area: 566 - do_bad_area(addr, fsr, regs); 563 + do_kernel_address_page_fault(current->mm, addr, fsr, regs); 564 + 567 565 return 0; 568 566 } 569 567 #else /* CONFIG_MMU */ ··· 584 580 static int 585 581 do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) 586 582 { 583 + /* 584 + * If this is a kernel address, but from user mode, then userspace 585 + * is trying bad stuff. Invoke the branch predictor handling. 586 + * Interrupts are disabled here. 587 + */ 588 + if (addr >= TASK_SIZE && user_mode(regs)) 589 + harden_branch_predictor(); 590 + 587 591 do_bad_area(addr, fsr, regs); 592 + 588 593 return 0; 589 594 } 590 595 #endif /* CONFIG_ARM_LPAE */

Configure Feed

Configure Feed