powerpc: mm: accelerate pagefault when badaccess

The access_[pkey]_error() of vma already checked under per-VMA lock, if it
is a bad access, directly handle error, no need to retry with mmap_lock
again. In order to release the correct lock, pass the mm_struct into
bad_access_pkey()/bad_access(), if mm is NULL, release vma lock, or
release mmap_lock. Since the page faut is handled under per-VMA lock,
count it as a vma lock event with VMA_LOCK_SUCCESS.

Link: https://lkml.kernel.org/r/20240403083805.1818160-5-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kefeng Wang and committed by

Andrew Morton 2 years ago 0cec9541 15e4a5f5

+20 -13

1 changed file

expand all

arch

powerpc

fault.c

+20 -13

arch/powerpc/mm/fault.c

··· 71 71 return __bad_area_nosemaphore(regs, address, SEGV_MAPERR); 72 72 } 73 73 74 - static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code) 74 + static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code, 75 + struct mm_struct *mm, struct vm_area_struct *vma) 75 76 { 76 - struct mm_struct *mm = current->mm; 77 77 78 78 /* 79 79 * Something tried to access memory that isn't in our memory map.. 80 80 * Fix it, but check if it's kernel or user first.. 81 81 */ 82 - mmap_read_unlock(mm); 82 + if (mm) 83 + mmap_read_unlock(mm); 84 + else 85 + vma_end_read(vma); 83 86 84 87 return __bad_area_nosemaphore(regs, address, si_code); 85 88 } 86 89 87 90 static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, 91 + struct mm_struct *mm, 88 92 struct vm_area_struct *vma) 89 93 { 90 - struct mm_struct *mm = current->mm; 91 94 int pkey; 92 95 93 96 /* ··· 112 109 */ 113 110 pkey = vma_pkey(vma); 114 111 115 - mmap_read_unlock(mm); 112 + if (mm) 113 + mmap_read_unlock(mm); 114 + else 115 + vma_end_read(vma); 116 116 117 117 /* 118 118 * If we are in kernel mode, bail out with a SEGV, this will ··· 130 124 return 0; 131 125 } 132 126 133 - static noinline int bad_access(struct pt_regs *regs, unsigned long address) 127 + static noinline int bad_access(struct pt_regs *regs, unsigned long address, 128 + struct mm_struct *mm, struct vm_area_struct *vma) 134 129 { 135 - return __bad_area(regs, address, SEGV_ACCERR); 130 + return __bad_area(regs, address, SEGV_ACCERR, mm, vma); 136 131 } 137 132 138 133 static int do_sigbus(struct pt_regs *regs, unsigned long address, ··· 486 479 487 480 if (unlikely(access_pkey_error(is_write, is_exec, 488 481 (error_code & DSISR_KEYFAULT), vma))) { 489 - vma_end_read(vma); 490 - goto lock_mmap; 482 + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); 483 + return bad_access_pkey(regs, address, NULL, vma); 491 484 } 492 485 493 486 if (unlikely(access_error(is_write, is_exec, vma))) { 494 - vma_end_read(vma); 495 - goto lock_mmap; 487 + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); 488 + return bad_access(regs, address, NULL, vma); 496 489 } 497 490 498 491 fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); ··· 528 521 529 522 if (unlikely(access_pkey_error(is_write, is_exec, 530 523 (error_code & DSISR_KEYFAULT), vma))) 531 - return bad_access_pkey(regs, address, vma); 524 + return bad_access_pkey(regs, address, mm, vma); 532 525 533 526 if (unlikely(access_error(is_write, is_exec, vma))) 534 - return bad_access(regs, address); 527 + return bad_access(regs, address, mm, vma); 535 528 536 529 /* 537 530 * If for any reason at all we couldn't handle the fault,

Configure Feed

Configure Feed