tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

x86: mm: accelerate pagefault when badaccess

The access_error() of vma is already checked under per-VMA lock, if it is
a bad access, directly handle error, no need to retry with mmap_lock
again. In order to release the correct lock, pass the mm_struct into
bad_area_access_error(). If mm is NULL, release vma lock, or release
mmap_lock. Since the page faut is handled under per-VMA lock, count it as
a vma lock event with VMA_LOCK_SUCCESS.

Link: https://lkml.kernel.org/r/20240403083805.1818160-8-wangkefeng.wang@huawei.com
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kefeng Wang and committed by
Andrew Morton bc7996c8 82b7a618

+14 -9
+14 -9
arch/x86/mm/fault.c
··· 866 866 867 867 static void 868 868 __bad_area(struct pt_regs *regs, unsigned long error_code, 869 - unsigned long address, u32 pkey, int si_code) 869 + unsigned long address, struct mm_struct *mm, 870 + struct vm_area_struct *vma, u32 pkey, int si_code) 870 871 { 871 - struct mm_struct *mm = current->mm; 872 872 /* 873 873 * Something tried to access memory that isn't in our memory map.. 874 874 * Fix it, but check if it's kernel or user first.. 875 875 */ 876 - mmap_read_unlock(mm); 876 + if (mm) 877 + mmap_read_unlock(mm); 878 + else 879 + vma_end_read(vma); 877 880 878 881 __bad_area_nosemaphore(regs, error_code, address, pkey, si_code); 879 882 } ··· 900 897 901 898 static noinline void 902 899 bad_area_access_error(struct pt_regs *regs, unsigned long error_code, 903 - unsigned long address, struct vm_area_struct *vma) 900 + unsigned long address, struct mm_struct *mm, 901 + struct vm_area_struct *vma) 904 902 { 905 903 /* 906 904 * This OSPKE check is not strictly necessary at runtime. ··· 931 927 */ 932 928 u32 pkey = vma_pkey(vma); 933 929 934 - __bad_area(regs, error_code, address, pkey, SEGV_PKUERR); 930 + __bad_area(regs, error_code, address, mm, vma, pkey, SEGV_PKUERR); 935 931 } else { 936 - __bad_area(regs, error_code, address, 0, SEGV_ACCERR); 932 + __bad_area(regs, error_code, address, mm, vma, 0, SEGV_ACCERR); 937 933 } 938 934 } 939 935 ··· 1361 1357 goto lock_mmap; 1362 1358 1363 1359 if (unlikely(access_error(error_code, vma))) { 1364 - vma_end_read(vma); 1365 - goto lock_mmap; 1360 + bad_area_access_error(regs, error_code, address, NULL, vma); 1361 + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); 1362 + return; 1366 1363 } 1367 1364 fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); 1368 1365 if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) ··· 1399 1394 * we can handle it.. 1400 1395 */ 1401 1396 if (unlikely(access_error(error_code, vma))) { 1402 - bad_area_access_error(regs, error_code, address, vma); 1397 + bad_area_access_error(regs, error_code, address, mm, vma); 1403 1398 return; 1404 1399 } 1405 1400