tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mmu_gather: move tlb flush for VM_PFNMAP/VM_MIXEDMAP vmas into free_pgtables()

Commit b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas") added a
forced tlbflush to tlb_vma_end(), which is required to avoid a race
between munmap() and unmap_mapping_range(). However it added some
overhead to other paths where tlb_vma_end() is used, but vmas are not
removed, e.g. madvise(MADV_DONTNEED).

Fix this by moving the tlb flush out of tlb_end_vma() into new
tlb_flush_vmas() called from free_pgtables(), somewhat similar to the
stable version of the original commit: commit 895428ee124a ("mm: Force TLB
flush for PFNMAP mappings before unlink_file_vma()").

Note, that if tlb->fullmm is set, no flush is required, as the whole mm is
about to be destroyed.

Link: https://lkml.kernel.org/r/20250522012838.163876-1-roman.gushchin@linux.dev
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
Reviewed-by: Jann Horn <jannh@google.com>
Acked-by: Hugh Dickins <hughd@google.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Will Deacon <will@kernel.org>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>
Cc: Nick Piggin <npiggin@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Roman Gushchin and committed by
Andrew Morton bfe125f1 28615e6e

+39 -10
+36 -10
include/asm-generic/tlb.h
··· 58 58 * Defaults to flushing at tlb_end_vma() to reset the range; helps when 59 59 * there's large holes between the VMAs. 60 60 * 61 + * - tlb_free_vmas() 62 + * 63 + * tlb_free_vmas() marks the start of unlinking of one or more vmas 64 + * and freeing page-tables. 65 + * 61 66 * - tlb_remove_table() 62 67 * 63 68 * tlb_remove_table() is the basic primitive to free page-table directories ··· 469 464 */ 470 465 tlb->vma_huge = is_vm_hugetlb_page(vma); 471 466 tlb->vma_exec = !!(vma->vm_flags & VM_EXEC); 472 - tlb->vma_pfn = !!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)); 467 + 468 + /* 469 + * Track if there's at least one VM_PFNMAP/VM_MIXEDMAP vma 470 + * in the tracked range, see tlb_free_vmas(). 471 + */ 472 + tlb->vma_pfn |= !!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)); 473 473 } 474 474 475 475 static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) ··· 558 548 559 549 static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) 560 550 { 551 + if (tlb->fullmm || IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) 552 + return; 553 + 554 + /* 555 + * Do a TLB flush and reset the range at VMA boundaries; this avoids 556 + * the ranges growing with the unused space between consecutive VMAs, 557 + * but also the mmu_gather::vma_* flags from tlb_start_vma() rely on 558 + * this. 559 + */ 560 + tlb_flush_mmu_tlbonly(tlb); 561 + } 562 + 563 + static inline void tlb_free_vmas(struct mmu_gather *tlb) 564 + { 561 565 if (tlb->fullmm) 562 566 return; 563 567 564 568 /* 565 569 * VM_PFNMAP is more fragile because the core mm will not track the 566 - * page mapcount -- there might not be page-frames for these PFNs after 567 - * all. Force flush TLBs for such ranges to avoid munmap() vs 568 - * unmap_mapping_range() races. 570 + * page mapcount -- there might not be page-frames for these PFNs 571 + * after all. 572 + * 573 + * Specifically() there is a race between munmap() and 574 + * unmap_mapping_range(), where munmap() will unlink the VMA, such 575 + * that unmap_mapping_range() will no longer observe the VMA and 576 + * no-op, without observing the TLBI, returning prematurely. 577 + * 578 + * So if we're about to unlink such a VMA, and we have pending 579 + * TLBI for such a vma, flush things now. 569 580 */ 570 - if (tlb->vma_pfn || !IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) { 571 - /* 572 - * Do a TLB flush and reset the range at VMA boundaries; this avoids 573 - * the ranges growing with the unused space between consecutive VMAs. 574 - */ 581 + if (tlb->vma_pfn) 575 582 tlb_flush_mmu_tlbonly(tlb); 576 - } 577 583 } 578 584 579 585 /*
+2
mm/memory.c
··· 358 358 { 359 359 struct unlink_vma_file_batch vb; 360 360 361 + tlb_free_vmas(tlb); 362 + 361 363 do { 362 364 unsigned long addr = vma->vm_start; 363 365 struct vm_area_struct *next;
+1
mm/mmu_gather.c
··· 424 424 #ifdef CONFIG_MMU_GATHER_PAGE_SIZE 425 425 tlb->page_size = 0; 426 426 #endif 427 + tlb->vma_pfn = 0; 427 428 428 429 __tlb_reset_range(tlb); 429 430 inc_tlb_flush_pending(tlb->mm);