Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

x86/mm: Only do broadcast flush from reclaim if pages were unmapped

Track whether pages were unmapped from any MM (even ones with a currently
empty mm_cpumask) by the reclaim code, to figure out whether or not
broadcast TLB flush should be done when reclaim finishes.

The reason any MM must be tracked, and not only ones contributing to the
tlbbatch cpumask, is that broadcast ASIDs are expected to be kept up to
date even on CPUs where the MM is not currently active.

This change allows reclaim to avoid doing TLB flushes when only clean page
cache pages and/or slab memory were reclaimed, which is fairly common.

( This is a simpler alternative to the code that was in my INVLPGB series
before, and it seems to capture most of the benefit due to how common
it is to reclaim only page cache. )

Signed-off-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lore.kernel.org/r/20250319132520.6b10ad90@fangorn

authored by

Rik van Riel and committed by
Ingo Molnar
0b7eb55c de844ef5

+8 -1
+5
arch/x86/include/asm/tlbbatch.h
··· 10 10 * the PFNs being flushed.. 11 11 */ 12 12 struct cpumask cpumask; 13 + /* 14 + * Set if pages were unmapped from any MM, even one that does not 15 + * have active CPUs in its cpumask. 16 + */ 17 + bool unmapped_pages; 13 18 }; 14 19 15 20 #endif /* _ARCH_X86_TLBBATCH_H */
+1
arch/x86/include/asm/tlbflush.h
··· 353 353 { 354 354 inc_mm_tlb_gen(mm); 355 355 cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); 356 + batch->unmapped_pages = true; 356 357 mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); 357 358 } 358 359
+2 -1
arch/x86/mm/tlb.c
··· 1633 1633 * a local TLB flush is needed. Optimize this use-case by calling 1634 1634 * flush_tlb_func_local() directly in this case. 1635 1635 */ 1636 - if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { 1636 + if (cpu_feature_enabled(X86_FEATURE_INVLPGB) && batch->unmapped_pages) { 1637 1637 invlpgb_flush_all_nonglobals(); 1638 + batch->unmapped_pages = false; 1638 1639 } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { 1639 1640 flush_tlb_multi(&batch->cpumask, info); 1640 1641 } else if (cpumask_test_cpu(cpu, &batch->cpumask)) {