tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs

This performs lazy tlb mm shootdown when doing the exit TLB flush when
all mm users go away and user mappings are removed, which avoids having
to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
references disappear.

powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
need to be invalidated (unless TLBIE is disabled), so this doesn't
necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
expensive.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
[mpe: Squash in preempt_disable/enable() fix from Nick]
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://msgid.link/20230524060821.148015-5-npiggin@gmail.com

authored by

Nicholas Piggin and committed by
Michael Ellerman e43c0a0c 177255af

+29 -1
+29 -1
arch/powerpc/mm/book3s64/radix_tlb.c
··· 1313 1313 * See the comment for radix in arch_exit_mmap(). 1314 1314 */ 1315 1315 if (tlb->fullmm) { 1316 - __flush_all_mm(mm, true); 1316 + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { 1317 + /* 1318 + * Shootdown based lazy tlb mm refcounting means we 1319 + * have to IPI everyone in the mm_cpumask anyway soon 1320 + * when the mm goes away, so might as well do it as 1321 + * part of the final flush now. 1322 + * 1323 + * If lazy shootdown was improved to reduce IPIs (e.g., 1324 + * by batching), then it may end up being better to use 1325 + * tlbies here instead. 1326 + */ 1327 + preempt_disable(); 1328 + 1329 + smp_mb(); /* see radix__flush_tlb_mm */ 1330 + exit_flush_lazy_tlbs(mm); 1331 + _tlbiel_pid(mm->context.id, RIC_FLUSH_ALL); 1332 + 1333 + /* 1334 + * It should not be possible to have coprocessors still 1335 + * attached here. 1336 + */ 1337 + if (WARN_ON_ONCE(atomic_read(&mm->context.copros) > 0)) 1338 + __flush_all_mm(mm, true); 1339 + 1340 + preempt_enable(); 1341 + } else { 1342 + __flush_all_mm(mm, true); 1343 + } 1344 + 1317 1345 } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { 1318 1346 if (!tlb->freed_tables) 1319 1347 radix__flush_tlb_mm(mm);