tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: introduce generic lazy_mmu helpers

The implementation of the lazy MMU mode is currently entirely
arch-specific; core code directly calls arch helpers:
arch_{enter,leave}_lazy_mmu_mode().

We are about to introduce support for nested lazy MMU sections. As things
stand we'd have to duplicate that logic in every arch implementing
lazy_mmu - adding to a fair amount of logic already duplicated across
lazy_mmu implementations.

This patch therefore introduces a new generic layer that calls the
existing arch_* helpers. Two pair of calls are introduced:

* lazy_mmu_mode_enable() ... lazy_mmu_mode_disable()
This is the standard case where the mode is enabled for a given
block of code by surrounding it with enable() and disable()
calls.

* lazy_mmu_mode_pause() ... lazy_mmu_mode_resume()
This is for situations where the mode is temporarily disabled
by first calling pause() and then resume() (e.g. to prevent any
batching from occurring in a critical section).

The documentation in <linux/pgtable.h> will be updated in a subsequent
patch.

No functional change should be introduced at this stage. The
implementation of enable()/resume() and disable()/pause() is currently
identical, but nesting support will change that.

Most of the call sites have been updated using the following Coccinelle
script:

@@
@@
{
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_enable();
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_disable();
...
}

@@
@@
{
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_pause();
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_resume();
...
}

A couple of notes regarding x86:

* Xen is currently the only case where explicit handling is required
for lazy MMU when context-switching. This is purely an
implementation detail and using the generic lazy_mmu_mode_*
functions would cause trouble when nesting support is introduced,
because the generic functions must be called from the current task.
For that reason we still use arch_leave() and arch_enter() there.

* x86 calls arch_flush_lazy_mmu_mode() unconditionally in a few
places, but only defines it if PARAVIRT_XXL is selected, and we
are removing the fallback in <linux/pgtable.h>. Add a new fallback
definition to <asm/pgtable.h> to keep things building.

Link: https://lkml.kernel.org/r/20251215150323.2218608-8-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky <kevin.brodsky@arm.com>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Reviewed-by: Yeoreum Yun <yeoreum.yun@arm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Borislav Betkov <bp@alien8.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Juegren Gross <jgross@suse.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Gleinxer <tglx@linutronix.de>
Cc: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kevin Brodsky and committed by
Andrew Morton 0a096ab7 7303ecbf

+83 -61
+4 -4
arch/arm64/mm/mmu.c
··· 800 800 return -EINVAL; 801 801 802 802 mutex_lock(&pgtable_split_lock); 803 - arch_enter_lazy_mmu_mode(); 803 + lazy_mmu_mode_enable(); 804 804 805 805 /* 806 806 * The split_kernel_leaf_mapping_locked() may sleep, it is not a ··· 822 822 ret = split_kernel_leaf_mapping_locked(end); 823 823 } 824 824 825 - arch_leave_lazy_mmu_mode(); 825 + lazy_mmu_mode_disable(); 826 826 mutex_unlock(&pgtable_split_lock); 827 827 return ret; 828 828 } ··· 883 883 { 884 884 int ret; 885 885 886 - arch_enter_lazy_mmu_mode(); 886 + lazy_mmu_mode_enable(); 887 887 ret = walk_kernel_page_table_range_lockless(start, end, 888 888 &split_to_ptes_ops, NULL, &gfp); 889 - arch_leave_lazy_mmu_mode(); 889 + lazy_mmu_mode_disable(); 890 890 891 891 return ret; 892 892 }
+2 -2
arch/arm64/mm/pageattr.c
··· 110 110 if (WARN_ON_ONCE(ret)) 111 111 return ret; 112 112 113 - arch_enter_lazy_mmu_mode(); 113 + lazy_mmu_mode_enable(); 114 114 115 115 /* 116 116 * The caller must ensure that the range we are operating on does not ··· 119 119 */ 120 120 ret = walk_kernel_page_table_range_lockless(start, start + size, 121 121 &pageattr_ops, NULL, &data); 122 - arch_leave_lazy_mmu_mode(); 122 + lazy_mmu_mode_disable(); 123 123 124 124 return ret; 125 125 }
+4 -4
arch/powerpc/mm/book3s64/hash_tlb.c
··· 205 205 * way to do things but is fine for our needs here. 206 206 */ 207 207 local_irq_save(flags); 208 - arch_enter_lazy_mmu_mode(); 208 + lazy_mmu_mode_enable(); 209 209 for (; start < end; start += PAGE_SIZE) { 210 210 pte_t *ptep = find_init_mm_pte(start, &hugepage_shift); 211 211 unsigned long pte; ··· 217 217 continue; 218 218 hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift); 219 219 } 220 - arch_leave_lazy_mmu_mode(); 220 + lazy_mmu_mode_disable(); 221 221 local_irq_restore(flags); 222 222 } 223 223 ··· 237 237 * way to do things but is fine for our needs here. 238 238 */ 239 239 local_irq_save(flags); 240 - arch_enter_lazy_mmu_mode(); 240 + lazy_mmu_mode_enable(); 241 241 start_pte = pte_offset_map(pmd, addr); 242 242 if (!start_pte) 243 243 goto out; ··· 249 249 } 250 250 pte_unmap(start_pte); 251 251 out: 252 - arch_leave_lazy_mmu_mode(); 252 + lazy_mmu_mode_disable(); 253 253 local_irq_restore(flags); 254 254 }
+2 -2
arch/powerpc/mm/book3s64/subpage_prot.c
··· 73 73 pte = pte_offset_map_lock(mm, pmd, addr, &ptl); 74 74 if (!pte) 75 75 return; 76 - arch_enter_lazy_mmu_mode(); 76 + lazy_mmu_mode_enable(); 77 77 for (; npages > 0; --npages) { 78 78 pte_update(mm, addr, pte, 0, 0, 0); 79 79 addr += PAGE_SIZE; 80 80 ++pte; 81 81 } 82 - arch_leave_lazy_mmu_mode(); 82 + lazy_mmu_mode_disable(); 83 83 pte_unmap_unlock(pte - 1, ptl); 84 84 } 85 85
+1
arch/x86/include/asm/pgtable.h
··· 118 118 #define __pte(x) native_make_pte(x) 119 119 120 120 #define arch_end_context_switch(prev) do {} while(0) 121 + static inline void arch_flush_lazy_mmu_mode(void) {} 121 122 #endif /* CONFIG_PARAVIRT_XXL */ 122 123 123 124 static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set)
+2 -2
fs/proc/task_mmu.c
··· 2739 2739 return 0; 2740 2740 } 2741 2741 2742 - arch_enter_lazy_mmu_mode(); 2742 + lazy_mmu_mode_enable(); 2743 2743 2744 2744 if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) { 2745 2745 /* Fast path for performing exclusive WP */ ··· 2809 2809 if (flush_end) 2810 2810 flush_tlb_range(vma, start, addr); 2811 2811 2812 - arch_leave_lazy_mmu_mode(); 2812 + lazy_mmu_mode_disable(); 2813 2813 pte_unmap_unlock(start_pte, ptl); 2814 2814 2815 2815 cond_resched();
+25 -4
include/linux/pgtable.h
··· 235 235 * 236 236 * Nesting is not permitted and the mode cannot be used in interrupt context. 237 237 */ 238 - #ifndef CONFIG_ARCH_HAS_LAZY_MMU_MODE 239 - static inline void arch_enter_lazy_mmu_mode(void) {} 240 - static inline void arch_leave_lazy_mmu_mode(void) {} 241 - static inline void arch_flush_lazy_mmu_mode(void) {} 238 + #ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE 239 + static inline void lazy_mmu_mode_enable(void) 240 + { 241 + arch_enter_lazy_mmu_mode(); 242 + } 243 + 244 + static inline void lazy_mmu_mode_disable(void) 245 + { 246 + arch_leave_lazy_mmu_mode(); 247 + } 248 + 249 + static inline void lazy_mmu_mode_pause(void) 250 + { 251 + arch_leave_lazy_mmu_mode(); 252 + } 253 + 254 + static inline void lazy_mmu_mode_resume(void) 255 + { 256 + arch_enter_lazy_mmu_mode(); 257 + } 258 + #else 259 + static inline void lazy_mmu_mode_enable(void) {} 260 + static inline void lazy_mmu_mode_disable(void) {} 261 + static inline void lazy_mmu_mode_pause(void) {} 262 + static inline void lazy_mmu_mode_resume(void) {} 242 263 #endif 243 264 244 265 #ifndef pte_batch_hint
+4 -4
mm/kasan/shadow.c
··· 305 305 pte_t pte; 306 306 int index; 307 307 308 - arch_leave_lazy_mmu_mode(); 308 + lazy_mmu_mode_pause(); 309 309 310 310 index = PFN_DOWN(addr - data->start); 311 311 page = data->pages[index]; ··· 319 319 } 320 320 spin_unlock(&init_mm.page_table_lock); 321 321 322 - arch_enter_lazy_mmu_mode(); 322 + lazy_mmu_mode_resume(); 323 323 324 324 return 0; 325 325 } ··· 471 471 pte_t pte; 472 472 int none; 473 473 474 - arch_leave_lazy_mmu_mode(); 474 + lazy_mmu_mode_pause(); 475 475 476 476 spin_lock(&init_mm.page_table_lock); 477 477 pte = ptep_get(ptep); ··· 483 483 if (likely(!none)) 484 484 __free_page(pfn_to_page(pte_pfn(pte))); 485 485 486 - arch_enter_lazy_mmu_mode(); 486 + lazy_mmu_mode_resume(); 487 487 488 488 return 0; 489 489 }
+9 -9
mm/madvise.c
··· 453 453 if (!start_pte) 454 454 return 0; 455 455 flush_tlb_batched_pending(mm); 456 - arch_enter_lazy_mmu_mode(); 456 + lazy_mmu_mode_enable(); 457 457 for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) { 458 458 nr = 1; 459 459 ptent = ptep_get(pte); ··· 461 461 if (++batch_count == SWAP_CLUSTER_MAX) { 462 462 batch_count = 0; 463 463 if (need_resched()) { 464 - arch_leave_lazy_mmu_mode(); 464 + lazy_mmu_mode_disable(); 465 465 pte_unmap_unlock(start_pte, ptl); 466 466 cond_resched(); 467 467 goto restart; ··· 497 497 if (!folio_trylock(folio)) 498 498 continue; 499 499 folio_get(folio); 500 - arch_leave_lazy_mmu_mode(); 500 + lazy_mmu_mode_disable(); 501 501 pte_unmap_unlock(start_pte, ptl); 502 502 start_pte = NULL; 503 503 err = split_folio(folio); ··· 508 508 if (!start_pte) 509 509 break; 510 510 flush_tlb_batched_pending(mm); 511 - arch_enter_lazy_mmu_mode(); 511 + lazy_mmu_mode_enable(); 512 512 if (!err) 513 513 nr = 0; 514 514 continue; ··· 556 556 } 557 557 558 558 if (start_pte) { 559 - arch_leave_lazy_mmu_mode(); 559 + lazy_mmu_mode_disable(); 560 560 pte_unmap_unlock(start_pte, ptl); 561 561 } 562 562 if (pageout) ··· 675 675 if (!start_pte) 676 676 return 0; 677 677 flush_tlb_batched_pending(mm); 678 - arch_enter_lazy_mmu_mode(); 678 + lazy_mmu_mode_enable(); 679 679 for (; addr != end; pte += nr, addr += PAGE_SIZE * nr) { 680 680 nr = 1; 681 681 ptent = ptep_get(pte); ··· 724 724 if (!folio_trylock(folio)) 725 725 continue; 726 726 folio_get(folio); 727 - arch_leave_lazy_mmu_mode(); 727 + lazy_mmu_mode_disable(); 728 728 pte_unmap_unlock(start_pte, ptl); 729 729 start_pte = NULL; 730 730 err = split_folio(folio); ··· 735 735 if (!start_pte) 736 736 break; 737 737 flush_tlb_batched_pending(mm); 738 - arch_enter_lazy_mmu_mode(); 738 + lazy_mmu_mode_enable(); 739 739 if (!err) 740 740 nr = 0; 741 741 continue; ··· 775 775 if (nr_swap) 776 776 add_mm_counter(mm, MM_SWAPENTS, nr_swap); 777 777 if (start_pte) { 778 - arch_leave_lazy_mmu_mode(); 778 + lazy_mmu_mode_disable(); 779 779 pte_unmap_unlock(start_pte, ptl); 780 780 } 781 781 cond_resched();
+8 -8
mm/memory.c
··· 1256 1256 spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); 1257 1257 orig_src_pte = src_pte; 1258 1258 orig_dst_pte = dst_pte; 1259 - arch_enter_lazy_mmu_mode(); 1259 + lazy_mmu_mode_enable(); 1260 1260 1261 1261 do { 1262 1262 nr = 1; ··· 1325 1325 } while (dst_pte += nr, src_pte += nr, addr += PAGE_SIZE * nr, 1326 1326 addr != end); 1327 1327 1328 - arch_leave_lazy_mmu_mode(); 1328 + lazy_mmu_mode_disable(); 1329 1329 pte_unmap_unlock(orig_src_pte, src_ptl); 1330 1330 add_mm_rss_vec(dst_mm, rss); 1331 1331 pte_unmap_unlock(orig_dst_pte, dst_ptl); ··· 1846 1846 return addr; 1847 1847 1848 1848 flush_tlb_batched_pending(mm); 1849 - arch_enter_lazy_mmu_mode(); 1849 + lazy_mmu_mode_enable(); 1850 1850 do { 1851 1851 bool any_skipped = false; 1852 1852 ··· 1878 1878 direct_reclaim = try_get_and_clear_pmd(mm, pmd, &pmdval); 1879 1879 1880 1880 add_mm_rss_vec(mm, rss); 1881 - arch_leave_lazy_mmu_mode(); 1881 + lazy_mmu_mode_disable(); 1882 1882 1883 1883 /* Do the actual TLB flush before dropping ptl */ 1884 1884 if (force_flush) { ··· 2816 2816 mapped_pte = pte = pte_alloc_map_lock(mm, pmd, addr, &ptl); 2817 2817 if (!pte) 2818 2818 return -ENOMEM; 2819 - arch_enter_lazy_mmu_mode(); 2819 + lazy_mmu_mode_enable(); 2820 2820 do { 2821 2821 BUG_ON(!pte_none(ptep_get(pte))); 2822 2822 if (!pfn_modify_allowed(pfn, prot)) { ··· 2826 2826 set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot))); 2827 2827 pfn++; 2828 2828 } while (pte++, addr += PAGE_SIZE, addr != end); 2829 - arch_leave_lazy_mmu_mode(); 2829 + lazy_mmu_mode_disable(); 2830 2830 pte_unmap_unlock(mapped_pte, ptl); 2831 2831 return err; 2832 2832 } ··· 3177 3177 return -EINVAL; 3178 3178 } 3179 3179 3180 - arch_enter_lazy_mmu_mode(); 3180 + lazy_mmu_mode_enable(); 3181 3181 3182 3182 if (fn) { 3183 3183 do { ··· 3190 3190 } 3191 3191 *mask |= PGTBL_PTE_MODIFIED; 3192 3192 3193 - arch_leave_lazy_mmu_mode(); 3193 + lazy_mmu_mode_disable(); 3194 3194 3195 3195 if (mm != &init_mm) 3196 3196 pte_unmap_unlock(mapped_pte, ptl);
+4 -4
mm/migrate_device.c
··· 271 271 ptep = pte_offset_map_lock(mm, pmdp, start, &ptl); 272 272 if (!ptep) 273 273 goto again; 274 - arch_enter_lazy_mmu_mode(); 274 + lazy_mmu_mode_enable(); 275 275 ptep += (addr - start) / PAGE_SIZE; 276 276 277 277 for (; addr < end; addr += PAGE_SIZE, ptep++) { ··· 313 313 if (folio_test_large(folio)) { 314 314 int ret; 315 315 316 - arch_leave_lazy_mmu_mode(); 316 + lazy_mmu_mode_disable(); 317 317 pte_unmap_unlock(ptep, ptl); 318 318 ret = migrate_vma_split_folio(folio, 319 319 migrate->fault_page); ··· 356 356 if (folio && folio_test_large(folio)) { 357 357 int ret; 358 358 359 - arch_leave_lazy_mmu_mode(); 359 + lazy_mmu_mode_disable(); 360 360 pte_unmap_unlock(ptep, ptl); 361 361 ret = migrate_vma_split_folio(folio, 362 362 migrate->fault_page); ··· 485 485 if (unmapped) 486 486 flush_tlb_range(walk->vma, start, end); 487 487 488 - arch_leave_lazy_mmu_mode(); 488 + lazy_mmu_mode_disable(); 489 489 pte_unmap_unlock(ptep - 1, ptl); 490 490 491 491 return 0;
+2 -2
mm/mprotect.c
··· 233 233 is_private_single_threaded = vma_is_single_threaded_private(vma); 234 234 235 235 flush_tlb_batched_pending(vma->vm_mm); 236 - arch_enter_lazy_mmu_mode(); 236 + lazy_mmu_mode_enable(); 237 237 do { 238 238 nr_ptes = 1; 239 239 oldpte = ptep_get(pte); ··· 379 379 } 380 380 } 381 381 } while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end); 382 - arch_leave_lazy_mmu_mode(); 382 + lazy_mmu_mode_disable(); 383 383 pte_unmap_unlock(pte - 1, ptl); 384 384 385 385 return pages;
+2 -2
mm/mremap.c
··· 260 260 if (new_ptl != old_ptl) 261 261 spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); 262 262 flush_tlb_batched_pending(vma->vm_mm); 263 - arch_enter_lazy_mmu_mode(); 263 + lazy_mmu_mode_enable(); 264 264 265 265 for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE, 266 266 new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) { ··· 305 305 } 306 306 } 307 307 308 - arch_leave_lazy_mmu_mode(); 308 + lazy_mmu_mode_disable(); 309 309 if (force_flush) 310 310 flush_tlb_range(vma, old_end - len, old_end); 311 311 if (new_ptl != old_ptl)
+2 -2
mm/userfaultfd.c
··· 1103 1103 /* It's safe to drop the reference now as the page-table is holding one. */ 1104 1104 folio_put(*first_src_folio); 1105 1105 *first_src_folio = NULL; 1106 - arch_enter_lazy_mmu_mode(); 1106 + lazy_mmu_mode_enable(); 1107 1107 1108 1108 while (true) { 1109 1109 orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte); ··· 1140 1140 break; 1141 1141 } 1142 1142 1143 - arch_leave_lazy_mmu_mode(); 1143 + lazy_mmu_mode_disable(); 1144 1144 if (src_addr > src_start) 1145 1145 flush_tlb_range(src_vma, src_start, src_addr); 1146 1146
+6 -6
mm/vmalloc.c
··· 108 108 if (!pte) 109 109 return -ENOMEM; 110 110 111 - arch_enter_lazy_mmu_mode(); 111 + lazy_mmu_mode_enable(); 112 112 113 113 do { 114 114 if (unlikely(!pte_none(ptep_get(pte)))) { ··· 134 134 pfn++; 135 135 } while (pte += PFN_DOWN(size), addr += size, addr != end); 136 136 137 - arch_leave_lazy_mmu_mode(); 137 + lazy_mmu_mode_disable(); 138 138 *mask |= PGTBL_PTE_MODIFIED; 139 139 return 0; 140 140 } ··· 371 371 unsigned long size = PAGE_SIZE; 372 372 373 373 pte = pte_offset_kernel(pmd, addr); 374 - arch_enter_lazy_mmu_mode(); 374 + lazy_mmu_mode_enable(); 375 375 376 376 do { 377 377 #ifdef CONFIG_HUGETLB_PAGE ··· 390 390 WARN_ON(!pte_none(ptent) && !pte_present(ptent)); 391 391 } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); 392 392 393 - arch_leave_lazy_mmu_mode(); 393 + lazy_mmu_mode_disable(); 394 394 *mask |= PGTBL_PTE_MODIFIED; 395 395 } 396 396 ··· 538 538 if (!pte) 539 539 return -ENOMEM; 540 540 541 - arch_enter_lazy_mmu_mode(); 541 + lazy_mmu_mode_enable(); 542 542 543 543 do { 544 544 struct page *page = pages[*nr]; ··· 560 560 (*nr)++; 561 561 } while (pte++, addr += PAGE_SIZE, addr != end); 562 562 563 - arch_leave_lazy_mmu_mode(); 563 + lazy_mmu_mode_disable(); 564 564 *mask |= PGTBL_PTE_MODIFIED; 565 565 566 566 return err;
+6 -6
mm/vmscan.c
··· 3516 3516 return false; 3517 3517 } 3518 3518 3519 - arch_enter_lazy_mmu_mode(); 3519 + lazy_mmu_mode_enable(); 3520 3520 restart: 3521 3521 for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { 3522 3522 unsigned long pfn; ··· 3557 3557 if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) 3558 3558 goto restart; 3559 3559 3560 - arch_leave_lazy_mmu_mode(); 3560 + lazy_mmu_mode_disable(); 3561 3561 pte_unmap_unlock(pte, ptl); 3562 3562 3563 3563 return suitable_to_scan(total, young); ··· 3598 3598 if (!spin_trylock(ptl)) 3599 3599 goto done; 3600 3600 3601 - arch_enter_lazy_mmu_mode(); 3601 + lazy_mmu_mode_enable(); 3602 3602 3603 3603 do { 3604 3604 unsigned long pfn; ··· 3645 3645 3646 3646 walk_update_folio(walk, last, gen, dirty); 3647 3647 3648 - arch_leave_lazy_mmu_mode(); 3648 + lazy_mmu_mode_disable(); 3649 3649 spin_unlock(ptl); 3650 3650 done: 3651 3651 *first = -1; ··· 4244 4244 } 4245 4245 } 4246 4246 4247 - arch_enter_lazy_mmu_mode(); 4247 + lazy_mmu_mode_enable(); 4248 4248 4249 4249 pte -= (addr - start) / PAGE_SIZE; 4250 4250 ··· 4278 4278 4279 4279 walk_update_folio(walk, last, gen, dirty); 4280 4280 4281 - arch_leave_lazy_mmu_mode(); 4281 + lazy_mmu_mode_disable(); 4282 4282 4283 4283 /* feedback from rmap walkers to page table walkers */ 4284 4284 if (mm_state && suitable_to_scan(i, young))