tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: add get_and_clear_ptes() and clear_ptes()

Patch series "Optimizations for khugepaged", v4.

If the underlying folio mapped by the ptes is large, we can process those
ptes in a batch using folio_pte_batch().

For arm64 specifically, this results in a 16x reduction in the number of
ptep_get() calls, since on a contig block, ptep_get() on arm64 will
iterate through all 16 entries to collect a/d bits. Next, ptep_clear()
will cause a TLBI for every contig block in the range via
contpte_try_unfold(). Instead, use clear_ptes() to only do the TLBI at
the first and last contig block of the range.

For split folios, there will be no pte batching; the batch size returned
by folio_pte_batch() will be 1. For pagetable split folios, the ptes will
still point to the same large folio; for arm64, this results in the
optimization described above, and for other arches, a minor improvement is
expected due to a reduction in the number of function calls and batching
atomic operations.


This patch (of 3):

Let's add variants to be used where "full" does not apply -- which will
be the majority of cases in the future. "full" really only applies if
we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will
be added next.

Link: https://lkml.kernel.org/r/20250724052301.23844-2-dev.jain@arm.com
Signed-off-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Mariano Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

David Hildenbrand and committed by
Andrew Morton 3dfde978 1623717b

+48 -3
+1 -1
arch/arm64/mm/mmu.c
··· 1528 1528 pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, 1529 1529 pte_t *ptep, unsigned int nr) 1530 1530 { 1531 - pte_t pte = get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full = */ 0); 1531 + pte_t pte = get_and_clear_ptes(vma->vm_mm, addr, ptep, nr); 1532 1532 1533 1533 if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) { 1534 1534 /*
+45
include/linux/pgtable.h
··· 736 736 } 737 737 #endif 738 738 739 + /** 740 + * get_and_clear_ptes - Clear present PTEs that map consecutive pages of 741 + * the same folio, collecting dirty/accessed bits. 742 + * @mm: Address space the pages are mapped into. 743 + * @addr: Address the first page is mapped at. 744 + * @ptep: Page table pointer for the first entry. 745 + * @nr: Number of entries to clear. 746 + * 747 + * Use this instead of get_and_clear_full_ptes() if it is known that we don't 748 + * need to clear the full mm, which is mostly the case. 749 + * 750 + * Note that PTE bits in the PTE range besides the PFN can differ. For example, 751 + * some PTEs might be write-protected. 752 + * 753 + * Context: The caller holds the page table lock. The PTEs map consecutive 754 + * pages that belong to the same folio. The PTEs are all in the same PMD. 755 + */ 756 + static inline pte_t get_and_clear_ptes(struct mm_struct *mm, unsigned long addr, 757 + pte_t *ptep, unsigned int nr) 758 + { 759 + return get_and_clear_full_ptes(mm, addr, ptep, nr, 0); 760 + } 761 + 739 762 #ifndef clear_full_ptes 740 763 /** 741 764 * clear_full_ptes - Clear present PTEs that map consecutive pages of the same ··· 790 767 } 791 768 } 792 769 #endif 770 + 771 + /** 772 + * clear_ptes - Clear present PTEs that map consecutive pages of the same folio. 773 + * @mm: Address space the pages are mapped into. 774 + * @addr: Address the first page is mapped at. 775 + * @ptep: Page table pointer for the first entry. 776 + * @nr: Number of entries to clear. 777 + * 778 + * Use this instead of clear_full_ptes() if it is known that we don't need to 779 + * clear the full mm, which is mostly the case. 780 + * 781 + * Note that PTE bits in the PTE range besides the PFN can differ. For example, 782 + * some PTEs might be write-protected. 783 + * 784 + * Context: The caller holds the page table lock. The PTEs map consecutive 785 + * pages that belong to the same folio. The PTEs are all in the same PMD. 786 + */ 787 + static inline void clear_ptes(struct mm_struct *mm, unsigned long addr, 788 + pte_t *ptep, unsigned int nr) 789 + { 790 + clear_full_ptes(mm, addr, ptep, nr, 0); 791 + } 793 792 794 793 /* 795 794 * If two threads concurrently fault at the same page, the thread that
+1 -1
mm/mremap.c
··· 280 280 old_pte, max_nr_ptes); 281 281 force_flush = true; 282 282 } 283 - pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0); 283 + pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes); 284 284 pte = move_pte(pte, old_addr, new_addr); 285 285 pte = move_soft_dirty_pte(pte); 286 286
+1 -1
mm/rmap.c
··· 2036 2036 flush_cache_range(vma, address, end_addr); 2037 2037 2038 2038 /* Nuke the page table entry. */ 2039 - pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0); 2039 + pteval = get_and_clear_ptes(mm, address, pvmw.pte, nr_pages); 2040 2040 /* 2041 2041 * We clear the PTE but do not flush so potentially 2042 2042 * a remote CPU could still be writing to the folio.