mm: introduce pmd_is_huge() and use where appropriate

The leaf entry PMD case is confusing as only migration entries and device
private entries are valid at PMD level, not true swap entries.

We repeatedly perform checks of the form is_swap_pmd() || pmd_trans_huge()
which is itself confusing - it implies that leaf entries at PMD level
exist and are different from huge entries.

Address this confusion by introduced pmd_is_huge() which checks for either
case. Sadly due to header dependency issues (huge_mm.h is included very
early on in headers and cannot really rely on much else) we cannot use
pmd_is_valid_softleaf() here.

However since these are the only valid, handled cases the function is
still achieving what it intends to do.

We then replace all instances of is_swap_pmd() || pmd_trans_huge() with
pmd_is_huge() invocations and adjust logic accordingly to accommodate
this.

No functional change intended.

Link: https://lkml.kernel.org/r/00f79db3b15293cac8f7040a48d69c52d00117e4.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Mathew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Wei Xu <weixugc@google.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes and committed by

Andrew Morton 6 months ago 15eabc89 0ac881ef

+47 -9

6 changed files

expand all

include

linux

huge_mm.h

swapops.h

huge_memory.c

memory.c

mprotect.c

mremap.c

+35 -4

include/linux/huge_mm.h

··· 419 419 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, 420 420 unsigned long address, bool freeze); 421 421 422 + /** 423 + * pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf entry? 424 + * @pmd: The PMD to check. 425 + * 426 + * A huge PMD entry is a non-empty entry which is present and marked huge or a 427 + * software leaf entry. This check be performed without the appropriate locks 428 + * held, in which case the condition should be rechecked after they are 429 + * acquired. 430 + * 431 + * Returns: true if this PMD is huge, false otherwise. 432 + */ 433 + static inline bool pmd_is_huge(pmd_t pmd) 434 + { 435 + if (pmd_present(pmd)) { 436 + return pmd_trans_huge(pmd); 437 + } else if (!pmd_none(pmd)) { 438 + /* 439 + * Non-present PMDs must be valid huge non-present entries. We 440 + * cannot assert that here due to header dependency issues. 441 + */ 442 + return true; 443 + } 444 + 445 + return false; 446 + } 447 + 422 448 #define split_huge_pmd(__vma, __pmd, __address) \ 423 449 do { \ 424 450 pmd_t *____pmd = (__pmd); \ 425 - if (is_swap_pmd(*____pmd) || pmd_trans_huge(*____pmd)) \ 451 + if (pmd_is_huge(*____pmd)) \ 426 452 __split_huge_pmd(__vma, __pmd, __address, \ 427 453 false); \ 428 454 } while (0) ··· 495 469 static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd, 496 470 struct vm_area_struct *vma) 497 471 { 498 - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd)) 472 + if (pmd_is_huge(*pmd)) 499 473 return __pmd_trans_huge_lock(pmd, vma); 500 - else 501 - return NULL; 474 + 475 + return NULL; 502 476 } 503 477 static inline spinlock_t *pud_trans_huge_lock(pud_t *pud, 504 478 struct vm_area_struct *vma) ··· 768 742 static inline struct folio *get_persistent_huge_zero_folio(void) 769 743 { 770 744 return NULL; 745 + } 746 + 747 + static inline bool pmd_is_huge(pmd_t pmd) 748 + { 749 + return false; 771 750 } 772 751 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ 773 752

include/linux/swapops.h

··· 471 471 } 472 472 473 473 #else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ 474 + static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, 475 + struct page *page) 476 + { 477 + BUILD_BUG(); 478 + } 479 + 474 480 static inline void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, 475 481 struct page *new) 476 482 {

+2 -1

mm/huge_memory.c

··· 2735 2735 spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma) 2736 2736 { 2737 2737 spinlock_t *ptl; 2738 + 2738 2739 ptl = pmd_lock(vma->vm_mm, pmd); 2739 - if (likely(is_swap_pmd(*pmd) || pmd_trans_huge(*pmd))) 2740 + if (likely(pmd_is_huge(*pmd))) 2740 2741 return ptl; 2741 2742 spin_unlock(ptl); 2742 2743 return NULL;

+2 -2

mm/memory.c

··· 1374 1374 src_pmd = pmd_offset(src_pud, addr); 1375 1375 do { 1376 1376 next = pmd_addr_end(addr, end); 1377 - if (is_swap_pmd(*src_pmd) || pmd_trans_huge(*src_pmd)) { 1377 + if (pmd_is_huge(*src_pmd)) { 1378 1378 int err; 1379 1379 1380 1380 VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma); ··· 1917 1917 pmd = pmd_offset(pud, addr); 1918 1918 do { 1919 1919 next = pmd_addr_end(addr, end); 1920 - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd)) { 1920 + if (pmd_is_huge(*pmd)) { 1921 1921 if (next - addr != HPAGE_PMD_SIZE) 1922 1922 __split_huge_pmd(vma, pmd, addr, false); 1923 1923 else if (zap_huge_pmd(tlb, vma, pmd, addr)) {

+1 -1

mm/mprotect.c

··· 474 474 goto next; 475 475 476 476 _pmd = pmdp_get_lockless(pmd); 477 - if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd)) { 477 + if (pmd_is_huge(_pmd)) { 478 478 if ((next - addr != HPAGE_PMD_SIZE) || 479 479 pgtable_split_needed(vma, cp_flags)) { 480 480 __split_huge_pmd(vma, pmd, addr, false);

+1 -1

mm/mremap.c

··· 850 850 if (!new_pmd) 851 851 break; 852 852 again: 853 - if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd)) { 853 + if (pmd_is_huge(*old_pmd)) { 854 854 if (extent == HPAGE_PMD_SIZE && 855 855 move_pgt_entry(pmc, HPAGE_PMD, old_pmd, new_pmd)) 856 856 continue;

Configure Feed

Configure Feed