mm/huge_memory: refactor change_huge_pmd() non-present logic

Similar to copy_huge_pmd(), there is a large mass of open-coded logic for
the CONFIG_ARCH_ENABLE_THP_MIGRATION non-present entry case that does not
use thp_migration_supported() consistently.

Resolve this by separating out this logic and introduce
change_non_present_huge_pmd().

No functional change intended.

Link: https://lkml.kernel.org/r/451b85636ad711e307fdfbff19af699fdab4d05f.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Mathew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Wei Xu <weixugc@google.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes and committed by

Andrew Morton 6 months ago 5dfa7916 e244d82d

+39 -33

1 changed file

expand all

huge_memory.c

+39 -33

mm/huge_memory.c

··· 2424 2424 return false; 2425 2425 } 2426 2426 2427 + static void change_non_present_huge_pmd(struct mm_struct *mm, 2428 + unsigned long addr, pmd_t *pmd, bool uffd_wp, 2429 + bool uffd_wp_resolve) 2430 + { 2431 + swp_entry_t entry = pmd_to_swp_entry(*pmd); 2432 + struct folio *folio = pfn_swap_entry_folio(entry); 2433 + pmd_t newpmd; 2434 + 2435 + VM_WARN_ON(!is_pmd_non_present_folio_entry(*pmd)); 2436 + if (is_writable_migration_entry(entry)) { 2437 + /* 2438 + * A protection check is difficult so 2439 + * just be safe and disable write 2440 + */ 2441 + if (folio_test_anon(folio)) 2442 + entry = make_readable_exclusive_migration_entry(swp_offset(entry)); 2443 + else 2444 + entry = make_readable_migration_entry(swp_offset(entry)); 2445 + newpmd = swp_entry_to_pmd(entry); 2446 + if (pmd_swp_soft_dirty(*pmd)) 2447 + newpmd = pmd_swp_mksoft_dirty(newpmd); 2448 + } else if (is_writable_device_private_entry(entry)) { 2449 + entry = make_readable_device_private_entry(swp_offset(entry)); 2450 + newpmd = swp_entry_to_pmd(entry); 2451 + } else { 2452 + newpmd = *pmd; 2453 + } 2454 + 2455 + if (uffd_wp) 2456 + newpmd = pmd_swp_mkuffd_wp(newpmd); 2457 + else if (uffd_wp_resolve) 2458 + newpmd = pmd_swp_clear_uffd_wp(newpmd); 2459 + if (!pmd_same(*pmd, newpmd)) 2460 + set_pmd_at(mm, addr, pmd, newpmd); 2461 + } 2462 + 2427 2463 /* 2428 2464 * Returns 2429 2465 * - 0 if PMD could not be locked ··· 2488 2452 if (!ptl) 2489 2453 return 0; 2490 2454 2491 - #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION 2492 - if (is_swap_pmd(*pmd)) { 2493 - swp_entry_t entry = pmd_to_swp_entry(*pmd); 2494 - struct folio *folio = pfn_swap_entry_folio(entry); 2495 - pmd_t newpmd; 2496 - 2497 - VM_WARN_ON(!is_pmd_non_present_folio_entry(*pmd)); 2498 - if (is_writable_migration_entry(entry)) { 2499 - /* 2500 - * A protection check is difficult so 2501 - * just be safe and disable write 2502 - */ 2503 - if (folio_test_anon(folio)) 2504 - entry = make_readable_exclusive_migration_entry(swp_offset(entry)); 2505 - else 2506 - entry = make_readable_migration_entry(swp_offset(entry)); 2507 - newpmd = swp_entry_to_pmd(entry); 2508 - if (pmd_swp_soft_dirty(*pmd)) 2509 - newpmd = pmd_swp_mksoft_dirty(newpmd); 2510 - } else if (is_writable_device_private_entry(entry)) { 2511 - entry = make_readable_device_private_entry(swp_offset(entry)); 2512 - newpmd = swp_entry_to_pmd(entry); 2513 - } else { 2514 - newpmd = *pmd; 2515 - } 2516 - 2517 - if (uffd_wp) 2518 - newpmd = pmd_swp_mkuffd_wp(newpmd); 2519 - else if (uffd_wp_resolve) 2520 - newpmd = pmd_swp_clear_uffd_wp(newpmd); 2521 - if (!pmd_same(*pmd, newpmd)) 2522 - set_pmd_at(mm, addr, pmd, newpmd); 2455 + if (thp_migration_supported() && is_swap_pmd(*pmd)) { 2456 + change_non_present_huge_pmd(mm, addr, pmd, uffd_wp, 2457 + uffd_wp_resolve); 2523 2458 goto unlock; 2524 2459 } 2525 - #endif 2526 2460 2527 2461 if (prot_numa) { 2528 2462

Configure Feed

Configure Feed