mm/huge_memory: add and use normal_or_softleaf_folio_pmd()

Now we have pmd_to_softleaf_folio() available to us which also raises a
CONFIG_DEBUG_VM warning if unexpectedly an invalid softleaf entry, we can
now abstract folio handling altogether.

vm_normal_folio() deals with the huge zero page (which is present), as well
as PFN map/mixed map mappings in both cases returning NULL.

Otherwise, we try to obtain the softleaf folio.

This makes the logic far easier to comprehend and has it use the standard
vm_normal_folio_pmd() path for decoding of present entries.

Finally, we have to update the flushing logic to only do so if a folio is
established.

This patch also makes the 'is_present' value more accurate - because PFN
map, mixed map and zero huge pages are present, just not present and
'normal'.

[ljs@kernel.org: avoid bisection hazard]
Link: https://lkml.kernel.org/r/d0cc6161-77a4-42ba-a411-96c23c78df1b@lucifer.local
Link: https://lkml.kernel.org/r/c2be872d64ef9573b80727d9ab5446cf002f17b5.1774029655.git.ljs@kernel.org
Signed-off-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Qi Zheng <zhengqi.arch@bytedance.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Lorenzo Stoakes (Oracle) and committed by

Andrew Morton 2 months ago d80a9cb1 64b7d889

+19 -28

1 changed file

expand all

huge_memory.c

+19 -28

mm/huge_memory.c

··· 2419 2419 add_mm_counter(mm, mm_counter_file(folio), 2420 2420 -HPAGE_PMD_NR); 2421 2421 2422 - /* 2423 - * Use flush_needed to indicate whether the PMD entry 2424 - * is present, instead of checking pmd_present() again. 2425 - */ 2426 2422 if (is_present && pmd_young(pmdval) && 2427 2423 likely(vma_has_recency(vma))) 2428 2424 folio_mark_accessed(folio); ··· 2427 2431 /* Device private folios are pinned. */ 2428 2432 if (is_device_private) 2429 2433 folio_put(folio); 2434 + } 2435 + 2436 + static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma, 2437 + unsigned long addr, pmd_t pmdval, bool is_present) 2438 + { 2439 + if (is_present) 2440 + return vm_normal_folio_pmd(vma, addr, pmdval); 2441 + 2442 + if (!thp_migration_supported()) 2443 + WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); 2444 + return pmd_to_softleaf_folio(pmdval); 2430 2445 } 2431 2446 2432 2447 /** ··· 2474 2467 tlb->fullmm); 2475 2468 arch_check_zapped_pmd(vma, orig_pmd); 2476 2469 tlb_remove_pmd_tlb_entry(tlb, pmd, addr); 2477 - if (vma_is_special_huge(vma)) 2478 - goto out; 2479 - if (is_huge_zero_pmd(orig_pmd)) { 2480 - if (!vma_is_dax(vma)) 2481 - has_deposit = true; 2482 - goto out; 2483 - } 2484 2470 2485 - if (pmd_present(orig_pmd)) { 2486 - folio = pmd_folio(orig_pmd); 2487 - is_present = true; 2488 - } else if (pmd_is_valid_softleaf(orig_pmd)) { 2489 - const softleaf_t entry = softleaf_from_pmd(orig_pmd); 2471 + is_present = pmd_present(orig_pmd); 2472 + folio = normal_or_softleaf_folio_pmd(vma, addr, orig_pmd, is_present); 2473 + if (folio) 2474 + zap_huge_pmd_folio(mm, vma, orig_pmd, folio, is_present, 2475 + &has_deposit); 2476 + else if (is_huge_zero_pmd(orig_pmd)) 2477 + has_deposit = has_deposit || !vma_is_dax(vma); 2490 2478 2491 - folio = softleaf_to_folio(entry); 2492 - if (!thp_migration_supported()) 2493 - WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); 2494 - } else { 2495 - WARN_ON_ONCE(true); 2496 - goto out; 2497 - } 2498 - 2499 - zap_huge_pmd_folio(mm, vma, orig_pmd, folio, is_present, &has_deposit); 2500 - 2501 - out: 2502 2479 if (has_deposit) 2503 2480 zap_deposited_table(mm, pmd); 2504 2481 2505 2482 spin_unlock(ptl); 2506 - if (is_present) 2483 + if (is_present && folio) 2507 2484 tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); 2508 2485 return true; 2509 2486 }

Configure Feed

Configure Feed