mm/migrate_device: handle partially mapped folios during collection

Extend migrate_vma_collect_pmd() to handle partially mapped large folios
that require splitting before migration can proceed.

During PTE walk in the collection phase, if a large folio is only
partially mapped in the migration range, it must be split to ensure the
folio is correctly migrated.

[matthew.brost@intel.com: handle partially mapped folios during split]
Link: https://lkml.kernel.org/r/20251120230825.181072-1-matthew.brost@intel.com
Link: https://lkml.kernel.org/r/20251001065707.920170-7-balbirs@nvidia.com
Signed-off-by: Balbir Singh <balbirs@nvidia.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lyude Paul <lyude@redhat.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Cc: Mika Penttilä <mpenttil@redhat.com>
Cc: Francois Dugast <francois.dugast@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Balbir Singh and committed by

Andrew Morton 6 months ago 022a12de 14628729

+69 -1

1 changed file

expand all

migrate_device.c

+69 -1

mm/migrate_device.c

··· 54 54 return 0; 55 55 } 56 56 57 + /** 58 + * migrate_vma_split_folio() - Helper function to split a THP folio 59 + * @folio: the folio to split 60 + * @fault_page: struct page associated with the fault if any 61 + * 62 + * Returns 0 on success 63 + */ 64 + static int migrate_vma_split_folio(struct folio *folio, 65 + struct page *fault_page) 66 + { 67 + int ret; 68 + struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL; 69 + struct folio *new_fault_folio = NULL; 70 + 71 + if (folio != fault_folio) { 72 + folio_get(folio); 73 + folio_lock(folio); 74 + } 75 + 76 + ret = split_folio(folio); 77 + if (ret) { 78 + if (folio != fault_folio) { 79 + folio_unlock(folio); 80 + folio_put(folio); 81 + } 82 + return ret; 83 + } 84 + 85 + new_fault_folio = fault_page ? page_folio(fault_page) : NULL; 86 + 87 + /* 88 + * Ensure the lock is held on the correct 89 + * folio after the split 90 + */ 91 + if (!new_fault_folio) { 92 + folio_unlock(folio); 93 + folio_put(folio); 94 + } else if (folio != new_fault_folio) { 95 + if (new_fault_folio != fault_folio) { 96 + folio_get(new_fault_folio); 97 + folio_lock(new_fault_folio); 98 + } 99 + folio_unlock(folio); 100 + folio_put(folio); 101 + } 102 + 103 + return 0; 104 + } 105 + 57 106 static int migrate_vma_collect_pmd(pmd_t *pmdp, 58 107 unsigned long start, 59 108 unsigned long end, ··· 156 107 } 157 108 } 158 109 159 - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); 110 + ptep = pte_offset_map_lock(mm, pmdp, start, &ptl); 160 111 if (!ptep) 161 112 goto again; 162 113 arch_enter_lazy_mmu_mode(); 114 + ptep += (addr - start) / PAGE_SIZE; 163 115 164 116 for (; addr < end; addr += PAGE_SIZE, ptep++) { 165 117 struct dev_pagemap *pgmap; ··· 220 170 MIGRATE_VMA_SELECT_DEVICE_COHERENT) || 221 171 pgmap->owner != migrate->pgmap_owner) 222 172 goto next; 173 + } 174 + folio = page ? page_folio(page) : NULL; 175 + if (folio && folio_test_large(folio)) { 176 + int ret; 177 + 178 + arch_leave_lazy_mmu_mode(); 179 + pte_unmap_unlock(ptep, ptl); 180 + ret = migrate_vma_split_folio(folio, 181 + migrate->fault_page); 182 + 183 + if (ret) { 184 + if (unmapped) 185 + flush_tlb_range(walk->vma, start, end); 186 + 187 + return migrate_vma_collect_skip(addr, end, walk); 188 + } 189 + 190 + goto again; 223 191 } 224 192 mpfn = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; 225 193 mpfn |= pte_write(pte) ? MIGRATE_PFN_WRITE : 0;

Configure Feed

Configure Feed