Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: khugepaged: skip lazy-free folios

For example, create three task: hot1 -> cold -> hot2. After all three
task are created, each allocate memory 128MB. the hot1/hot2 task
continuously access 128 MB memory, while the cold task only accesses its
memory briefly and then call madvise(MADV_FREE). However, khugepaged
still prioritizes scanning the cold task and only scans the hot2 task
after completing the scan of the cold task.

All folios in VM_DROPPABLE are lazyfree, Collapsing maintains that
property, so we can just collapse and memory pressure in the future will
free it up. In contrast, collapsing in !VM_DROPPABLE does not maintain
that property, the collapsed folio will not be lazyfree and memory
pressure in the future will not be able to free it up.

So if the user has explicitly informed us via MADV_FREE that this memory
will be freed, and this vma does not have VM_DROPPABLE flags, it is
appropriate for khugepaged to skip it only, thereby avoiding unnecessary
scan and collapse operations to reducing CPU wastage.

Here are the performance test results:
(Throughput bigger is better, other smaller is better)

Testing on x86_64 machine:

| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.14 sec | 2.93 sec | -6.69% |
| cycles per access | 4.96 | 2.21 | -55.44% |
| Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% |
| dTLB-load-misses | 284814532 | 69597236 | -75.56% |

Testing on qemu-system-x86_64 -enable-kvm:

| task hot2 | without patch | with patch | delta |
|---------------------|---------------|---------------|---------|
| total accesses time | 3.35 sec | 2.96 sec | -11.64% |
| cycles per access | 7.29 | 2.07 | -71.60% |
| Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% |
| dTLB-load-misses | 241600871 | 3216108 | -98.67% |

[vernon2gm@gmail.com: add comment about VM_DROPPABLE in code, make it clearer]
Link: https://lkml.kernel.org/r/i4uowkt4h2ev47obm5h2vtd4zbk6fyw5g364up7kkjn2vmcikq@auepvqethj5r
Link: https://lkml.kernel.org/r/20260221093918.1456187-5-vernon2gm@gmail.com
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Acked-by: David Hildenbrand (arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Liam Howlett <Liam.Howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Vernon Yang and committed by
Andrew Morton
05620419 6cc153f9

+22
+1
include/trace/events/huge_memory.h
··· 25 25 EM( SCAN_PAGE_LRU, "page_not_in_lru") \ 26 26 EM( SCAN_PAGE_LOCK, "page_locked") \ 27 27 EM( SCAN_PAGE_ANON, "page_not_anon") \ 28 + EM( SCAN_PAGE_LAZYFREE, "page_lazyfree") \ 28 29 EM( SCAN_PAGE_COMPOUND, "page_compound") \ 29 30 EM( SCAN_ANY_PROCESS, "no_process_for_page") \ 30 31 EM( SCAN_VMA_NULL, "vma_null") \
+21
mm/khugepaged.c
··· 46 46 SCAN_PAGE_LRU, 47 47 SCAN_PAGE_LOCK, 48 48 SCAN_PAGE_ANON, 49 + SCAN_PAGE_LAZYFREE, 49 50 SCAN_PAGE_COMPOUND, 50 51 SCAN_ANY_PROCESS, 51 52 SCAN_VMA_NULL, ··· 577 576 578 577 folio = page_folio(page); 579 578 VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); 579 + 580 + /* 581 + * If the vma has the VM_DROPPABLE flag, the collapse will 582 + * preserve the lazyfree property without needing to skip. 583 + */ 584 + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && 585 + folio_test_lazyfree(folio) && !pte_dirty(pteval)) { 586 + result = SCAN_PAGE_LAZYFREE; 587 + goto out; 588 + } 580 589 581 590 /* See hpage_collapse_scan_pmd(). */ 582 591 if (folio_maybe_mapped_shared(folio)) { ··· 1335 1324 goto out_unmap; 1336 1325 } 1337 1326 folio = page_folio(page); 1327 + 1328 + /* 1329 + * If the vma has the VM_DROPPABLE flag, the collapse will 1330 + * preserve the lazyfree property without needing to skip. 1331 + */ 1332 + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && 1333 + folio_test_lazyfree(folio) && !pte_dirty(pteval)) { 1334 + result = SCAN_PAGE_LAZYFREE; 1335 + goto out_unmap; 1336 + } 1338 1337 1339 1338 if (!folio_test_anon(folio)) { 1340 1339 result = SCAN_PAGE_ANON;