tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

gup: optimize longterm pin_user_pages() for large folio

In the current implementation of longterm pin_user_pages(), we invoke
collect_longterm_unpinnable_folios(). This function iterates through the
list to check whether each folio belongs to the "longterm_unpinnabled"
category. The folios in this list essentially correspond to a contiguous
region of userspace addresses, with each folio representing a physical
address in increments of PAGESIZE.

If this userspace address range is mapped with large folio, we can
optimize the performance of function collect_longterm_unpinnable_folios()
by reducing the using of READ_ONCE() invoked in
pofs_get_folio()->page_folio()->_compound_head().

Also, we can simplify the logic of collect_longterm_unpinnable_folios().
Instead of comparing with prev_folio after calling pofs_get_folio(), we
can check whether the next page is within the same folio.

The performance test results, based on v6.15, obtained through the
gup_test tool from the kernel source tree are as follows. We achieve an
improvement of over 66% for large folio with pagesize=2M. For small
folio, we have only observed a very slight degradation in performance.

Without this patch:

[root@localhost ~] ./gup_test -HL -m 8192 -n 512
TAP version 13
1..1
# PIN_LONGTERM_BENCHMARK: Time: get:14391 put:10858 us#
ok 1 ioctl status 0
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[root@localhost ~]# ./gup_test -LT -m 8192 -n 512
TAP version 13
1..1
# PIN_LONGTERM_BENCHMARK: Time: get:130538 put:31676 us#
ok 1 ioctl status 0
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

With this patch:

[root@localhost ~] ./gup_test -HL -m 8192 -n 512
TAP version 13
1..1
# PIN_LONGTERM_BENCHMARK: Time: get:4867 put:10516 us#
ok 1 ioctl status 0
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[root@localhost ~]# ./gup_test -LT -m 8192 -n 512
TAP version 13
1..1
# PIN_LONGTERM_BENCHMARK: Time: get:131798 put:31328 us#
ok 1 ioctl status 0
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

[lizhe.67@bytedance.com: whitespace fix, per David]
Link: https://lkml.kernel.org/r/20250606091917.91384-1-lizhe.67@bytedance.com
Link: https://lkml.kernel.org/r/20250606023742.58344-1-lizhe.67@bytedance.com
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Peter Xu <peterx@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Li Zhe and committed by
Andrew Morton a03db236 96d81e47

+30 -8
+30 -8
mm/gup.c
··· 2296 2296 unpin_user_pages(pofs->pages, pofs->nr_entries); 2297 2297 } 2298 2298 2299 + static struct folio *pofs_next_folio(struct folio *folio, 2300 + struct pages_or_folios *pofs, long *index_ptr) 2301 + { 2302 + long i = *index_ptr + 1; 2303 + 2304 + if (!pofs->has_folios && folio_test_large(folio)) { 2305 + const unsigned long start_pfn = folio_pfn(folio); 2306 + const unsigned long end_pfn = start_pfn + folio_nr_pages(folio); 2307 + 2308 + for (; i < pofs->nr_entries; i++) { 2309 + unsigned long pfn = page_to_pfn(pofs->pages[i]); 2310 + 2311 + /* Is this page part of this folio? */ 2312 + if (pfn < start_pfn || pfn >= end_pfn) 2313 + break; 2314 + } 2315 + } 2316 + 2317 + if (unlikely(i == pofs->nr_entries)) 2318 + return NULL; 2319 + *index_ptr = i; 2320 + 2321 + return pofs_get_folio(pofs, i); 2322 + } 2323 + 2299 2324 /* 2300 2325 * Returns the number of collected folios. Return value is always >= 0. 2301 2326 */ ··· 2328 2303 struct list_head *movable_folio_list, 2329 2304 struct pages_or_folios *pofs) 2330 2305 { 2331 - unsigned long i, collected = 0; 2332 - struct folio *prev_folio = NULL; 2306 + unsigned long collected = 0; 2333 2307 bool drain_allow = true; 2308 + struct folio *folio; 2309 + long i = 0; 2334 2310 2335 - for (i = 0; i < pofs->nr_entries; i++) { 2336 - struct folio *folio = pofs_get_folio(pofs, i); 2337 - 2338 - if (folio == prev_folio) 2339 - continue; 2340 - prev_folio = folio; 2311 + for (folio = pofs_get_folio(pofs, i); folio; 2312 + folio = pofs_next_folio(folio, pofs, &i)) { 2341 2313 2342 2314 if (folio_is_longterm_pinnable(folio)) 2343 2315 continue;