Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm/gup: local lru_add_drain() to avoid lru_add_drain_all()

In many cases, if collect_longterm_unpinnable_folios() does need to drain
the LRU cache to release a reference, the cache in question is on this
same CPU, and much more efficiently drained by a preliminary local
lru_add_drain(), than the later cross-CPU lru_add_drain_all().

Marked for stable, to counter the increase in lru_add_drain_all()s from
"mm/gup: check ref_count instead of lru before migration". Note for clean
backports: can take 6.16 commit a03db236aebf ("gup: optimize longterm
pin_user_pages() for large folio") first.

Link: https://lkml.kernel.org/r/66f2751f-283e-816d-9530-765db7edc465@google.com
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: David Hildenbrand <david@redhat.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keir Fraser <keirf@google.com>
Cc: Konstantin Khlebnikov <koct9i@gmail.com>
Cc: Li Zhe <lizhe.67@bytedance.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shivank Garg <shivankg@amd.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Wei Xu <weixugc@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: yangge <yangge1116@126.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Hugh Dickins and committed by
Andrew Morton
a09a8a1f 98c6d259

+11 -4
+11 -4
mm/gup.c
··· 2287 2287 struct pages_or_folios *pofs) 2288 2288 { 2289 2289 unsigned long collected = 0; 2290 - bool drain_allow = true; 2291 2290 struct folio *folio; 2291 + int drained = 0; 2292 2292 long i = 0; 2293 2293 2294 2294 for (folio = pofs_get_folio(pofs, i); folio; ··· 2307 2307 continue; 2308 2308 } 2309 2309 2310 - if (drain_allow && folio_ref_count(folio) != 2311 - folio_expected_ref_count(folio) + 1) { 2310 + if (drained == 0 && 2311 + folio_ref_count(folio) != 2312 + folio_expected_ref_count(folio) + 1) { 2313 + lru_add_drain(); 2314 + drained = 1; 2315 + } 2316 + if (drained == 1 && 2317 + folio_ref_count(folio) != 2318 + folio_expected_ref_count(folio) + 1) { 2312 2319 lru_add_drain_all(); 2313 - drain_allow = false; 2320 + drained = 2; 2314 2321 } 2315 2322 2316 2323 if (!folio_isolate_lru(folio))