tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: page_alloc: tighten up find_suitable_fallback()

find_suitable_fallback() is not as efficient as it could be, and somewhat
difficult to follow.

1. should_try_claim_block() is a loop invariant. There is no point in
checking fallback areas if the caller is interested in claimable
blocks but the order and the migratetype don't allow for that.

2. __rmqueue_steal() doesn't care about claimability, so it shouldn't
have to run those tests.

Different callers want different things from this helper:

1. __compact_finished() scans orders up until it finds a claimable block
2. __rmqueue_claim() scans orders down as long as blocks are claimable
3. __rmqueue_steal() doesn't care about claimability at all

Move should_try_claim_block() out of the loop. Only test it for the
two callers who care in the first place. Distinguish "no blocks" from
"order + mt are not claimable" in the return value; __rmqueue_claim()
can stop once order becomes unclaimable, __compact_finished() can keep
advancing until order becomes claimable.

Before:

Performance counter stats for './run case-lru-file-mmap-read' (5 runs):

85,294.85 msec task-clock # 5.644 CPUs utilized ( +- 0.32% )
15,968 context-switches # 187.209 /sec ( +- 3.81% )
153 cpu-migrations # 1.794 /sec ( +- 3.29% )
801,808 page-faults # 9.400 K/sec ( +- 0.10% )
733,358,331,786 instructions # 1.87 insn per cycle ( +- 0.20% ) (64.94%)
392,622,904,199 cycles # 4.603 GHz ( +- 0.31% ) (64.84%)
148,563,488,531 branches # 1.742 G/sec ( +- 0.18% ) (63.86%)
152,143,228 branch-misses # 0.10% of all branches ( +- 1.19% ) (62.82%)

15.1128 +- 0.0637 seconds time elapsed ( +- 0.42% )

After:

Performance counter stats for './run case-lru-file-mmap-read' (5 runs):

84,380.21 msec task-clock # 5.664 CPUs utilized ( +- 0.21% )
16,656 context-switches # 197.392 /sec ( +- 3.27% )
151 cpu-migrations # 1.790 /sec ( +- 3.28% )
801,703 page-faults # 9.501 K/sec ( +- 0.09% )
731,914,183,060 instructions # 1.88 insn per cycle ( +- 0.38% ) (64.90%)
388,673,535,116 cycles # 4.606 GHz ( +- 0.24% ) (65.06%)
148,251,482,143 branches # 1.757 G/sec ( +- 0.37% ) (63.92%)
149,766,550 branch-misses # 0.10% of all branches ( +- 1.22% ) (62.88%)

14.8968 +- 0.0486 seconds time elapsed ( +- 0.33% )

Link: https://lkml.kernel.org/r/20250407180154.63348-2-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Brendan Jackman <jackmanb@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Carlos Song <carlos.song@nxp.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Johannes Weiner and committed by
Andrew Morton ee414bd9 6e3092d7

+15 -22
+1 -3
mm/compaction.c
··· 2344 2344 ret = COMPACT_NO_SUITABLE_PAGE; 2345 2345 for (order = cc->order; order < NR_PAGE_ORDERS; order++) { 2346 2346 struct free_area *area = &cc->zone->free_area[order]; 2347 - bool claim_block; 2348 2347 2349 2348 /* Job done if page is free of the right migratetype */ 2350 2349 if (!free_area_empty(area, migratetype)) ··· 2359 2360 * Job done if allocation would steal freepages from 2360 2361 * other migratetype buddy lists. 2361 2362 */ 2362 - if (find_suitable_fallback(area, order, migratetype, 2363 - true, &claim_block) != -1) 2363 + if (find_suitable_fallback(area, order, migratetype, true) >= 0) 2364 2364 /* 2365 2365 * Movable pages are OK in any pageblock. If we are 2366 2366 * stealing for a non-movable allocation, make sure
+1 -1
mm/internal.h
··· 910 910 911 911 912 912 int find_suitable_fallback(struct free_area *area, unsigned int order, 913 - int migratetype, bool claim_only, bool *claim_block); 913 + int migratetype, bool claimable); 914 914 915 915 static inline bool free_area_empty(struct free_area *area, int migratetype) 916 916 {
+13 -18
mm/page_alloc.c
··· 2068 2068 2069 2069 /* 2070 2070 * Check whether there is a suitable fallback freepage with requested order. 2071 - * Sets *claim_block to instruct the caller whether it should convert a whole 2072 - * pageblock to the returned migratetype. 2073 - * If only_claim is true, this function returns fallback_mt only if 2071 + * If claimable is true, this function returns fallback_mt only if 2074 2072 * we would do this whole-block claiming. This would help to reduce 2075 2073 * fragmentation due to mixed migratetype pages in one pageblock. 2076 2074 */ 2077 2075 int find_suitable_fallback(struct free_area *area, unsigned int order, 2078 - int migratetype, bool only_claim, bool *claim_block) 2076 + int migratetype, bool claimable) 2079 2077 { 2080 2078 int i; 2081 - int fallback_mt; 2079 + 2080 + if (claimable && !should_try_claim_block(order, migratetype)) 2081 + return -2; 2082 2082 2083 2083 if (area->nr_free == 0) 2084 2084 return -1; 2085 2085 2086 - *claim_block = false; 2087 2086 for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) { 2088 - fallback_mt = fallbacks[migratetype][i]; 2089 - if (free_area_empty(area, fallback_mt)) 2090 - continue; 2087 + int fallback_mt = fallbacks[migratetype][i]; 2091 2088 2092 - if (should_try_claim_block(order, migratetype)) 2093 - *claim_block = true; 2094 - 2095 - if (*claim_block || !only_claim) 2089 + if (!free_area_empty(area, fallback_mt)) 2096 2090 return fallback_mt; 2097 2091 } 2098 2092 ··· 2183 2189 int min_order = order; 2184 2190 struct page *page; 2185 2191 int fallback_mt; 2186 - bool claim_block; 2187 2192 2188 2193 /* 2189 2194 * Do not steal pages from freelists belonging to other pageblocks ··· 2201 2208 --current_order) { 2202 2209 area = &(zone->free_area[current_order]); 2203 2210 fallback_mt = find_suitable_fallback(area, current_order, 2204 - start_migratetype, false, &claim_block); 2211 + start_migratetype, true); 2212 + 2213 + /* No block in that order */ 2205 2214 if (fallback_mt == -1) 2206 2215 continue; 2207 2216 2208 - if (!claim_block) 2217 + /* Advanced into orders too low to claim, abort */ 2218 + if (fallback_mt == -2) 2209 2219 break; 2210 2220 2211 2221 page = get_page_from_free_area(area, fallback_mt); ··· 2236 2240 int current_order; 2237 2241 struct page *page; 2238 2242 int fallback_mt; 2239 - bool claim_block; 2240 2243 2241 2244 for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) { 2242 2245 area = &(zone->free_area[current_order]); 2243 2246 fallback_mt = find_suitable_fallback(area, current_order, 2244 - start_migratetype, false, &claim_block); 2247 + start_migratetype, false); 2245 2248 if (fallback_mt == -1) 2246 2249 continue; 2247 2250