mm: swap: add a fragment cluster list

Now swap cluster allocator arranges the clusters in LRU style, so the
"cold" cluster stay at the head of nonfull lists are the ones that were
used for allocation long time ago and still partially occupied. So if
allocator can't find enough contiguous slots to satisfy an high order
allocation, it's unlikely there will be slot being free on them to satisfy
the allocation, at least in a short period.

As a result, nonfull cluster scanning will waste time repeatly scanning
the unusable head of the list.

Also, multiple CPUs could content on the same head cluster of nonfull
list. Unlike free clusters which are removed from the list when any CPU
starts using it, nonfull cluster stays on the head.

So introduce a new list frag list, all scanned nonfull clusters will be
moved to this list. Both for avoiding repeated scanning and contention.

Frag list is still used as fallback for allocations, so if one CPU failed
to allocate one order of slots, it can still steal other CPU's clusters.
And order 0 will favor the fragmented clusters to better protect nonfull
clusters

If any slots on a fragment list are being freed, move the fragment list
back to nonfull list indicating it worth another scan on the cluster.
Compared to scan upon freeing a slot, this keep the scanning lazy and save
some CPU if there are still other clusters to use.

It may seems unneccessay to keep the fragmented cluster on list at all if
they can't be used for specific order allocation. But this will start to
make sense once reclaim dring scanning is ready.

Link: https://lkml.kernel.org/r/20240730-swap-allocator-v5-7-cb9c148b9297@kernel.org
Signed-off-by: Kairui Song <kasong@tencent.com>
Reported-by: Barry Song <21cnbao@gmail.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: "Huang, Ying" <ying.huang@intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Kalesh Singh <kaleshsingh@google.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kairui Song and committed by

Andrew Morton 2 years ago 477cb7ba 862590ac

+40 -4

2 changed files

expand all

include

linux

swap.h

swapfile.c

include/linux/swap.h

··· 259 259 }; 260 260 #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ 261 261 #define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ 262 + #define CLUSTER_FLAG_FRAG 4 /* This cluster is on nonfull list */ 262 263 263 264 /* 264 265 * The first page in the swap file is the swap header, which is always marked ··· 299 298 struct list_head free_clusters; /* free clusters list */ 300 299 struct list_head nonfull_clusters[SWAP_NR_ORDERS]; 301 300 /* list of cluster that contains at least one free slot */ 301 + struct list_head frag_clusters[SWAP_NR_ORDERS]; 302 + /* list of cluster that are fragmented or contented */ 302 303 unsigned int lowest_bit; /* index of first free in swap_map */ 303 304 unsigned int highest_bit; /* index of last free in swap_map */ 304 305 unsigned int pages; /* total of usable pages of swap */

+37 -4

mm/swapfile.c

··· 572 572 573 573 if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { 574 574 VM_BUG_ON(ci->flags & CLUSTER_FLAG_FREE); 575 - list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); 575 + if (ci->flags & CLUSTER_FLAG_FRAG) 576 + list_move_tail(&ci->list, &p->nonfull_clusters[ci->order]); 577 + else 578 + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); 576 579 ci->flags = CLUSTER_FLAG_NONFULL; 577 580 } 578 581 } ··· 613 610 ci->count += nr_pages; 614 611 615 612 if (ci->count == SWAPFILE_CLUSTER) { 616 - VM_BUG_ON(!(ci->flags & (CLUSTER_FLAG_FREE | CLUSTER_FLAG_NONFULL))); 613 + VM_BUG_ON(!(ci->flags & 614 + (CLUSTER_FLAG_FREE | CLUSTER_FLAG_NONFULL | CLUSTER_FLAG_FRAG))); 617 615 list_del(&ci->list); 618 616 ci->flags = 0; 619 617 } ··· 670 666 struct percpu_cluster *cluster; 671 667 struct swap_cluster_info *ci, *n; 672 668 unsigned int offset, found = 0; 669 + LIST_HEAD(fraged); 673 670 674 671 new_cluster: 675 672 lockdep_assert_held(&si->lock); ··· 691 686 692 687 if (order < PMD_ORDER) { 693 688 list_for_each_entry_safe(ci, n, &si->nonfull_clusters[order], list) { 689 + list_move_tail(&ci->list, &fraged); 690 + ci->flags = CLUSTER_FLAG_FRAG; 694 691 offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), 695 692 &found, order, usage); 696 693 if (found) 697 - goto done; 694 + break; 698 695 } 696 + 697 + if (!found) { 698 + list_for_each_entry_safe(ci, n, &si->frag_clusters[order], list) { 699 + offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), 700 + &found, order, usage); 701 + if (found) 702 + break; 703 + } 704 + } 705 + 706 + list_splice_tail(&fraged, &si->frag_clusters[order]); 699 707 } 708 + 709 + if (found) 710 + goto done; 700 711 701 712 if (!list_empty(&si->discard_clusters)) { 702 713 /* ··· 727 706 if (order) 728 707 goto done; 729 708 709 + /* Order 0 stealing from higher order */ 730 710 for (int o = 1; o < SWAP_NR_ORDERS; o++) { 711 + if (!list_empty(&si->frag_clusters[o])) { 712 + ci = list_first_entry(&si->frag_clusters[o], 713 + struct swap_cluster_info, list); 714 + offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), &found, 715 + 0, usage); 716 + VM_BUG_ON(!found); 717 + goto done; 718 + } 719 + 731 720 if (!list_empty(&si->nonfull_clusters[o])) { 732 721 ci = list_first_entry(&si->nonfull_clusters[o], struct swap_cluster_info, 733 722 list); ··· 3039 3008 INIT_LIST_HEAD(&p->free_clusters); 3040 3009 INIT_LIST_HEAD(&p->discard_clusters); 3041 3010 3042 - for (i = 0; i < SWAP_NR_ORDERS; i++) 3011 + for (i = 0; i < SWAP_NR_ORDERS; i++) { 3043 3012 INIT_LIST_HEAD(&p->nonfull_clusters[i]); 3013 + INIT_LIST_HEAD(&p->frag_clusters[i]); 3014 + } 3044 3015 3045 3016 for (i = 0; i < swap_header->info.nr_badpages; i++) { 3046 3017 unsigned int page_nr = swap_header->info.badpages[i];

Configure Feed

Configure Feed