tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm/mincore: use a helper for checking the swap cache

Introduce a mincore_swap helper for checking swap entries. Move all swap
related logic and sanity debug check into it, and separate them from page
cache checking.

The performance is better after this commit. mincore_page is never called
on a swap cache space now, so the logic can be simpler. The sanity check
also covers more potential cases now, previously the WARN_ON only catches
potentially corrupted page table, now if shmem contains a swap entry with
!CONFIG_SWAP, a WARN will be triggered. This changes the mincore value
when the WARN is triggered, but this shouldn't matter. The WARN_ON means
the data is already corrupted or something is very wrong, so it really
should not happen.

Before this series:
mincore on a swaped out 16G anon mmap range:
Took 488220 us
mincore on 16G shmem mmap range:
Took 530272 us.

After this commit:
mincore on a swaped out 16G anon mmap range:
Took 446763 us
mincore on 16G shmem mmap range:
Took 460496 us.

About ~10% faster.

Link: https://lkml.kernel.org/r/20250811172018.48901-3-ryncsn@gmail.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kairui Song and committed by
Andrew Morton 1f205275 27763eda

+49 -41
+49 -41
mm/mincore.c
··· 47 47 return 0; 48 48 } 49 49 50 + static unsigned char mincore_swap(swp_entry_t entry, bool shmem) 51 + { 52 + struct swap_info_struct *si; 53 + struct folio *folio = NULL; 54 + unsigned char present = 0; 55 + 56 + if (!IS_ENABLED(CONFIG_SWAP)) { 57 + WARN_ON(1); 58 + return 0; 59 + } 60 + 61 + /* 62 + * Shmem mapping may contain swapin error entries, which are 63 + * absent. Page table may contain migration or hwpoison 64 + * entries which are always uptodate. 65 + */ 66 + if (non_swap_entry(entry)) 67 + return !shmem; 68 + 69 + /* 70 + * Shmem mapping lookup is lockless, so we need to grab the swap 71 + * device. mincore page table walk locks the PTL, and the swap 72 + * device is stable, avoid touching the si for better performance. 73 + */ 74 + if (shmem) { 75 + si = get_swap_device(entry); 76 + if (!si) 77 + return 0; 78 + } 79 + folio = filemap_get_entry(swap_address_space(entry), 80 + swap_cache_index(entry)); 81 + if (shmem) 82 + put_swap_device(si); 83 + /* The swap cache space contains either folio, shadow or NULL */ 84 + if (folio && !xa_is_value(folio)) { 85 + present = folio_test_uptodate(folio); 86 + folio_put(folio); 87 + } 88 + 89 + return present; 90 + } 91 + 50 92 /* 51 93 * Later we can get more picky about what "in core" means precisely. 52 94 * For now, simply check to see if the page is in the page cache, ··· 106 64 * any other file mapping (ie. marked !present and faulted in with 107 65 * tmpfs's .fault). So swapped out tmpfs mappings are tested here. 108 66 */ 109 - if (IS_ENABLED(CONFIG_SWAP) && shmem_mapping(mapping)) { 110 - folio = filemap_get_entry(mapping, index); 111 - /* 112 - * shmem/tmpfs may return swap: account for swapcache 113 - * page too. 114 - */ 67 + folio = filemap_get_entry(mapping, index); 68 + if (folio) { 115 69 if (xa_is_value(folio)) { 116 - struct swap_info_struct *si; 117 - swp_entry_t swp = radix_to_swp_entry(folio); 118 - /* There might be swapin error entries in shmem mapping. */ 119 - if (non_swap_entry(swp)) 70 + if (shmem_mapping(mapping)) 71 + return mincore_swap(radix_to_swp_entry(folio), 72 + true); 73 + else 120 74 return 0; 121 - /* Prevent swap device to being swapoff under us */ 122 - si = get_swap_device(swp); 123 - if (si) { 124 - folio = filemap_get_folio(swap_address_space(swp), 125 - swap_cache_index(swp)); 126 - put_swap_device(si); 127 - } else { 128 - return 0; 129 - } 130 75 } 131 - } else { 132 - folio = filemap_get_folio(mapping, index); 133 - } 134 - 135 - if (!IS_ERR_OR_NULL(folio)) { 136 76 present = folio_test_uptodate(folio); 137 77 folio_put(folio); 138 78 } ··· 192 168 for (i = 0; i < step; i++) 193 169 vec[i] = 1; 194 170 } else { /* pte is a swap entry */ 195 - swp_entry_t entry = pte_to_swp_entry(pte); 196 - 197 - if (non_swap_entry(entry)) { 198 - /* 199 - * migration or hwpoison entries are always 200 - * uptodate 201 - */ 202 - *vec = 1; 203 - } else { 204 - #ifdef CONFIG_SWAP 205 - *vec = mincore_page(swap_address_space(entry), 206 - swap_cache_index(entry)); 207 - #else 208 - WARN_ON(1); 209 - *vec = 1; 210 - #endif 211 - } 171 + *vec = mincore_swap(pte_to_swp_entry(pte), false); 212 172 } 213 173 vec += step; 214 174 }