tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: drop fake head checks

With fake head pages eliminated in the previous commit, remove the
supporting infrastructure:

- page_fixed_fake_head(): no longer needed to detect fake heads;
- page_is_fake_head(): no longer needed;
- page_count_writable(): no longer needed for RCU protection;
- RCU read_lock in page_ref_add_unless(): no longer needed;

This substantially simplifies compound_head() and page_ref_add_unless(),
removing both branches and RCU overhead from these hot paths.

RCU was required to serialize allocation of hugetlb page against
get_page_unless_zero() and prevent writing to read-only fake head. It is
redundant without fake heads.

See bd225530a4c7 ("mm/hugetlb_vmemmap: fix race with speculative PFN
walkers") for more details.

synchronize_rcu() in mm/hugetlb_vmemmap.c will be removed by a separate
patch.

Link: https://lkml.kernel.org/r/20260227194302.274384-14-kas@kernel.org
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baoquan He <bhe@redhat.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Harry Yoo <harry.yoo@oracle.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kiryl Shutsemau and committed by
Andrew Morton 32c440d6 622026e8

+4 -97
+3 -90
include/linux/page-flags.h
··· 221 221 return is_power_of_2(sizeof(struct page)); 222 222 } 223 223 224 - #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP 225 224 DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); 226 - 227 - /* 228 - * Return the real head page struct iff the @page is a fake head page, otherwise 229 - * return the @page itself. See Documentation/mm/vmemmap_dedup.rst. 230 - */ 231 - static __always_inline const struct page *page_fixed_fake_head(const struct page *page) 232 - { 233 - /* Fake heads only exists if compound_info_has_mask() is true */ 234 - if (!compound_info_has_mask()) 235 - return page; 236 - 237 - if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key)) 238 - return page; 239 - 240 - /* 241 - * Only addresses aligned with PAGE_SIZE of struct page may be fake head 242 - * struct page. The alignment check aims to avoid access the fields ( 243 - * e.g. compound_info) of the @page[1]. It can avoid touch a (possibly) 244 - * cold cacheline in some cases. 245 - */ 246 - if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && 247 - test_bit(PG_head, &page->flags.f)) { 248 - /* 249 - * We can safely access the field of the @page[1] with PG_head 250 - * because the @page is a compound page composed with at least 251 - * two contiguous pages. 252 - */ 253 - unsigned long info = READ_ONCE(page[1].compound_info); 254 - 255 - /* See set_compound_head() */ 256 - if (likely(info & 1)) { 257 - unsigned long p = (unsigned long)page; 258 - 259 - return (const struct page *)(p & info); 260 - } 261 - } 262 - return page; 263 - } 264 - 265 - static __always_inline bool page_count_writable(const struct page *page, int u) 266 - { 267 - if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key)) 268 - return true; 269 - 270 - /* 271 - * The refcount check is ordered before the fake-head check to prevent 272 - * the following race: 273 - * CPU 1 (HVO) CPU 2 (speculative PFN walker) 274 - * 275 - * page_ref_freeze() 276 - * synchronize_rcu() 277 - * rcu_read_lock() 278 - * page_is_fake_head() is false 279 - * vmemmap_remap_pte() 280 - * XXX: struct page[] becomes r/o 281 - * 282 - * page_ref_unfreeze() 283 - * page_ref_count() is not zero 284 - * 285 - * atomic_add_unless(&page->_refcount) 286 - * XXX: try to modify r/o struct page[] 287 - * 288 - * The refcount check also prevents modification attempts to other (r/o) 289 - * tail pages that are not fake heads. 290 - */ 291 - if (atomic_read_acquire(&page->_refcount) == u) 292 - return false; 293 - 294 - return page_fixed_fake_head(page) == page; 295 - } 296 - #else 297 - static inline const struct page *page_fixed_fake_head(const struct page *page) 298 - { 299 - return page; 300 - } 301 - 302 - static inline bool page_count_writable(const struct page *page, int u) 303 - { 304 - return true; 305 - } 306 - #endif 307 - 308 - static __always_inline int page_is_fake_head(const struct page *page) 309 - { 310 - return page_fixed_fake_head(page) != page; 311 - } 312 225 313 226 static __always_inline unsigned long _compound_head(const struct page *page) 314 227 { ··· 229 316 230 317 /* Bit 0 encodes PageTail() */ 231 318 if (!(info & 1)) 232 - return (unsigned long)page_fixed_fake_head(page); 319 + return (unsigned long)page; 233 320 234 321 /* 235 322 * If compound_info_has_mask() is false, the rest of compound_info is ··· 309 396 310 397 static __always_inline int PageTail(const struct page *page) 311 398 { 312 - return READ_ONCE(page->compound_info) & 1 || page_is_fake_head(page); 399 + return READ_ONCE(page->compound_info) & 1; 313 400 } 314 401 315 402 static __always_inline int PageCompound(const struct page *page) ··· 841 928 static __always_inline int PageHead(const struct page *page) 842 929 { 843 930 PF_POISONED_CHECK(page); 844 - return test_bit(PG_head, &page->flags.f) && !page_is_fake_head(page); 931 + return test_bit(PG_head, &page->flags.f); 845 932 } 846 933 847 934 __SETPAGEFLAG(Head, head, PF_ANY)
+1 -7
include/linux/page_ref.h
··· 230 230 231 231 static inline bool page_ref_add_unless(struct page *page, int nr, int u) 232 232 { 233 - bool ret = false; 234 - 235 - rcu_read_lock(); 236 - /* avoid writing to the vmemmap area being remapped */ 237 - if (page_count_writable(page, u)) 238 - ret = atomic_add_unless(&page->_refcount, nr, u); 239 - rcu_read_unlock(); 233 + bool ret = atomic_add_unless(&page->_refcount, nr, u); 240 234 241 235 if (page_ref_tracepoint_active(page_ref_mod_unless)) 242 236 __page_ref_mod_unless(page, nr, ret);