tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: remove the branch from compound_head()

The compound_head() function is a hot path. For example, the zap path
calls it for every leaf page table entry.

Rewrite the helper function in a branchless manner to eliminate the risk
of CPU branch misprediction.

Link: https://lkml.kernel.org/r/20260227194302.274384-17-kas@kernel.org
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baoquan He <bhe@redhat.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Harry Yoo <harry.yoo@oracle.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kiryl Shutsemau and committed by
Andrew Morton 66b2a3d9 da3e2d1c

+17 -10
+17 -10
include/linux/page-flags.h
··· 224 224 static __always_inline unsigned long _compound_head(const struct page *page) 225 225 { 226 226 unsigned long info = READ_ONCE(page->compound_info); 227 + unsigned long mask; 227 228 228 - /* Bit 0 encodes PageTail() */ 229 - if (!(info & 1)) 229 + if (!compound_info_has_mask()) { 230 + /* Bit 0 encodes PageTail() */ 231 + if (info & 1) 232 + return info - 1; 233 + 230 234 return (unsigned long)page; 231 - 232 - /* 233 - * If compound_info_has_mask() is false, the rest of compound_info is 234 - * the pointer to the head page. 235 - */ 236 - if (!compound_info_has_mask()) 237 - return info - 1; 235 + } 238 236 239 237 /* 240 238 * If compound_info_has_mask() is true the rest of the info encodes 241 239 * the mask that converts the address of the tail page to the head page. 242 240 * 243 241 * No need to clear bit 0 in the mask as 'page' always has it clear. 242 + * 243 + * Let's do it in a branchless manner. 244 244 */ 245 - return (unsigned long)page & info; 245 + 246 + /* Non-tail: -1UL, Tail: 0 */ 247 + mask = (info & 1) - 1; 248 + 249 + /* Non-tail: -1UL, Tail: info */ 250 + mask |= info; 251 + 252 + return (unsigned long)page & mask; 246 253 } 247 254 248 255 #define compound_head(page) ((typeof(page))_compound_head(page))