tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

hugetlb: remove VMEMMAP_SYNCHRONIZE_RCU

The VMEMMAP_SYNCHRONIZE_RCU flag triggered synchronize_rcu() calls to
prevent a race between HVO remapping and page_ref_add_unless(). The race
could occur when a speculative PFN walker tried to modify the refcount on
a struct page that was in the process of being remapped to a fake head.

With fake heads eliminated, page_ref_add_unless() no longer needs RCU
protection.

Remove the flag and synchronize_rcu() calls.

Link: https://lkml.kernel.org/r/20260227194302.274384-15-kas@kernel.org
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baoquan He <bhe@redhat.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Harry Yoo <harry.yoo@oracle.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Kiryl Shutsemau and committed by
Andrew Morton 01b1d0ff 32c440d6

+4 -16
+4 -16
mm/hugetlb_vmemmap.c
··· 47 47 #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) 48 48 /* Skip the TLB flush when we remap the PTE */ 49 49 #define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) 50 - /* synchronize_rcu() to avoid writes from page_ref_add_unless() */ 51 - #define VMEMMAP_SYNCHRONIZE_RCU BIT(2) 52 50 unsigned long flags; 53 51 }; 54 52 ··· 407 409 if (!folio_test_hugetlb_vmemmap_optimized(folio)) 408 410 return 0; 409 411 410 - if (flags & VMEMMAP_SYNCHRONIZE_RCU) 411 - synchronize_rcu(); 412 - 413 412 vmemmap_start = (unsigned long)&folio->page; 414 413 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); 415 414 ··· 439 444 */ 440 445 int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) 441 446 { 442 - return __hugetlb_vmemmap_restore_folio(h, folio, VMEMMAP_SYNCHRONIZE_RCU); 447 + return __hugetlb_vmemmap_restore_folio(h, folio, 0); 443 448 } 444 449 445 450 /** ··· 462 467 struct folio *folio, *t_folio; 463 468 long restored = 0; 464 469 long ret = 0; 465 - unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH | VMEMMAP_SYNCHRONIZE_RCU; 470 + unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH; 466 471 467 472 list_for_each_entry_safe(folio, t_folio, folio_list, lru) { 468 473 if (folio_test_hugetlb_vmemmap_optimized(folio)) { 469 474 ret = __hugetlb_vmemmap_restore_folio(h, folio, flags); 470 - /* only need to synchronize_rcu() once for each batch */ 471 - flags &= ~VMEMMAP_SYNCHRONIZE_RCU; 472 - 473 475 if (ret) 474 476 break; 475 477 restored++; ··· 546 554 547 555 static_branch_inc(&hugetlb_optimize_vmemmap_key); 548 556 549 - if (flags & VMEMMAP_SYNCHRONIZE_RCU) 550 - synchronize_rcu(); 551 557 /* 552 558 * Very Subtle 553 559 * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed ··· 603 613 { 604 614 LIST_HEAD(vmemmap_pages); 605 615 606 - __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, VMEMMAP_SYNCHRONIZE_RCU); 616 + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0); 607 617 free_vmemmap_page_list(&vmemmap_pages); 608 618 } 609 619 ··· 631 641 struct folio *folio; 632 642 int nr_to_optimize; 633 643 LIST_HEAD(vmemmap_pages); 634 - unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH | VMEMMAP_SYNCHRONIZE_RCU; 644 + unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH; 635 645 636 646 nr_to_optimize = 0; 637 647 list_for_each_entry(folio, folio_list, lru) { ··· 684 694 int ret; 685 695 686 696 ret = __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, flags); 687 - /* only need to synchronize_rcu() once for each batch */ 688 - flags &= ~VMEMMAP_SYNCHRONIZE_RCU; 689 697 690 698 /* 691 699 * Pages to be freed may have been accumulated. If we