mm/huge_memory: Fix initialization of huge zero folio

The recent fix to properly initialize the tags of the huge zero folio
had an unfortunate not-so-subtle side effect: it caused the actual
*contents* of the huge zero folio to not be initialized at all when the
hardware didn't support the memory tagging.

The reason was the unfortunate semantics of tag_clear_highpage(): on
hardware that didn't do the tagging, it would silently just not do
anything at all. And since this is done only on arm64 with MTE support,
that basically meant most hardware.

It wasn't necessarily immediately obvious since the huge zero page isn't
necessarily very heavily used - or because it might already be zero
because all-zeroes is the most common pattern. But it ends up causing
random odd user space failures when you do hit it.

The unfortunate semantics have been around for a while, but became a
real bug only when we started actively using __GFP_ZEROTAGS in the
generic get_huge_zero_folio() function - before that, it had only ever
been used in code that checked that the hardware supported it.

Fix this by simply changing the semantics of tag_clear_highpage() to
return whether it actually successfully did something or not. While at
it, also make it initialize multiple pages in one go, since that's
actually what the only caller wants it to do and it simplifies the whole
logic.

Fixes: adfb6609c680 ("mm/huge_memory: initialise the tags of the huge zero folio")
Link: https://lore.kernel.org/all/20251117082023.90176-1-00107082@163.com/
Reviewed-by: David Hildenbrand (Red Hat) <david@kernel.org>
Reported-and-tested-by: David Wang <00107082@163.com>
Reported-and-tested-by: Carlos Llamas <cmllamas@google.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Linus Torvalds 5 months ago 5bebe8de e7c375b1

+19 -21

4 changed files

expand all

arch

arm64

include

asm

page.h

fault.c

include

linux

highmem.h

page_alloc.c

+2 -2

arch/arm64/include/asm/page.h

··· 33 33 unsigned long vaddr); 34 34 #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio 35 35 36 - void tag_clear_highpage(struct page *to); 37 - #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE 36 + bool tag_clear_highpages(struct page *to, int numpages); 37 + #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGES 38 38 39 39 #define clear_user_page(page, vaddr, pg) clear_page(page) 40 40 #define copy_user_page(to, from, vaddr, pg) copy_page(to, from)

+11 -10

arch/arm64/mm/fault.c

··· 967 967 return vma_alloc_folio(flags, 0, vma, vaddr); 968 968 } 969 969 970 - void tag_clear_highpage(struct page *page) 970 + bool tag_clear_highpages(struct page *page, int numpages) 971 971 { 972 972 /* 973 973 * Check if MTE is supported and fall back to clear_highpage(). 974 974 * get_huge_zero_folio() unconditionally passes __GFP_ZEROTAGS and 975 - * post_alloc_hook() will invoke tag_clear_highpage(). 975 + * post_alloc_hook() will invoke tag_clear_highpages(). 976 976 */ 977 - if (!system_supports_mte()) { 978 - clear_highpage(page); 979 - return; 980 - } 977 + if (!system_supports_mte()) 978 + return false; 981 979 982 - /* Newly allocated page, shouldn't have been tagged yet */ 983 - WARN_ON_ONCE(!try_page_mte_tagging(page)); 984 - mte_zero_clear_page_tags(page_address(page)); 985 - set_page_mte_tagged(page); 980 + /* Newly allocated pages, shouldn't have been tagged yet */ 981 + for (int i = 0; i < numpages; i++, page++) { 982 + WARN_ON_ONCE(!try_page_mte_tagging(page)); 983 + mte_zero_clear_page_tags(page_address(page)); 984 + set_page_mte_tagged(page); 985 + } 986 + return true; 986 987 }

+4 -2

include/linux/highmem.h

··· 249 249 kunmap_local(kaddr); 250 250 } 251 251 252 - #ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGE 252 + #ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGES 253 253 254 - static inline void tag_clear_highpage(struct page *page) 254 + /* Return false to let people know we did not initialize the pages */ 255 + static inline bool tag_clear_highpages(struct page *page, int numpages) 255 256 { 257 + return false; 256 258 } 257 259 258 260 #endif

+2 -7

mm/page_alloc.c

··· 1822 1822 * If memory tags should be zeroed 1823 1823 * (which happens only when memory should be initialized as well). 1824 1824 */ 1825 - if (zero_tags) { 1826 - /* Initialize both memory and memory tags. */ 1827 - for (i = 0; i != 1 << order; ++i) 1828 - tag_clear_highpage(page + i); 1825 + if (zero_tags) 1826 + init = !tag_clear_highpages(page, 1 << order); 1829 1827 1830 - /* Take note that memory was initialized by the loop above. */ 1831 - init = false; 1832 - } 1833 1828 if (!should_skip_kasan_unpoison(gfp_flags) && 1834 1829 kasan_unpoison_pages(page, order, init)) { 1835 1830 /* Take note that memory was initialized by KASAN. */

Configure Feed

Configure Feed