tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

mm: add folio_xor_flags_has_waiters()

Optimise folio_end_read() by setting the uptodate bit at the same time we
clear the unlock bit. This saves at least one memory barrier and one
write-after-write hazard.

Link: https://lkml.kernel.org/r/20231004165317.1061855-16-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Andreas Dilger <adilger.kernel@dilger.ca>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: "Theodore Ts'o" <tytso@mit.edu>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

authored by

Matthew Wilcox (Oracle) and committed by
Andrew Morton 0410cd84 f12fb73b

+30 -3
+19
include/linux/page-flags.h
··· 693 693 u64 stable_page_flags(struct page *page); 694 694 695 695 /** 696 + * folio_xor_flags_has_waiters - Change some folio flags. 697 + * @folio: The folio. 698 + * @mask: Bits set in this word will be changed. 699 + * 700 + * This must only be used for flags which are changed with the folio 701 + * lock held. For example, it is unsafe to use for PG_dirty as that 702 + * can be set without the folio lock held. It can also only be used 703 + * on flags which are in the range 0-6 as some of the implementations 704 + * only affect those bits. 705 + * 706 + * Return: Whether there are tasks waiting on the folio. 707 + */ 708 + static inline bool folio_xor_flags_has_waiters(struct folio *folio, 709 + unsigned long mask) 710 + { 711 + return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0)); 712 + } 713 + 714 + /** 696 715 * folio_test_uptodate - Is this folio up to date? 697 716 * @folio: The folio. 698 717 *
+11 -3
mm/filemap.c
··· 1497 1497 BUILD_BUG_ON(PG_waiters != 7); 1498 1498 BUILD_BUG_ON(PG_locked > 7); 1499 1499 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); 1500 - if (xor_unlock_is_negative_byte(1 << PG_locked, folio_flags(folio, 0))) 1500 + if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) 1501 1501 folio_wake_bit(folio, PG_locked); 1502 1502 } 1503 1503 EXPORT_SYMBOL(folio_unlock); ··· 1518 1518 */ 1519 1519 void folio_end_read(struct folio *folio, bool success) 1520 1520 { 1521 + unsigned long mask = 1 << PG_locked; 1522 + 1523 + /* Must be in bottom byte for x86 to work */ 1524 + BUILD_BUG_ON(PG_uptodate > 7); 1525 + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); 1526 + VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio); 1527 + 1521 1528 if (likely(success)) 1522 - folio_mark_uptodate(folio); 1523 - folio_unlock(folio); 1529 + mask |= 1 << PG_uptodate; 1530 + if (folio_xor_flags_has_waiters(folio, mask)) 1531 + folio_wake_bit(folio, PG_locked); 1524 1532 } 1525 1533 EXPORT_SYMBOL(folio_end_read); 1526 1534