tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

srcu: Make SRCU-fast readers enforce use of SRCU-fast definition/init

This commit makes CONFIG_PROVE_RCU=y kernels enforce the new rule
that srcu_struct structures that are passed to srcu_read_lock_fast()
and other SRCU-fast read-side markers be either initialized with
init_srcu_struct_fast() on the one hand or defined with DEFINE_SRCU_FAST()
or DEFINE_STATIC_SRCU_FAST() on the other.

This eliminates the read-side test that was formerly included in
srcu_read_lock_fast() and friends, speeding these primitives up by
about 25% (admittedly only about half of a nanosecond, but when tracing
on fastpaths...)

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: <bpf@vger.kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

authored by

Paul E. McKenney and committed by
Frederic Weisbecker ac51c40c 8235bcfd

+4 -19
+3 -3
include/linux/srcu.h
··· 297 297 struct srcu_ctr __percpu *retval; 298 298 299 299 RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast()."); 300 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); 300 + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); 301 301 retval = __srcu_read_lock_fast(ssp); 302 302 rcu_try_lock_acquire(&ssp->dep_map); 303 303 return retval; ··· 312 312 { 313 313 struct srcu_ctr __percpu *retval; 314 314 315 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); 315 + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); 316 316 retval = __srcu_read_lock_fast(ssp); 317 317 return retval; 318 318 } ··· 333 333 { 334 334 WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); 335 335 RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read_fast()."); 336 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); 336 + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); 337 337 return __srcu_read_lock_fast(ssp); 338 338 } 339 339
-1
include/linux/srcutiny.h
··· 112 112 113 113 static inline void srcu_expedite_current(struct srcu_struct *ssp) { } 114 114 #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) 115 - #define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) 116 115 117 116 /* Defined here to avoid size increase for non-torture kernels. */ 118 117 static inline void srcu_torture_stats_print(struct srcu_struct *ssp,
+1 -15
include/linux/srcutree.h
··· 307 307 308 308 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); 309 309 310 - // Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is 311 - // needed only for flavors that require grace-period smp_mb() calls to be 312 - // promoted to synchronize_rcu(). 313 - static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, int read_flavor) 314 - { 315 - struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); 316 - 317 - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) 318 - return; 319 - 320 - // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. 321 - __srcu_check_read_flavor(ssp, read_flavor); 322 - } 323 - 324 - // Record non-_lite() usage only for CONFIG_PROVE_RCU=y kernels. 310 + // Record SRCU-reader usage type only for CONFIG_PROVE_RCU=y kernels. 325 311 static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) 326 312 { 327 313 if (IS_ENABLED(CONFIG_PROVE_RCU))