tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

srcu: Require special srcu_struct define/init for SRCU-fast readers

This commit adds CONFIG_PROVE_RCU=y checking to enforce the new rule that
srcu_struct structures passed to srcu_read_lock_fast() and other SRCU-fast
read-side markers be either initialized with init_srcu_struct_fast()
on the one hand or defined using either DEFINE_SRCU_FAST() or
DEFINE_STATIC_SRCU_FAST(). This will enable removal of the non-debug
read-side checks from srcu_read_lock_fast() and friends, which on my
laptop provides a 25% speedup (which admittedly amounts to about half
a nanosecond, but when tracing fastpaths...)

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: <bpf@vger.kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

authored by

Paul E. McKenney and committed by
Frederic Weisbecker 8235bcfd e4ed20c1

+21 -10
+20 -10
include/linux/srcu.h
··· 271 271 * @ssp: srcu_struct in which to register the new reader. 272 272 * 273 273 * Enter an SRCU read-side critical section, but for a light-weight 274 - * smp_mb()-free reader. See srcu_read_lock() for more information. 274 + * smp_mb()-free reader. See srcu_read_lock() for more information. This 275 + * function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe(). 275 276 * 276 - * If srcu_read_lock_fast() is ever used on an srcu_struct structure, 277 - * then none of the other flavors may be used, whether before, during, 278 - * or after. Note that grace-period auto-expediting is disabled for _fast 279 - * srcu_struct structures because auto-expedited grace periods invoke 280 - * synchronize_rcu_expedited(), IPIs and all. 277 + * For srcu_read_lock_fast() to be used on an srcu_struct structure, 278 + * that structure must have been defined using either DEFINE_SRCU_FAST() 279 + * or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with 280 + * init_srcu_struct_fast() on the other. Such an srcu_struct structure 281 + * cannot be passed to any non-fast variant of srcu_read_{,un}lock() or 282 + * srcu_{down,up}_read(). In kernels built with CONFIG_PROVE_RCU=y, 283 + * __srcu_check_read_flavor() will complain bitterly if you ignore this 284 + * restriction. 281 285 * 282 - * Note that srcu_read_lock_fast() can be invoked only from those contexts 283 - * where RCU is watching, that is, from contexts where it would be legal 284 - * to invoke rcu_read_lock(). Otherwise, lockdep will complain. 286 + * Grace-period auto-expediting is disabled for SRCU-fast srcu_struct 287 + * structures because SRCU-fast expedited grace periods invoke 288 + * synchronize_rcu_expedited(), IPIs and all. If you need expedited 289 + * SRCU-fast grace periods, use synchronize_srcu_expedited(). 290 + * 291 + * The srcu_read_lock_fast() function can be invoked only from those 292 + * contexts where RCU is watching, that is, from contexts where it would 293 + * be legal to invoke rcu_read_lock(). Otherwise, lockdep will complain. 285 294 */ 286 295 static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_struct *ssp) __acquires(ssp) 287 296 { ··· 326 317 * srcu_down_read() for more information. 327 318 * 328 319 * The same srcu_struct may be used concurrently by srcu_down_read_fast() 329 - * and srcu_read_lock_fast(). 320 + * and srcu_read_lock_fast(). However, the same definition/initialization 321 + * requirements called out for srcu_read_lock_safe() apply. 330 322 */ 331 323 static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_struct *ssp) __acquires(ssp) 332 324 {
+1
kernel/rcu/srcutree.c
··· 766 766 WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor != ssp->srcu_reader_flavor); 767 767 WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && 768 768 old_read_flavor != ssp->srcu_reader_flavor); 769 + WARN_ON_ONCE(read_flavor == SRCU_READ_FLAVOR_FAST && !ssp->srcu_reader_flavor); 769 770 if (!old_read_flavor) { 770 771 old_read_flavor = cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); 771 772 if (!old_read_flavor)