Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
srcu: Document __srcu_read_{,un}lock_fast() implicit RCU readers
This commit documents the implicit RCU readers that are implied by the
this_cpu_inc() and atomic_long_inc() operations in __srcu_read_lock_fast()
and __srcu_read_unlock_fast(). While in the area, fix the documentation
of the memory pairing of atomic_long_inc() in __srcu_read_lock_fast().
[ paulmck: Apply Joel Fernandes feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: <bpf@vger.kernel.org>
+27
-15
+27
-15
include/linux/srcutree.h
+27
-15
include/linux/srcutree.h
···
232
232
* srcu_read_unlock_fast().
233
233
*
234
234
* Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side
235
-
* critical sections either because they disables interrupts, because they
236
-
* are a single instruction, or because they are a read-modify-write atomic
237
-
* operation, depending on the whims of the architecture.
235
+
* critical sections either because they disables interrupts, because
236
+
* they are a single instruction, or because they are read-modify-write
237
+
* atomic operations, depending on the whims of the architecture.
238
+
* This matters because the SRCU-fast grace-period mechanism uses either
239
+
* synchronize_rcu() or synchronize_rcu_expedited(), that is, RCU,
240
+
* *not* SRCU, in order to eliminate the need for the read-side smp_mb()
241
+
* invocations that are used by srcu_read_lock() and srcu_read_unlock().
242
+
* The __srcu_read_unlock_fast() function also relies on this same RCU
243
+
* (again, *not* SRCU) trick to eliminate the need for smp_mb().
244
+
*
245
+
* The key point behind this RCU trick is that if any part of a given
246
+
* RCU reader precedes the beginning of a given RCU grace period, then
247
+
* the entirety of that RCU reader and everything preceding it happens
248
+
* before the end of that same RCU grace period. Similarly, if any part
249
+
* of a given RCU reader follows the end of a given RCU grace period,
250
+
* then the entirety of that RCU reader and everything following it
251
+
* happens after the beginning of that same RCU grace period. Therefore,
252
+
* the operations labeled Y in __srcu_read_lock_fast() and those labeled Z
253
+
* in __srcu_read_unlock_fast() are ordered against the corresponding SRCU
254
+
* read-side critical section from the viewpoint of the SRCU grace period.
255
+
* This is all the ordering that is required, hence no calls to smp_mb().
238
256
*
239
257
* This means that __srcu_read_lock_fast() is not all that fast
240
258
* on architectures that support NMIs but do not supply NMI-safe
···
263
245
struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
264
246
265
247
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
266
-
this_cpu_inc(scp->srcu_locks.counter); /* Y */
248
+
this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader.
267
249
else
268
-
atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */
250
+
atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU reader.
269
251
barrier(); /* Avoid leaking the critical section. */
270
252
return scp;
271
253
}
···
276
258
* different CPU than that which was incremented by the corresponding
277
259
* srcu_read_lock_fast(), but it must be within the same task.
278
260
*
279
-
* Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side
280
-
* critical sections either because they disables interrupts, because they
281
-
* are a single instruction, or because they are a read-modify-write atomic
282
-
* operation, depending on the whims of the architecture.
283
-
*
284
-
* This means that __srcu_read_unlock_fast() is not all that fast
285
-
* on architectures that support NMIs but do not supply NMI-safe
286
-
* implementations of this_cpu_inc().
261
+
* Please see the __srcu_read_lock_fast() function's header comment for
262
+
* information on implicit RCU readers and NMI safety.
287
263
*/
288
264
static inline void notrace
289
265
__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
290
266
{
291
267
barrier(); /* Avoid leaking the critical section. */
292
268
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
293
-
this_cpu_inc(scp->srcu_unlocks.counter); /* Z */
269
+
this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader.
294
270
else
295
-
atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */
271
+
atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RCU reader.
296
272
}
297
273
298
274
void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);