tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

doc: Update rcubarrier.rst

This commit updates rcubarrier.txt to reflect RCU additions and changes
over the past few years.

[ paulmck: Apply Stephen Rothwell feedback. ]

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Paul E. McKenney 42d689ec 8750dfe6

+108 -84
+108 -84
Documentation/RCU/rcubarrier.rst
··· 5 5 6 6 [Originally published in LWN Jan. 14, 2007: http://lwn.net/Articles/217484/] 7 7 8 - RCU (read-copy update) is a synchronization mechanism that can be thought 9 - of as a replacement for read-writer locking (among other things), but with 10 - very low-overhead readers that are immune to deadlock, priority inversion, 11 - and unbounded latency. RCU read-side critical sections are delimited 12 - by rcu_read_lock() and rcu_read_unlock(), which, in non-CONFIG_PREEMPTION 13 - kernels, generate no code whatsoever. 14 - 15 - This means that RCU writers are unaware of the presence of concurrent 16 - readers, so that RCU updates to shared data must be undertaken quite 17 - carefully, leaving an old version of the data structure in place until all 18 - pre-existing readers have finished. These old versions are needed because 19 - such readers might hold a reference to them. RCU updates can therefore be 20 - rather expensive, and RCU is thus best suited for read-mostly situations. 21 - 22 - How can an RCU writer possibly determine when all readers are finished, 23 - given that readers might well leave absolutely no trace of their 24 - presence? There is a synchronize_rcu() primitive that blocks until all 25 - pre-existing readers have completed. An updater wishing to delete an 26 - element p from a linked list might do the following, while holding an 27 - appropriate lock, of course:: 28 - 29 - list_del_rcu(p); 30 - synchronize_rcu(); 31 - kfree(p); 32 - 33 - But the above code cannot be used in IRQ context -- the call_rcu() 34 - primitive must be used instead. This primitive takes a pointer to an 35 - rcu_head struct placed within the RCU-protected data structure and 36 - another pointer to a function that may be invoked later to free that 37 - structure. Code to delete an element p from the linked list from IRQ 38 - context might then be as follows:: 8 + RCU updaters sometimes use call_rcu() to initiate an asynchronous wait for 9 + a grace period to elapse. This primitive takes a pointer to an rcu_head 10 + struct placed within the RCU-protected data structure and another pointer 11 + to a function that may be invoked later to free that structure. Code to 12 + delete an element p from the linked list from IRQ context might then be 13 + as follows:: 39 14 40 15 list_del_rcu(p); 41 16 call_rcu(&p->rcu, p_callback); ··· 29 54 Unloading Modules That Use call_rcu() 30 55 ------------------------------------- 31 56 32 - But what if p_callback is defined in an unloadable module? 57 + But what if the p_callback() function is defined in an unloadable module? 33 58 34 59 If we unload the module while some RCU callbacks are pending, 35 60 the CPUs executing these callbacks are going to be severely ··· 42 67 43 68 One might be tempted to try several back-to-back synchronize_rcu() 44 69 calls, but this is still not guaranteed to work. If there is a very 45 - heavy RCU-callback load, then some of the callbacks might be deferred 46 - in order to allow other processing to proceed. Such deferral is required 47 - in realtime kernels in order to avoid excessive scheduling latencies. 70 + heavy RCU-callback load, then some of the callbacks might be deferred in 71 + order to allow other processing to proceed. For but one example, such 72 + deferral is required in realtime kernels in order to avoid excessive 73 + scheduling latencies. 48 74 49 75 50 76 rcu_barrier() 51 77 ------------- 52 78 53 - We instead need the rcu_barrier() primitive. Rather than waiting for 54 - a grace period to elapse, rcu_barrier() waits for all outstanding RCU 55 - callbacks to complete. Please note that rcu_barrier() does **not** imply 56 - synchronize_rcu(), in particular, if there are no RCU callbacks queued 57 - anywhere, rcu_barrier() is within its rights to return immediately, 58 - without waiting for a grace period to elapse. 79 + This situation can be handled by the rcu_barrier() primitive. Rather 80 + than waiting for a grace period to elapse, rcu_barrier() waits for all 81 + outstanding RCU callbacks to complete. Please note that rcu_barrier() 82 + does **not** imply synchronize_rcu(), in particular, if there are no RCU 83 + callbacks queued anywhere, rcu_barrier() is within its rights to return 84 + immediately, without waiting for anything, let alone a grace period. 59 85 60 86 Pseudo-code using rcu_barrier() is as follows: 61 87 ··· 65 89 3. Allow the module to be unloaded. 66 90 67 91 There is also an srcu_barrier() function for SRCU, and you of course 68 - must match the flavor of rcu_barrier() with that of call_rcu(). If your 69 - module uses multiple flavors of call_rcu(), then it must also use multiple 70 - flavors of rcu_barrier() when unloading that module. For example, if 71 - it uses call_rcu(), call_srcu() on srcu_struct_1, and call_srcu() on 72 - srcu_struct_2, then the following three lines of code will be required 73 - when unloading:: 92 + must match the flavor of srcu_barrier() with that of call_srcu(). 93 + If your module uses multiple srcu_struct structures, then it must also 94 + use multiple invocations of srcu_barrier() when unloading that module. 95 + For example, if it uses call_rcu(), call_srcu() on srcu_struct_1, and 96 + call_srcu() on srcu_struct_2, then the following three lines of code 97 + will be required when unloading:: 74 98 75 99 1 rcu_barrier(); 76 100 2 srcu_barrier(&srcu_struct_1); 77 101 3 srcu_barrier(&srcu_struct_2); 78 102 79 - The rcutorture module makes use of rcu_barrier() in its exit function 80 - as follows:: 103 + If latency is of the essence, workqueues could be used to run these 104 + three functions concurrently. 105 + 106 + An ancient version of the rcutorture module makes use of rcu_barrier() 107 + in its exit function as follows:: 81 108 82 109 1 static void 83 110 2 rcu_torture_cleanup(void) ··· 169 190 :ref:`Answer to Quick Quiz #1 <answer_rcubarrier_quiz_1>` 170 191 171 192 Your module might have additional complications. For example, if your 172 - module invokes call_rcu() from timers, you will need to first cancel all 173 - the timers, and only then invoke rcu_barrier() to wait for any remaining 193 + module invokes call_rcu() from timers, you will need to first refrain 194 + from posting new timers, cancel (or wait for) all the already-posted 195 + timers, and only then invoke rcu_barrier() to wait for any remaining 174 196 RCU callbacks to complete. 175 197 176 - Of course, if you module uses call_rcu(), you will need to invoke 198 + Of course, if your module uses call_rcu(), you will need to invoke 177 199 rcu_barrier() before unloading. Similarly, if your module uses 178 200 call_srcu(), you will need to invoke srcu_barrier() before unloading, 179 201 and on the same srcu_struct structure. If your module uses call_rcu() 180 - **and** call_srcu(), then you will need to invoke rcu_barrier() **and** 181 - srcu_barrier(). 202 + **and** call_srcu(), then (as noted above) you will need to invoke 203 + rcu_barrier() **and** srcu_barrier(). 182 204 183 205 184 206 Implementing rcu_barrier() ··· 191 211 callback queues, and then waits until they have all started executing, at 192 212 which point, all earlier RCU callbacks are guaranteed to have completed. 193 213 194 - The original code for rcu_barrier() was as follows:: 214 + The original code for rcu_barrier() was roughly as follows:: 195 215 196 - 1 void rcu_barrier(void) 197 - 2 { 198 - 3 BUG_ON(in_interrupt()); 199 - 4 /* Take cpucontrol mutex to protect against CPU hotplug */ 200 - 5 mutex_lock(&rcu_barrier_mutex); 201 - 6 init_completion(&rcu_barrier_completion); 202 - 7 atomic_set(&rcu_barrier_cpu_count, 0); 203 - 8 on_each_cpu(rcu_barrier_func, NULL, 0, 1); 204 - 9 wait_for_completion(&rcu_barrier_completion); 205 - 10 mutex_unlock(&rcu_barrier_mutex); 206 - 11 } 216 + 1 void rcu_barrier(void) 217 + 2 { 218 + 3 BUG_ON(in_interrupt()); 219 + 4 /* Take cpucontrol mutex to protect against CPU hotplug */ 220 + 5 mutex_lock(&rcu_barrier_mutex); 221 + 6 init_completion(&rcu_barrier_completion); 222 + 7 atomic_set(&rcu_barrier_cpu_count, 1); 223 + 8 on_each_cpu(rcu_barrier_func, NULL, 0, 1); 224 + 9 if (atomic_dec_and_test(&rcu_barrier_cpu_count)) 225 + 10 complete(&rcu_barrier_completion); 226 + 11 wait_for_completion(&rcu_barrier_completion); 227 + 12 mutex_unlock(&rcu_barrier_mutex); 228 + 13 } 207 229 208 - Line 3 verifies that the caller is in process context, and lines 5 and 10 230 + Line 3 verifies that the caller is in process context, and lines 5 and 12 209 231 use rcu_barrier_mutex to ensure that only one rcu_barrier() is using the 210 232 global completion and counters at a time, which are initialized on lines 211 233 6 and 7. Line 8 causes each CPU to invoke rcu_barrier_func(), which is 212 234 shown below. Note that the final "1" in on_each_cpu()'s argument list 213 235 ensures that all the calls to rcu_barrier_func() will have completed 214 - before on_each_cpu() returns. Line 9 then waits for the completion. 236 + before on_each_cpu() returns. Line 9 removes the initial count from 237 + rcu_barrier_cpu_count, and if this count is now zero, line 10 finalizes 238 + the completion, which prevents line 11 from blocking. Either way, 239 + line 11 then waits (if needed) for the completion. 240 + 241 + .. _rcubarrier_quiz_2: 242 + 243 + Quick Quiz #2: 244 + Why doesn't line 8 initialize rcu_barrier_cpu_count to zero, 245 + thereby avoiding the need for lines 9 and 10? 246 + 247 + :ref:`Answer to Quick Quiz #2 <answer_rcubarrier_quiz_2>` 215 248 216 249 This code was rewritten in 2008 and several times thereafter, but this 217 250 still gives the general idea. ··· 246 253 Lines 3 and 4 locate RCU's internal per-CPU rcu_data structure, 247 254 which contains the struct rcu_head that needed for the later call to 248 255 call_rcu(). Line 7 picks up a pointer to this struct rcu_head, and line 249 - 8 increments a global counter. This counter will later be decremented 256 + 8 increments the global counter. This counter will later be decremented 250 257 by the callback. Line 9 then registers the rcu_barrier_callback() on 251 258 the current CPU's queue. 252 259 ··· 260 267 4 complete(&rcu_barrier_completion); 261 268 5 } 262 269 263 - .. _rcubarrier_quiz_2: 270 + .. _rcubarrier_quiz_3: 264 271 265 - Quick Quiz #2: 272 + Quick Quiz #3: 266 273 What happens if CPU 0's rcu_barrier_func() executes 267 274 immediately (thus incrementing rcu_barrier_cpu_count to the 268 275 value one), but the other CPU's rcu_barrier_func() invocations 269 276 are delayed for a full grace period? Couldn't this result in 270 277 rcu_barrier() returning prematurely? 271 278 272 - :ref:`Answer to Quick Quiz #2 <answer_rcubarrier_quiz_2>` 279 + :ref:`Answer to Quick Quiz #3 <answer_rcubarrier_quiz_3>` 273 280 274 281 The current rcu_barrier() implementation is more complex, due to the need 275 282 to avoid disturbing idle CPUs (especially on battery-powered systems) 276 283 and the need to minimally disturb non-idle CPUs in real-time systems. 277 - However, the code above illustrates the concepts. 284 + In addition, a great many optimizations have been applied. However, 285 + the code above illustrates the concepts. 278 286 279 287 280 288 rcu_barrier() Summary 281 289 --------------------- 282 290 283 - The rcu_barrier() primitive has seen relatively little use, since most 291 + The rcu_barrier() primitive is used relatively infrequently, since most 284 292 code using RCU is in the core kernel rather than in modules. However, if 285 293 you are using RCU from an unloadable module, you need to use rcu_barrier() 286 294 so that your module may be safely unloaded. ··· 312 318 .. _answer_rcubarrier_quiz_2: 313 319 314 320 Quick Quiz #2: 321 + Why doesn't line 8 initialize rcu_barrier_cpu_count to zero, 322 + thereby avoiding the need for lines 9 and 10? 323 + 324 + Answer: Suppose that the on_each_cpu() function shown on line 8 was 325 + delayed, so that CPU 0's rcu_barrier_func() executed and 326 + the corresponding grace period elapsed, all before CPU 1's 327 + rcu_barrier_func() started executing. This would result in 328 + rcu_barrier_cpu_count being decremented to zero, so that line 329 + 11's wait_for_completion() would return immediately, failing to 330 + wait for CPU 1's callbacks to be invoked. 331 + 332 + Note that this was not a problem when the rcu_barrier() code 333 + was first added back in 2005. This is because on_each_cpu() 334 + disables preemption, which acted as an RCU read-side critical 335 + section, thus preventing CPU 0's grace period from completing 336 + until on_each_cpu() had dealt with all of the CPUs. However, 337 + with the advent of preemptible RCU, rcu_barrier() no longer 338 + waited on nonpreemptible regions of code in preemptible kernels, 339 + that being the job of the new rcu_barrier_sched() function. 340 + 341 + However, with the RCU flavor consolidation around v4.20, this 342 + possibility was once again ruled out, because the consolidated 343 + RCU once again waits on nonpreemptible regions of code. 344 + 345 + Nevertheless, that extra count might still be a good idea. 346 + Relying on these sort of accidents of implementation can result 347 + in later surprise bugs when the implementation changes. 348 + 349 + :ref:`Back to Quick Quiz #2 <rcubarrier_quiz_2>` 350 + 351 + .. _answer_rcubarrier_quiz_3: 352 + 353 + Quick Quiz #3: 315 354 What happens if CPU 0's rcu_barrier_func() executes 316 355 immediately (thus incrementing rcu_barrier_cpu_count to the 317 356 value one), but the other CPU's rcu_barrier_func() invocations ··· 363 336 364 337 Therefore, on_each_cpu() disables preemption across its call 365 338 to smp_call_function() and also across the local call to 366 - rcu_barrier_func(). This prevents the local CPU from context 367 - switching, again preventing grace periods from completing. This 339 + rcu_barrier_func(). Because recent RCU implementations treat 340 + preemption-disabled regions of code as RCU read-side critical 341 + sections, this prevents grace periods from completing. This 368 342 means that all CPUs have executed rcu_barrier_func() before 369 343 the first rcu_barrier_callback() can possibly execute, in turn 370 344 preventing rcu_barrier_cpu_count from prematurely reaching zero. 371 345 372 - Currently, -rt implementations of RCU keep but a single global 373 - queue for RCU callbacks, and thus do not suffer from this 374 - problem. However, when the -rt RCU eventually does have per-CPU 375 - callback queues, things will have to change. One simple change 376 - is to add an rcu_read_lock() before line 8 of rcu_barrier() 377 - and an rcu_read_unlock() after line 8 of this same function. If 378 - you can think of a better change, please let me know! 346 + But if on_each_cpu() ever decides to forgo disabling preemption, 347 + as might well happen due to real-time latency considerations, 348 + initializing rcu_barrier_cpu_count to one will save the day. 379 349 380 - :ref:`Back to Quick Quiz #2 <rcubarrier_quiz_2>` 350 + :ref:`Back to Quick Quiz #3 <rcubarrier_quiz_3>`