Merge tag 'rcu.2024.07.12a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu

+3 -3

Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst

··· 149 149 ``atomic_add_return()`` read-modify-write atomic operation that 150 150 is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry 151 151 time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. 152 - The grace-period kthread invokes ``rcu_dynticks_snap()`` and 153 - ``rcu_dynticks_in_eqs_since()`` (both of which invoke 154 - an ``atomic_add_return()`` of zero) to detect idle CPUs. 152 + The grace-period kthread invokes first ``ct_dynticks_cpu_acquire()`` 153 + (preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()`` 154 + (both of which rely on acquire semantics) to detect idle CPUs. 155 155 156 156 +-----------------------------------------------------------------------+ 157 157 | **Quick Quiz**: |

+16

Documentation/RCU/Design/Requirements/Requirements.rst

··· 2357 2357 #. `Sched Flavor (Historical)`_ 2358 2358 #. `Sleepable RCU`_ 2359 2359 #. `Tasks RCU`_ 2360 + #. `Tasks Trace RCU`_ 2360 2361 2361 2362 Bottom-Half Flavor (Historical) 2362 2363 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ··· 2611 2610 is, calls to schedule(), cond_resched(), and 2612 2611 synchronize_rcu_tasks(). In addition, transitions to and from 2613 2612 userspace execution also delimit tasks-RCU read-side critical sections. 2613 + Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to 2614 + interact with them. 2615 + 2616 + Note well that involuntary context switches are *not* Tasks-RCU quiescent 2617 + states. After all, in preemptible kernels, a task executing code in a 2618 + trampoline might be preempted. In this case, the Tasks-RCU grace period 2619 + clearly cannot end until that task resumes and its execution leaves that 2620 + trampoline. This means, among other things, that cond_resched() does 2621 + not provide a Tasks RCU quiescent state. (Instead, use rcu_softirq_qs() 2622 + from softirq or rcu_tasks_classic_qs() otherwise.) 2614 2623 2615 2624 The tasks-RCU API is quite compact, consisting only of 2616 2625 call_rcu_tasks(), synchronize_rcu_tasks(), and ··· 2642 2631 moniker. And this operation is considered to be quite rude by real-time 2643 2632 workloads that don't want their ``nohz_full`` CPUs receiving IPIs and 2644 2633 by battery-powered systems that don't want their idle CPUs to be awakened. 2634 + 2635 + Once kernel entry/exit and deep-idle functions have been properly tagged 2636 + ``noinstr``, Tasks RCU can start paying attention to idle tasks (except 2637 + those that are idle from RCU's perspective) and then Tasks Rude RCU can 2638 + be removed from the kernel. 2645 2639 2646 2640 The tasks-rude-RCU API is also reader-marking-free and thus quite compact, 2647 2641 consisting of call_rcu_tasks_rude(), synchronize_rcu_tasks_rude(),

+18 -10

Documentation/RCU/whatisRCU.rst

··· 250 250 ^^^^^^^^^^^^^^^^^^^^ 251 251 void rcu_assign_pointer(p, typeof(p) v); 252 252 253 - Yes, rcu_assign_pointer() **is** implemented as a macro, though it 254 - would be cool to be able to declare a function in this manner. 255 - (Compiler experts will no doubt disagree.) 253 + Yes, rcu_assign_pointer() **is** implemented as a macro, though 254 + it would be cool to be able to declare a function in this manner. 255 + (And there has been some discussion of adding overloaded functions 256 + to the C language, so who knows?) 256 257 257 258 The updater uses this spatial macro to assign a new value to an 258 259 RCU-protected pointer, in order to safely communicate the change 259 260 in value from the updater to the reader. This is a spatial (as 260 261 opposed to temporal) macro. It does not evaluate to an rvalue, 261 - but it does execute any memory-barrier instructions required 262 - for a given CPU architecture. Its ordering properties are that 263 - of a store-release operation. 262 + but it does provide any compiler directives and memory-barrier 263 + instructions required for a given compile or CPU architecture. 264 + Its ordering properties are that of a store-release operation, 265 + that is, any prior loads and stores required to initialize the 266 + structure are ordered before the store that publishes the pointer 267 + to that structure. 264 268 265 - Perhaps just as important, it serves to document (1) which 266 - pointers are protected by RCU and (2) the point at which a 267 - given structure becomes accessible to other CPUs. That said, 269 + Perhaps just as important, rcu_assign_pointer() serves to document 270 + (1) which pointers are protected by RCU and (2) the point at which 271 + a given structure becomes accessible to other CPUs. That said, 268 272 rcu_assign_pointer() is most frequently used indirectly, via 269 273 the _rcu list-manipulation primitives such as list_add_rcu(). 270 274 ··· 287 283 executes any needed memory-barrier instructions for a given 288 284 CPU architecture. Currently, only Alpha needs memory barriers 289 285 within rcu_dereference() -- on other CPUs, it compiles to a 290 - volatile load. 286 + volatile load. However, no mainstream C compilers respect 287 + address dependencies, so rcu_dereference() uses volatile casts, 288 + which, in combination with the coding guidelines listed in 289 + rcu_dereference.rst, prevent current compilers from breaking 290 + these dependencies. 291 291 292 292 Common coding practice uses rcu_dereference() to copy an 293 293 RCU-protected pointer to a local variable, then dereferences

+8

Documentation/admin-guide/kernel-parameters.txt

··· 5015 5015 the ->nocb_bypass queue. The definition of "too 5016 5016 many" is supplied by this kernel boot parameter. 5017 5017 5018 + rcutree.nohz_full_patience_delay= [KNL] 5019 + On callback-offloaded (rcu_nocbs) CPUs, avoid 5020 + disturbing RCU unless the grace period has 5021 + reached the specified age in milliseconds. 5022 + Defaults to zero. Large values will be capped 5023 + at five seconds. All values will be rounded down 5024 + to the nearest value representable by jiffies. 5025 + 5018 5026 rcutree.qhimark= [KNL] 5019 5027 Set threshold of queued RCU callbacks beyond which 5020 5028 batch limiting is disabled.

+1

MAINTAINERS

··· 18863 18863 M: Joel Fernandes <joel@joelfernandes.org> 18864 18864 M: Josh Triplett <josh@joshtriplett.org> 18865 18865 M: Boqun Feng <boqun.feng@gmail.com> 18866 + M: Uladzislau Rezki <urezki@gmail.com> 18866 18867 R: Steven Rostedt <rostedt@goodmis.org> 18867 18868 R: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> 18868 18869 R: Lai Jiangshan <jiangshanlai@gmail.com>

+38 -50

include/linux/rcu_segcblist.h

··· 80 80 * | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED | 81 81 * | | 82 82 * | Callbacks processed by rcu_core() from softirqs or local | 83 - * | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, | 84 - * | allowing nocb_timer to be armed. | 83 + * | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads. | 85 84 * ---------------------------------------------------------------------------- 86 85 * | 87 86 * v 88 - * ----------------------------------- 89 - * | | 90 - * v v 91 - * --------------------------------------- ----------------------------------| 92 - * | SEGCBLIST_RCU_CORE | | | SEGCBLIST_RCU_CORE | | 93 - * | SEGCBLIST_LOCKING | | | SEGCBLIST_LOCKING | | 94 - * | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | | 95 - * | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP | 96 - * | | | | 97 - * | | | | 98 - * | CB kthread woke up and | | GP kthread woke up and | 99 - * | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED| 100 - * | Processes callbacks concurrently | | | 101 - * | with rcu_core(), holding | | | 102 - * | nocb_lock. | | | 103 - * --------------------------------------- ----------------------------------- 104 - * | | 105 - * ----------------------------------- 87 + * ---------------------------------------------------------------------------- 88 + * | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED | 89 + * | + unparked CB kthread | 90 + * | | 91 + * | CB kthread got unparked and processes callbacks concurrently with | 92 + * | rcu_core(), holding nocb_lock. | 93 + * --------------------------------------------------------------------------- 94 + * | 95 + * v 96 + * ---------------------------------------------------------------------------| 97 + * | SEGCBLIST_RCU_CORE | | 98 + * | SEGCBLIST_LOCKING | | 99 + * | SEGCBLIST_OFFLOADED | | 100 + * | SEGCBLIST_KTHREAD_GP | 101 + * | + unparked CB kthread | 102 + * | | 103 + * | GP kthread woke up and acknowledged nocb_lock. | 104 + * ---------------------------------------- ----------------------------------- 106 105 * | 107 106 * v 108 107 * |--------------------------------------------------------------------------| 109 - * | SEGCBLIST_LOCKING | | 110 - * | SEGCBLIST_OFFLOADED | | 108 + * | SEGCBLIST_LOCKING | | 109 + * | SEGCBLIST_OFFLOADED | | 111 110 * | SEGCBLIST_KTHREAD_GP | | 112 - * | SEGCBLIST_KTHREAD_CB | 111 + * | + unparked CB kthread | 113 112 * | | 114 113 * | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops | 115 114 * | handling callbacks. Enable bypass queueing. | ··· 124 125 * |--------------------------------------------------------------------------| 125 126 * | SEGCBLIST_LOCKING | | 126 127 * | SEGCBLIST_OFFLOADED | | 127 - * | SEGCBLIST_KTHREAD_CB | | 128 128 * | SEGCBLIST_KTHREAD_GP | 129 + * | + unparked CB kthread | 129 130 * | | 130 131 * | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() | 131 132 * | ignores callbacks. Bypass enqueue is enabled. | ··· 136 137 * | SEGCBLIST_RCU_CORE | | 137 138 * | SEGCBLIST_LOCKING | | 138 139 * | SEGCBLIST_OFFLOADED | | 139 - * | SEGCBLIST_KTHREAD_CB | | 140 140 * | SEGCBLIST_KTHREAD_GP | 141 + * | + unparked CB kthread | 141 142 * | | 142 143 * | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() | 143 - * | handles callbacks concurrently. Bypass enqueue is enabled. | 144 + * | handles callbacks concurrently. Bypass enqueue is disabled. | 144 145 * | Invoke RCU core so we make sure not to preempt it in the middle with | 145 146 * | leaving some urgent work unattended within a jiffy. | 146 147 * ---------------------------------------------------------------------------- ··· 149 150 * |--------------------------------------------------------------------------| 150 151 * | SEGCBLIST_RCU_CORE | | 151 152 * | SEGCBLIST_LOCKING | | 152 - * | SEGCBLIST_KTHREAD_CB | | 153 153 * | SEGCBLIST_KTHREAD_GP | 154 + * | + unparked CB kthread | 154 155 * | | 155 156 * | CB/GP kthreads and local rcu_core() handle callbacks concurrently | 156 - * | holding nocb_lock. Wake up CB and GP kthreads if necessary. Disable | 157 - * | bypass enqueue. | 157 + * | holding nocb_lock. Wake up GP kthread if necessary. | 158 158 * ---------------------------------------------------------------------------- 159 159 * | 160 160 * v 161 - * ----------------------------------- 162 - * | | 163 - * v v 164 - * ---------------------------------------------------------------------------| 165 - * | | | 166 - * | SEGCBLIST_RCU_CORE | | SEGCBLIST_RCU_CORE | | 167 - * | SEGCBLIST_LOCKING | | SEGCBLIST_LOCKING | | 168 - * | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP | 169 - * | | | 170 - * | GP kthread woke up and | CB kthread woke up and | 171 - * | acknowledged the fact that | acknowledged the fact that | 172 - * | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. | 173 - * | | The CB kthread goes to sleep | 174 - * | The callbacks from the target CPU | until it ever gets re-offloaded. | 175 - * | will be ignored from the GP kthread | | 176 - * | loop. | | 161 + * |--------------------------------------------------------------------------| 162 + * | SEGCBLIST_RCU_CORE | | 163 + * | SEGCBLIST_LOCKING | | 164 + * | + unparked CB kthread | 165 + * | | 166 + * | GP kthread woke up and acknowledged the fact that SEGCBLIST_OFFLOADED | 167 + * | got cleared. The callbacks from the target CPU will be ignored from the| 168 + * | GP kthread loop. | 177 169 * ---------------------------------------------------------------------------- 178 - * | | 179 - * ----------------------------------- 180 170 * | 181 171 * v 182 172 * ---------------------------------------------------------------------------- 183 173 * | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | 174 + * | + parked CB kthread | 184 175 * | | 185 - * | Callbacks processed by rcu_core() from softirqs or local | 186 - * | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. | 187 - * | Flush pending nocb_timer. Flush nocb bypass callbacks. | 176 + * | CB kthread is parked. Callbacks processed by rcu_core() from softirqs or | 177 + * | local rcuc kthread, while holding nocb_lock. | 188 178 * ---------------------------------------------------------------------------- 189 179 * | 190 180 * v

+60 -2

include/linux/rcupdate.h

··· 209 209 210 210 #define rcu_note_voluntary_context_switch(t) rcu_tasks_qs(t, false) 211 211 void exit_tasks_rcu_start(void); 212 - void exit_tasks_rcu_stop(void); 213 212 void exit_tasks_rcu_finish(void); 214 213 #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ 215 214 #define rcu_tasks_classic_qs(t, preempt) do { } while (0) ··· 217 218 #define call_rcu_tasks call_rcu 218 219 #define synchronize_rcu_tasks synchronize_rcu 219 220 static inline void exit_tasks_rcu_start(void) { } 220 - static inline void exit_tasks_rcu_stop(void) { } 221 221 static inline void exit_tasks_rcu_finish(void) { } 222 222 #endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ 223 223 ··· 419 421 "Illegal context switch in RCU-sched read-side critical section"); \ 420 422 } while (0) 421 423 424 + // See RCU_LOCKDEP_WARN() for an explanation of the double call to 425 + // debug_lockdep_rcu_enabled(). 426 + static inline bool lockdep_assert_rcu_helper(bool c) 427 + { 428 + return debug_lockdep_rcu_enabled() && 429 + (c || !rcu_is_watching() || !rcu_lockdep_current_cpu_online()) && 430 + debug_lockdep_rcu_enabled(); 431 + } 432 + 433 + /** 434 + * lockdep_assert_in_rcu_read_lock - WARN if not protected by rcu_read_lock() 435 + * 436 + * Splats if lockdep is enabled and there is no rcu_read_lock() in effect. 437 + */ 438 + #define lockdep_assert_in_rcu_read_lock() \ 439 + WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map))) 440 + 441 + /** 442 + * lockdep_assert_in_rcu_read_lock_bh - WARN if not protected by rcu_read_lock_bh() 443 + * 444 + * Splats if lockdep is enabled and there is no rcu_read_lock_bh() in effect. 445 + * Note that local_bh_disable() and friends do not suffice here, instead an 446 + * actual rcu_read_lock_bh() is required. 447 + */ 448 + #define lockdep_assert_in_rcu_read_lock_bh() \ 449 + WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_bh_lock_map))) 450 + 451 + /** 452 + * lockdep_assert_in_rcu_read_lock_sched - WARN if not protected by rcu_read_lock_sched() 453 + * 454 + * Splats if lockdep is enabled and there is no rcu_read_lock_sched() 455 + * in effect. Note that preempt_disable() and friends do not suffice here, 456 + * instead an actual rcu_read_lock_sched() is required. 457 + */ 458 + #define lockdep_assert_in_rcu_read_lock_sched() \ 459 + WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_sched_lock_map))) 460 + 461 + /** 462 + * lockdep_assert_in_rcu_reader - WARN if not within some type of RCU reader 463 + * 464 + * Splats if lockdep is enabled and there is no RCU reader of any 465 + * type in effect. Note that regions of code protected by things like 466 + * preempt_disable, local_bh_disable(), and local_irq_disable() all qualify 467 + * as RCU readers. 468 + * 469 + * Note that this will never trigger in PREEMPT_NONE or PREEMPT_VOLUNTARY 470 + * kernels that are not also built with PREEMPT_COUNT. But if you have 471 + * lockdep enabled, you might as well also enable PREEMPT_COUNT. 472 + */ 473 + #define lockdep_assert_in_rcu_reader() \ 474 + WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map) && \ 475 + !lock_is_held(&rcu_bh_lock_map) && \ 476 + !lock_is_held(&rcu_sched_lock_map) && \ 477 + preemptible())) 478 + 422 479 #else /* #ifdef CONFIG_PROVE_RCU */ 423 480 424 481 #define RCU_LOCKDEP_WARN(c, s) do { } while (0 && (c)) 425 482 #define rcu_sleep_check() do { } while (0) 483 + 484 + #define lockdep_assert_in_rcu_read_lock() do { } while (0) 485 + #define lockdep_assert_in_rcu_read_lock_bh() do { } while (0) 486 + #define lockdep_assert_in_rcu_read_lock_sched() do { } while (0) 487 + #define lockdep_assert_in_rcu_reader() do { } while (0) 426 488 427 489 #endif /* #else #ifdef CONFIG_PROVE_RCU */ 428 490

+35

include/linux/srcu.h

··· 57 57 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); 58 58 void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); 59 59 void synchronize_srcu(struct srcu_struct *ssp); 60 + 61 + #define SRCU_GET_STATE_COMPLETED 0x1 62 + 63 + /** 64 + * get_completed_synchronize_srcu - Return a pre-completed polled state cookie 65 + * 66 + * Returns a value that poll_state_synchronize_srcu() will always treat 67 + * as a cookie whose grace period has already completed. 68 + */ 69 + static inline unsigned long get_completed_synchronize_srcu(void) 70 + { 71 + return SRCU_GET_STATE_COMPLETED; 72 + } 73 + 60 74 unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp); 61 75 unsigned long start_poll_synchronize_srcu(struct srcu_struct *ssp); 62 76 bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie); 77 + 78 + // Maximum number of unsigned long values corresponding to 79 + // not-yet-completed SRCU grace periods. 80 + #define NUM_ACTIVE_SRCU_POLL_OLDSTATE 2 81 + 82 + /** 83 + * same_state_synchronize_srcu - Are two old-state values identical? 84 + * @oldstate1: First old-state value. 85 + * @oldstate2: Second old-state value. 86 + * 87 + * The two old-state values must have been obtained from either 88 + * get_state_synchronize_srcu(), start_poll_synchronize_srcu(), or 89 + * get_completed_synchronize_srcu(). Returns @true if the two values are 90 + * identical and @false otherwise. This allows structures whose lifetimes 91 + * are tracked by old-state values to push these values to a list header, 92 + * allowing those structures to be slightly smaller. 93 + */ 94 + static inline bool same_state_synchronize_srcu(unsigned long oldstate1, unsigned long oldstate2) 95 + { 96 + return oldstate1 == oldstate2; 97 + } 63 98 64 99 #ifdef CONFIG_NEED_SRCU_NMI_SAFE 65 100 int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp);

-17

kernel/pid_namespace.c

··· 249 249 set_current_state(TASK_INTERRUPTIBLE); 250 250 if (pid_ns->pid_allocated == init_pids) 251 251 break; 252 - /* 253 - * Release tasks_rcu_exit_srcu to avoid following deadlock: 254 - * 255 - * 1) TASK A unshare(CLONE_NEWPID) 256 - * 2) TASK A fork() twice -> TASK B (child reaper for new ns) 257 - * and TASK C 258 - * 3) TASK B exits, kills TASK C, waits for TASK A to reap it 259 - * 4) TASK A calls synchronize_rcu_tasks() 260 - * -> synchronize_srcu(tasks_rcu_exit_srcu) 261 - * 5) *DEADLOCK* 262 - * 263 - * It is considered safe to release tasks_rcu_exit_srcu here 264 - * because we assume the current task can not be concurrently 265 - * reaped at this point. 266 - */ 267 - exit_tasks_rcu_stop(); 268 252 schedule(); 269 - exit_tasks_rcu_start(); 270 253 } 271 254 __set_current_state(TASK_RUNNING); 272 255

+1

kernel/rcu/rcuscale.c

··· 42 42 43 43 #include "rcu.h" 44 44 45 + MODULE_DESCRIPTION("Read-Copy Update module-based scalability-test facility"); 45 46 MODULE_LICENSE("GPL"); 46 47 MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com>"); 47 48

+28 -21

kernel/rcu/rcutorture.c

··· 51 51 52 52 #include "rcu.h" 53 53 54 + MODULE_DESCRIPTION("Read-Copy Update module-based torture test facility"); 54 55 MODULE_LICENSE("GPL"); 55 56 MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com> and Josh Triplett <josh@joshtriplett.org>"); 56 57 ··· 391 390 int extendables; 392 391 int slow_gps; 393 392 int no_pi_lock; 393 + int debug_objects; 394 394 const char *name; 395 395 }; 396 396 ··· 579 577 .irq_capable = 1, 580 578 .can_boost = IS_ENABLED(CONFIG_RCU_BOOST), 581 579 .extendables = RCUTORTURE_MAX_EXTEND, 580 + .debug_objects = 1, 582 581 .name = "rcu" 583 582 }; 584 583 ··· 750 747 .cbflood_max = 50000, 751 748 .irq_capable = 1, 752 749 .no_pi_lock = IS_ENABLED(CONFIG_TINY_SRCU), 750 + .debug_objects = 1, 753 751 .name = "srcu" 754 752 }; 755 753 ··· 790 786 .cbflood_max = 50000, 791 787 .irq_capable = 1, 792 788 .no_pi_lock = IS_ENABLED(CONFIG_TINY_SRCU), 789 + .debug_objects = 1, 793 790 .name = "srcud" 794 791 }; 795 792 ··· 2631 2626 spin_lock_irqsave(&rfp->rcu_fwd_lock, flags); 2632 2627 rfcpp = rfp->rcu_fwd_cb_tail; 2633 2628 rfp->rcu_fwd_cb_tail = &rfcp->rfc_next; 2634 - WRITE_ONCE(*rfcpp, rfcp); 2629 + smp_store_release(rfcpp, rfcp); 2635 2630 WRITE_ONCE(rfp->n_launders_cb, rfp->n_launders_cb + 1); 2636 2631 i = ((jiffies - rfp->rcu_fwd_startat) / (HZ / FWD_CBS_HIST_DIV)); 2637 2632 if (i >= ARRAY_SIZE(rfp->n_launders_hist)) ··· 3460 3455 cur_ops->gp_slow_unregister(NULL); 3461 3456 } 3462 3457 3463 - #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD 3464 3458 static void rcu_torture_leak_cb(struct rcu_head *rhp) 3465 3459 { 3466 3460 } ··· 3477 3473 */ 3478 3474 pr_alert("%s: duplicated callback was invoked.\n", KBUILD_MODNAME); 3479 3475 } 3480 - #endif /* #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD */ 3481 3476 3482 3477 /* 3483 3478 * Verify that double-free causes debug-objects to complain, but only ··· 3485 3482 */ 3486 3483 static void rcu_test_debug_objects(void) 3487 3484 { 3488 - #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD 3489 3485 struct rcu_head rh1; 3490 3486 struct rcu_head rh2; 3487 + int idx; 3488 + 3489 + if (!IS_ENABLED(CONFIG_DEBUG_OBJECTS_RCU_HEAD)) { 3490 + pr_alert("%s: !CONFIG_DEBUG_OBJECTS_RCU_HEAD, not testing duplicate call_%s()\n", 3491 + KBUILD_MODNAME, cur_ops->name); 3492 + return; 3493 + } 3494 + 3495 + if (WARN_ON_ONCE(cur_ops->debug_objects && 3496 + (!cur_ops->call || !cur_ops->cb_barrier))) 3497 + return; 3498 + 3491 3499 struct rcu_head *rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); 3492 3500 3493 3501 init_rcu_head_on_stack(&rh1); 3494 3502 init_rcu_head_on_stack(&rh2); 3495 - pr_alert("%s: WARN: Duplicate call_rcu() test starting.\n", KBUILD_MODNAME); 3503 + pr_alert("%s: WARN: Duplicate call_%s() test starting.\n", KBUILD_MODNAME, cur_ops->name); 3496 3504 3497 3505 /* Try to queue the rh2 pair of callbacks for the same grace period. */ 3498 - preempt_disable(); /* Prevent preemption from interrupting test. */ 3499 - rcu_read_lock(); /* Make it impossible to finish a grace period. */ 3500 - call_rcu_hurry(&rh1, rcu_torture_leak_cb); /* Start grace period. */ 3501 - local_irq_disable(); /* Make it harder to start a new grace period. */ 3502 - call_rcu_hurry(&rh2, rcu_torture_leak_cb); 3503 - call_rcu_hurry(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ 3506 + idx = cur_ops->readlock(); /* Make it impossible to finish a grace period. */ 3507 + cur_ops->call(&rh1, rcu_torture_leak_cb); /* Start grace period. */ 3508 + cur_ops->call(&rh2, rcu_torture_leak_cb); 3509 + cur_ops->call(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ 3504 3510 if (rhp) { 3505 - call_rcu_hurry(rhp, rcu_torture_leak_cb); 3506 - call_rcu_hurry(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ 3511 + cur_ops->call(rhp, rcu_torture_leak_cb); 3512 + cur_ops->call(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ 3507 3513 } 3508 - local_irq_enable(); 3509 - rcu_read_unlock(); 3510 - preempt_enable(); 3514 + cur_ops->readunlock(idx); 3511 3515 3512 3516 /* Wait for them all to get done so we can safely return. */ 3513 - rcu_barrier(); 3514 - pr_alert("%s: WARN: Duplicate call_rcu() test complete.\n", KBUILD_MODNAME); 3517 + cur_ops->cb_barrier(); 3518 + pr_alert("%s: WARN: Duplicate call_%s() test complete.\n", KBUILD_MODNAME, cur_ops->name); 3515 3519 destroy_rcu_head_on_stack(&rh1); 3516 3520 destroy_rcu_head_on_stack(&rh2); 3517 3521 kfree(rhp); 3518 - #else /* #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD */ 3519 - pr_alert("%s: !CONFIG_DEBUG_OBJECTS_RCU_HEAD, not testing duplicate call_rcu()\n", KBUILD_MODNAME); 3520 - #endif /* #else #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD */ 3521 3522 } 3522 3523 3523 3524 static void rcutorture_sync(void)

+1

kernel/rcu/refscale.c

··· 63 63 64 64 #define SCALEOUT_ERRSTRING(s, x...) pr_alert("%s" SCALE_FLAG "!!! " s "\n", scale_type, ## x) 65 65 66 + MODULE_DESCRIPTION("Scalability test for object reference mechanisms"); 66 67 MODULE_LICENSE("GPL"); 67 68 MODULE_AUTHOR("Joel Fernandes (Google) <joel@joelfernandes.org>"); 68 69

+2 -1

kernel/rcu/srcutiny.c

··· 277 277 unsigned long cur_s = READ_ONCE(ssp->srcu_idx); 278 278 279 279 barrier(); 280 - return ULONG_CMP_GE(cur_s, cookie) || ULONG_CMP_LT(cur_s, cookie - 3); 280 + return cookie == SRCU_GET_STATE_COMPLETED || 281 + ULONG_CMP_GE(cur_s, cookie) || ULONG_CMP_LT(cur_s, cookie - 3); 281 282 } 282 283 EXPORT_SYMBOL_GPL(poll_state_synchronize_srcu); 283 284

+8 -5

kernel/rcu/srcutree.c

··· 667 667 pr_info("%s: Active srcu_struct %p read state: %d gp state: %lu/%lu\n", 668 668 __func__, ssp, rcu_seq_state(READ_ONCE(sup->srcu_gp_seq)), 669 669 rcu_seq_current(&sup->srcu_gp_seq), sup->srcu_gp_seq_needed); 670 - return; /* Caller forgot to stop doing call_srcu()? */ 670 + return; // Caller forgot to stop doing call_srcu()? 671 + // Or caller invoked start_poll_synchronize_srcu() 672 + // and then cleanup_srcu_struct() before that grace 673 + // period ended? 671 674 } 672 675 kfree(sup->node); 673 676 sup->node = NULL; ··· 848 845 bool cbs; 849 846 bool last_lvl; 850 847 int cpu; 851 - unsigned long flags; 852 848 unsigned long gpseq; 853 849 int idx; 854 850 unsigned long mask; ··· 909 907 if (!(gpseq & counter_wrap_check)) 910 908 for_each_possible_cpu(cpu) { 911 909 sdp = per_cpu_ptr(ssp->sda, cpu); 912 - spin_lock_irqsave_rcu_node(sdp, flags); 910 + spin_lock_irq_rcu_node(sdp); 913 911 if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed + 100)) 914 912 sdp->srcu_gp_seq_needed = gpseq; 915 913 if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed_exp + 100)) 916 914 sdp->srcu_gp_seq_needed_exp = gpseq; 917 - spin_unlock_irqrestore_rcu_node(sdp, flags); 915 + spin_unlock_irq_rcu_node(sdp); 918 916 } 919 917 920 918 /* Callback initiation done, allow grace periods after next. */ ··· 1542 1540 */ 1543 1541 bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie) 1544 1542 { 1545 - if (!rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, cookie)) 1543 + if (cookie != SRCU_GET_STATE_COMPLETED && 1544 + !rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, cookie)) 1546 1545 return false; 1547 1546 // Ensure that the end of the SRCU grace period happens before 1548 1547 // any subsequent code that the caller might execute.

+4 -8

kernel/rcu/sync.c

··· 122 122 * we are called at early boot time but this shouldn't happen. 123 123 */ 124 124 } 125 - WRITE_ONCE(rsp->gp_count, rsp->gp_count + 1); 125 + rsp->gp_count++; 126 126 spin_unlock_irq(&rsp->rss_lock); 127 127 128 128 if (gp_state == GP_IDLE) { ··· 151 151 */ 152 152 void rcu_sync_exit(struct rcu_sync *rsp) 153 153 { 154 - int gpc; 155 - 156 154 WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_IDLE); 157 - WARN_ON_ONCE(READ_ONCE(rsp->gp_count) == 0); 158 155 159 156 spin_lock_irq(&rsp->rss_lock); 160 - gpc = rsp->gp_count - 1; 161 - WRITE_ONCE(rsp->gp_count, gpc); 162 - if (!gpc) { 157 + WARN_ON_ONCE(rsp->gp_count == 0); 158 + if (!--rsp->gp_count) { 163 159 if (rsp->gp_state == GP_PASSED) { 164 160 WRITE_ONCE(rsp->gp_state, GP_EXIT); 165 161 rcu_sync_call(rsp); ··· 174 178 { 175 179 int gp_state; 176 180 177 - WARN_ON_ONCE(READ_ONCE(rsp->gp_count)); 178 181 WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_PASSED); 179 182 180 183 spin_lock_irq(&rsp->rss_lock); 184 + WARN_ON_ONCE(rsp->gp_count); 181 185 if (rsp->gp_state == GP_REPLAY) 182 186 WRITE_ONCE(rsp->gp_state, GP_EXIT); 183 187 gp_state = rsp->gp_state;

+13 -13

kernel/rcu/tasks.h

··· 858 858 // not know to synchronize with this RCU Tasks grace period) have 859 859 // completed exiting. The synchronize_rcu() in rcu_tasks_postgp() 860 860 // will take care of any tasks stuck in the non-preemptible region 861 - // of do_exit() following its call to exit_tasks_rcu_stop(). 861 + // of do_exit() following its call to exit_tasks_rcu_finish(). 862 862 // check_all_holdout_tasks(), repeatedly until holdout list is empty: 863 863 // Scans the holdout list, attempting to identify a quiescent state 864 864 // for each task on the list. If there is a quiescent state, the ··· 1220 1220 * Remove the task from the "yet another list" because do_exit() is now 1221 1221 * non-preemptible, allowing synchronize_rcu() to wait beyond this point. 1222 1222 */ 1223 - void exit_tasks_rcu_stop(void) 1223 + void exit_tasks_rcu_finish(void) 1224 1224 { 1225 1225 unsigned long flags; 1226 1226 struct rcu_tasks_percpu *rtpcp; ··· 1231 1231 raw_spin_lock_irqsave_rcu_node(rtpcp, flags); 1232 1232 list_del_init(&t->rcu_tasks_exit_list); 1233 1233 raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); 1234 - } 1235 1234 1236 - /* 1237 - * Contribute to protect against tasklist scan blind spot while the 1238 - * task is exiting and may be removed from the tasklist. See 1239 - * corresponding synchronize_srcu() for further details. 1240 - */ 1241 - void exit_tasks_rcu_finish(void) 1242 - { 1243 - exit_tasks_rcu_stop(); 1244 - exit_tasks_rcu_finish_trace(current); 1235 + exit_tasks_rcu_finish_trace(t); 1245 1236 } 1246 1237 1247 1238 #else /* #ifdef CONFIG_TASKS_RCU */ 1248 1239 void exit_tasks_rcu_start(void) { } 1249 - void exit_tasks_rcu_stop(void) { } 1250 1240 void exit_tasks_rcu_finish(void) { exit_tasks_rcu_finish_trace(current); } 1251 1241 #endif /* #else #ifdef CONFIG_TASKS_RCU */ 1252 1242 ··· 1747 1757 // allow safe access to the hop list. 1748 1758 for_each_online_cpu(cpu) { 1749 1759 rcu_read_lock(); 1760 + // Note that cpu_curr_snapshot() picks up the target 1761 + // CPU's current task while its runqueue is locked with 1762 + // an smp_mb__after_spinlock(). This ensures that either 1763 + // the grace-period kthread will see that task's read-side 1764 + // critical section or the task will see the updater's pre-GP 1765 + // accesses. The trailing smp_mb() in cpu_curr_snapshot() 1766 + // does not currently play a role other than simplify 1767 + // that function's ordering semantics. If these simplified 1768 + // ordering semantics continue to be redundant, that smp_mb() 1769 + // might be removed. 1750 1770 t = cpu_curr_snapshot(cpu); 1751 1771 if (rcu_tasks_trace_pertask_prep(t, true)) 1752 1772 trc_add_holdout(t, hop);

+67 -25

kernel/rcu/tree.c

··· 96 96 .ofl_lock = __ARCH_SPIN_LOCK_UNLOCKED, 97 97 .srs_cleanup_work = __WORK_INITIALIZER(rcu_state.srs_cleanup_work, 98 98 rcu_sr_normal_gp_cleanup_work), 99 + .srs_cleanups_pending = ATOMIC_INIT(0), 99 100 }; 100 101 101 102 /* Dump rcu_node combining tree at boot to verify correct setup. */ ··· 176 175 module_param(gp_init_delay, int, 0444); 177 176 static int gp_cleanup_delay; 178 177 module_param(gp_cleanup_delay, int, 0444); 178 + static int nohz_full_patience_delay; 179 + module_param(nohz_full_patience_delay, int, 0444); 180 + static int nohz_full_patience_delay_jiffies; 179 181 180 182 // Add delay to rcu_read_unlock() for strict grace periods. 181 183 static int rcu_unlock_delay; ··· 300 296 } 301 297 302 298 /* 303 - * Snapshot the ->dynticks counter with full ordering so as to allow 304 - * stable comparison of this counter with past and future snapshots. 305 - */ 306 - static int rcu_dynticks_snap(int cpu) 307 - { 308 - smp_mb(); // Fundamental RCU ordering guarantee. 309 - return ct_dynticks_cpu_acquire(cpu); 310 - } 311 - 312 - /* 313 299 * Return true if the snapshot returned from rcu_dynticks_snap() 314 300 * indicates that RCU is in an extended quiescent state. 315 301 */ ··· 315 321 */ 316 322 static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) 317 323 { 318 - return snap != rcu_dynticks_snap(rdp->cpu); 324 + /* 325 + * The first failing snapshot is already ordered against the accesses 326 + * performed by the remote CPU after it exits idle. 327 + * 328 + * The second snapshot therefore only needs to order against accesses 329 + * performed by the remote CPU prior to entering idle and therefore can 330 + * rely solely on acquire semantics. 331 + */ 332 + return snap != ct_dynticks_cpu_acquire(rdp->cpu); 319 333 } 320 334 321 335 /* ··· 771 769 */ 772 770 static int dyntick_save_progress_counter(struct rcu_data *rdp) 773 771 { 774 - rdp->dynticks_snap = rcu_dynticks_snap(rdp->cpu); 772 + /* 773 + * Full ordering between remote CPU's post idle accesses and updater's 774 + * accesses prior to current GP (and also the started GP sequence number) 775 + * is enforced by rcu_seq_start() implicit barrier and even further by 776 + * smp_mb__after_unlock_lock() barriers chained all the way throughout the 777 + * rnp locking tree since rcu_gp_init() and up to the current leaf rnp 778 + * locking. 779 + * 780 + * Ordering between remote CPU's pre idle accesses and post grace period 781 + * updater's accesses is enforced by the below acquire semantic. 782 + */ 783 + rdp->dynticks_snap = ct_dynticks_cpu_acquire(rdp->cpu); 775 784 if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { 776 785 trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); 777 786 rcu_gpnum_ovf(rdp->mynode, rdp); ··· 1673 1660 1674 1661 rcu_sr_put_wait_head(rcu); 1675 1662 } 1663 + 1664 + /* Order list manipulations with atomic access. */ 1665 + atomic_dec_return_release(&rcu_state.srs_cleanups_pending); 1676 1666 } 1677 1667 1678 1668 /* ··· 1683 1667 */ 1684 1668 static void rcu_sr_normal_gp_cleanup(void) 1685 1669 { 1686 - struct llist_node *wait_tail, *next, *rcu; 1670 + struct llist_node *wait_tail, *next = NULL, *rcu = NULL; 1687 1671 int done = 0; 1688 1672 1689 1673 wait_tail = rcu_state.srs_wait_tail; ··· 1709 1693 break; 1710 1694 } 1711 1695 1712 - // concurrent sr_normal_gp_cleanup work might observe this update. 1713 - smp_store_release(&rcu_state.srs_done_tail, wait_tail); 1696 + /* 1697 + * Fast path, no more users to process except putting the second last 1698 + * wait head if no inflight-workers. If there are in-flight workers, 1699 + * they will remove the last wait head. 1700 + * 1701 + * Note that the ACQUIRE orders atomic access with list manipulation. 1702 + */ 1703 + if (wait_tail->next && wait_tail->next->next == NULL && 1704 + rcu_sr_is_wait_head(wait_tail->next) && 1705 + !atomic_read_acquire(&rcu_state.srs_cleanups_pending)) { 1706 + rcu_sr_put_wait_head(wait_tail->next); 1707 + wait_tail->next = NULL; 1708 + } 1709 + 1710 + /* Concurrent sr_normal_gp_cleanup work might observe this update. */ 1714 1711 ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_done_tail); 1712 + smp_store_release(&rcu_state.srs_done_tail, wait_tail); 1715 1713 1716 1714 /* 1717 1715 * We schedule a work in order to perform a final processing 1718 1716 * of outstanding users(if still left) and releasing wait-heads 1719 1717 * added by rcu_sr_normal_gp_init() call. 1720 1718 */ 1721 - queue_work(sync_wq, &rcu_state.srs_cleanup_work); 1719 + if (wait_tail->next) { 1720 + atomic_inc(&rcu_state.srs_cleanups_pending); 1721 + if (!queue_work(sync_wq, &rcu_state.srs_cleanup_work)) 1722 + atomic_dec(&rcu_state.srs_cleanups_pending); 1723 + } 1722 1724 } 1723 1725 1724 1726 /* ··· 1844 1810 WRITE_ONCE(rcu_state.gp_state, RCU_GP_ONOFF); 1845 1811 /* Exclude CPU hotplug operations. */ 1846 1812 rcu_for_each_leaf_node(rnp) { 1847 - local_irq_save(flags); 1813 + local_irq_disable(); 1848 1814 arch_spin_lock(&rcu_state.ofl_lock); 1849 1815 raw_spin_lock_rcu_node(rnp); 1850 1816 if (rnp->qsmaskinit == rnp->qsmaskinitnext && ··· 1852 1818 /* Nothing to do on this leaf rcu_node structure. */ 1853 1819 raw_spin_unlock_rcu_node(rnp); 1854 1820 arch_spin_unlock(&rcu_state.ofl_lock); 1855 - local_irq_restore(flags); 1821 + local_irq_enable(); 1856 1822 continue; 1857 1823 } 1858 1824 ··· 1889 1855 1890 1856 raw_spin_unlock_rcu_node(rnp); 1891 1857 arch_spin_unlock(&rcu_state.ofl_lock); 1892 - local_irq_restore(flags); 1858 + local_irq_enable(); 1893 1859 } 1894 1860 rcu_gp_slow(gp_preinit_delay); /* Races with CPU hotplug. */ 1895 1861 ··· 4347 4313 return 1; 4348 4314 4349 4315 /* Is this a nohz_full CPU in userspace or idle? (Ignore RCU if so.) */ 4350 - if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu()) 4316 + gp_in_progress = rcu_gp_in_progress(); 4317 + if ((user || rcu_is_cpu_rrupt_from_idle() || 4318 + (gp_in_progress && 4319 + time_before(jiffies, READ_ONCE(rcu_state.gp_start) + 4320 + nohz_full_patience_delay_jiffies))) && 4321 + rcu_nohz_full_cpu()) 4351 4322 return 0; 4352 4323 4353 4324 /* Is the RCU core waiting for a quiescent state from this CPU? */ 4354 - gp_in_progress = rcu_gp_in_progress(); 4355 4325 if (rdp->core_needs_qs && !rdp->cpu_no_qs.b.norm && gp_in_progress) 4356 4326 return 1; 4357 4327 ··· 4805 4767 rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); 4806 4768 INIT_WORK(&rdp->strict_work, strict_work_handler); 4807 4769 WARN_ON_ONCE(ct->dynticks_nesting != 1); 4808 - WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu))); 4770 + WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu))); 4809 4771 rdp->barrier_seq_snap = rcu_state.barrier_sequence; 4810 4772 rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; 4811 4773 rdp->rcu_ofl_gp_state = RCU_GP_CLEANED; ··· 5148 5110 struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); 5149 5111 bool needwake; 5150 5112 5151 - if (rcu_rdp_is_offloaded(rdp) || 5152 - rcu_segcblist_empty(&rdp->cblist)) 5153 - return; /* No callbacks to migrate. */ 5113 + if (rcu_rdp_is_offloaded(rdp)) 5114 + return; 5154 5115 5155 5116 raw_spin_lock_irqsave(&rcu_state.barrier_lock, flags); 5117 + if (rcu_segcblist_empty(&rdp->cblist)) { 5118 + raw_spin_unlock_irqrestore(&rcu_state.barrier_lock, flags); 5119 + return; /* No callbacks to migrate. */ 5120 + } 5121 + 5156 5122 WARN_ON_ONCE(rcu_rdp_cpu_online(rdp)); 5157 5123 rcu_barrier_entrain(rdp); 5158 5124 my_rdp = this_cpu_ptr(&rcu_data);

+1 -1

kernel/rcu/tree.h

··· 223 223 struct swait_queue_head nocb_state_wq; /* For offloading state changes */ 224 224 struct task_struct *nocb_gp_kthread; 225 225 raw_spinlock_t nocb_lock; /* Guard following pair of fields. */ 226 - atomic_t nocb_lock_contended; /* Contention experienced. */ 227 226 int nocb_defer_wakeup; /* Defer wakeup of nocb_kthread. */ 228 227 struct timer_list nocb_timer; /* Enforce finite deferral. */ 229 228 unsigned long nocb_gp_adv_time; /* Last call_rcu() CB adv (jiffies). */ ··· 419 420 struct llist_node *srs_done_tail; /* ready for GP users. */ 420 421 struct sr_wait_node srs_wait_nodes[SR_NORMAL_GP_WAIT_HEAD_MAX]; 421 422 struct work_struct srs_cleanup_work; 423 + atomic_t srs_cleanups_pending; /* srs inflight worker cleanups. */ 422 424 }; 423 425 424 426 /* Values for rcu_state structure's gp_flags field. */

+21 -3

kernel/rcu/tree_exp.h

··· 265 265 { 266 266 if (rcu_exp_gp_seq_done(s)) { 267 267 trace_rcu_exp_grace_period(rcu_state.name, s, TPS("done")); 268 - smp_mb(); /* Ensure test happens before caller kfree(). */ 268 + /* 269 + * Order GP completion with preceding accesses. Order also GP 270 + * completion with post GP update side accesses. Pairs with 271 + * rcu_seq_end(). 272 + */ 273 + smp_mb(); 269 274 return true; 270 275 } 271 276 return false; ··· 362 357 !(rnp->qsmaskinitnext & mask)) { 363 358 mask_ofl_test |= mask; 364 359 } else { 365 - snap = rcu_dynticks_snap(cpu); 360 + /* 361 + * Full ordering between remote CPU's post idle accesses 362 + * and updater's accesses prior to current GP (and also 363 + * the started GP sequence number) is enforced by 364 + * rcu_seq_start() implicit barrier, relayed by kworkers 365 + * locking and even further by smp_mb__after_unlock_lock() 366 + * barriers chained all the way throughout the rnp locking 367 + * tree since sync_exp_reset_tree() and up to the current 368 + * leaf rnp locking. 369 + * 370 + * Ordering between remote CPU's pre idle accesses and 371 + * post grace period updater's accesses is enforced by the 372 + * below acquire semantic. 373 + */ 374 + snap = ct_dynticks_cpu_acquire(cpu); 366 375 if (rcu_dynticks_in_eqs(snap)) 367 376 mask_ofl_test |= mask; 368 377 else ··· 972 953 rnp = rcu_get_root(); 973 954 wait_event(rnp->exp_wq[rcu_seq_ctr(s) & 0x3], 974 955 sync_exp_work_done(s)); 975 - smp_mb(); /* Work actions happen before return. */ 976 956 977 957 /* Let the next expedited grace period start. */ 978 958 mutex_unlock(&rcu_state.exp_mutex);

+41 -108

kernel/rcu/tree_nocb.h

··· 91 91 92 92 /* 93 93 * Acquire the specified rcu_data structure's ->nocb_bypass_lock. If the 94 - * lock isn't immediately available, increment ->nocb_lock_contended to 95 - * flag the contention. 94 + * lock isn't immediately available, perform minimal sanity check. 96 95 */ 97 96 static void rcu_nocb_bypass_lock(struct rcu_data *rdp) 98 97 __acquires(&rdp->nocb_bypass_lock) ··· 99 100 lockdep_assert_irqs_disabled(); 100 101 if (raw_spin_trylock(&rdp->nocb_bypass_lock)) 101 102 return; 102 - atomic_inc(&rdp->nocb_lock_contended); 103 + /* 104 + * Contention expected only when local enqueue collide with 105 + * remote flush from kthreads. 106 + */ 103 107 WARN_ON_ONCE(smp_processor_id() != rdp->cpu); 104 - smp_mb__after_atomic(); /* atomic_inc() before lock. */ 105 108 raw_spin_lock(&rdp->nocb_bypass_lock); 106 - smp_mb__before_atomic(); /* atomic_dec() after lock. */ 107 - atomic_dec(&rdp->nocb_lock_contended); 108 - } 109 - 110 - /* 111 - * Spinwait until the specified rcu_data structure's ->nocb_lock is 112 - * not contended. Please note that this is extremely special-purpose, 113 - * relying on the fact that at most two kthreads and one CPU contend for 114 - * this lock, and also that the two kthreads are guaranteed to have frequent 115 - * grace-period-duration time intervals between successive acquisitions 116 - * of the lock. This allows us to use an extremely simple throttling 117 - * mechanism, and further to apply it only to the CPU doing floods of 118 - * call_rcu() invocations. Don't try this at home! 119 - */ 120 - static void rcu_nocb_wait_contended(struct rcu_data *rdp) 121 - { 122 - WARN_ON_ONCE(smp_processor_id() != rdp->cpu); 123 - while (WARN_ON_ONCE(atomic_read(&rdp->nocb_lock_contended))) 124 - cpu_relax(); 125 109 } 126 110 127 111 /* ··· 492 510 } 493 511 494 512 // We need to use the bypass. 495 - rcu_nocb_wait_contended(rdp); 496 513 rcu_nocb_bypass_lock(rdp); 497 514 ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); 498 515 rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ ··· 616 635 } 617 636 } 618 637 619 - static int nocb_gp_toggle_rdp(struct rcu_data *rdp, 620 - bool *wake_state) 638 + static int nocb_gp_toggle_rdp(struct rcu_data *rdp) 621 639 { 622 640 struct rcu_segcblist *cblist = &rdp->cblist; 623 641 unsigned long flags; ··· 630 650 * We will handle this rdp until it ever gets de-offloaded. 631 651 */ 632 652 rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_GP); 633 - if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) 634 - *wake_state = true; 635 653 ret = 1; 636 654 } else if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED) && 637 655 rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)) { ··· 638 660 * We will ignore this rdp until it ever gets re-offloaded. 639 661 */ 640 662 rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_GP); 641 - if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) 642 - *wake_state = true; 643 663 ret = 0; 644 664 } else { 645 665 WARN_ON_ONCE(1); ··· 853 877 } 854 878 855 879 if (rdp_toggling) { 856 - bool wake_state = false; 857 880 int ret; 858 881 859 - ret = nocb_gp_toggle_rdp(rdp_toggling, &wake_state); 882 + ret = nocb_gp_toggle_rdp(rdp_toggling); 860 883 if (ret == 1) 861 884 list_add_tail(&rdp_toggling->nocb_entry_rdp, &my_rdp->nocb_head_rdp); 862 885 else if (ret == 0) 863 886 list_del(&rdp_toggling->nocb_entry_rdp); 864 - if (wake_state) 865 - swake_up_one(&rdp_toggling->nocb_state_wq); 887 + 888 + swake_up_one(&rdp_toggling->nocb_state_wq); 866 889 } 867 890 868 891 my_rdp->nocb_gp_seq = -1; ··· 888 913 return 0; 889 914 } 890 915 891 - static inline bool nocb_cb_can_run(struct rcu_data *rdp) 892 - { 893 - u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_CB; 894 - 895 - return rcu_segcblist_test_flags(&rdp->cblist, flags); 896 - } 897 - 898 916 static inline bool nocb_cb_wait_cond(struct rcu_data *rdp) 899 917 { 900 - return nocb_cb_can_run(rdp) && !READ_ONCE(rdp->nocb_cb_sleep); 918 + return !READ_ONCE(rdp->nocb_cb_sleep) || kthread_should_park(); 901 919 } 902 920 903 921 /* ··· 902 934 struct rcu_segcblist *cblist = &rdp->cblist; 903 935 unsigned long cur_gp_seq; 904 936 unsigned long flags; 905 - bool needwake_state = false; 906 937 bool needwake_gp = false; 907 - bool can_sleep = true; 908 938 struct rcu_node *rnp = rdp->mynode; 909 939 910 - do { 911 - swait_event_interruptible_exclusive(rdp->nocb_cb_wq, 912 - nocb_cb_wait_cond(rdp)); 940 + swait_event_interruptible_exclusive(rdp->nocb_cb_wq, 941 + nocb_cb_wait_cond(rdp)); 942 + if (kthread_should_park()) { 943 + kthread_parkme(); 944 + } else if (READ_ONCE(rdp->nocb_cb_sleep)) { 945 + WARN_ON(signal_pending(current)); 946 + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty")); 947 + } 913 948 914 - if (READ_ONCE(rdp->nocb_cb_sleep)) { 915 - WARN_ON(signal_pending(current)); 916 - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty")); 917 - } 918 - } while (!nocb_cb_can_run(rdp)); 919 - 949 + WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp)); 920 950 921 951 local_irq_save(flags); 922 952 rcu_momentary_dyntick_idle(); ··· 937 971 raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ 938 972 } 939 973 940 - if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) { 941 - if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) { 942 - rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_CB); 943 - if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)) 944 - needwake_state = true; 945 - } 946 - if (rcu_segcblist_ready_cbs(cblist)) 947 - can_sleep = false; 948 - } else { 949 - /* 950 - * De-offloading. Clear our flag and notify the de-offload worker. 951 - * We won't touch the callbacks and keep sleeping until we ever 952 - * get re-offloaded. 953 - */ 954 - WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)); 955 - rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_CB); 956 - if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)) 957 - needwake_state = true; 958 - } 959 - 960 - WRITE_ONCE(rdp->nocb_cb_sleep, can_sleep); 961 - 962 - if (rdp->nocb_cb_sleep) 974 + if (!rcu_segcblist_ready_cbs(cblist)) { 975 + WRITE_ONCE(rdp->nocb_cb_sleep, true); 963 976 trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep")); 977 + } else { 978 + WRITE_ONCE(rdp->nocb_cb_sleep, false); 979 + } 964 980 965 981 rcu_nocb_unlock_irqrestore(rdp, flags); 966 982 if (needwake_gp) 967 983 rcu_gp_kthread_wake(); 968 - 969 - if (needwake_state) 970 - swake_up_one(&rdp->nocb_state_wq); 971 984 } 972 985 973 986 /* ··· 1039 1094 bool wake_gp = false; 1040 1095 1041 1096 rcu_segcblist_offload(cblist, offload); 1042 - 1043 - if (rdp->nocb_cb_sleep) 1044 - rdp->nocb_cb_sleep = false; 1045 1097 rcu_nocb_unlock_irqrestore(rdp, flags); 1046 - 1047 - /* 1048 - * Ignore former value of nocb_cb_sleep and force wake up as it could 1049 - * have been spuriously set to false already. 1050 - */ 1051 - swake_up_one(&rdp->nocb_cb_wq); 1052 1098 1053 1099 raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); 1054 1100 // Queue this rdp for add/del to/from the list to iterate on rcuog ··· 1097 1161 if (wake_gp) 1098 1162 wake_up_process(rdp_gp->nocb_gp_kthread); 1099 1163 1100 - /* 1101 - * If rcuo[p] kthread spawn failed, directly remove SEGCBLIST_KTHREAD_CB. 1102 - * Just wait SEGCBLIST_KTHREAD_GP to be cleared by rcuog. 1103 - */ 1104 - if (!rdp->nocb_cb_kthread) { 1105 - rcu_nocb_lock_irqsave(rdp, flags); 1106 - rcu_segcblist_clear_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB); 1107 - rcu_nocb_unlock_irqrestore(rdp, flags); 1108 - } 1109 - 1110 1164 swait_event_exclusive(rdp->nocb_state_wq, 1111 - !rcu_segcblist_test_flags(cblist, 1112 - SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP)); 1165 + !rcu_segcblist_test_flags(cblist, 1166 + SEGCBLIST_KTHREAD_GP)); 1167 + if (rdp->nocb_cb_kthread) 1168 + kthread_park(rdp->nocb_cb_kthread); 1113 1169 } else { 1114 1170 /* 1115 1171 * No kthread to clear the flags for us or remove the rdp from the nocb list ··· 1109 1181 * but we stick to paranoia in this rare path. 1110 1182 */ 1111 1183 rcu_nocb_lock_irqsave(rdp, flags); 1112 - rcu_segcblist_clear_flags(&rdp->cblist, 1113 - SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP); 1184 + rcu_segcblist_clear_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP); 1114 1185 rcu_nocb_unlock_irqrestore(rdp, flags); 1115 1186 1116 1187 list_del(&rdp->nocb_entry_rdp); ··· 1209 1282 wake_gp = rdp_offload_toggle(rdp, true, flags); 1210 1283 if (wake_gp) 1211 1284 wake_up_process(rdp_gp->nocb_gp_kthread); 1285 + 1286 + kthread_unpark(rdp->nocb_cb_kthread); 1287 + 1212 1288 swait_event_exclusive(rdp->nocb_state_wq, 1213 - rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB) && 1214 1289 rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)); 1215 1290 1216 1291 /* ··· 1397 1468 if (rcu_segcblist_empty(&rdp->cblist)) 1398 1469 rcu_segcblist_init(&rdp->cblist); 1399 1470 rcu_segcblist_offload(&rdp->cblist, true); 1400 - rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP); 1471 + rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP); 1401 1472 rcu_segcblist_clear_flags(&rdp->cblist, SEGCBLIST_RCU_CORE); 1402 1473 } 1403 1474 rcu_organize_nocb_kthreads(); ··· 1455 1526 mutex_unlock(&rdp_gp->nocb_gp_kthread_mutex); 1456 1527 1457 1528 /* Spawn the kthread for this CPU. */ 1458 - t = kthread_run(rcu_nocb_cb_kthread, rdp, 1459 - "rcuo%c/%d", rcu_state.abbr, cpu); 1529 + t = kthread_create(rcu_nocb_cb_kthread, rdp, 1530 + "rcuo%c/%d", rcu_state.abbr, cpu); 1460 1531 if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo CB kthread, OOM is now expected behavior\n", __func__)) 1461 1532 goto end; 1533 + 1534 + if (rcu_rdp_is_offloaded(rdp)) 1535 + wake_up_process(t); 1536 + else 1537 + kthread_park(t); 1462 1538 1463 1539 if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_CB_BOOST) && kthread_prio) 1464 1540 sched_setscheduler_nocheck(t, SCHED_FIFO, &sp); ··· 1612 1678 1613 1679 sprintf(bufw, "%ld", rsclp->gp_seq[RCU_WAIT_TAIL]); 1614 1680 sprintf(bufr, "%ld", rsclp->gp_seq[RCU_NEXT_READY_TAIL]); 1615 - pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n", 1681 + pr_info(" CB %d^%d->%d %c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n", 1616 1682 rdp->cpu, rdp->nocb_gp_rdp->cpu, 1617 1683 nocb_next_rdp ? nocb_next_rdp->cpu : -1, 1618 1684 "kK"[!!rdp->nocb_cb_kthread], 1619 1685 "bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)], 1620 - "cC"[!!atomic_read(&rdp->nocb_lock_contended)], 1621 1686 "lL"[raw_spin_is_locked(&rdp->nocb_lock)], 1622 1687 "sS"[!!rdp->nocb_cb_sleep], 1623 1688 ".W"[swait_active(&rdp->nocb_cb_wq)],

+12 -2

kernel/rcu/tree_plugin.h

··· 28 28 !(lockdep_is_held(&rcu_state.barrier_mutex) || 29 29 (IS_ENABLED(CONFIG_HOTPLUG_CPU) && lockdep_is_cpus_held()) || 30 30 rcu_lockdep_is_held_nocb(rdp) || 31 - (rdp == this_cpu_ptr(&rcu_data) && 32 - !(IS_ENABLED(CONFIG_PREEMPT_COUNT) && preemptible())) || 31 + (!(IS_ENABLED(CONFIG_PREEMPT_COUNT) && preemptible()) && 32 + rdp == this_cpu_ptr(&rcu_data)) || 33 33 rcu_current_is_nocb_kthread(rdp)), 34 34 "Unsafe read of RCU_NOCB offloaded state" 35 35 ); ··· 93 93 pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay); 94 94 if (gp_cleanup_delay) 95 95 pr_info("\tRCU debug GP cleanup slowdown %d jiffies.\n", gp_cleanup_delay); 96 + if (nohz_full_patience_delay < 0) { 97 + pr_info("\tRCU NOCB CPU patience negative (%d), resetting to zero.\n", nohz_full_patience_delay); 98 + nohz_full_patience_delay = 0; 99 + } else if (nohz_full_patience_delay > 5 * MSEC_PER_SEC) { 100 + pr_info("\tRCU NOCB CPU patience too large (%d), resetting to %ld.\n", nohz_full_patience_delay, 5 * MSEC_PER_SEC); 101 + nohz_full_patience_delay = 5 * MSEC_PER_SEC; 102 + } else if (nohz_full_patience_delay) { 103 + pr_info("\tRCU NOCB CPU patience set to %d milliseconds.\n", nohz_full_patience_delay); 104 + } 105 + nohz_full_patience_delay_jiffies = msecs_to_jiffies(nohz_full_patience_delay); 96 106 if (!use_softirq) 97 107 pr_info("\tRCU_SOFTIRQ processing moved to rcuc kthreads.\n"); 98 108 if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG))

+2 -2

kernel/rcu/tree_stall.h

··· 501 501 } 502 502 delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); 503 503 falsepositive = rcu_is_gp_kthread_starving(NULL) && 504 - rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu)); 504 + rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu)); 505 505 rcuc_starved = rcu_is_rcuc_kthread_starving(rdp, &j); 506 506 if (rcuc_starved) 507 507 // Print signed value, as negative values indicate a probable bug. ··· 515 515 rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : 516 516 "!."[!delta], 517 517 ticks_value, ticks_title, 518 - rcu_dynticks_snap(cpu) & 0xffff, 518 + ct_dynticks_cpu(cpu) & 0xffff, 519 519 ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), 520 520 rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), 521 521 data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,

+7 -7

kernel/sched/core.c

··· 4466 4466 * @cpu: The CPU on which to snapshot the task. 4467 4467 * 4468 4468 * Returns the task_struct pointer of the task "currently" running on 4469 - * the specified CPU. If the same task is running on that CPU throughout, 4470 - * the return value will be a pointer to that task's task_struct structure. 4471 - * If the CPU did any context switches even vaguely concurrently with the 4472 - * execution of this function, the return value will be a pointer to the 4473 - * task_struct structure of a randomly chosen task that was running on 4474 - * that CPU somewhere around the time that this function was executing. 4469 + * the specified CPU. 4475 4470 * 4476 4471 * If the specified CPU was offline, the return value is whatever it 4477 4472 * is, perhaps a pointer to the task_struct structure of that CPU's idle ··· 4480 4485 */ 4481 4486 struct task_struct *cpu_curr_snapshot(int cpu) 4482 4487 { 4488 + struct rq *rq = cpu_rq(cpu); 4483 4489 struct task_struct *t; 4490 + struct rq_flags rf; 4484 4491 4485 - smp_mb(); /* Pairing determined by caller's synchronization design. */ 4492 + rq_lock_irqsave(rq, &rf); 4493 + smp_mb__after_spinlock(); /* Pairing determined by caller's synchronization design. */ 4486 4494 t = rcu_dereference(cpu_curr(cpu)); 4495 + rq_unlock_irqrestore(rq, &rf); 4487 4496 smp_mb(); /* Pairing determined by caller's synchronization design. */ 4497 + 4488 4498 return t; 4489 4499 } 4490 4500

+52

tools/rcu/rcu-updaters.sh

··· 1 + #!/bin/sh 2 + # SPDX-License-Identifier: GPL-2.0+ 3 + # 4 + # Run bpftrace to obtain a histogram of the types of primitives used to 5 + # initiate RCU grace periods. The count associated with rcu_gp_init() 6 + # is the number of normal (non-expedited) grace periods. 7 + # 8 + # Usage: rcu-updaters.sh [ duration-in-seconds ] 9 + # 10 + # Note that not all kernel builds have all of these functions. In those 11 + # that do not, this script will issue a diagnostic for each that is not 12 + # found, but continue normally for the rest of the functions. 13 + 14 + duration=${1} 15 + if test -n "${duration}" 16 + then 17 + exitclause='interval:s:'"${duration}"' { exit(); }' 18 + else 19 + echo 'Hit control-C to end sample and print results.' 20 + fi 21 + bpftrace -e 'kprobe:kvfree_call_rcu, 22 + kprobe:call_rcu, 23 + kprobe:call_rcu_tasks, 24 + kprobe:call_rcu_tasks_rude, 25 + kprobe:call_rcu_tasks_trace, 26 + kprobe:call_srcu, 27 + kprobe:rcu_barrier, 28 + kprobe:rcu_barrier_tasks, 29 + kprobe:rcu_barrier_tasks_rude, 30 + kprobe:rcu_barrier_tasks_trace, 31 + kprobe:srcu_barrier, 32 + kprobe:synchronize_rcu, 33 + kprobe:synchronize_rcu_expedited, 34 + kprobe:synchronize_rcu_tasks, 35 + kprobe:synchronize_rcu_tasks_rude, 36 + kprobe:synchronize_rcu_tasks_trace, 37 + kprobe:synchronize_srcu, 38 + kprobe:synchronize_srcu_expedited, 39 + kprobe:get_state_synchronize_rcu, 40 + kprobe:get_state_synchronize_rcu_full, 41 + kprobe:start_poll_synchronize_rcu, 42 + kprobe:start_poll_synchronize_rcu_expedited, 43 + kprobe:start_poll_synchronize_rcu_full, 44 + kprobe:start_poll_synchronize_rcu_expedited_full, 45 + kprobe:poll_state_synchronize_rcu, 46 + kprobe:poll_state_synchronize_rcu_full, 47 + kprobe:cond_synchronize_rcu, 48 + kprobe:cond_synchronize_rcu_full, 49 + kprobe:start_poll_synchronize_srcu, 50 + kprobe:poll_state_synchronize_srcu, 51 + kprobe:rcu_gp_init 52 + { @counts[func] = count(); } '"${exitclause}"

Configure Feed

Configure Feed