+24 -28

Documentation/RCU/Design/Requirements/Requirements.rst

reviewed

··· 1973 1973 Note that grace period initialization (rcu_gp_init()) must carefully sequence 1974 1974 CPU hotplug scanning with grace period state changes. For example, the 1975 1975 following race could occur in rcu_gp_init() if rcu_seq_start() were to happen 1976 1976 - after the CPU hotplug scanning. 1977 1977 - 1978 1978 - .. code-block:: none 1976 1976 + after the CPU hotplug scanning:: 1979 1977 1980 1978 CPU0 (rcu_gp_init) CPU1 CPU2 1981 1979 --------------------- ---- ---- ··· 2006 2008 kfree(r1); 2007 2009 r2 = *r0; // USE-AFTER-FREE! 2008 2010 2009 2009 - By incrementing gp_seq first, CPU1's RCU read-side critical section 2011 2011 + By incrementing ``gp_seq`` first, CPU1's RCU read-side critical section 2010 2012 is guaranteed to not be missed by CPU2. 2011 2013 2012 2012 - **Concurrent Quiescent State Reporting for Offline CPUs** 2014 2014 + Concurrent Quiescent State Reporting for Offline CPUs 2015 2015 + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2013 2016 2014 2017 RCU must ensure that CPUs going offline report quiescent states to avoid 2015 2018 blocking grace periods. This requires careful synchronization to handle 2016 2019 race conditions 2017 2020 2018 2018 - **Race condition causing Offline CPU to hang GP** 2021 2021 + Race condition causing Offline CPU to hang GP 2022 2022 + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2019 2023 2020 2020 - A race between CPU offlining and new GP initialization (gp_init) may occur 2021 2021 - because `rcu_report_qs_rnp()` in `rcutree_report_cpu_dead()` must temporarily 2022 2022 - release the `rcu_node` lock to wake the RCU grace-period kthread: 2023 2023 - 2024 2024 - .. code-block:: none 2024 2024 + A race between CPU offlining and new GP initialization (gp_init()) may occur 2025 2025 + because rcu_report_qs_rnp() in rcutree_report_cpu_dead() must temporarily 2026 2026 + release the ``rcu_node`` lock to wake the RCU grace-period kthread:: 2025 2027 2026 2028 CPU1 (going offline) CPU0 (GP kthread) 2027 2029 -------------------- ----------------- ··· 2042 2044 // Reacquire lock (but too late) 2043 2045 rnp->qsmaskinitnext &= ~mask // Finally clears bit 2044 2046 2045 2045 - Without `ofl_lock`, the new grace period includes the offline CPU and waits 2047 2047 + Without ``ofl_lock``, the new grace period includes the offline CPU and waits 2046 2048 forever for its quiescent state causing a GP hang. 2047 2049 2048 2048 - **A solution with ofl_lock** 2050 2050 + A solution with ofl_lock 2051 2051 + ^^^^^^^^^^^^^^^^^^^^^^^^ 2049 2052 2050 2050 - The `ofl_lock` (offline lock) prevents `rcu_gp_init()` from running during 2051 2051 - the vulnerable window when `rcu_report_qs_rnp()` has released `rnp->lock`: 2052 2052 - 2053 2053 - .. code-block:: none 2053 2053 + The ``ofl_lock`` (offline lock) prevents rcu_gp_init() from running during 2054 2054 + the vulnerable window when rcu_report_qs_rnp() has released ``rnp->lock``:: 2054 2055 2055 2056 CPU0 (rcu_gp_init) CPU1 (rcutree_report_cpu_dead) 2056 2057 ------------------ ------------------------------ ··· 2062 2065 arch_spin_unlock(&ofl_lock) ---> // Now CPU1 can proceed 2063 2066 } // But snapshot already taken 2064 2067 2065 2065 - **Another race causing GP hangs in rcu_gpu_init(): Reporting QS for Now-offline CPUs** 2068 2068 + Another race causing GP hangs in rcu_gpu_init(): Reporting QS for Now-offline CPUs 2069 2069 + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2066 2070 2067 2071 After the first loop takes an atomic snapshot of online CPUs, as shown above, 2068 2068 - the second loop in `rcu_gp_init()` detects CPUs that went offline between 2069 2069 - releasing `ofl_lock` and acquiring the per-node `rnp->lock`. This detection is 2070 2070 - crucial because: 2072 2072 + the second loop in rcu_gp_init() detects CPUs that went offline between 2073 2073 + releasing ``ofl_lock`` and acquiring the per-node ``rnp->lock``. 2074 2074 + This detection is crucial because: 2071 2075 2072 2076 1. The CPU might have gone offline after the snapshot but before the second loop 2073 2077 2. The offline CPU cannot report its own QS if it's already dead 2074 2078 3. Without this detection, the grace period would wait forever for CPUs that 2075 2079 are now offline. 2076 2080 2077 2077 - The second loop performs this detection safely: 2078 2078 - 2079 2079 - .. code-block:: none 2081 2081 + The second loop performs this detection safely:: 2080 2082 2081 2083 rcu_for_each_node_breadth_first(rnp) { 2082 2084 raw_spin_lock_irqsave_rcu_node(rnp, flags); ··· 2089 2093 } 2090 2094 2091 2095 This approach ensures atomicity: quiescent state reporting for offline CPUs 2092 2092 - happens either in `rcu_gp_init()` (second loop) or in `rcutree_report_cpu_dead()`, 2093 2093 - never both and never neither. The `rnp->lock` held throughout the sequence 2094 2094 - prevents races - `rcutree_report_cpu_dead()` also acquires this lock when 2095 2095 - clearing `qsmaskinitnext`, ensuring mutual exclusion. 2096 2096 + happens either in rcu_gp_init() (second loop) or in rcutree_report_cpu_dead(), 2097 2097 + never both and never neither. The ``rnp->lock`` held throughout the sequence 2098 2098 + prevents races - rcutree_report_cpu_dead() also acquires this lock when 2099 2099 + clearing ``qsmaskinitnext``, ensuring mutual exclusion. 2096 2100 2097 2101 Scheduler and RCU 2098 2102 ~~~~~~~~~~~~~~~~~

+3 -3

Documentation/RCU/RTFP.txt

reviewed

··· 641 641 ,Month="July" 642 642 ,Year="2001" 643 643 ,note="Available: 644 644 - \url{http://www.linuxsymposium.org/2001/abstracts/readcopy.php} 644 644 + \url{https://kernel.org/doc/ols/2001/read-copy.pdf} 645 645 \url{http://www.rdrop.com/users/paulmck/RCU/rclock_OLS.2001.05.01c.pdf} 646 646 [Viewed June 23, 2004]" 647 647 ,annotation={ ··· 1480 1480 ,Year="2006" 1481 1481 ,pages="v2 123-138" 1482 1482 ,note="Available: 1483 1483 - \url{http://www.linuxsymposium.org/2006/view_abstract.php?content_key=184} 1483 1483 + \url{https://kernel.org/doc/ols/2006/ols2006v2-pages-131-146.pdf} 1484 1484 \url{http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf} 1485 1485 [Viewed January 1, 2007]" 1486 1486 ,annotation={ ··· 1511 1511 ,Year="2006" 1512 1512 ,pages="v2 249-254" 1513 1513 ,note="Available: 1514 1514 - \url{http://www.linuxsymposium.org/2006/view_abstract.php?content_key=184} 1514 1514 + \url{https://kernel.org/doc/ols/2006/ols2006v2-pages-249-262.pdf} 1515 1515 [Viewed January 11, 2009]" 1516 1516 ,annotation={ 1517 1517 Uses RCU-protected radix tree for a lockless page cache.

+19 -8

Documentation/RCU/checklist.rst

reviewed

··· 69 69 Explicit disabling of preemption (preempt_disable(), for example) 70 70 can serve as rcu_read_lock_sched(), but is less readable and 71 71 prevents lockdep from detecting locking issues. Acquiring a 72 72 - spinlock also enters an RCU read-side critical section. 72 72 + raw spinlock also enters an RCU read-side critical section. 73 73 + 74 74 + The guard(rcu)() and scoped_guard(rcu) primitives designate 75 75 + the remainder of the current scope or the next statement, 76 76 + respectively, as the RCU read-side critical section. Use of 77 77 + these guards can be less error-prone than rcu_read_lock(), 78 78 + rcu_read_unlock(), and friends. 73 79 74 80 Please note that you *cannot* rely on code known to be built 75 81 only in non-preemptible kernels. Such code can and will break, ··· 411 405 13. Unlike most flavors of RCU, it *is* permissible to block in an 412 406 SRCU read-side critical section (demarked by srcu_read_lock() 413 407 and srcu_read_unlock()), hence the "SRCU": "sleepable RCU". 414 414 - Please note that if you don't need to sleep in read-side critical 415 415 - sections, you should be using RCU rather than SRCU, because RCU 416 416 - is almost always faster and easier to use than is SRCU. 408 408 + As with RCU, guard(srcu)() and scoped_guard(srcu) forms are 409 409 + available, and often provide greater ease of use. Please note 410 410 + that if you don't need to sleep in read-side critical sections, 411 411 + you should be using RCU rather than SRCU, because RCU is almost 412 412 + always faster and easier to use than is SRCU. 417 413 418 414 Also unlike other forms of RCU, explicit initialization and 419 415 cleanup is required either at build time via DEFINE_SRCU() ··· 451 443 real-time workloads than is synchronize_rcu_expedited(). 452 444 453 445 It is also permissible to sleep in RCU Tasks Trace read-side 454 454 - critical section, which are delimited by rcu_read_lock_trace() and 455 455 - rcu_read_unlock_trace(). However, this is a specialized flavor 456 456 - of RCU, and you should not use it without first checking with 457 457 - its current users. In most cases, you should instead use SRCU. 446 446 + critical section, which are delimited by rcu_read_lock_trace() 447 447 + and rcu_read_unlock_trace(). However, this is a specialized 448 448 + flavor of RCU, and you should not use it without first checking 449 449 + with its current users. In most cases, you should instead 450 450 + use SRCU. As with RCU and SRCU, guard(rcu_tasks_trace)() and 451 451 + scoped_guard(rcu_tasks_trace) are available, and often provide 452 452 + greater ease of use. 458 453 459 454 Note that rcu_assign_pointer() relates to SRCU just as it does to 460 455 other forms of RCU, but instead of rcu_dereference() you should

+3 -3

Documentation/RCU/index.rst

reviewed

··· 1 1 .. SPDX-License-Identifier: GPL-2.0 2 2 3 3 - .. _rcu_concepts: 3 3 + .. _rcu_handbook: 4 4 5 5 ============ 6 6 - RCU concepts 6 6 + RCU Handbook 7 7 ============ 8 8 9 9 .. toctree:: 10 10 - :maxdepth: 3 10 10 + :maxdepth: 2 11 11 12 12 checklist 13 13 lockdep

+2 -2

Documentation/RCU/torture.rst

reviewed

··· 344 344 345 345 And this is why the kvm-remote.sh script exists. 346 346 347 347 - If you the following command works:: 347 347 + If the following command works:: 348 348 349 349 ssh system0 date 350 350 ··· 364 364 The kvm.sh ``--dryrun scenarios`` argument is useful for working out 365 365 how many scenarios may be run in one batch across a group of systems. 366 366 367 367 - You can also re-run a previous remote run in a manner similar to kvm.sh: 367 367 + You can also re-run a previous remote run in a manner similar to kvm.sh:: 368 368 369 369 kvm-remote.sh "system0 system1 system2 system3 system4 system5" \ 370 370 tools/testing/selftests/rcutorture/res/2022.11.03-11.26.28-remote \

+116 -30

Documentation/RCU/whatisRCU.rst

reviewed

··· 1021 1021 list_entry_rcu 1022 1022 list_entry_lockless 1023 1023 list_first_entry_rcu 1024 1024 + list_first_or_null_rcu 1025 1025 + list_tail_rcu 1024 1026 list_next_rcu 1027 1027 + list_next_or_null_rcu 1025 1028 list_for_each_entry_rcu 1026 1029 list_for_each_entry_continue_rcu 1027 1030 list_for_each_entry_from_rcu 1028 1028 - list_first_or_null_rcu 1029 1029 - list_next_or_null_rcu 1031 1031 + list_for_each_entry_lockless 1030 1032 hlist_first_rcu 1031 1033 hlist_next_rcu 1032 1034 hlist_pprev_rcu 1033 1035 hlist_for_each_entry_rcu 1036 1036 + hlist_for_each_entry_rcu_notrace 1034 1037 hlist_for_each_entry_rcu_bh 1035 1038 hlist_for_each_entry_from_rcu 1036 1039 hlist_for_each_entry_continue_rcu 1037 1040 hlist_for_each_entry_continue_rcu_bh 1038 1041 hlist_nulls_first_rcu 1042 1042 + hlist_nulls_next_rcu 1039 1043 hlist_nulls_for_each_entry_rcu 1044 1044 + hlist_nulls_for_each_entry_safe 1040 1045 hlist_bl_first_rcu 1041 1046 hlist_bl_for_each_entry_rcu 1042 1047 1043 1048 RCU pointer/list update:: 1044 1049 1045 1050 rcu_assign_pointer 1051 1051 + rcu_replace_pointer 1052 1052 + INIT_LIST_HEAD_RCU 1046 1053 list_add_rcu 1047 1054 list_add_tail_rcu 1048 1055 list_del_rcu 1049 1056 list_replace_rcu 1057 1057 + list_splice_init_rcu 1058 1058 + list_splice_tail_init_rcu 1050 1059 hlist_add_behind_rcu 1051 1060 hlist_add_before_rcu 1052 1061 hlist_add_head_rcu ··· 1063 1054 hlist_del_rcu 1064 1055 hlist_del_init_rcu 1065 1056 hlist_replace_rcu 1066 1066 - list_splice_init_rcu 1067 1067 - list_splice_tail_init_rcu 1068 1057 hlist_nulls_del_init_rcu 1069 1058 hlist_nulls_del_rcu 1070 1059 hlist_nulls_add_head_rcu 1060 1060 + hlist_nulls_add_tail_rcu 1061 1061 + hlist_nulls_add_fake 1062 1062 + hlists_swap_heads_rcu 1071 1063 hlist_bl_add_head_rcu 1072 1072 - hlist_bl_del_init_rcu 1073 1064 hlist_bl_del_rcu 1074 1065 hlist_bl_set_first_rcu 1075 1066 1076 1067 RCU:: 1077 1068 1078 1078 - Critical sections Grace period Barrier 1069 1069 + Critical sections Grace period Barrier 1079 1070 1080 1080 - rcu_read_lock synchronize_net rcu_barrier 1081 1081 - rcu_read_unlock synchronize_rcu 1082 1082 - rcu_dereference synchronize_rcu_expedited 1083 1083 - rcu_read_lock_held call_rcu 1084 1084 - rcu_dereference_check kfree_rcu 1085 1085 - rcu_dereference_protected 1071 1071 + rcu_read_lock synchronize_net rcu_barrier 1072 1072 + rcu_read_unlock synchronize_rcu 1073 1073 + guard(rcu)() synchronize_rcu_expedited 1074 1074 + scoped_guard(rcu) synchronize_rcu_mult 1075 1075 + rcu_dereference call_rcu 1076 1076 + rcu_dereference_check call_rcu_hurry 1077 1077 + rcu_dereference_protected kfree_rcu 1078 1078 + rcu_read_lock_held kvfree_rcu 1079 1079 + rcu_read_lock_any_held kfree_rcu_mightsleep 1080 1080 + rcu_pointer_handoff cond_synchronize_rcu 1081 1081 + unrcu_pointer cond_synchronize_rcu_full 1082 1082 + cond_synchronize_rcu_expedited 1083 1083 + cond_synchronize_rcu_expedited_full 1084 1084 + get_completed_synchronize_rcu 1085 1085 + get_completed_synchronize_rcu_full 1086 1086 + get_state_synchronize_rcu 1087 1087 + get_state_synchronize_rcu_full 1088 1088 + poll_state_synchronize_rcu 1089 1089 + poll_state_synchronize_rcu_full 1090 1090 + same_state_synchronize_rcu 1091 1091 + same_state_synchronize_rcu_full 1092 1092 + start_poll_synchronize_rcu 1093 1093 + start_poll_synchronize_rcu_full 1094 1094 + start_poll_synchronize_rcu_expedited 1095 1095 + start_poll_synchronize_rcu_expedited_full 1086 1096 1087 1097 bh:: 1088 1098 1089 1099 Critical sections Grace period Barrier 1090 1100 1091 1091 - rcu_read_lock_bh call_rcu rcu_barrier 1092 1092 - rcu_read_unlock_bh synchronize_rcu 1093 1093 - [local_bh_disable] synchronize_rcu_expedited 1101 1101 + rcu_read_lock_bh [Same as RCU] [Same as RCU] 1102 1102 + rcu_read_unlock_bh 1103 1103 + [local_bh_disable] 1094 1104 [and friends] 1095 1105 rcu_dereference_bh 1096 1106 rcu_dereference_bh_check ··· 1120 1092 1121 1093 Critical sections Grace period Barrier 1122 1094 1123 1123 - rcu_read_lock_sched call_rcu rcu_barrier 1124 1124 - rcu_read_unlock_sched synchronize_rcu 1125 1125 - [preempt_disable] synchronize_rcu_expedited 1095 1095 + rcu_read_lock_sched [Same as RCU] [Same as RCU] 1096 1096 + rcu_read_unlock_sched 1097 1097 + [preempt_disable] 1126 1098 [and friends] 1127 1099 rcu_read_lock_sched_notrace 1128 1100 rcu_read_unlock_sched_notrace ··· 1132 1104 rcu_read_lock_sched_held 1133 1105 1134 1106 1107 1107 + RCU: Initialization/cleanup/ordering:: 1108 1108 + 1109 1109 + RCU_INIT_POINTER 1110 1110 + RCU_INITIALIZER 1111 1111 + RCU_POINTER_INITIALIZER 1112 1112 + init_rcu_head 1113 1113 + destroy_rcu_head 1114 1114 + init_rcu_head_on_stack 1115 1115 + destroy_rcu_head_on_stack 1116 1116 + SLAB_TYPESAFE_BY_RCU 1117 1117 + 1118 1118 + 1119 1119 + RCU: Quiescents states and control:: 1120 1120 + 1121 1121 + cond_resched_tasks_rcu_qs 1122 1122 + rcu_all_qs 1123 1123 + rcu_softirq_qs_periodic 1124 1124 + rcu_end_inkernel_boot 1125 1125 + rcu_expedite_gp 1126 1126 + rcu_gp_is_expedited 1127 1127 + rcu_unexpedite_gp 1128 1128 + rcu_cpu_stall_reset 1129 1129 + rcu_head_after_call_rcu 1130 1130 + rcu_is_watching 1131 1131 + 1132 1132 + 1133 1133 + RCU-sync primitive:: 1134 1134 + 1135 1135 + rcu_sync_is_idle 1136 1136 + rcu_sync_init 1137 1137 + rcu_sync_enter 1138 1138 + rcu_sync_exit 1139 1139 + rcu_sync_dtor 1140 1140 + 1141 1141 + 1135 1142 RCU-Tasks:: 1136 1143 1137 1137 - Critical sections Grace period Barrier 1144 1144 + Critical sections Grace period Barrier 1138 1145 1139 1139 - N/A call_rcu_tasks rcu_barrier_tasks 1146 1146 + N/A call_rcu_tasks rcu_barrier_tasks 1140 1147 synchronize_rcu_tasks 1141 1148 1142 1149 1143 1150 RCU-Tasks-Rude:: 1144 1151 1145 1145 - Critical sections Grace period Barrier 1152 1152 + Critical sections Grace period Barrier 1146 1153 1147 1147 - N/A N/A 1148 1148 - synchronize_rcu_tasks_rude 1154 1154 + N/A synchronize_rcu_tasks_rude rcu_barrier_tasks_rude 1155 1155 + call_rcu_tasks_rude 1149 1156 1150 1157 1151 1158 RCU-Tasks-Trace:: 1152 1159 1153 1153 - Critical sections Grace period Barrier 1160 1160 + Critical sections Grace period Barrier 1154 1161 1155 1155 - rcu_read_lock_trace call_rcu_tasks_trace rcu_barrier_tasks_trace 1162 1162 + rcu_read_lock_trace call_rcu_tasks_trace rcu_barrier_tasks_trace 1156 1163 rcu_read_unlock_trace synchronize_rcu_tasks_trace 1164 1164 + guard(rcu_tasks_trace)() 1165 1165 + scoped_guard(rcu_tasks_trace) 1166 1166 + 1167 1167 + 1168 1168 + SRCU list traversal:: 1169 1169 + list_for_each_entry_srcu 1170 1170 + hlist_for_each_entry_srcu 1157 1171 1158 1172 1159 1173 SRCU:: 1160 1174 1161 1161 - Critical sections Grace period Barrier 1175 1175 + Critical sections Grace period Barrier 1162 1176 1163 1163 - srcu_read_lock call_srcu srcu_barrier 1164 1164 - srcu_read_unlock synchronize_srcu 1165 1165 - srcu_dereference synchronize_srcu_expedited 1177 1177 + srcu_read_lock call_srcu srcu_barrier 1178 1178 + srcu_read_unlock synchronize_srcu 1179 1179 + srcu_read_lock_fast synchronize_srcu_expedited 1180 1180 + srcu_read_unlock_fast get_state_synchronize_srcu 1181 1181 + srcu_read_lock_nmisafe start_poll_synchronize_srcu 1182 1182 + srcu_read_unlock_nmisafe start_poll_synchronize_srcu_expedited 1183 1183 + srcu_read_lock_notrace poll_state_synchronize_srcu 1184 1184 + srcu_read_unlock_notrace 1185 1185 + srcu_down_read 1186 1186 + srcu_up_read 1187 1187 + srcu_down_read_fast 1188 1188 + srcu_up_read_fast 1189 1189 + guard(srcu)() 1190 1190 + scoped_guard(srcu) 1191 1191 + srcu_read_lock_held 1192 1192 + srcu_dereference 1166 1193 srcu_dereference_check 1194 1194 + srcu_dereference_notrace 1167 1195 srcu_read_lock_held 1168 1196 1169 1169 - SRCU: Initialization/cleanup:: 1197 1197 + 1198 1198 + SRCU: Initialization/cleanup/ordering:: 1170 1199 1171 1200 DEFINE_SRCU 1172 1201 DEFINE_STATIC_SRCU 1173 1202 init_srcu_struct 1174 1203 cleanup_srcu_struct 1204 1204 + smp_mb__after_srcu_read_unlock 1175 1205 1176 1206 All: lockdep-checked RCU utility APIs:: 1177 1207