Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

+20 -1

Documentation/RCU/rcuref.txt

··· 12 12 Reference counting on elements of lists which are protected by traditional 13 13 reader/writer spinlocks or semaphores are straightforward: 14 14 15 + CODE LISTING A: 15 16 1. 2. 16 17 add() search_and_reference() 17 18 { { ··· 29 28 release_referenced() delete() 30 29 { { 31 30 ... write_lock(&list_lock); 32 - atomic_dec(&el->rc, relfunc) ... 31 + if(atomic_dec_and_test(&el->rc)) ... 32 + kfree(el); 33 33 ... remove_element 34 34 } write_unlock(&list_lock); 35 35 ... ··· 46 44 has already been deleted from the list/array. Use atomic_inc_not_zero() 47 45 in this scenario as follows: 48 46 47 + CODE LISTING B: 49 48 1. 2. 50 49 add() search_and_reference() 51 50 { { ··· 82 79 atomic_dec_and_test() may be moved from delete() to el_free() 83 80 as follows: 84 81 82 + CODE LISTING C: 85 83 1. 2. 86 84 add() search_and_reference() 87 85 { { ··· 118 114 any reader finds the element, that reader may safely acquire a reference 119 115 without checking the value of the reference counter. 120 116 117 + A clear advantage of the RCU-based pattern in listing C over the one 118 + in listing B is that any call to search_and_reference() that locates 119 + a given object will succeed in obtaining a reference to that object, 120 + even given a concurrent invocation of delete() for that same object. 121 + Similarly, a clear advantage of both listings B and C over listing A is 122 + that a call to delete() is not delayed even if there are an arbitrarily 123 + large number of calls to search_and_reference() searching for the same 124 + object that delete() was invoked on. Instead, all that is delayed is 125 + the eventual invocation of kfree(), which is usually not a problem on 126 + modern computer systems, even the small ones. 127 + 121 128 In cases where delete() can sleep, synchronize_rcu() can be called from 122 129 delete(), so that el_free() can be subsumed into delete as follows: 123 130 ··· 145 130 kfree(el); 146 131 ... 147 132 } 133 + 134 + As additional examples in the kernel, the pattern in listing C is used by 135 + reference counting of struct pid, while the pattern in listing B is used by 136 + struct posix_acl.

+1 -1

Documentation/RCU/stallwarn.txt

··· 153 153 This boot/sysfs parameter controls the RCU-tasks stall warning 154 154 interval. A value of zero or less suppresses RCU-tasks stall 155 155 warnings. A positive value sets the stall-warning interval 156 - in jiffies. An RCU-tasks stall warning starts with the line: 156 + in seconds. An RCU-tasks stall warning starts with the line: 157 157 158 158 INFO: rcu_tasks detected stalls on tasks: 159 159

+4 -4

Documentation/RCU/whatisRCU.txt

··· 212 212 213 213 rcu_assign_pointer() 214 214 215 - typeof(p) rcu_assign_pointer(p, typeof(p) v); 215 + void rcu_assign_pointer(p, typeof(p) v); 216 216 217 217 Yes, rcu_assign_pointer() -is- implemented as a macro, though it 218 218 would be cool to be able to declare a function in this manner. ··· 220 220 221 221 The updater uses this function to assign a new value to an 222 222 RCU-protected pointer, in order to safely communicate the change 223 - in value from the updater to the reader. This function returns 224 - the new value, and also executes any memory-barrier instructions 225 - required for a given CPU architecture. 223 + in value from the updater to the reader. This macro does not 224 + evaluate to an rvalue, but it does execute any memory-barrier 225 + instructions required for a given CPU architecture. 226 226 227 227 Perhaps just as important, it serves to document (1) which 228 228 pointers are protected by RCU and (2) the point at which a

+6

Documentation/admin-guide/kernel-parameters.txt

··· 3752 3752 the propagation of recent CPU-hotplug changes up 3753 3753 the rcu_node combining tree. 3754 3754 3755 + rcutree.use_softirq= [KNL] 3756 + If set to zero, move all RCU_SOFTIRQ processing to 3757 + per-CPU rcuc kthreads. Defaults to a non-zero 3758 + value, meaning that RCU_SOFTIRQ is used by default. 3759 + Specify rcutree.use_softirq=0 to use rcuc kthreads. 3760 + 3755 3761 rcutree.rcu_fanout_exact= [KNL] 3756 3762 Disable autobalancing of the rcu_node combining 3757 3763 tree. This is used by rcutorture, and might

+13 -4

Documentation/atomic_t.txt

··· 187 187 188 188 smp_mb__{before,after}_atomic() 189 189 190 - only apply to the RMW ops and can be used to augment/upgrade the ordering 191 - inherent to the used atomic op. These barriers provide a full smp_mb(). 190 + only apply to the RMW atomic ops and can be used to augment/upgrade the 191 + ordering inherent to the op. These barriers act almost like a full smp_mb(): 192 + smp_mb__before_atomic() orders all earlier accesses against the RMW op 193 + itself and all accesses following it, and smp_mb__after_atomic() orders all 194 + later accesses against the RMW op and all accesses preceding it. However, 195 + accesses between the smp_mb__{before,after}_atomic() and the RMW op are not 196 + ordered, so it is advisable to place the barrier right next to the RMW atomic 197 + op whenever possible. 192 198 193 199 These helper barriers exist because architectures have varying implicit 194 200 ordering on their SMP atomic primitives. For example our TSO architectures ··· 218 212 atomic_dec(&X); 219 213 220 214 is a 'typical' RELEASE pattern, the barrier is strictly stronger than 221 - a RELEASE. Similarly for something like: 215 + a RELEASE because it orders preceding instructions against both the read 216 + and write parts of the atomic_dec(), and against all following instructions 217 + as well. Similarly, something like: 222 218 223 219 atomic_inc(&X); 224 220 smp_mb__after_atomic(); ··· 252 244 253 245 This should not happen; but a hypothetical atomic_inc_acquire() -- 254 246 (void)atomic_fetch_inc_acquire() for instance -- would allow the outcome, 255 - since then: 247 + because it would not order the W part of the RMW against the following 248 + WRITE_ONCE. Thus: 256 249 257 250 P1 P2 258 251

+1 -1

Documentation/core-api/circular-buffers.rst

··· 3 3 ================ 4 4 5 5 :Author: David Howells <dhowells@redhat.com> 6 - :Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com> 6 + :Author: Paul E. McKenney <paulmck@linux.ibm.com> 7 7 8 8 9 9 Linux provides a number of features that can be used to implement circular

+1 -1

Documentation/memory-barriers.txt

··· 3 3 ============================ 4 4 5 5 By: David Howells <dhowells@redhat.com> 6 - Paul E. McKenney <paulmck@linux.vnet.ibm.com> 6 + Paul E. McKenney <paulmck@linux.ibm.com> 7 7 Will Deacon <will.deacon@arm.com> 8 8 Peter Zijlstra <peterz@infradead.org> 9 9

+1 -1

Documentation/translations/ko_KR/memory-barriers.txt

··· 24 24 ========================= 25 25 26 26 저자: David Howells <dhowells@redhat.com> 27 - Paul E. McKenney <paulmck@linux.vnet.ibm.com> 27 + Paul E. McKenney <paulmck@linux.ibm.com> 28 28 Will Deacon <will.deacon@arm.com> 29 29 Peter Zijlstra <peterz@infradead.org> 30 30

+7

include/linux/lockdep.h

··· 632 632 "IRQs not disabled as expected\n"); \ 633 633 } while (0) 634 634 635 + #define lockdep_assert_in_irq() do { \ 636 + WARN_ONCE(debug_locks && !current->lockdep_recursion && \ 637 + !current->hardirq_context, \ 638 + "Not in hardirq as expected\n"); \ 639 + } while (0) 640 + 635 641 #else 636 642 # define might_lock(lock) do { } while (0) 637 643 # define might_lock_read(lock) do { } while (0) 638 644 # define lockdep_assert_irqs_enabled() do { } while (0) 639 645 # define lockdep_assert_irqs_disabled() do { } while (0) 646 + # define lockdep_assert_in_irq() do { } while (0) 640 647 #endif 641 648 642 649 #ifdef CONFIG_LOCKDEP

+5

include/linux/module.h

··· 21 21 #include <linux/rbtree_latch.h> 22 22 #include <linux/error-injection.h> 23 23 #include <linux/tracepoint-defs.h> 24 + #include <linux/srcu.h> 24 25 25 26 #include <linux/percpu.h> 26 27 #include <asm/module.h> ··· 450 449 #ifdef CONFIG_TRACEPOINTS 451 450 unsigned int num_tracepoints; 452 451 tracepoint_ptr_t *tracepoints_ptrs; 452 + #endif 453 + #ifdef CONFIG_TREE_SRCU 454 + unsigned int num_srcu_structs; 455 + struct srcu_struct **srcu_struct_ptrs; 453 456 #endif 454 457 #ifdef CONFIG_BPF_EVENTS 455 458 unsigned int num_bpf_raw_events;

+7 -3

include/linux/percpu-rwsem.h

··· 17 17 int readers_block; 18 18 }; 19 19 20 - #define DEFINE_STATIC_PERCPU_RWSEM(name) \ 20 + #define __DEFINE_PERCPU_RWSEM(name, is_static) \ 21 21 static DEFINE_PER_CPU(unsigned int, __percpu_rwsem_rc_##name); \ 22 - static struct percpu_rw_semaphore name = { \ 23 - .rss = __RCU_SYNC_INITIALIZER(name.rss, RCU_SCHED_SYNC), \ 22 + is_static struct percpu_rw_semaphore name = { \ 23 + .rss = __RCU_SYNC_INITIALIZER(name.rss), \ 24 24 .read_count = &__percpu_rwsem_rc_##name, \ 25 25 .rw_sem = __RWSEM_INITIALIZER(name.rw_sem), \ 26 26 .writer = __RCUWAIT_INITIALIZER(name.writer), \ 27 27 } 28 + #define DEFINE_PERCPU_RWSEM(name) \ 29 + __DEFINE_PERCPU_RWSEM(name, /* not static */) 30 + #define DEFINE_STATIC_PERCPU_RWSEM(name) \ 31 + __DEFINE_PERCPU_RWSEM(name, static) 28 32 29 33 extern int __percpu_down_read(struct percpu_rw_semaphore *, int); 30 34 extern void __percpu_up_read(struct percpu_rw_semaphore *);

+11 -29

include/linux/rcu_sync.h

··· 13 13 #include <linux/wait.h> 14 14 #include <linux/rcupdate.h> 15 15 16 - enum rcu_sync_type { RCU_SYNC, RCU_SCHED_SYNC, RCU_BH_SYNC }; 17 - 18 16 /* Structure to mediate between updaters and fastpath-using readers. */ 19 17 struct rcu_sync { 20 18 int gp_state; 21 19 int gp_count; 22 20 wait_queue_head_t gp_wait; 23 21 24 - int cb_state; 25 22 struct rcu_head cb_head; 26 - 27 - enum rcu_sync_type gp_type; 28 23 }; 29 - 30 - extern void rcu_sync_lockdep_assert(struct rcu_sync *); 31 24 32 25 /** 33 26 * rcu_sync_is_idle() - Are readers permitted to use their fastpaths? 34 27 * @rsp: Pointer to rcu_sync structure to use for synchronization 35 28 * 36 - * Returns true if readers are permitted to use their fastpaths. 37 - * Must be invoked within an RCU read-side critical section whose 38 - * flavor matches that of the rcu_sync struture. 29 + * Returns true if readers are permitted to use their fastpaths. Must be 30 + * invoked within some flavor of RCU read-side critical section. 39 31 */ 40 32 static inline bool rcu_sync_is_idle(struct rcu_sync *rsp) 41 33 { 42 - #ifdef CONFIG_PROVE_RCU 43 - rcu_sync_lockdep_assert(rsp); 44 - #endif 45 - return !rsp->gp_state; /* GP_IDLE */ 34 + RCU_LOCKDEP_WARN(!rcu_read_lock_held() && 35 + !rcu_read_lock_bh_held() && 36 + !rcu_read_lock_sched_held(), 37 + "suspicious rcu_sync_is_idle() usage"); 38 + return !READ_ONCE(rsp->gp_state); /* GP_IDLE */ 46 39 } 47 40 48 - extern void rcu_sync_init(struct rcu_sync *, enum rcu_sync_type); 41 + extern void rcu_sync_init(struct rcu_sync *); 49 42 extern void rcu_sync_enter_start(struct rcu_sync *); 50 43 extern void rcu_sync_enter(struct rcu_sync *); 51 44 extern void rcu_sync_exit(struct rcu_sync *); 52 45 extern void rcu_sync_dtor(struct rcu_sync *); 53 46 54 - #define __RCU_SYNC_INITIALIZER(name, type) { \ 47 + #define __RCU_SYNC_INITIALIZER(name) { \ 55 48 .gp_state = 0, \ 56 49 .gp_count = 0, \ 57 50 .gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait), \ 58 - .cb_state = 0, \ 59 - .gp_type = type, \ 60 51 } 61 52 62 - #define __DEFINE_RCU_SYNC(name, type) \ 63 - struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type) 64 - 65 - #define DEFINE_RCU_SYNC(name) \ 66 - __DEFINE_RCU_SYNC(name, RCU_SYNC) 67 - 68 - #define DEFINE_RCU_SCHED_SYNC(name) \ 69 - __DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC) 70 - 71 - #define DEFINE_RCU_BH_SYNC(name) \ 72 - __DEFINE_RCU_SYNC(name, RCU_BH_SYNC) 53 + #define DEFINE_RCU_SYNC(name) \ 54 + struct rcu_sync name = __RCU_SYNC_INITIALIZER(name) 73 55 74 56 #endif /* _LINUX_RCU_SYNC_H_ */

+12 -9

include/linux/rcupdate.h

··· 365 365 * other macros that it invokes. 366 366 */ 367 367 #define rcu_assign_pointer(p, v) \ 368 - ({ \ 368 + do { \ 369 369 uintptr_t _r_a_p__v = (uintptr_t)(v); \ 370 - rcu_check_sparse(p, __rcu); \ 370 + rcu_check_sparse(p, __rcu); \ 371 371 \ 372 372 if (__builtin_constant_p(v) && (_r_a_p__v) == (uintptr_t)NULL) \ 373 373 WRITE_ONCE((p), (typeof(p))(_r_a_p__v)); \ 374 374 else \ 375 375 smp_store_release(&p, RCU_INITIALIZER((typeof(p))_r_a_p__v)); \ 376 - _r_a_p__v; \ 377 - }) 376 + } while (0) 378 377 379 378 /** 380 379 * rcu_swap_protected() - swap an RCU and a regular pointer ··· 585 586 * read-side critical sections may be preempted and they may also block, but 586 587 * only when acquiring spinlocks that are subject to priority inheritance. 587 588 */ 588 - static inline void rcu_read_lock(void) 589 + static __always_inline void rcu_read_lock(void) 589 590 { 590 591 __rcu_read_lock(); 591 592 __acquire(RCU); ··· 802 803 /** 803 804 * kfree_rcu() - kfree an object after a grace period. 804 805 * @ptr: pointer to kfree 805 - * @rcu_head: the name of the struct rcu_head within the type of @ptr. 806 + * @rhf: the name of the struct rcu_head within the type of @ptr. 806 807 * 807 808 * Many rcu callbacks functions just call kfree() on the base structure. 808 809 * These functions are trivial, but their size adds up, and furthermore ··· 825 826 * The BUILD_BUG_ON check must not involve any function calls, hence the 826 827 * checks are done in macros here. 827 828 */ 828 - #define kfree_rcu(ptr, rcu_head) \ 829 - __kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head)) 830 - 829 + #define kfree_rcu(ptr, rhf) \ 830 + do { \ 831 + typeof (ptr) ___p = (ptr); \ 832 + \ 833 + if (___p) \ 834 + __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ 835 + } while (0) 831 836 832 837 /* 833 838 * Place this after a lock-acquisition primitive to guarantee that

+1 -1

include/linux/sched.h

··· 565 565 u8 blocked; 566 566 u8 need_qs; 567 567 u8 exp_hint; /* Hint for performance. */ 568 - u8 pad; /* No garbage from compiler! */ 568 + u8 deferred_qs; 569 569 } b; /* Bits. */ 570 570 u32 s; /* Set of bits. */ 571 571 };

+11 -3

include/linux/srcutree.h

··· 120 120 * 121 121 * See include/linux/percpu-defs.h for the rules on per-CPU variables. 122 122 */ 123 - #define __DEFINE_SRCU(name, is_static) \ 124 - static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data);\ 125 - is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name##_srcu_data) 123 + #ifdef MODULE 124 + # define __DEFINE_SRCU(name, is_static) \ 125 + is_static struct srcu_struct name; \ 126 + struct srcu_struct * const __srcu_struct_##name \ 127 + __section("___srcu_struct_ptrs") = &name 128 + #else 129 + # define __DEFINE_SRCU(name, is_static) \ 130 + static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ 131 + is_static struct srcu_struct name = \ 132 + __SRCU_STRUCT_INIT(name, name##_srcu_data) 133 + #endif 126 134 #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) 127 135 #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) 128 136

+1 -1

include/linux/torture.h

··· 66 66 67 67 /* Task stuttering, which forces load/no-load transitions. */ 68 68 bool stutter_wait(const char *title); 69 - int torture_stutter_init(int s); 69 + int torture_stutter_init(int s, int sgap); 70 70 71 71 /* Initialization and cleanup. */ 72 72 bool torture_init_begin(char *ttype, int v);

+1 -2

kernel/cgroup/cgroup.c

··· 101 101 */ 102 102 static DEFINE_SPINLOCK(cgroup_file_kn_lock); 103 103 104 - struct percpu_rw_semaphore cgroup_threadgroup_rwsem; 104 + DEFINE_PERCPU_RWSEM(cgroup_threadgroup_rwsem); 105 105 106 106 #define cgroup_assert_mutex_or_rcu_locked() \ 107 107 RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \ ··· 5666 5666 int ssid; 5667 5667 5668 5668 BUILD_BUG_ON(CGROUP_SUBSYS_COUNT > 16); 5669 - BUG_ON(percpu_init_rwsem(&cgroup_threadgroup_rwsem)); 5670 5669 BUG_ON(cgroup_init_cftypes(NULL, cgroup_base_files)); 5671 5670 BUG_ON(cgroup_init_cftypes(NULL, cgroup1_base_files)); 5672 5671

+1 -3

kernel/events/uprobes.c

··· 46 46 static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ]; 47 47 #define uprobes_mmap_hash(v) (&uprobes_mmap_mutex[((unsigned long)(v)) % UPROBES_HASH_SZ]) 48 48 49 - static struct percpu_rw_semaphore dup_mmap_sem; 49 + DEFINE_STATIC_PERCPU_RWSEM(dup_mmap_sem); 50 50 51 51 /* Have a copy of original instruction */ 52 52 #define UPROBE_COPY_INSN 0 ··· 2301 2301 2302 2302 for (i = 0; i < UPROBES_HASH_SZ; i++) 2303 2303 mutex_init(&uprobes_mmap_mutex[i]); 2304 - 2305 - BUG_ON(percpu_init_rwsem(&dup_mmap_sem)); 2306 2304 2307 2305 BUG_ON(register_die_notifier(&uprobe_exception_nb)); 2308 2306 }

+1 -1

kernel/locking/locktorture.c

··· 975 975 goto unwind; 976 976 } 977 977 if (stutter > 0) { 978 - firsterr = torture_stutter_init(stutter); 978 + firsterr = torture_stutter_init(stutter, stutter); 979 979 if (firsterr) 980 980 goto unwind; 981 981 }

+1 -1

kernel/locking/percpu-rwsem.c

··· 18 18 return -ENOMEM; 19 19 20 20 /* ->rw_sem represents the whole percpu_rw_semaphore for lockdep */ 21 - rcu_sync_init(&sem->rss, RCU_SCHED_SYNC); 21 + rcu_sync_init(&sem->rss); 22 22 __init_rwsem(&sem->rw_sem, name, rwsem_key); 23 23 rcuwait_init(&sem->writer); 24 24 sem->readers_block = 0;

+5

kernel/module.c

··· 3083 3083 sizeof(*mod->tracepoints_ptrs), 3084 3084 &mod->num_tracepoints); 3085 3085 #endif 3086 + #ifdef CONFIG_TREE_SRCU 3087 + mod->srcu_struct_ptrs = section_objs(info, "___srcu_struct_ptrs", 3088 + sizeof(*mod->srcu_struct_ptrs), 3089 + &mod->num_srcu_structs); 3090 + #endif 3086 3091 #ifdef CONFIG_BPF_EVENTS 3087 3092 mod->bpf_raw_events = section_objs(info, "__bpf_raw_tp_map", 3088 3093 sizeof(*mod->bpf_raw_events),

+5

kernel/rcu/rcu.h

··· 446 446 enum rcutorture_type { 447 447 RCU_FLAVOR, 448 448 RCU_TASKS_FLAVOR, 449 + RCU_TRIVIAL_FLAVOR, 449 450 SRCU_FLAVOR, 450 451 INVALID_RCU_FLAVOR 451 452 }; ··· 478 477 #define do_trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \ 479 478 do { } while (0) 480 479 #endif 480 + #endif 481 + 482 + #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TEST) 483 + long rcutorture_sched_setaffinity(pid_t pid, const struct cpumask *in_mask); 481 484 #endif 482 485 483 486 #ifdef CONFIG_TINY_SRCU

+85 -11

kernel/rcu/rcutorture.c

··· 299 299 int irq_capable; 300 300 int can_boost; 301 301 int extendables; 302 + int slow_gps; 302 303 const char *name; 303 304 }; 304 305 ··· 668 667 .fqs = NULL, 669 668 .stats = NULL, 670 669 .irq_capable = 1, 670 + .slow_gps = 1, 671 671 .name = "tasks" 672 + }; 673 + 674 + /* 675 + * Definitions for trivial CONFIG_PREEMPT=n-only torture testing. 676 + * This implementation does not necessarily work well with CPU hotplug. 677 + */ 678 + 679 + static void synchronize_rcu_trivial(void) 680 + { 681 + int cpu; 682 + 683 + for_each_online_cpu(cpu) { 684 + rcutorture_sched_setaffinity(current->pid, cpumask_of(cpu)); 685 + WARN_ON_ONCE(raw_smp_processor_id() != cpu); 686 + } 687 + } 688 + 689 + static int rcu_torture_read_lock_trivial(void) __acquires(RCU) 690 + { 691 + preempt_disable(); 692 + return 0; 693 + } 694 + 695 + static void rcu_torture_read_unlock_trivial(int idx) __releases(RCU) 696 + { 697 + preempt_enable(); 698 + } 699 + 700 + static struct rcu_torture_ops trivial_ops = { 701 + .ttype = RCU_TRIVIAL_FLAVOR, 702 + .init = rcu_sync_torture_init, 703 + .readlock = rcu_torture_read_lock_trivial, 704 + .read_delay = rcu_read_delay, /* just reuse rcu's version. */ 705 + .readunlock = rcu_torture_read_unlock_trivial, 706 + .get_gp_seq = rcu_no_completed, 707 + .sync = synchronize_rcu_trivial, 708 + .exp_sync = synchronize_rcu_trivial, 709 + .fqs = NULL, 710 + .stats = NULL, 711 + .irq_capable = 1, 712 + .name = "trivial" 672 713 }; 673 714 674 715 static unsigned long rcutorture_seq_diff(unsigned long new, unsigned long old) ··· 1053 1010 !rcu_gp_is_normal(); 1054 1011 } 1055 1012 rcu_torture_writer_state = RTWS_STUTTER; 1056 - if (stutter_wait("rcu_torture_writer")) 1013 + if (stutter_wait("rcu_torture_writer") && 1014 + !READ_ONCE(rcu_fwd_cb_nodelay) && 1015 + !cur_ops->slow_gps && 1016 + !torture_must_stop()) 1057 1017 for (i = 0; i < ARRAY_SIZE(rcu_tortures); i++) 1058 - if (list_empty(&rcu_tortures[i].rtort_free)) 1059 - WARN_ON_ONCE(1); 1018 + if (list_empty(&rcu_tortures[i].rtort_free) && 1019 + rcu_access_pointer(rcu_torture_current) != 1020 + &rcu_tortures[i]) { 1021 + rcu_ftrace_dump(DUMP_ALL); 1022 + WARN(1, "%s: rtort_pipe_count: %d\n", __func__, rcu_tortures[i].rtort_pipe_count); 1023 + } 1060 1024 } while (!torture_must_stop()); 1061 1025 /* Reset expediting back to unexpedited. */ 1062 1026 if (expediting > 0) ··· 1408 1358 } 1409 1359 1410 1360 pr_alert("%s%s ", torture_type, TORTURE_FLAG); 1411 - pr_cont("rtc: %p ver: %lu tfle: %d rta: %d rtaf: %d rtf: %d ", 1361 + pr_cont("rtc: %p %s: %lu tfle: %d rta: %d rtaf: %d rtf: %d ", 1412 1362 rcu_torture_current, 1363 + rcu_torture_current ? "ver" : "VER", 1413 1364 rcu_torture_current_version, 1414 1365 list_empty(&rcu_torture_freelist), 1415 1366 atomic_read(&n_rcu_torture_alloc), ··· 1712 1661 spin_unlock_irqrestore(&rcu_fwd_lock, flags); 1713 1662 } 1714 1663 1664 + // Give the scheduler a chance, even on nohz_full CPUs. 1665 + static void rcu_torture_fwd_prog_cond_resched(void) 1666 + { 1667 + if (IS_ENABLED(CONFIG_PREEMPT) && IS_ENABLED(CONFIG_NO_HZ_FULL)) { 1668 + if (need_resched()) 1669 + schedule(); 1670 + } else { 1671 + cond_resched(); 1672 + } 1673 + } 1674 + 1715 1675 /* 1716 1676 * Free all callbacks on the rcu_fwd_cb_head list, either because the 1717 1677 * test is over or because we hit an OOM event. ··· 1736 1674 for (;;) { 1737 1675 spin_lock_irqsave(&rcu_fwd_lock, flags); 1738 1676 rfcp = rcu_fwd_cb_head; 1739 - if (!rfcp) 1677 + if (!rfcp) { 1678 + spin_unlock_irqrestore(&rcu_fwd_lock, flags); 1740 1679 break; 1680 + } 1741 1681 rcu_fwd_cb_head = rfcp->rfc_next; 1742 1682 if (!rcu_fwd_cb_head) 1743 1683 rcu_fwd_cb_tail = &rcu_fwd_cb_head; 1744 1684 spin_unlock_irqrestore(&rcu_fwd_lock, flags); 1745 1685 kfree(rfcp); 1746 1686 freed++; 1687 + rcu_torture_fwd_prog_cond_resched(); 1747 1688 } 1748 - spin_unlock_irqrestore(&rcu_fwd_lock, flags); 1749 1689 return freed; 1750 1690 } 1751 1691 ··· 1771 1707 } 1772 1708 1773 1709 /* Tight loop containing cond_resched(). */ 1710 + WRITE_ONCE(rcu_fwd_cb_nodelay, true); 1711 + cur_ops->sync(); /* Later readers see above write. */ 1774 1712 if (selfpropcb) { 1775 1713 WRITE_ONCE(fcs.stop, 0); 1776 1714 cur_ops->call(&fcs.rh, rcu_torture_fwd_prog_cb); ··· 1790 1724 udelay(10); 1791 1725 cur_ops->readunlock(idx); 1792 1726 if (!fwd_progress_need_resched || need_resched()) 1793 - cond_resched(); 1727 + rcu_torture_fwd_prog_cond_resched(); 1794 1728 } 1795 1729 (*tested_tries)++; 1796 1730 if (!time_before(jiffies, stopat) && ··· 1811 1745 WARN_ON(READ_ONCE(fcs.stop) != 2); 1812 1746 destroy_rcu_head_on_stack(&fcs.rh); 1813 1747 } 1748 + schedule_timeout_uninterruptible(HZ / 10); /* Let kthreads recover. */ 1749 + WRITE_ONCE(rcu_fwd_cb_nodelay, false); 1814 1750 } 1815 1751 1816 1752 /* Carry out call_rcu() forward-progress testing. */ ··· 1833 1765 1834 1766 if (READ_ONCE(rcu_fwd_emergency_stop)) 1835 1767 return; /* Get out of the way quickly, no GP wait! */ 1768 + if (!cur_ops->call) 1769 + return; /* Can't do call_rcu() fwd prog without ->call. */ 1836 1770 1837 1771 /* Loop continuously posting RCU callbacks. */ 1838 1772 WRITE_ONCE(rcu_fwd_cb_nodelay, true); ··· 1875 1805 rfcp->rfc_gps = 0; 1876 1806 } 1877 1807 cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr); 1878 - cond_resched(); 1808 + rcu_torture_fwd_prog_cond_resched(); 1879 1809 } 1880 1810 stoppedat = jiffies; 1881 1811 n_launders_cb_snap = READ_ONCE(n_launders_cb); ··· 1884 1814 cur_ops->cb_barrier(); /* Wait for callbacks to be invoked. */ 1885 1815 (void)rcu_torture_fwd_prog_cbfree(); 1886 1816 1887 - WRITE_ONCE(rcu_fwd_cb_nodelay, false); 1888 1817 if (!torture_must_stop() && !READ_ONCE(rcu_fwd_emergency_stop)) { 1889 1818 WARN_ON(n_max_gps < MIN_FWD_CBS_LAUNDERED); 1890 1819 pr_alert("%s Duration %lu barrier: %lu pending %ld n_launders: %ld n_launders_sa: %ld n_max_gps: %ld n_max_cbs: %ld cver %ld gps %ld\n", ··· 1894 1825 n_max_gps, n_max_cbs, cver, gps); 1895 1826 rcu_torture_fwd_cb_hist(); 1896 1827 } 1828 + schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */ 1829 + WRITE_ONCE(rcu_fwd_cb_nodelay, false); 1897 1830 } 1898 1831 1899 1832 ··· 2311 2240 int firsterr = 0; 2312 2241 static struct rcu_torture_ops *torture_ops[] = { 2313 2242 &rcu_ops, &rcu_busted_ops, &srcu_ops, &srcud_ops, 2314 - &busted_srcud_ops, &tasks_ops, 2243 + &busted_srcud_ops, &tasks_ops, &trivial_ops, 2315 2244 }; 2316 2245 2317 2246 if (!torture_init_begin(torture_type, verbose)) ··· 2434 2363 if (stutter < 0) 2435 2364 stutter = 0; 2436 2365 if (stutter) { 2437 - firsterr = torture_stutter_init(stutter * HZ); 2366 + int t; 2367 + 2368 + t = cur_ops->stall_dur ? cur_ops->stall_dur() : stutter * HZ; 2369 + firsterr = torture_stutter_init(stutter * HZ, t); 2438 2370 if (firsterr) 2439 2371 goto unwind; 2440 2372 }

+67 -2

kernel/rcu/srcutree.c

··· 831 831 * srcu_read_lock(), and srcu_read_unlock() that are all passed the same 832 832 * srcu_struct structure. 833 833 */ 834 - void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp, 835 - rcu_callback_t func, bool do_norm) 834 + static void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp, 835 + rcu_callback_t func, bool do_norm) 836 836 { 837 837 unsigned long flags; 838 838 int idx; ··· 1310 1310 queue_work(rcu_gp_wq, &ssp->work.work); 1311 1311 } 1312 1312 } 1313 + 1314 + #ifdef CONFIG_MODULES 1315 + 1316 + /* Initialize any global-scope srcu_struct structures used by this module. */ 1317 + static int srcu_module_coming(struct module *mod) 1318 + { 1319 + int i; 1320 + struct srcu_struct **sspp = mod->srcu_struct_ptrs; 1321 + int ret; 1322 + 1323 + for (i = 0; i < mod->num_srcu_structs; i++) { 1324 + ret = init_srcu_struct(*(sspp++)); 1325 + if (WARN_ON_ONCE(ret)) 1326 + return ret; 1327 + } 1328 + return 0; 1329 + } 1330 + 1331 + /* Clean up any global-scope srcu_struct structures used by this module. */ 1332 + static void srcu_module_going(struct module *mod) 1333 + { 1334 + int i; 1335 + struct srcu_struct **sspp = mod->srcu_struct_ptrs; 1336 + 1337 + for (i = 0; i < mod->num_srcu_structs; i++) 1338 + cleanup_srcu_struct(*(sspp++)); 1339 + } 1340 + 1341 + /* Handle one module, either coming or going. */ 1342 + static int srcu_module_notify(struct notifier_block *self, 1343 + unsigned long val, void *data) 1344 + { 1345 + struct module *mod = data; 1346 + int ret = 0; 1347 + 1348 + switch (val) { 1349 + case MODULE_STATE_COMING: 1350 + ret = srcu_module_coming(mod); 1351 + break; 1352 + case MODULE_STATE_GOING: 1353 + srcu_module_going(mod); 1354 + break; 1355 + default: 1356 + break; 1357 + } 1358 + return ret; 1359 + } 1360 + 1361 + static struct notifier_block srcu_module_nb = { 1362 + .notifier_call = srcu_module_notify, 1363 + .priority = 0, 1364 + }; 1365 + 1366 + static __init int init_srcu_module_notifier(void) 1367 + { 1368 + int ret; 1369 + 1370 + ret = register_module_notifier(&srcu_module_nb); 1371 + if (ret) 1372 + pr_warn("Failed to register srcu module notifier\n"); 1373 + return ret; 1374 + } 1375 + late_initcall(init_srcu_module_notifier); 1376 + 1377 + #endif /* #ifdef CONFIG_MODULES */

+110 -130

kernel/rcu/sync.c

··· 10 10 #include <linux/rcu_sync.h> 11 11 #include <linux/sched.h> 12 12 13 - #ifdef CONFIG_PROVE_RCU 14 - #define __INIT_HELD(func) .held = func, 15 - #else 16 - #define __INIT_HELD(func) 17 - #endif 18 - 19 - static const struct { 20 - void (*sync)(void); 21 - void (*call)(struct rcu_head *, void (*)(struct rcu_head *)); 22 - void (*wait)(void); 23 - #ifdef CONFIG_PROVE_RCU 24 - int (*held)(void); 25 - #endif 26 - } gp_ops[] = { 27 - [RCU_SYNC] = { 28 - .sync = synchronize_rcu, 29 - .call = call_rcu, 30 - .wait = rcu_barrier, 31 - __INIT_HELD(rcu_read_lock_held) 32 - }, 33 - [RCU_SCHED_SYNC] = { 34 - .sync = synchronize_rcu, 35 - .call = call_rcu, 36 - .wait = rcu_barrier, 37 - __INIT_HELD(rcu_read_lock_sched_held) 38 - }, 39 - [RCU_BH_SYNC] = { 40 - .sync = synchronize_rcu, 41 - .call = call_rcu, 42 - .wait = rcu_barrier, 43 - __INIT_HELD(rcu_read_lock_bh_held) 44 - }, 45 - }; 46 - 47 - enum { GP_IDLE = 0, GP_PENDING, GP_PASSED }; 48 - enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY }; 13 + enum { GP_IDLE = 0, GP_ENTER, GP_PASSED, GP_EXIT, GP_REPLAY }; 49 14 50 15 #define rss_lock gp_wait.lock 51 - 52 - #ifdef CONFIG_PROVE_RCU 53 - void rcu_sync_lockdep_assert(struct rcu_sync *rsp) 54 - { 55 - RCU_LOCKDEP_WARN(!gp_ops[rsp->gp_type].held(), 56 - "suspicious rcu_sync_is_idle() usage"); 57 - } 58 - 59 - EXPORT_SYMBOL_GPL(rcu_sync_lockdep_assert); 60 - #endif 61 16 62 17 /** 63 18 * rcu_sync_init() - Initialize an rcu_sync structure 64 19 * @rsp: Pointer to rcu_sync structure to be initialized 65 - * @type: Flavor of RCU with which to synchronize rcu_sync structure 66 20 */ 67 - void rcu_sync_init(struct rcu_sync *rsp, enum rcu_sync_type type) 21 + void rcu_sync_init(struct rcu_sync *rsp) 68 22 { 69 23 memset(rsp, 0, sizeof(*rsp)); 70 24 init_waitqueue_head(&rsp->gp_wait); 71 - rsp->gp_type = type; 72 25 } 73 26 74 27 /** ··· 37 84 { 38 85 rsp->gp_count++; 39 86 rsp->gp_state = GP_PASSED; 87 + } 88 + 89 + 90 + static void rcu_sync_func(struct rcu_head *rhp); 91 + 92 + static void rcu_sync_call(struct rcu_sync *rsp) 93 + { 94 + call_rcu(&rsp->cb_head, rcu_sync_func); 95 + } 96 + 97 + /** 98 + * rcu_sync_func() - Callback function managing reader access to fastpath 99 + * @rhp: Pointer to rcu_head in rcu_sync structure to use for synchronization 100 + * 101 + * This function is passed to call_rcu() function by rcu_sync_enter() and 102 + * rcu_sync_exit(), so that it is invoked after a grace period following the 103 + * that invocation of enter/exit. 104 + * 105 + * If it is called by rcu_sync_enter() it signals that all the readers were 106 + * switched onto slow path. 107 + * 108 + * If it is called by rcu_sync_exit() it takes action based on events that 109 + * have taken place in the meantime, so that closely spaced rcu_sync_enter() 110 + * and rcu_sync_exit() pairs need not wait for a grace period. 111 + * 112 + * If another rcu_sync_enter() is invoked before the grace period 113 + * ended, reset state to allow the next rcu_sync_exit() to let the 114 + * readers back onto their fastpaths (after a grace period). If both 115 + * another rcu_sync_enter() and its matching rcu_sync_exit() are invoked 116 + * before the grace period ended, re-invoke call_rcu() on behalf of that 117 + * rcu_sync_exit(). Otherwise, set all state back to idle so that readers 118 + * can again use their fastpaths. 119 + */ 120 + static void rcu_sync_func(struct rcu_head *rhp) 121 + { 122 + struct rcu_sync *rsp = container_of(rhp, struct rcu_sync, cb_head); 123 + unsigned long flags; 124 + 125 + WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_IDLE); 126 + WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_PASSED); 127 + 128 + spin_lock_irqsave(&rsp->rss_lock, flags); 129 + if (rsp->gp_count) { 130 + /* 131 + * We're at least a GP after the GP_IDLE->GP_ENTER transition. 132 + */ 133 + WRITE_ONCE(rsp->gp_state, GP_PASSED); 134 + wake_up_locked(&rsp->gp_wait); 135 + } else if (rsp->gp_state == GP_REPLAY) { 136 + /* 137 + * A new rcu_sync_exit() has happened; requeue the callback to 138 + * catch a later GP. 139 + */ 140 + WRITE_ONCE(rsp->gp_state, GP_EXIT); 141 + rcu_sync_call(rsp); 142 + } else { 143 + /* 144 + * We're at least a GP after the last rcu_sync_exit(); eveybody 145 + * will now have observed the write side critical section. 146 + * Let 'em rip!. 147 + */ 148 + WRITE_ONCE(rsp->gp_state, GP_IDLE); 149 + } 150 + spin_unlock_irqrestore(&rsp->rss_lock, flags); 40 151 } 41 152 42 153 /** ··· 120 103 */ 121 104 void rcu_sync_enter(struct rcu_sync *rsp) 122 105 { 123 - bool need_wait, need_sync; 106 + int gp_state; 124 107 125 108 spin_lock_irq(&rsp->rss_lock); 126 - need_wait = rsp->gp_count++; 127 - need_sync = rsp->gp_state == GP_IDLE; 128 - if (need_sync) 129 - rsp->gp_state = GP_PENDING; 109 + gp_state = rsp->gp_state; 110 + if (gp_state == GP_IDLE) { 111 + WRITE_ONCE(rsp->gp_state, GP_ENTER); 112 + WARN_ON_ONCE(rsp->gp_count); 113 + /* 114 + * Note that we could simply do rcu_sync_call(rsp) here and 115 + * avoid the "if (gp_state == GP_IDLE)" block below. 116 + * 117 + * However, synchronize_rcu() can be faster if rcu_expedited 118 + * or rcu_blocking_is_gp() is true. 119 + * 120 + * Another reason is that we can't wait for rcu callback if 121 + * we are called at early boot time but this shouldn't happen. 122 + */ 123 + } 124 + rsp->gp_count++; 130 125 spin_unlock_irq(&rsp->rss_lock); 131 126 132 - WARN_ON_ONCE(need_wait && need_sync); 133 - if (need_sync) { 134 - gp_ops[rsp->gp_type].sync(); 135 - rsp->gp_state = GP_PASSED; 136 - wake_up_all(&rsp->gp_wait); 137 - } else if (need_wait) { 138 - wait_event(rsp->gp_wait, rsp->gp_state == GP_PASSED); 139 - } else { 127 + if (gp_state == GP_IDLE) { 140 128 /* 141 - * Possible when there's a pending CB from a rcu_sync_exit(). 142 - * Nobody has yet been allowed the 'fast' path and thus we can 143 - * avoid doing any sync(). The callback will get 'dropped'. 129 + * See the comment above, this simply does the "synchronous" 130 + * call_rcu(rcu_sync_func) which does GP_ENTER -> GP_PASSED. 144 131 */ 145 - WARN_ON_ONCE(rsp->gp_state != GP_PASSED); 132 + synchronize_rcu(); 133 + rcu_sync_func(&rsp->cb_head); 134 + /* Not really needed, wait_event() would see GP_PASSED. */ 135 + return; 146 136 } 137 + 138 + wait_event(rsp->gp_wait, READ_ONCE(rsp->gp_state) >= GP_PASSED); 147 139 } 148 140 149 141 /** 150 - * rcu_sync_func() - Callback function managing reader access to fastpath 151 - * @rhp: Pointer to rcu_head in rcu_sync structure to use for synchronization 152 - * 153 - * This function is passed to one of the call_rcu() functions by 154 - * rcu_sync_exit(), so that it is invoked after a grace period following the 155 - * that invocation of rcu_sync_exit(). It takes action based on events that 156 - * have taken place in the meantime, so that closely spaced rcu_sync_enter() 157 - * and rcu_sync_exit() pairs need not wait for a grace period. 158 - * 159 - * If another rcu_sync_enter() is invoked before the grace period 160 - * ended, reset state to allow the next rcu_sync_exit() to let the 161 - * readers back onto their fastpaths (after a grace period). If both 162 - * another rcu_sync_enter() and its matching rcu_sync_exit() are invoked 163 - * before the grace period ended, re-invoke call_rcu() on behalf of that 164 - * rcu_sync_exit(). Otherwise, set all state back to idle so that readers 165 - * can again use their fastpaths. 166 - */ 167 - static void rcu_sync_func(struct rcu_head *rhp) 168 - { 169 - struct rcu_sync *rsp = container_of(rhp, struct rcu_sync, cb_head); 170 - unsigned long flags; 171 - 172 - WARN_ON_ONCE(rsp->gp_state != GP_PASSED); 173 - WARN_ON_ONCE(rsp->cb_state == CB_IDLE); 174 - 175 - spin_lock_irqsave(&rsp->rss_lock, flags); 176 - if (rsp->gp_count) { 177 - /* 178 - * A new rcu_sync_begin() has happened; drop the callback. 179 - */ 180 - rsp->cb_state = CB_IDLE; 181 - } else if (rsp->cb_state == CB_REPLAY) { 182 - /* 183 - * A new rcu_sync_exit() has happened; requeue the callback 184 - * to catch a later GP. 185 - */ 186 - rsp->cb_state = CB_PENDING; 187 - gp_ops[rsp->gp_type].call(&rsp->cb_head, rcu_sync_func); 188 - } else { 189 - /* 190 - * We're at least a GP after rcu_sync_exit(); eveybody will now 191 - * have observed the write side critical section. Let 'em rip!. 192 - */ 193 - rsp->cb_state = CB_IDLE; 194 - rsp->gp_state = GP_IDLE; 195 - } 196 - spin_unlock_irqrestore(&rsp->rss_lock, flags); 197 - } 198 - 199 - /** 200 - * rcu_sync_exit() - Allow readers back onto fast patch after grace period 142 + * rcu_sync_exit() - Allow readers back onto fast path after grace period 201 143 * @rsp: Pointer to rcu_sync structure to use for synchronization 202 144 * 203 145 * This function is used by updaters who have completed, and can therefore ··· 167 191 */ 168 192 void rcu_sync_exit(struct rcu_sync *rsp) 169 193 { 194 + WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_IDLE); 195 + WARN_ON_ONCE(READ_ONCE(rsp->gp_count) == 0); 196 + 170 197 spin_lock_irq(&rsp->rss_lock); 171 198 if (!--rsp->gp_count) { 172 - if (rsp->cb_state == CB_IDLE) { 173 - rsp->cb_state = CB_PENDING; 174 - gp_ops[rsp->gp_type].call(&rsp->cb_head, rcu_sync_func); 175 - } else if (rsp->cb_state == CB_PENDING) { 176 - rsp->cb_state = CB_REPLAY; 199 + if (rsp->gp_state == GP_PASSED) { 200 + WRITE_ONCE(rsp->gp_state, GP_EXIT); 201 + rcu_sync_call(rsp); 202 + } else if (rsp->gp_state == GP_EXIT) { 203 + WRITE_ONCE(rsp->gp_state, GP_REPLAY); 177 204 } 178 205 } 179 206 spin_unlock_irq(&rsp->rss_lock); ··· 188 209 */ 189 210 void rcu_sync_dtor(struct rcu_sync *rsp) 190 211 { 191 - int cb_state; 212 + int gp_state; 192 213 193 - WARN_ON_ONCE(rsp->gp_count); 214 + WARN_ON_ONCE(READ_ONCE(rsp->gp_count)); 215 + WARN_ON_ONCE(READ_ONCE(rsp->gp_state) == GP_PASSED); 194 216 195 217 spin_lock_irq(&rsp->rss_lock); 196 - if (rsp->cb_state == CB_REPLAY) 197 - rsp->cb_state = CB_PENDING; 198 - cb_state = rsp->cb_state; 218 + if (rsp->gp_state == GP_REPLAY) 219 + WRITE_ONCE(rsp->gp_state, GP_EXIT); 220 + gp_state = rsp->gp_state; 199 221 spin_unlock_irq(&rsp->rss_lock); 200 222 201 - if (cb_state != CB_IDLE) { 202 - gp_ops[rsp->gp_type].wait(); 203 - WARN_ON_ONCE(rsp->cb_state != CB_IDLE); 223 + if (gp_state != GP_IDLE) { 224 + rcu_barrier(); 225 + WARN_ON_ONCE(rsp->gp_state != GP_IDLE); 204 226 } 205 227 }

+140 -28

kernel/rcu/tree.c

··· 51 51 #include <linux/tick.h> 52 52 #include <linux/sysrq.h> 53 53 #include <linux/kprobes.h> 54 + #include <linux/gfp.h> 55 + #include <linux/oom.h> 56 + #include <linux/smpboot.h> 57 + #include <linux/jiffies.h> 58 + #include <linux/sched/isolation.h> 59 + #include "../time/tick-internal.h" 54 60 55 61 #include "tree.h" 56 62 #include "rcu.h" ··· 98 92 /* Dump rcu_node combining tree at boot to verify correct setup. */ 99 93 static bool dump_tree; 100 94 module_param(dump_tree, bool, 0444); 95 + /* By default, use RCU_SOFTIRQ instead of rcuc kthreads. */ 96 + static bool use_softirq = 1; 97 + module_param(use_softirq, bool, 0444); 101 98 /* Control rcu_node-tree auto-balancing at boot time. */ 102 99 static bool rcu_fanout_exact; 103 100 module_param(rcu_fanout_exact, bool, 0444); ··· 147 138 static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf); 148 139 static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu); 149 140 static void invoke_rcu_core(void); 150 - static void invoke_rcu_callbacks(struct rcu_data *rdp); 151 141 static void rcu_report_exp_rdp(struct rcu_data *rdp); 152 142 static void sync_sched_exp_online_cleanup(int cpu); 153 143 ··· 376 368 } 377 369 378 370 /** 379 - * rcu_is_cpu_rrupt_from_idle - see if idle or immediately interrupted from idle 371 + * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle 380 372 * 381 - * If the current CPU is idle or running at a first-level (not nested) 373 + * If the current CPU is idle and running at a first-level (not nested) 382 374 * interrupt from idle, return true. The caller must have at least 383 375 * disabled preemption. 384 376 */ 385 377 static int rcu_is_cpu_rrupt_from_idle(void) 386 378 { 387 - return __this_cpu_read(rcu_data.dynticks_nesting) <= 0 && 388 - __this_cpu_read(rcu_data.dynticks_nmi_nesting) <= 1; 379 + /* Called only from within the scheduling-clock interrupt */ 380 + lockdep_assert_in_irq(); 381 + 382 + /* Check for counter underflows */ 383 + RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0, 384 + "RCU dynticks_nesting counter underflow!"); 385 + RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <= 0, 386 + "RCU dynticks_nmi_nesting counter underflow/zero!"); 387 + 388 + /* Are we at first interrupt nesting level? */ 389 + if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1) 390 + return false; 391 + 392 + /* Does CPU appear to be idle from an RCU standpoint? */ 393 + return __this_cpu_read(rcu_data.dynticks_nesting) == 0; 389 394 } 390 395 391 - #define DEFAULT_RCU_BLIMIT 10 /* Maximum callbacks per rcu_do_batch. */ 396 + #define DEFAULT_RCU_BLIMIT 10 /* Maximum callbacks per rcu_do_batch ... */ 397 + #define DEFAULT_MAX_RCU_BLIMIT 10000 /* ... even during callback flood. */ 392 398 static long blimit = DEFAULT_RCU_BLIMIT; 393 399 #define DEFAULT_RCU_QHIMARK 10000 /* If this many pending, ignore blimit. */ 394 400 static long qhimark = DEFAULT_RCU_QHIMARK; ··· 2135 2113 2136 2114 /* Reinstate batch limit if we have worked down the excess. */ 2137 2115 count = rcu_segcblist_n_cbs(&rdp->cblist); 2138 - if (rdp->blimit == LONG_MAX && count <= qlowmark) 2116 + if (rdp->blimit >= DEFAULT_MAX_RCU_BLIMIT && count <= qlowmark) 2139 2117 rdp->blimit = blimit; 2140 2118 2141 2119 /* Reset ->qlen_last_fqs_check trigger if enough CBs have drained. */ ··· 2275 2253 EXPORT_SYMBOL_GPL(rcu_force_quiescent_state); 2276 2254 2277 2255 /* Perform RCU core processing work for the current CPU. */ 2278 - static __latent_entropy void rcu_core(struct softirq_action *unused) 2256 + static __latent_entropy void rcu_core(void) 2279 2257 { 2280 2258 unsigned long flags; 2281 2259 struct rcu_data *rdp = raw_cpu_ptr(&rcu_data); ··· 2309 2287 rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check()); 2310 2288 2311 2289 /* If there are callbacks ready, invoke them. */ 2312 - if (rcu_segcblist_ready_cbs(&rdp->cblist)) 2313 - invoke_rcu_callbacks(rdp); 2290 + if (rcu_segcblist_ready_cbs(&rdp->cblist) && 2291 + likely(READ_ONCE(rcu_scheduler_fully_active))) 2292 + rcu_do_batch(rdp); 2314 2293 2315 2294 /* Do any needed deferred wakeups of rcuo kthreads. */ 2316 2295 do_nocb_deferred_wakeup(rdp); 2317 2296 trace_rcu_utilization(TPS("End RCU core")); 2318 2297 } 2319 2298 2320 - /* 2321 - * Schedule RCU callback invocation. If the running implementation of RCU 2322 - * does not support RCU priority boosting, just do a direct call, otherwise 2323 - * wake up the per-CPU kernel kthread. Note that because we are running 2324 - * on the current CPU with softirqs disabled, the rcu_cpu_kthread_task 2325 - * cannot disappear out from under us. 2326 - */ 2327 - static void invoke_rcu_callbacks(struct rcu_data *rdp) 2299 + static void rcu_core_si(struct softirq_action *h) 2328 2300 { 2329 - if (unlikely(!READ_ONCE(rcu_scheduler_fully_active))) 2330 - return; 2331 - if (likely(!rcu_state.boost)) { 2332 - rcu_do_batch(rdp); 2333 - return; 2334 - } 2335 - invoke_rcu_callbacks_kthread(); 2301 + rcu_core(); 2336 2302 } 2337 2303 2304 + static void rcu_wake_cond(struct task_struct *t, int status) 2305 + { 2306 + /* 2307 + * If the thread is yielding, only wake it when this 2308 + * is invoked from idle 2309 + */ 2310 + if (t && (status != RCU_KTHREAD_YIELDING || is_idle_task(current))) 2311 + wake_up_process(t); 2312 + } 2313 + 2314 + static void invoke_rcu_core_kthread(void) 2315 + { 2316 + struct task_struct *t; 2317 + unsigned long flags; 2318 + 2319 + local_irq_save(flags); 2320 + __this_cpu_write(rcu_data.rcu_cpu_has_work, 1); 2321 + t = __this_cpu_read(rcu_data.rcu_cpu_kthread_task); 2322 + if (t != NULL && t != current) 2323 + rcu_wake_cond(t, __this_cpu_read(rcu_data.rcu_cpu_kthread_status)); 2324 + local_irq_restore(flags); 2325 + } 2326 + 2327 + /* 2328 + * Wake up this CPU's rcuc kthread to do RCU core processing. 2329 + */ 2338 2330 static void invoke_rcu_core(void) 2339 2331 { 2340 - if (cpu_online(smp_processor_id())) 2332 + if (!cpu_online(smp_processor_id())) 2333 + return; 2334 + if (use_softirq) 2341 2335 raise_softirq(RCU_SOFTIRQ); 2336 + else 2337 + invoke_rcu_core_kthread(); 2342 2338 } 2339 + 2340 + static void rcu_cpu_kthread_park(unsigned int cpu) 2341 + { 2342 + per_cpu(rcu_data.rcu_cpu_kthread_status, cpu) = RCU_KTHREAD_OFFCPU; 2343 + } 2344 + 2345 + static int rcu_cpu_kthread_should_run(unsigned int cpu) 2346 + { 2347 + return __this_cpu_read(rcu_data.rcu_cpu_has_work); 2348 + } 2349 + 2350 + /* 2351 + * Per-CPU kernel thread that invokes RCU callbacks. This replaces 2352 + * the RCU softirq used in configurations of RCU that do not support RCU 2353 + * priority boosting. 2354 + */ 2355 + static void rcu_cpu_kthread(unsigned int cpu) 2356 + { 2357 + unsigned int *statusp = this_cpu_ptr(&rcu_data.rcu_cpu_kthread_status); 2358 + char work, *workp = this_cpu_ptr(&rcu_data.rcu_cpu_has_work); 2359 + int spincnt; 2360 + 2361 + for (spincnt = 0; spincnt < 10; spincnt++) { 2362 + trace_rcu_utilization(TPS("Start CPU kthread@rcu_wait")); 2363 + local_bh_disable(); 2364 + *statusp = RCU_KTHREAD_RUNNING; 2365 + local_irq_disable(); 2366 + work = *workp; 2367 + *workp = 0; 2368 + local_irq_enable(); 2369 + if (work) 2370 + rcu_core(); 2371 + local_bh_enable(); 2372 + if (*workp == 0) { 2373 + trace_rcu_utilization(TPS("End CPU kthread@rcu_wait")); 2374 + *statusp = RCU_KTHREAD_WAITING; 2375 + return; 2376 + } 2377 + } 2378 + *statusp = RCU_KTHREAD_YIELDING; 2379 + trace_rcu_utilization(TPS("Start CPU kthread@rcu_yield")); 2380 + schedule_timeout_interruptible(2); 2381 + trace_rcu_utilization(TPS("End CPU kthread@rcu_yield")); 2382 + *statusp = RCU_KTHREAD_WAITING; 2383 + } 2384 + 2385 + static struct smp_hotplug_thread rcu_cpu_thread_spec = { 2386 + .store = &rcu_data.rcu_cpu_kthread_task, 2387 + .thread_should_run = rcu_cpu_kthread_should_run, 2388 + .thread_fn = rcu_cpu_kthread, 2389 + .thread_comm = "rcuc/%u", 2390 + .setup = rcu_cpu_kthread_setup, 2391 + .park = rcu_cpu_kthread_park, 2392 + }; 2393 + 2394 + /* 2395 + * Spawn per-CPU RCU core processing kthreads. 2396 + */ 2397 + static int __init rcu_spawn_core_kthreads(void) 2398 + { 2399 + int cpu; 2400 + 2401 + for_each_possible_cpu(cpu) 2402 + per_cpu(rcu_data.rcu_cpu_has_work, cpu) = 0; 2403 + if (!IS_ENABLED(CONFIG_RCU_BOOST) && use_softirq) 2404 + return 0; 2405 + WARN_ONCE(smpboot_register_percpu_thread(&rcu_cpu_thread_spec), 2406 + "%s: Could not start rcuc kthread, OOM is now expected behavior\n", __func__); 2407 + return 0; 2408 + } 2409 + early_initcall(rcu_spawn_core_kthreads); 2343 2410 2344 2411 /* 2345 2412 * Handle any core-RCU processing required by a call_rcu() invocation. ··· 2465 2354 rcu_accelerate_cbs_unlocked(rdp->mynode, rdp); 2466 2355 } else { 2467 2356 /* Give the grace period a kick. */ 2468 - rdp->blimit = LONG_MAX; 2357 + rdp->blimit = DEFAULT_MAX_RCU_BLIMIT; 2469 2358 if (rcu_state.n_force_qs == rdp->n_force_qs_snap && 2470 2359 rcu_segcblist_first_pend_cb(&rdp->cblist) != head) 2471 2360 rcu_force_quiescent_state(); ··· 3466 3355 rcu_init_one(); 3467 3356 if (dump_tree) 3468 3357 rcu_dump_rcu_node_tree(); 3469 - open_softirq(RCU_SOFTIRQ, rcu_core); 3358 + if (use_softirq) 3359 + open_softirq(RCU_SOFTIRQ, rcu_core_si); 3470 3360 3471 3361 /* 3472 3362 * We don't need protection against CPU-hotplug here because

+4 -2

kernel/rcu/tree.h

··· 154 154 bool core_needs_qs; /* Core waits for quiesc state. */ 155 155 bool beenonline; /* CPU online at least once. */ 156 156 bool gpwrap; /* Possible ->gp_seq wrap. */ 157 - bool deferred_qs; /* This CPU awaiting a deferred QS? */ 157 + bool exp_deferred_qs; /* This CPU awaiting a deferred QS? */ 158 158 struct rcu_node *mynode; /* This CPU's leaf of hierarchy */ 159 159 unsigned long grpmask; /* Mask to apply to leaf qsmask. */ 160 160 unsigned long ticks_this_gp; /* The number of scheduling-clock */ 161 161 /* ticks this CPU has handled */ 162 162 /* during and after the last grace */ 163 163 /* period it is aware of. */ 164 + struct irq_work defer_qs_iw; /* Obtain later scheduler attention. */ 165 + bool defer_qs_iw_pending; /* Scheduler attention pending? */ 164 166 165 167 /* 2) batch handling */ 166 168 struct rcu_segcblist cblist; /* Segmented callback list, with */ ··· 409 407 static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck); 410 408 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags); 411 409 static void rcu_preempt_boost_start_gp(struct rcu_node *rnp); 412 - static void invoke_rcu_callbacks_kthread(void); 413 410 static bool rcu_is_callbacks_kthread(void); 411 + static void rcu_cpu_kthread_setup(unsigned int cpu); 414 412 static void __init rcu_spawn_boost_kthreads(void); 415 413 static void rcu_prepare_kthreads(int cpu); 416 414 static void rcu_cleanup_after_idle(void);

+39 -14

kernel/rcu/tree_exp.h

··· 250 250 */ 251 251 static void rcu_report_exp_rdp(struct rcu_data *rdp) 252 252 { 253 - WRITE_ONCE(rdp->deferred_qs, false); 253 + WRITE_ONCE(rdp->exp_deferred_qs, false); 254 254 rcu_report_exp_cpu_mult(rdp->mynode, rdp->grpmask, true); 255 255 } 256 256 ··· 259 259 { 260 260 if (rcu_exp_gp_seq_done(s)) { 261 261 trace_rcu_exp_grace_period(rcu_state.name, s, TPS("done")); 262 - /* Ensure test happens before caller kfree(). */ 263 - smp_mb__before_atomic(); /* ^^^ */ 262 + smp_mb(); /* Ensure test happens before caller kfree(). */ 264 263 return true; 265 264 } 266 265 return false; ··· 383 384 mask_ofl_test |= mask; 384 385 continue; 385 386 } 387 + if (get_cpu() == cpu) { 388 + put_cpu(); 389 + continue; 390 + } 386 391 ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0); 392 + put_cpu(); 387 393 if (!ret) { 388 394 mask_ofl_ipi &= ~mask; 389 395 continue; ··· 615 611 rcu_dynticks_curr_cpu_in_eqs()) { 616 612 rcu_report_exp_rdp(rdp); 617 613 } else { 618 - rdp->deferred_qs = true; 614 + rdp->exp_deferred_qs = true; 619 615 set_tsk_need_resched(t); 620 616 set_preempt_need_resched(); 621 617 } ··· 637 633 if (t->rcu_read_lock_nesting > 0) { 638 634 raw_spin_lock_irqsave_rcu_node(rnp, flags); 639 635 if (rnp->expmask & rdp->grpmask) { 640 - rdp->deferred_qs = true; 636 + rdp->exp_deferred_qs = true; 641 637 t->rcu_read_unlock_special.b.exp_hint = true; 642 638 } 643 639 raw_spin_unlock_irqrestore_rcu_node(rnp, flags); ··· 660 656 * 661 657 * Otherwise, force a context switch after the CPU enables everything. 662 658 */ 663 - rdp->deferred_qs = true; 659 + rdp->exp_deferred_qs = true; 664 660 if (!(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK)) || 665 661 WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs())) { 666 662 rcu_preempt_deferred_qs(t); ··· 698 694 699 695 #else /* #ifdef CONFIG_PREEMPT_RCU */ 700 696 697 + /* Request an expedited quiescent state. */ 698 + static void rcu_exp_need_qs(void) 699 + { 700 + __this_cpu_write(rcu_data.cpu_no_qs.b.exp, true); 701 + /* Store .exp before .rcu_urgent_qs. */ 702 + smp_store_release(this_cpu_ptr(&rcu_data.rcu_urgent_qs), true); 703 + set_tsk_need_resched(current); 704 + set_preempt_need_resched(); 705 + } 706 + 701 707 /* Invoked on each online non-idle CPU for expedited quiescent state. */ 702 708 static void rcu_exp_handler(void *unused) 703 709 { ··· 723 709 rcu_report_exp_rdp(this_cpu_ptr(&rcu_data)); 724 710 return; 725 711 } 726 - __this_cpu_write(rcu_data.cpu_no_qs.b.exp, true); 727 - /* Store .exp before .rcu_urgent_qs. */ 728 - smp_store_release(this_cpu_ptr(&rcu_data.rcu_urgent_qs), true); 729 - set_tsk_need_resched(current); 730 - set_preempt_need_resched(); 712 + rcu_exp_need_qs(); 731 713 } 732 714 733 715 /* Send IPI for expedited cleanup if needed at end of CPU-hotplug operation. */ 734 716 static void sync_sched_exp_online_cleanup(int cpu) 735 717 { 718 + unsigned long flags; 719 + int my_cpu; 736 720 struct rcu_data *rdp; 737 721 int ret; 738 722 struct rcu_node *rnp; 739 723 740 724 rdp = per_cpu_ptr(&rcu_data, cpu); 741 725 rnp = rdp->mynode; 742 - if (!(READ_ONCE(rnp->expmask) & rdp->grpmask)) 726 + my_cpu = get_cpu(); 727 + /* Quiescent state either not needed or already requested, leave. */ 728 + if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) || 729 + __this_cpu_read(rcu_data.cpu_no_qs.b.exp)) { 730 + put_cpu(); 743 731 return; 732 + } 733 + /* Quiescent state needed on current CPU, so set it up locally. */ 734 + if (my_cpu == cpu) { 735 + local_irq_save(flags); 736 + rcu_exp_need_qs(); 737 + local_irq_restore(flags); 738 + put_cpu(); 739 + return; 740 + } 741 + /* Quiescent state needed on some other CPU, send IPI. */ 744 742 ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0); 743 + put_cpu(); 745 744 WARN_ON_ONCE(ret); 746 745 } 747 746 ··· 792 765 */ 793 766 void synchronize_rcu_expedited(void) 794 767 { 795 - struct rcu_data *rdp; 796 768 struct rcu_exp_work rew; 797 769 struct rcu_node *rnp; 798 770 unsigned long s; ··· 828 802 } 829 803 830 804 /* Wait for expedited grace period to complete. */ 831 - rdp = per_cpu_ptr(&rcu_data, raw_smp_processor_id()); 832 805 rnp = rcu_get_root(); 833 806 wait_event(rnp->exp_wq[rcu_seq_ctr(s) & 0x3], 834 807 sync_exp_work_done(s));

+64 -133

kernel/rcu/tree_plugin.h

··· 11 11 * Paul E. McKenney <paulmck@linux.ibm.com> 12 12 */ 13 13 14 - #include <linux/delay.h> 15 - #include <linux/gfp.h> 16 - #include <linux/oom.h> 17 - #include <linux/sched/debug.h> 18 - #include <linux/smpboot.h> 19 - #include <linux/sched/isolation.h> 20 - #include <uapi/linux/sched/types.h> 21 - #include "../time/tick-internal.h" 22 - 23 - #ifdef CONFIG_RCU_BOOST 24 14 #include "../locking/rtmutex_common.h" 25 - #else /* #ifdef CONFIG_RCU_BOOST */ 26 - 27 - /* 28 - * Some architectures do not define rt_mutexes, but if !CONFIG_RCU_BOOST, 29 - * all uses are in dead code. Provide a definition to keep the compiler 30 - * happy, but add WARN_ON_ONCE() to complain if used in the wrong place. 31 - * This probably needs to be excluded from -rt builds. 32 - */ 33 - #define rt_mutex_owner(a) ({ WARN_ON_ONCE(1); NULL; }) 34 - #define rt_mutex_futex_unlock(x) WARN_ON_ONCE(1) 35 - 36 - #endif /* #else #ifdef CONFIG_RCU_BOOST */ 37 15 38 16 #ifdef CONFIG_RCU_NOCB_CPU 39 17 static cpumask_var_t rcu_nocb_mask; /* CPUs to have callbacks offloaded. */ ··· 72 94 pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay); 73 95 if (gp_cleanup_delay) 74 96 pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_cleanup_delay); 97 + if (!use_softirq) 98 + pr_info("\tRCU_SOFTIRQ processing moved to rcuc kthreads.\n"); 75 99 if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG)) 76 100 pr_info("\tRCU debug extended QS entry/exit.\n"); 77 101 rcupdate_announce_bootup_oddness(); ··· 237 257 * no need to check for a subsequent expedited GP. (Though we are 238 258 * still in a quiescent state in any case.) 239 259 */ 240 - if (blkd_state & RCU_EXP_BLKD && rdp->deferred_qs) 260 + if (blkd_state & RCU_EXP_BLKD && rdp->exp_deferred_qs) 241 261 rcu_report_exp_rdp(rdp); 242 262 else 243 - WARN_ON_ONCE(rdp->deferred_qs); 263 + WARN_ON_ONCE(rdp->exp_deferred_qs); 244 264 } 245 265 246 266 /* ··· 337 357 * means that we continue to block the current grace period. 338 358 */ 339 359 rcu_qs(); 340 - if (rdp->deferred_qs) 360 + if (rdp->exp_deferred_qs) 341 361 rcu_report_exp_rdp(rdp); 342 362 trace_rcu_utilization(TPS("End context switch")); 343 363 barrier(); /* Avoid RCU read-side critical sections leaking up. */ ··· 451 471 */ 452 472 special = t->rcu_read_unlock_special; 453 473 rdp = this_cpu_ptr(&rcu_data); 454 - if (!special.s && !rdp->deferred_qs) { 474 + if (!special.s && !rdp->exp_deferred_qs) { 455 475 local_irq_restore(flags); 456 476 return; 457 477 } 478 + t->rcu_read_unlock_special.b.deferred_qs = false; 458 479 if (special.b.need_qs) { 459 480 rcu_qs(); 460 481 t->rcu_read_unlock_special.b.need_qs = false; 461 - if (!t->rcu_read_unlock_special.s && !rdp->deferred_qs) { 482 + if (!t->rcu_read_unlock_special.s && !rdp->exp_deferred_qs) { 462 483 local_irq_restore(flags); 463 484 return; 464 485 } ··· 471 490 * tasks are handled when removing the task from the 472 491 * blocked-tasks list below. 473 492 */ 474 - if (rdp->deferred_qs) { 493 + if (rdp->exp_deferred_qs) { 475 494 rcu_report_exp_rdp(rdp); 476 495 if (!t->rcu_read_unlock_special.s) { 477 496 local_irq_restore(flags); ··· 560 579 */ 561 580 static bool rcu_preempt_need_deferred_qs(struct task_struct *t) 562 581 { 563 - return (__this_cpu_read(rcu_data.deferred_qs) || 582 + return (__this_cpu_read(rcu_data.exp_deferred_qs) || 564 583 READ_ONCE(t->rcu_read_unlock_special.s)) && 565 584 t->rcu_read_lock_nesting <= 0; 566 585 } ··· 588 607 } 589 608 590 609 /* 610 + * Minimal handler to give the scheduler a chance to re-evaluate. 611 + */ 612 + static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp) 613 + { 614 + struct rcu_data *rdp; 615 + 616 + rdp = container_of(iwp, struct rcu_data, defer_qs_iw); 617 + rdp->defer_qs_iw_pending = false; 618 + } 619 + 620 + /* 591 621 * Handle special cases during rcu_read_unlock(), such as needing to 592 622 * notify RCU core processing or task having blocked during the RCU 593 623 * read-side critical section. ··· 617 625 local_irq_save(flags); 618 626 irqs_were_disabled = irqs_disabled_flags(flags); 619 627 if (preempt_bh_were_disabled || irqs_were_disabled) { 620 - WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false); 621 - /* Need to defer quiescent state until everything is enabled. */ 622 - if (irqs_were_disabled) { 623 - /* Enabling irqs does not reschedule, so... */ 628 + bool exp; 629 + struct rcu_data *rdp = this_cpu_ptr(&rcu_data); 630 + struct rcu_node *rnp = rdp->mynode; 631 + 632 + t->rcu_read_unlock_special.b.exp_hint = false; 633 + exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) || 634 + (rdp->grpmask & rnp->expmask) || 635 + tick_nohz_full_cpu(rdp->cpu); 636 + // Need to defer quiescent state until everything is enabled. 637 + if ((exp || in_irq()) && irqs_were_disabled && use_softirq && 638 + (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) { 639 + // Using softirq, safe to awaken, and we get 640 + // no help from enabling irqs, unlike bh/preempt. 624 641 raise_softirq_irqoff(RCU_SOFTIRQ); 642 + } else if (exp && irqs_were_disabled && !use_softirq && 643 + !t->rcu_read_unlock_special.b.deferred_qs) { 644 + // Safe to awaken and we get no help from enabling 645 + // irqs, unlike bh/preempt. 646 + invoke_rcu_core(); 625 647 } else { 626 - /* Enabling BH or preempt does reschedule, so... */ 648 + // Enabling BH or preempt does reschedule, so... 649 + // Also if no expediting or NO_HZ_FULL, slow is OK. 627 650 set_tsk_need_resched(current); 628 651 set_preempt_need_resched(); 652 + if (IS_ENABLED(CONFIG_IRQ_WORK) && 653 + !rdp->defer_qs_iw_pending && exp) { 654 + // Get scheduler to re-evaluate and call hooks. 655 + // If !IRQ_WORK, FQS scan will eventually IPI. 656 + init_irq_work(&rdp->defer_qs_iw, 657 + rcu_preempt_deferred_qs_handler); 658 + rdp->defer_qs_iw_pending = true; 659 + irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu); 660 + } 629 661 } 662 + t->rcu_read_unlock_special.b.deferred_qs = true; 630 663 local_irq_restore(flags); 631 664 return; 632 665 } ··· 777 760 i = 0; 778 761 list_for_each(lhp, &rnp->blkd_tasks) { 779 762 pr_cont(" %p", lhp); 780 - if (++i >= 10) 763 + if (++i >= ncheck) 781 764 break; 782 765 } 783 766 pr_cont("\n"); ··· 961 944 962 945 #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ 963 946 964 - #ifdef CONFIG_RCU_BOOST 965 - 966 - static void rcu_wake_cond(struct task_struct *t, int status) 947 + /* 948 + * If boosting, set rcuc kthreads to realtime priority. 949 + */ 950 + static void rcu_cpu_kthread_setup(unsigned int cpu) 967 951 { 968 - /* 969 - * If the thread is yielding, only wake it when this 970 - * is invoked from idle 971 - */ 972 - if (status != RCU_KTHREAD_YIELDING || is_idle_task(current)) 973 - wake_up_process(t); 952 + #ifdef CONFIG_RCU_BOOST 953 + struct sched_param sp; 954 + 955 + sp.sched_priority = kthread_prio; 956 + sched_setscheduler_nocheck(current, SCHED_FIFO, &sp); 957 + #endif /* #ifdef CONFIG_RCU_BOOST */ 974 958 } 959 + 960 + #ifdef CONFIG_RCU_BOOST 975 961 976 962 /* 977 963 * Carry out RCU priority boosting on the task indicated by ->exp_tasks ··· 1111 1091 } 1112 1092 1113 1093 /* 1114 - * Wake up the per-CPU kthread to invoke RCU callbacks. 1115 - */ 1116 - static void invoke_rcu_callbacks_kthread(void) 1117 - { 1118 - unsigned long flags; 1119 - 1120 - local_irq_save(flags); 1121 - __this_cpu_write(rcu_data.rcu_cpu_has_work, 1); 1122 - if (__this_cpu_read(rcu_data.rcu_cpu_kthread_task) != NULL && 1123 - current != __this_cpu_read(rcu_data.rcu_cpu_kthread_task)) { 1124 - rcu_wake_cond(__this_cpu_read(rcu_data.rcu_cpu_kthread_task), 1125 - __this_cpu_read(rcu_data.rcu_cpu_kthread_status)); 1126 - } 1127 - local_irq_restore(flags); 1128 - } 1129 - 1130 - /* 1131 1094 * Is the current CPU running the RCU-callbacks kthread? 1132 1095 * Caller must have preemption disabled. 1133 1096 */ ··· 1163 1160 return 0; 1164 1161 } 1165 1162 1166 - static void rcu_cpu_kthread_setup(unsigned int cpu) 1167 - { 1168 - struct sched_param sp; 1169 - 1170 - sp.sched_priority = kthread_prio; 1171 - sched_setscheduler_nocheck(current, SCHED_FIFO, &sp); 1172 - } 1173 - 1174 - static void rcu_cpu_kthread_park(unsigned int cpu) 1175 - { 1176 - per_cpu(rcu_data.rcu_cpu_kthread_status, cpu) = RCU_KTHREAD_OFFCPU; 1177 - } 1178 - 1179 - static int rcu_cpu_kthread_should_run(unsigned int cpu) 1180 - { 1181 - return __this_cpu_read(rcu_data.rcu_cpu_has_work); 1182 - } 1183 - 1184 - /* 1185 - * Per-CPU kernel thread that invokes RCU callbacks. This replaces 1186 - * the RCU softirq used in configurations of RCU that do not support RCU 1187 - * priority boosting. 1188 - */ 1189 - static void rcu_cpu_kthread(unsigned int cpu) 1190 - { 1191 - unsigned int *statusp = this_cpu_ptr(&rcu_data.rcu_cpu_kthread_status); 1192 - char work, *workp = this_cpu_ptr(&rcu_data.rcu_cpu_has_work); 1193 - int spincnt; 1194 - 1195 - for (spincnt = 0; spincnt < 10; spincnt++) { 1196 - trace_rcu_utilization(TPS("Start CPU kthread@rcu_wait")); 1197 - local_bh_disable(); 1198 - *statusp = RCU_KTHREAD_RUNNING; 1199 - local_irq_disable(); 1200 - work = *workp; 1201 - *workp = 0; 1202 - local_irq_enable(); 1203 - if (work) 1204 - rcu_do_batch(this_cpu_ptr(&rcu_data)); 1205 - local_bh_enable(); 1206 - if (*workp == 0) { 1207 - trace_rcu_utilization(TPS("End CPU kthread@rcu_wait")); 1208 - *statusp = RCU_KTHREAD_WAITING; 1209 - return; 1210 - } 1211 - } 1212 - *statusp = RCU_KTHREAD_YIELDING; 1213 - trace_rcu_utilization(TPS("Start CPU kthread@rcu_yield")); 1214 - schedule_timeout_interruptible(2); 1215 - trace_rcu_utilization(TPS("End CPU kthread@rcu_yield")); 1216 - *statusp = RCU_KTHREAD_WAITING; 1217 - } 1218 - 1219 1163 /* 1220 1164 * Set the per-rcu_node kthread's affinity to cover all CPUs that are 1221 1165 * served by the rcu_node in question. The CPU hotplug lock is still ··· 1193 1243 free_cpumask_var(cm); 1194 1244 } 1195 1245 1196 - static struct smp_hotplug_thread rcu_cpu_thread_spec = { 1197 - .store = &rcu_data.rcu_cpu_kthread_task, 1198 - .thread_should_run = rcu_cpu_kthread_should_run, 1199 - .thread_fn = rcu_cpu_kthread, 1200 - .thread_comm = "rcuc/%u", 1201 - .setup = rcu_cpu_kthread_setup, 1202 - .park = rcu_cpu_kthread_park, 1203 - }; 1204 - 1205 1246 /* 1206 1247 * Spawn boost kthreads -- called as soon as the scheduler is running. 1207 1248 */ 1208 1249 static void __init rcu_spawn_boost_kthreads(void) 1209 1250 { 1210 1251 struct rcu_node *rnp; 1211 - int cpu; 1212 1252 1213 - for_each_possible_cpu(cpu) 1214 - per_cpu(rcu_data.rcu_cpu_has_work, cpu) = 0; 1215 - if (WARN_ONCE(smpboot_register_percpu_thread(&rcu_cpu_thread_spec), "%s: Could not start rcub kthread, OOM is now expected behavior\n", __func__)) 1216 - return; 1217 1253 rcu_for_each_leaf_node(rnp) 1218 1254 (void)rcu_spawn_one_boost_kthread(rnp); 1219 1255 } ··· 1220 1284 __releases(rnp->lock) 1221 1285 { 1222 1286 raw_spin_unlock_irqrestore_rcu_node(rnp, flags); 1223 - } 1224 - 1225 - static void invoke_rcu_callbacks_kthread(void) 1226 - { 1227 - WARN_ON_ONCE(1); 1228 1287 } 1229 1288 1230 1289 static bool rcu_is_callbacks_kthread(void)

+3 -1

kernel/rcu/tree_stall.h

··· 630 630 time_before(j, rcu_state.gp_req_activity + gpssdelay) || 631 631 time_before(j, rcu_state.gp_activity + gpssdelay) || 632 632 atomic_xchg(&warned, 1)) { 633 - raw_spin_unlock_rcu_node(rnp_root); /* irqs remain disabled. */ 633 + if (rnp_root != rnp) 634 + /* irqs remain disabled. */ 635 + raw_spin_unlock_rcu_node(rnp_root); 634 636 raw_spin_unlock_irqrestore_rcu_node(rnp, flags); 635 637 return; 636 638 }

+13

kernel/rcu/update.c

··· 423 423 do { } while (0) 424 424 #endif 425 425 426 + #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TEST) 427 + /* Get rcutorture access to sched_setaffinity(). */ 428 + long rcutorture_sched_setaffinity(pid_t pid, const struct cpumask *in_mask) 429 + { 430 + int ret; 431 + 432 + ret = sched_setaffinity(pid, in_mask); 433 + WARN_ONCE(ret, "%s: sched_setaffinity() returned %d\n", __func__, ret); 434 + return ret; 435 + } 436 + EXPORT_SYMBOL_GPL(rcutorture_sched_setaffinity); 437 + #endif 438 + 426 439 #ifdef CONFIG_RCU_STALL_COMMON 427 440 int rcu_cpu_stall_suppress __read_mostly; /* 1 = suppress stall warnings. */ 428 441 EXPORT_SYMBOL_GPL(rcu_cpu_stall_suppress);

+17 -6

kernel/torture.c

··· 570 570 static struct task_struct *stutter_task; 571 571 static int stutter_pause_test; 572 572 static int stutter; 573 + static int stutter_gap; 573 574 574 575 /* 575 576 * Block until the stutter interval ends. This must be called periodically ··· 579 578 bool stutter_wait(const char *title) 580 579 { 581 580 int spt; 581 + bool ret = false; 582 582 583 583 cond_resched_tasks_rcu_qs(); 584 584 spt = READ_ONCE(stutter_pause_test); 585 585 for (; spt; spt = READ_ONCE(stutter_pause_test)) { 586 + ret = true; 586 587 if (spt == 1) { 587 588 schedule_timeout_interruptible(1); 588 589 } else if (spt == 2) { ··· 595 592 } 596 593 torture_shutdown_absorb(title); 597 594 } 598 - return !!spt; 595 + return ret; 599 596 } 600 597 EXPORT_SYMBOL_GPL(stutter_wait); 601 598 ··· 605 602 */ 606 603 static int torture_stutter(void *arg) 607 604 { 605 + int wtime; 606 + 608 607 VERBOSE_TOROUT_STRING("torture_stutter task started"); 609 608 do { 610 609 if (!torture_must_stop() && stutter > 1) { 611 - WRITE_ONCE(stutter_pause_test, 1); 612 - schedule_timeout_interruptible(stutter - 1); 610 + wtime = stutter; 611 + if (stutter > HZ + 1) { 612 + WRITE_ONCE(stutter_pause_test, 1); 613 + wtime = stutter - HZ - 1; 614 + schedule_timeout_interruptible(wtime); 615 + wtime = HZ + 1; 616 + } 613 617 WRITE_ONCE(stutter_pause_test, 2); 614 - schedule_timeout_interruptible(1); 618 + schedule_timeout_interruptible(wtime); 615 619 } 616 620 WRITE_ONCE(stutter_pause_test, 0); 617 621 if (!torture_must_stop()) 618 - schedule_timeout_interruptible(stutter); 622 + schedule_timeout_interruptible(stutter_gap); 619 623 torture_shutdown_absorb("torture_stutter"); 620 624 } while (!torture_must_stop()); 621 625 torture_kthread_stopping("torture_stutter"); ··· 632 622 /* 633 623 * Initialize and kick off the torture_stutter kthread. 634 624 */ 635 - int torture_stutter_init(const int s) 625 + int torture_stutter_init(const int s, const int sgap) 636 626 { 637 627 stutter = s; 628 + stutter_gap = sgap; 638 629 return torture_create_kthread(torture_stutter, NULL, stutter_task); 639 630 } 640 631 EXPORT_SYMBOL_GPL(torture_stutter_init);

+2 -2

tools/include/linux/rcu.h

··· 19 19 return false; 20 20 } 21 21 22 - #define rcu_assign_pointer(p, v) ((p) = (v)) 23 - #define RCU_INIT_POINTER(p, v) p=(v) 22 + #define rcu_assign_pointer(p, v) do { (p) = (v); } while (0) 23 + #define RCU_INIT_POINTER(p, v) do { (p) = (v); } while (0) 24 24 25 25 #endif

+6

tools/memory-model/linux-kernel.bell

··· 24 24 enum Barriers = 'wmb (*smp_wmb*) || 25 25 'rmb (*smp_rmb*) || 26 26 'mb (*smp_mb*) || 27 + 'barrier (*barrier*) || 27 28 'rcu-lock (*rcu_read_lock*) || 28 29 'rcu-unlock (*rcu_read_unlock*) || 29 30 'sync-rcu (*synchronize_rcu*) || ··· 77 76 78 77 (* Validate SRCU dynamic match *) 79 78 flag ~empty different-values(srcu-rscs) as srcu-bad-nesting 79 + 80 + (* Compute marked and plain memory accesses *) 81 + let Marked = (~M) | IW | Once | Release | Acquire | domain(rmw) | range(rmw) | 82 + LKR | LKW | UL | LF | RL | RU 83 + let Plain = M \ Marked

+80 -22

tools/memory-model/linux-kernel.cat

··· 24 24 (* Basic relations *) 25 25 (*******************) 26 26 27 + (* Release Acquire *) 28 + let acq-po = [Acquire] ; po ; [M] 29 + let po-rel = [M] ; po ; [Release] 30 + let po-unlock-rf-lock-po = po ; [UL] ; rf ; [LKR] ; po 31 + 27 32 (* Fences *) 28 - let rmb = [R \ Noreturn] ; fencerel(Rmb) ; [R \ Noreturn] 33 + let R4rmb = R \ Noreturn (* Reads for which rmb works *) 34 + let rmb = [R4rmb] ; fencerel(Rmb) ; [R4rmb] 29 35 let wmb = [W] ; fencerel(Wmb) ; [W] 30 36 let mb = ([M] ; fencerel(Mb) ; [M]) | 31 37 ([M] ; fencerel(Before-atomic) ; [RMW] ; po? ; [M]) | ··· 40 34 ([M] ; po ; [UL] ; (co | po) ; [LKW] ; 41 35 fencerel(After-unlock-lock) ; [M]) 42 36 let gp = po ; [Sync-rcu | Sync-srcu] ; po? 43 - 44 37 let strong-fence = mb | gp 45 38 46 - (* Release Acquire *) 47 - let acq-po = [Acquire] ; po ; [M] 48 - let po-rel = [M] ; po ; [Release] 49 - let po-unlock-rf-lock-po = po ; [UL] ; rf ; [LKR] ; po 39 + let nonrw-fence = strong-fence | po-rel | acq-po 40 + let fence = nonrw-fence | wmb | rmb 41 + let barrier = fencerel(Barrier | Rmb | Wmb | Mb | Sync-rcu | Sync-srcu | 42 + Before-atomic | After-atomic | Acquire | Release | 43 + Rcu-lock | Rcu-unlock | Srcu-lock | Srcu-unlock) | 44 + (po ; [Release]) | ([Acquire] ; po) 50 45 51 46 (**********************************) 52 47 (* Fundamental coherence ordering *) ··· 68 61 let dep = addr | data 69 62 let rwdep = (dep | ctrl) ; [W] 70 63 let overwrite = co | fr 71 - let to-w = rwdep | (overwrite & int) 72 - let to-r = addr | (dep ; rfi) 73 - let fence = strong-fence | wmb | po-rel | rmb | acq-po 64 + let to-w = rwdep | (overwrite & int) | (addr ; [Plain] ; wmb) 65 + let to-r = addr | (dep ; [Marked] ; rfi) 74 66 let ppo = to-r | to-w | fence | (po-unlock-rf-lock-po & int) 75 67 76 68 (* Propagation: Ordering from release operations and strong fences. *) 77 - let A-cumul(r) = rfe? ; r 78 - let cumul-fence = A-cumul(strong-fence | po-rel) | wmb | po-unlock-rf-lock-po 79 - let prop = (overwrite & ext)? ; cumul-fence* ; rfe? 69 + let A-cumul(r) = (rfe ; [Marked])? ; r 70 + let cumul-fence = [Marked] ; (A-cumul(strong-fence | po-rel) | wmb | 71 + po-unlock-rf-lock-po) ; [Marked] 72 + let prop = [Marked] ; (overwrite & ext)? ; cumul-fence* ; 73 + [Marked] ; rfe? ; [Marked] 80 74 81 75 (* 82 76 * Happens Before: Ordering from the passage of time. 83 77 * No fences needed here for prop because relation confined to one process. 84 78 *) 85 - let hb = ppo | rfe | ((prop \ id) & int) 79 + let hb = [Marked] ; (ppo | rfe | ((prop \ id) & int)) ; [Marked] 86 80 acyclic hb as happens-before 87 81 88 82 (****************************************) ··· 91 83 (****************************************) 92 84 93 85 (* Propagation: Each non-rf link needs a strong fence. *) 94 - let pb = prop ; strong-fence ; hb* 86 + let pb = prop ; strong-fence ; hb* ; [Marked] 95 87 acyclic pb as propagation 96 88 97 89 (*******) ··· 122 114 123 115 (* 124 116 * Any sequence containing at least as many grace periods as RCU read-side 125 - * critical sections (joined by rcu-link) acts as a generalized strong fence. 117 + * critical sections (joined by rcu-link) induces order like a generalized 118 + * inter-CPU strong fence. 126 119 * Likewise for SRCU grace periods and read-side critical sections, provided 127 120 * the synchronize_srcu() and srcu_read_[un]lock() calls refer to the same 128 121 * struct srcu_struct location. 129 122 *) 130 - let rec rcu-fence = rcu-gp | srcu-gp | 123 + let rec rcu-order = rcu-gp | srcu-gp | 131 124 (rcu-gp ; rcu-link ; rcu-rscsi) | 132 125 ((srcu-gp ; rcu-link ; srcu-rscsi) & loc) | 133 126 (rcu-rscsi ; rcu-link ; rcu-gp) | 134 127 ((srcu-rscsi ; rcu-link ; srcu-gp) & loc) | 135 - (rcu-gp ; rcu-link ; rcu-fence ; rcu-link ; rcu-rscsi) | 136 - ((srcu-gp ; rcu-link ; rcu-fence ; rcu-link ; srcu-rscsi) & loc) | 137 - (rcu-rscsi ; rcu-link ; rcu-fence ; rcu-link ; rcu-gp) | 138 - ((srcu-rscsi ; rcu-link ; rcu-fence ; rcu-link ; srcu-gp) & loc) | 139 - (rcu-fence ; rcu-link ; rcu-fence) 128 + (rcu-gp ; rcu-link ; rcu-order ; rcu-link ; rcu-rscsi) | 129 + ((srcu-gp ; rcu-link ; rcu-order ; rcu-link ; srcu-rscsi) & loc) | 130 + (rcu-rscsi ; rcu-link ; rcu-order ; rcu-link ; rcu-gp) | 131 + ((srcu-rscsi ; rcu-link ; rcu-order ; rcu-link ; srcu-gp) & loc) | 132 + (rcu-order ; rcu-link ; rcu-order) 133 + let rcu-fence = po ; rcu-order ; po? 134 + let fence = fence | rcu-fence 135 + let strong-fence = strong-fence | rcu-fence 140 136 141 137 (* rb orders instructions just as pb does *) 142 - let rb = prop ; po ; rcu-fence ; po? ; hb* ; pb* 138 + let rb = prop ; rcu-fence ; hb* ; pb* ; [Marked] 143 139 144 140 irreflexive rb as rcu 145 141 ··· 155 143 * let xb = hb | pb | rb 156 144 * acyclic xb as executes-before 157 145 *) 146 + 147 + (*********************************) 148 + (* Plain accesses and data races *) 149 + (*********************************) 150 + 151 + (* Warn about plain writes and marked accesses in the same region *) 152 + let mixed-accesses = ([Plain & W] ; (po-loc \ barrier) ; [Marked]) | 153 + ([Marked] ; (po-loc \ barrier) ; [Plain & W]) 154 + flag ~empty mixed-accesses as mixed-accesses 155 + 156 + (* Executes-before and visibility *) 157 + let xbstar = (hb | pb | rb)* 158 + let vis = cumul-fence* ; rfe? ; [Marked] ; 159 + ((strong-fence ; [Marked] ; xbstar) | (xbstar & int)) 160 + 161 + (* Boundaries for lifetimes of plain accesses *) 162 + let w-pre-bounded = [Marked] ; (addr | fence)? 163 + let r-pre-bounded = [Marked] ; (addr | nonrw-fence | 164 + ([R4rmb] ; fencerel(Rmb) ; [~Noreturn]))? 165 + let w-post-bounded = fence? ; [Marked] 166 + let r-post-bounded = (nonrw-fence | ([~Noreturn] ; fencerel(Rmb) ; [R4rmb]))? ; 167 + [Marked] 168 + 169 + (* Visibility and executes-before for plain accesses *) 170 + let ww-vis = fence | (strong-fence ; xbstar ; w-pre-bounded) | 171 + (w-post-bounded ; vis ; w-pre-bounded) 172 + let wr-vis = fence | (strong-fence ; xbstar ; r-pre-bounded) | 173 + (w-post-bounded ; vis ; r-pre-bounded) 174 + let rw-xbstar = fence | (r-post-bounded ; xbstar ; w-pre-bounded) 175 + 176 + (* Potential races *) 177 + let pre-race = ext & ((Plain * M) | ((M \ IW) * Plain)) 178 + 179 + (* Coherence requirements for plain accesses *) 180 + let wr-incoh = pre-race & rf & rw-xbstar^-1 181 + let rw-incoh = pre-race & fr & wr-vis^-1 182 + let ww-incoh = pre-race & co & ww-vis^-1 183 + empty (wr-incoh | rw-incoh | ww-incoh) as plain-coherence 184 + 185 + (* Actual races *) 186 + let ww-nonrace = ww-vis & ((Marked * W) | rw-xbstar) & ((W * Marked) | wr-vis) 187 + let ww-race = (pre-race & co) \ ww-nonrace 188 + let wr-race = (pre-race & (co? ; rf)) \ wr-vis 189 + let rw-race = (pre-race & fr) \ rw-xbstar 190 + 191 + flag ~empty (ww-race | wr-race | rw-race) as data-race

+1

tools/memory-model/linux-kernel.def

··· 24 24 smp_mb__after_atomic() { __fence{after-atomic}; } 25 25 smp_mb__after_spinlock() { __fence{after-spinlock}; } 26 26 smp_mb__after_unlock_lock() { __fence{after-unlock-lock}; } 27 + barrier() { __fence{barrier}; } 27 28 28 29 // Exchange 29 30 xchg(X,V) __xchg{mb}(X,V)

+1 -1

tools/memory-model/litmus-tests/MP+poonceonces.litmus

··· 1 1 C MP+poonceonces 2 2 3 3 (* 4 - * Result: Maybe 4 + * Result: Sometimes 5 5 * 6 6 * Can the counter-intuitive message-passing outcome be prevented with 7 7 * no ordering at all?

+1 -1

tools/memory-model/litmus-tests/README

··· 244 244 Adding the ".litmus" suffix: SB+rfionceonce-poonceonces.litmus 245 245 246 246 The descriptors that describe connections between consecutive accesses 247 - within the cycle through a given litmus test can be provided by the herd 247 + within the cycle through a given litmus test can be provided by the herd7 248 248 tool (Rfi, Po, Fre, and so on) or by the linux-kernel.bell file (Once, 249 249 Release, Acquire, and so on). 250 250

+1 -1

tools/memory-model/lock.cat

··· 11 11 include "cross.cat" 12 12 13 13 (* 14 - * The lock-related events generated by herd are as follows: 14 + * The lock-related events generated by herd7 are as follows: 15 15 * 16 16 * LKR Lock-Read: the read part of a spin_lock() or successful 17 17 * spin_trylock() read-modify-write event pair

+2 -2

tools/memory-model/scripts/README

··· 22 22 23 23 Run all litmus tests having .litmus.out files from previous 24 24 initlitmushist.sh or newlitmushist.sh runs, comparing the 25 - herd output to that of the original runs. 25 + herd7 output to that of the original runs. 26 26 27 27 checklitmus.sh 28 28 ··· 43 43 44 44 judgelitmus.sh 45 45 46 - Given a .litmus file and its .litmus.out herd output, check the 46 + Given a .litmus file and its .litmus.out herd7 output, check the 47 47 .litmus.out file against the .litmus file's "Result:" comment to 48 48 judge whether the test ran correctly. Not normally run manually, 49 49 provided instead for use by other scripts.

+1 -1

tools/memory-model/scripts/checkalllitmus.sh

··· 1 1 #!/bin/sh 2 2 # SPDX-License-Identifier: GPL-2.0+ 3 3 # 4 - # Run herd tests on all .litmus files in the litmus-tests directory 4 + # Run herd7 tests on all .litmus files in the litmus-tests directory 5 5 # and check each file's result against a "Result:" comment within that 6 6 # litmus test. If the verification result does not match that specified 7 7 # in the litmus test, this script prints an error message prefixed with

+1 -1

tools/memory-model/scripts/checklitmus.sh

··· 1 1 #!/bin/sh 2 2 # SPDX-License-Identifier: GPL-2.0+ 3 3 # 4 - # Run a herd test and invokes judgelitmus.sh to check the result against 4 + # Run a herd7 test and invokes judgelitmus.sh to check the result against 5 5 # a "Result:" comment within the litmus test. It also outputs verification 6 6 # results to a file whose name is that of the specified litmus test, but 7 7 # with ".out" appended.

+1 -1

tools/memory-model/scripts/parseargs.sh

··· 91 91 shift 92 92 ;; 93 93 --herdopts|--herdopt) 94 - checkarg --destdir "(herd options)" "$#" "$2" '.*' '^--' 94 + checkarg --destdir "(herd7 options)" "$#" "$2" '.*' '^--' 95 95 LKMM_HERD_OPTIONS="$2" 96 96 shift 97 97 ;;

+1 -1

tools/memory-model/scripts/runlitmushist.sh

··· 79 79 echo ' ---' Summary: 1>&2 80 80 grep '!!!' $T/*.sh.out 1>&2 81 81 nfail="`grep '!!!' $T/*.sh.out | wc -l`" 82 - echo 'Number of failed herd runs (e.g., timeout): ' $nfail 1>&2 82 + echo 'Number of failed herd7 runs (e.g., timeout): ' $nfail 1>&2 83 83 exit 1 84 84 else 85 85 echo All runs completed successfully. 1>&2

+1 -1

tools/testing/radix-tree/linux/rcupdate.h

··· 7 7 #define rcu_dereference_raw(p) rcu_dereference(p) 8 8 #define rcu_dereference_protected(p, cond) rcu_dereference(p) 9 9 #define rcu_dereference_check(p, cond) rcu_dereference(p) 10 - #define RCU_INIT_POINTER(p, v) (p) = (v) 10 + #define RCU_INIT_POINTER(p, v) do { (p) = (v); } while (0) 11 11 12 12 #endif

+3

tools/testing/selftests/rcutorture/Makefile

··· 1 + # SPDX-License-Identifier: GPL-2.0+ 2 + all: 3 + ( cd ../../../..; tools/testing/selftests/rcutorture/bin/kvm.sh --duration 10 --configs TREE01 )

+12 -27

tools/testing/selftests/rcutorture/bin/configinit.sh

··· 1 1 #!/bin/bash 2 2 # SPDX-License-Identifier: GPL-2.0+ 3 3 # 4 - # Usage: configinit.sh config-spec-file build-output-dir results-dir 4 + # Usage: configinit.sh config-spec-file results-dir 5 5 # 6 6 # Create a .config file from the spec file. Run from the kernel source tree. 7 7 # Exits with 0 if all went well, with 1 if all went well but the config ··· 10 10 # The first argument is the .config specification file, which contains 11 11 # desired settings, for example, "CONFIG_NO_HZ=y". For best results, 12 12 # this should be a full pathname. 13 - # 14 - # The second argument is a optional path to a build output directory, 15 - # for example, "O=/tmp/foo". If this argument is omitted, the .config 16 - # file will be generated directly in the current directory. 17 13 # 18 14 # Copyright (C) IBM Corporation, 2013 19 15 # ··· 22 26 # Capture config spec file. 23 27 24 28 c=$1 25 - buildloc=$2 26 - resdir=$3 27 - builddir= 28 - if echo $buildloc | grep -q '^O=' 29 - then 30 - builddir=`echo $buildloc | sed -e 's/^O=//'` 31 - if test ! -d $builddir 32 - then 33 - mkdir $builddir 34 - fi 35 - else 36 - echo Bad build directory: \"$buildloc\" 37 - exit 2 38 - fi 29 + resdir=$2 39 30 40 31 sed -e 's/^$CONFIG[0-9A-Z_]*$=.*$/grep -v "^# \1" |/' < $c > $T/u.sh 41 32 sed -e 's/^$CONFIG[0-9A-Z_]*=$.*$/grep -v \1 |/' < $c >> $T/u.sh 42 33 grep '^grep' < $T/u.sh > $T/upd.sh 43 34 echo "cat - $c" >> $T/upd.sh 44 - make mrproper 45 - make $buildloc distclean > $resdir/Make.distclean 2>&1 46 - make $buildloc $TORTURE_DEFCONFIG > $resdir/Make.defconfig.out 2>&1 47 - mv $builddir/.config $builddir/.config.sav 48 - sh $T/upd.sh < $builddir/.config.sav > $builddir/.config 49 - cp $builddir/.config $builddir/.config.new 50 - yes '' | make $buildloc oldconfig > $resdir/Make.oldconfig.out 2> $resdir/Make.oldconfig.err 35 + if test -z "$TORTURE_TRUST_MAKE" 36 + then 37 + make clean > $resdir/Make.clean 2>&1 38 + fi 39 + make $TORTURE_DEFCONFIG > $resdir/Make.defconfig.out 2>&1 40 + mv .config .config.sav 41 + sh $T/upd.sh < .config.sav > .config 42 + cp .config .config.new 43 + yes '' | make oldconfig > $resdir/Make.oldconfig.out 2> $resdir/Make.oldconfig.err 51 44 52 45 # verify new config matches specification. 53 - configcheck.sh $builddir/.config $c 46 + configcheck.sh .config $c 54 47 55 48 exit 0

+5

tools/testing/selftests/rcutorture/bin/cpus2use.sh

··· 9 9 # 10 10 # Authors: Paul E. McKenney <paulmck@linux.ibm.com> 11 11 12 + if test -n "$TORTURE_ALLOTED_CPUS" 13 + then 14 + echo $TORTURE_ALLOTED_CPUS 15 + exit 0 16 + fi 12 17 ncpus=`grep '^processor' /proc/cpuinfo | wc -l` 13 18 idlecpus=`mpstat | tail -1 | \ 14 19 awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`

+12 -1

tools/testing/selftests/rcutorture/bin/functions.sh

··· 172 172 local console=ttyS0 173 173 case "$1" in 174 174 qemu-system-x86_64|qemu-system-i386) 175 - echo noapic selinux=0 initcall_debug debug 175 + echo selinux=0 initcall_debug debug 176 176 ;; 177 177 qemu-system-aarch64) 178 178 console=ttyAMA0 ··· 191 191 # Output arguments for qemu arguments based on the TORTURE_QEMU_MAC 192 192 # and TORTURE_QEMU_INTERACTIVE environment variables. 193 193 identify_qemu_args () { 194 + local KVM_CPU="" 195 + case "$1" in 196 + qemu-system-x86_64) 197 + KVM_CPU=kvm64 198 + ;; 199 + qemu-system-i386) 200 + KVM_CPU=kvm32 201 + ;; 202 + esac 194 203 case "$1" in 195 204 qemu-system-x86_64|qemu-system-i386) 205 + echo -machine q35,accel=kvm 206 + echo -cpu ${KVM_CPU} 196 207 ;; 197 208 qemu-system-aarch64) 198 209 echo -machine virt,gic-version=host -cpu host

+9 -4

tools/testing/selftests/rcutorture/bin/jitter.sh

··· 34 34 exit 0; 35 35 fi 36 36 37 - # Set affinity to randomly selected CPU 38 - cpus=`ls /sys/devices/system/cpu/*/online | 39 - sed -e 's,/[^/]*$,,' -e 's/^[^0-9]*//' | 40 - grep -v '^0*$'` 37 + # Set affinity to randomly selected online CPU 38 + cpus=`grep 1 /sys/devices/system/cpu/*/online | 39 + sed -e 's,/[^/]*$,,' -e 's/^[^0-9]*//'` 40 + 41 + # Do not leave out poor old cpu0 which may not be hot-pluggable 42 + if [ ! -f "/sys/devices/system/cpu/cpu0/online" ]; then 43 + cpus="0 $cpus" 44 + fi 45 + 41 46 cpumask=`awk -v cpus="$cpus" -v me=$me -v n=$n 'BEGIN { 42 47 srand(n + me + systime()); 43 48 ncpus = split(cpus, ca);

+4 -5

tools/testing/selftests/rcutorture/bin/kvm-build.sh

··· 3 3 # 4 4 # Build a kvm-ready Linux kernel from the tree in the current directory. 5 5 # 6 - # Usage: kvm-build.sh config-template build-dir resdir 6 + # Usage: kvm-build.sh config-template resdir 7 7 # 8 8 # Copyright (C) IBM Corporation, 2011 9 9 # ··· 15 15 echo "kvm-build.sh :$config_template: Not a readable file" 16 16 exit 1 17 17 fi 18 - builddir=${2} 19 - resdir=${3} 18 + resdir=${2} 20 19 21 20 T=${TMPDIR-/tmp}/test-linux.sh.$$ 22 21 trap 'rm -rf $T' 0 ··· 28 29 CONFIG_VIRTIO_CONSOLE=y 29 30 ___EOF___ 30 31 31 - configinit.sh $T/config O=$builddir $resdir 32 + configinit.sh $T/config $resdir 32 33 retval=$? 33 34 if test $retval -gt 1 34 35 then 35 36 exit 2 36 37 fi 37 38 ncpus=`cpus2use.sh` 38 - make O=$builddir -j$ncpus $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1 39 + make -j$ncpus $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1 39 40 retval=$? 40 41 if test $retval -ne 0 || grep "rcu[^/]*": < $resdir/Make.out | egrep -q "Stop|Error|error:|warning:" || egrep -q "Stop|Error|error:" < $resdir/Make.out 41 42 then

+3

tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh

··· 11 11 # 12 12 # The "directory" above should end with the date/time directory, for example, 13 13 # "tools/testing/selftests/rcutorture/res/2018.02.25-14:27:27". 14 + # Returns error status reflecting the success (or not) of the specified run. 14 15 # 15 16 # Copyright (C) IBM Corporation, 2018 16 17 # ··· 57 56 if test -n "$files" 58 57 then 59 58 $editor $files 59 + exit 1 60 60 else 61 61 echo No errors in console logs. 62 + exit 0 62 63 fi

+12 -1

tools/testing/selftests/rcutorture/bin/kvm-recheck.sh

··· 7 7 # 8 8 # Usage: kvm-recheck.sh resdir ... 9 9 # 10 + # Returns status reflecting the success or not of the last run specified. 11 + # 10 12 # Copyright (C) IBM Corporation, 2011 11 13 # 12 14 # Authors: Paul E. McKenney <paulmck@linux.ibm.com> ··· 30 28 TORTURE_SUITE="`cat $i/../TORTURE_SUITE`" 31 29 rm -f $i/console.log.*.diags 32 30 kvm-recheck-${TORTURE_SUITE}.sh $i 33 - if test -f "$i/console.log" 31 + if test -f "$i/qemu-retval" && test "`cat $i/qemu-retval`" -ne 0 && test "`cat $i/qemu-retval`" -ne 137 34 32 then 33 + echo QEMU error, output: 34 + cat $i/qemu-output 35 + elif test -f "$i/console.log" 36 + then 37 + if test -f "$i/qemu-retval" && test "`cat $i/qemu-retval`" -eq 137 38 + then 39 + echo QEMU killed 40 + fi 35 41 configcheck.sh $i/.config $i/ConfigFragment 36 42 if test -r $i/Make.oldconfig.err 37 43 then ··· 68 58 fi 69 59 done 70 60 done 61 + EDITOR=echo kvm-find-errors.sh "${@: -1}" > /dev/null 2>&1

+9 -14

tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh

··· 36 36 config_dir=`echo $config_template | sed -e 's,/[^/]*$,,'` 37 37 title=`echo $config_template | sed -e 's/^.*\///'` 38 38 builddir=${2} 39 - if test -z "$builddir" -o ! -d "$builddir" -o ! -w "$builddir" 40 - then 41 - echo "kvm-test-1-run.sh :$builddir: Not a writable directory, cannot build into it" 42 - exit 1 43 - fi 44 39 resdir=${3} 45 40 if test -z "$resdir" -o ! -d "$resdir" -o ! -w "$resdir" 46 41 then ··· 80 85 ln -s $base_resdir/.config $resdir # for kvm-recheck.sh 81 86 # Arch-independent indicator 82 87 touch $resdir/builtkernel 83 - elif kvm-build.sh $T/Kc2 $builddir $resdir 88 + elif kvm-build.sh $T/Kc2 $resdir 84 89 then 85 90 # Had to build a kernel for this test. 86 - QEMU="`identify_qemu $builddir/vmlinux`" 91 + QEMU="`identify_qemu vmlinux`" 87 92 BOOT_IMAGE="`identify_boot_image $QEMU`" 88 - cp $builddir/vmlinux $resdir 89 - cp $builddir/.config $resdir 90 - cp $builddir/Module.symvers $resdir > /dev/null || : 91 - cp $builddir/System.map $resdir > /dev/null || : 93 + cp vmlinux $resdir 94 + cp .config $resdir 95 + cp Module.symvers $resdir > /dev/null || : 96 + cp System.map $resdir > /dev/null || : 92 97 if test -n "$BOOT_IMAGE" 93 98 then 94 - cp $builddir/$BOOT_IMAGE $resdir 99 + cp $BOOT_IMAGE $resdir 95 100 KERNEL=$resdir/${BOOT_IMAGE##*/} 96 101 # Arch-independent indicator 97 102 touch $resdir/builtkernel ··· 102 107 parse-build.sh $resdir/Make.out $title 103 108 else 104 109 # Build failed. 105 - cp $builddir/.config $resdir || : 110 + cp .config $resdir || : 106 111 echo Build failed, not running KVM, see $resdir. 107 112 if test -f $builddir.wait 108 113 then ··· 160 165 fi 161 166 echo "NOTE: $QEMU either did not run or was interactive" > $resdir/console.log 162 167 echo $QEMU $qemu_args -m $TORTURE_QEMU_MEM -kernel $KERNEL -append \"$qemu_append $boot_args\" > $resdir/qemu-cmd 163 - ( $QEMU $qemu_args -m $TORTURE_QEMU_MEM -kernel $KERNEL -append "$qemu_append $boot_args"& echo $! > $resdir/qemu_pid; wait `cat $resdir/qemu_pid`; echo $? > $resdir/qemu-retval ) & 168 + ( $QEMU $qemu_args -m $TORTURE_QEMU_MEM -kernel $KERNEL -append "$qemu_append $boot_args" > $resdir/qemu-output 2>&1 & echo $! > $resdir/qemu_pid; wait `cat $resdir/qemu_pid`; echo $? > $resdir/qemu-retval ) & 164 169 commandcompleted=0 165 170 sleep 10 # Give qemu's pid a chance to reach the file 166 171 if test -s "$resdir/qemu_pid"

+12 -2

tools/testing/selftests/rcutorture/bin/kvm.sh

··· 24 24 dryrun="" 25 25 KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM 26 26 PATH=${KVM}/bin:$PATH; export PATH 27 + TORTURE_ALLOTED_CPUS="" 27 28 TORTURE_DEFCONFIG=defconfig 28 29 TORTURE_BOOT_IMAGE="" 29 30 TORTURE_INITRD="$KVM/initrd"; export TORTURE_INITRD ··· 33 32 TORTURE_QEMU_MEM=512 34 33 TORTURE_SHUTDOWN_GRACE=180 35 34 TORTURE_SUITE=rcu 35 + TORTURE_TRUST_MAKE="" 36 36 resdir="" 37 37 configs="" 38 38 cpus=0 ··· 64 62 echo " --qemu-cmd qemu-system-..." 65 63 echo " --results absolute-pathname" 66 64 echo " --torture rcu" 65 + echo " --trust-make" 67 66 exit 1 68 67 } 69 68 ··· 92 89 --cpus) 93 90 checkarg --cpus "(number)" "$#" "$2" '^[0-9]*$' '^--' 94 91 cpus=$2 92 + TORTURE_ALLOTED_CPUS="$2" 95 93 shift 96 94 ;; 97 95 --datestamp) ··· 176 172 # it after specifying rcuperf. (But why?) 177 173 jitter=0 178 174 fi 175 + ;; 176 + --trust-make) 177 + TORTURE_TRUST_MAKE="y" 179 178 ;; 180 179 *) 181 180 echo Unknown argument $1 ··· 292 285 CONFIGFRAG="$CONFIGFRAG"; export CONFIGFRAG 293 286 KVM="$KVM"; export KVM 294 287 PATH="$PATH"; export PATH 288 + TORTURE_ALLOTED_CPUS="$TORTURE_ALLOTED_CPUS"; export TORTURE_ALLOTED_CPUS 295 289 TORTURE_BOOT_IMAGE="$TORTURE_BOOT_IMAGE"; export TORTURE_BOOT_IMAGE 296 290 TORTURE_BUILDONLY="$TORTURE_BUILDONLY"; export TORTURE_BUILDONLY 297 291 TORTURE_DEFCONFIG="$TORTURE_DEFCONFIG"; export TORTURE_DEFCONFIG ··· 305 297 TORTURE_QEMU_MEM="$TORTURE_QEMU_MEM"; export TORTURE_QEMU_MEM 306 298 TORTURE_SHUTDOWN_GRACE="$TORTURE_SHUTDOWN_GRACE"; export TORTURE_SHUTDOWN_GRACE 307 299 TORTURE_SUITE="$TORTURE_SUITE"; export TORTURE_SUITE 300 + TORTURE_TRUST_MAKE="$TORTURE_TRUST_MAKE"; export TORTURE_TRUST_MAKE 308 301 if ! test -e $resdir 309 302 then 310 303 mkdir -p "$resdir" || : ··· 351 342 print "needqemurun=" 352 343 jn=1 353 344 for (j = first; j < pastlast; j++) { 354 - builddir=KVM "/b1" 345 + builddir=KVM "/b" j - first + 1 355 346 cpusr[jn] = cpus[j]; 356 347 if (cfrep[cf[j]] == "") { 357 348 cfr[jn] = cf[j]; ··· 367 358 print "echo ", cfr[jn], cpusr[jn] ovf ": Starting build. `date` | tee -a " rd "log"; 368 359 print "rm -f " builddir ".*"; 369 360 print "touch " builddir ".wait"; 370 - print "mkdir " builddir " > /dev/null 2>&1 || :"; 371 361 print "mkdir " rd cfr[jn] " || :"; 372 362 print "kvm-test-1-run.sh " CONFIGDIR cf[j], builddir, rd cfr[jn], dur " \"" TORTURE_QEMU_ARG "\" \"" TORTURE_BOOTARGS "\" > " rd cfr[jn] "/kvm-test-1-run.sh.out 2>&1 &" 373 363 print "echo ", cfr[jn], cpusr[jn] ovf ": Waiting for build to complete. `date` | tee -a " rd "log"; ··· 472 464 fi 473 465 474 466 # Tracing: trace_event=rcu:rcu_grace_period,rcu:rcu_future_grace_period,rcu:rcu_grace_period_init,rcu:rcu_nocb_wake,rcu:rcu_preempt_task,rcu:rcu_unlock_preempted_task,rcu:rcu_quiescent_state_report,rcu:rcu_fqs,rcu:rcu_callback,rcu:rcu_kfree_callback,rcu:rcu_batch_start,rcu:rcu_invoke_callback,rcu:rcu_invoke_kfree_callback,rcu:rcu_batch_end,rcu:rcu_torture_read,rcu:rcu_barrier 467 + # Function-graph tracing: ftrace=function_graph ftrace_graph_filter=sched_setaffinity,migration_cpu_stop 468 + # Also --kconfig "CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y"

+1 -1

tools/testing/selftests/rcutorture/bin/parse-build.sh

··· 21 21 22 22 . functions.sh 23 23 24 - if grep -q CC < $F 24 + if grep -q CC < $F || test -n "$TORTURE_TRUST_MAKE" 25 25 then 26 26 : 27 27 else

+1

tools/testing/selftests/rcutorture/bin/parse-console.sh

+3

tools/testing/selftests/rcutorture/configs/rcu/CFcommon

··· 1 1 CONFIG_RCU_TORTURE_TEST=y 2 2 CONFIG_PRINTK_TIME=y 3 + CONFIG_HYPERVISOR_GUEST=y 4 + CONFIG_PARAVIRT=y 5 + CONFIG_KVM_GUEST=y

+1

tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot

··· 3 3 rcutree.gp_init_delay=3 4 4 rcutree.gp_cleanup_delay=3 5 5 rcu_nocbs=0 6 + rcutorture.fwd_progress=0

+14

tools/testing/selftests/rcutorture/configs/rcu/TRIVIAL

··· 1 + CONFIG_SMP=y 2 + CONFIG_NR_CPUS=8 3 + CONFIG_PREEMPT_NONE=y 4 + CONFIG_PREEMPT_VOLUNTARY=n 5 + CONFIG_PREEMPT=n 6 + CONFIG_HZ_PERIODIC=n 7 + CONFIG_NO_HZ_IDLE=y 8 + CONFIG_NO_HZ_FULL=n 9 + CONFIG_HOTPLUG_CPU=n 10 + CONFIG_SUSPEND=n 11 + CONFIG_HIBERNATION=n 12 + CONFIG_DEBUG_LOCK_ALLOC=n 13 + CONFIG_DEBUG_OBJECTS_RCU_HEAD=n 14 + CONFIG_RCU_EXPERT=y

+3

tools/testing/selftests/rcutorture/configs/rcu/TRIVIAL.boot

··· 1 + rcutorture.torture_type=trivial 2 + rcutorture.onoff_interval=0 3 + rcutorture.shuffle_interval=0

Configure Feed

Configure Feed