tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'rcu.2026.03.31a' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux

Pull RCU updates from Joel Fernandes:
"NOCB CPU management:

- Consolidate rcu_nocb_cpu_offload() and rcu_nocb_cpu_deoffload() to
reduce code duplication

- Extract nocb_bypass_needs_flush() helper to reduce duplication in
NOCB bypass path

rcutorture/torture infrastructure:

- Add NOCB01 config for RCU_LAZY torture testing

- Add NOCB02 config for NOCB poll mode testing

- Add TRIVIAL-PREEMPT config for textbook-style preemptible RCU
torture

- Test call_srcu() with preemption both disabled and enabled

- Remove kvm-check-branches.sh in favor of kvm-series.sh

- Make hangs more visible in torture.sh output

- Add informative message for tests without a recheck file

- Fix numeric test comparison in srcu_lockdep.sh

- Use torture_shutdown_init() in refscale and rcuscale instead of
open-coded shutdown functions

- Fix modulo-zero error in torture_hrtimeout_ns().

SRCU:

- Fix SRCU read flavor macro comments

- Fix s/they disables/they disable/ typo in srcu_read_unlock_fast()

RCU Tasks:

- Document that RCU Tasks Trace grace periods now imply RCU grace
periods

- Remove unnecessary smp_store_release() in cblist_init_generic()"

* tag 'rcu.2026.03.31a' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux:
rcutorture: Test call_srcu() with preemption disabled and not
rcu: Add BOOTPARAM_RCU_STALL_PANIC Kconfig option
torture: Avoid modulo-zero error in torture_hrtimeout_ns()
rcu/nocb: Extract nocb_bypass_needs_flush() to reduce duplication
rcu/nocb: Consolidate rcu_nocb_cpu_offload/deoffload functions
rcu-tasks: Remove unnecessary smp_store_release() in cblist_init_generic()
rcutorture: Add NOCB02 config for nocb poll mode testing
rcutorture: Add NOCB01 config for RCU_LAZY torture testing
rcu-tasks: Document that RCU Tasks Trace grace periods now imply RCU grace periods
srcu: Fix s/they disables/they disable/ typo in srcu_read_unlock_fast()
srcu: Fix SRCU read flavor macro comments
rcuscale: Ditch rcu_scale_shutdown in favor of torture_shutdown_init()
refscale: Ditch ref_scale_shutdown in favor of torture_shutdown_init()
rcutorture: Fix numeric "test" comparison in srcu_lockdep.sh
torture: Print informative message for test without recheck file
torture: Make hangs more visible in torture.sh output
kvm-check-branches.sh: Remove in favor of kvm-series.sh
rcutorture: Add a textbook-style trivial preemptible RCU

Linus Torvalds 28483203 599bbba5

+301 -267
+7
Documentation/RCU/Design/Requirements/Requirements.rst
··· 2787 2787 that apply noinstr to kernel entry/exit code (or that build with 2788 2788 ``CONFIG_TASKS_TRACE_RCU_NO_MB=y``. 2789 2789 2790 + Now that the implementation is based on SRCU-fast, a call 2791 + to synchronize_rcu_tasks_trace() implies at least one call to 2792 + synchronize_rcu(), that is, every Tasks Trace RCU grace period contains 2793 + at least one plain vanilla RCU grace period. Should there ever 2794 + be a synchronize_rcu_tasks_trace_expedited(), this guarantee would 2795 + *not* necessarily apply to this hypothetical API member. 2796 + 2790 2797 The tasks-trace-RCU API is also reasonably compact, 2791 2798 consisting of rcu_read_lock_trace(), rcu_read_unlock_trace(), 2792 2799 rcu_read_lock_trace_held(), call_rcu_tasks_trace(),
+3 -6
include/linux/rcupdate.h
··· 208 208 /** 209 209 * rcu_trace_implies_rcu_gp - does an RCU Tasks Trace grace period imply an RCU grace period? 210 210 * 211 - * As an accident of implementation, an RCU Tasks Trace grace period also 212 - * acts as an RCU grace period. However, this could change at any time. 213 - * Code relying on this accident must call this function to verify that 214 - * this accident is still happening. 215 - * 216 - * You have been warned! 211 + * Now that RCU Tasks Trace is implemented in terms of SRCU-fast, a 212 + * call to synchronize_rcu_tasks_trace() is guaranteed to imply at least 213 + * one call to synchronize_rcu(). 217 214 */ 218 215 static inline bool rcu_trace_implies_rcu_gp(void) { return true; } 219 216
+4
include/linux/sched.h
··· 949 949 struct srcu_ctr __percpu *trc_reader_scp; 950 950 #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ 951 951 952 + #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 953 + int rcu_trivial_preempt_nesting; 954 + #endif /* #ifdef CONFIG_TRIVIAL_PREEMPT_RCU */ 955 + 952 956 struct sched_info sched_info; 953 957 954 958 struct list_head tasks;
+2 -2
include/linux/srcu.h
··· 69 69 #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). 70 70 #define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). 71 71 // 0x4 // SRCU-lite is no longer with us. 72 - #define SRCU_READ_FLAVOR_FAST 0x4 // srcu_read_lock_fast(). 73 - #define SRCU_READ_FLAVOR_FAST_UPDOWN 0x8 // srcu_read_lock_fast(). 72 + #define SRCU_READ_FLAVOR_FAST 0x4 // srcu_read_lock_fast(), also NMI-safe. 73 + #define SRCU_READ_FLAVOR_FAST_UPDOWN 0x8 // srcu_read_lock_fast_updown(). 74 74 #define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \ 75 75 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN) 76 76 // All of the above.
+1 -1
include/linux/srcutree.h
··· 260 260 * srcu_read_unlock_fast(). 261 261 * 262 262 * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side 263 - * critical sections either because they disables interrupts, because 263 + * critical sections either because they disable interrupts, because 264 264 * they are a single instruction, or because they are read-modify-write 265 265 * atomic operations, depending on the whims of the architecture. 266 266 * This matters because the SRCU-fast grace-period mechanism uses either
+11
kernel/rcu/Kconfig.debug
··· 228 228 229 229 This has no value for production and is only for testing. 230 230 231 + config TRIVIAL_PREEMPT_RCU 232 + bool "Textbook trivial preemptible RCU in rcutorture" 233 + depends on RCU_EXPERT && RCU_TORTURE_TEST 234 + default n 235 + help 236 + This option enables a textbook preemptible RCU that is 237 + implemented in rcutorture. Its sole purpose is to validate 238 + code used in books, papers, and presentations. 239 + 240 + This has no value for production and is only for testing. 241 + 231 242 endmenu # "RCU Debugging"
+4
kernel/rcu/rcu.h
··· 691 691 static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { return NOTIFY_DONE; } 692 692 #endif // #else // #if defined(CONFIG_RCU_STALL_COMMON) && defined(CONFIG_RCU_CPU_STALL_NOTIFIER) 693 693 694 + #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 695 + void synchronize_rcu_trivial_preempt(void); 696 + #endif // #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 697 + 694 698 #endif /* __LINUX_RCU_H */
+21 -57
kernel/rcu/rcuscale.c
··· 79 79 * test-end checks, and the pair of calls through pointers. 80 80 */ 81 81 82 - #ifdef MODULE 83 - # define RCUSCALE_SHUTDOWN 0 84 - #else 85 - # define RCUSCALE_SHUTDOWN 1 86 - #endif 87 - 88 82 torture_param(bool, gp_async, false, "Use asynchronous GP wait primitives"); 89 83 torture_param(int, gp_async_max, 1000, "Max # outstanding waits per writer"); 90 84 torture_param(bool, gp_exp, false, "Use expedited GP wait primitives"); ··· 86 92 torture_param(int, minruntime, 0, "Minimum run time (s)"); 87 93 torture_param(int, nreaders, -1, "Number of RCU reader threads"); 88 94 torture_param(int, nwriters, -1, "Number of RCU updater threads"); 89 - torture_param(bool, shutdown, RCUSCALE_SHUTDOWN, 90 - "Shutdown at end of scalability tests."); 95 + torture_param(int, shutdown_secs, !IS_MODULE(CONFIG_RCU_SCALE_TEST) * 300, 96 + "Shutdown at end of scalability tests or at specified timeout (s)."); 91 97 torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); 92 98 torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); 93 99 torture_param(int, writer_holdoff_jiffies, 0, "Holdoff (jiffies) between GPs, zero to disable"); ··· 117 123 static int nrealwriters; 118 124 static struct task_struct **writer_tasks; 119 125 static struct task_struct **reader_tasks; 120 - static struct task_struct *shutdown_task; 121 126 122 127 static u64 **writer_durations; 123 128 static bool *writer_done; ··· 125 132 static atomic_t n_rcu_scale_reader_started; 126 133 static atomic_t n_rcu_scale_writer_started; 127 134 static atomic_t n_rcu_scale_writer_finished; 128 - static wait_queue_head_t shutdown_wq; 129 135 static u64 t_rcu_scale_writer_started; 130 136 static u64 t_rcu_scale_writer_finished; 131 137 static unsigned long b_rcu_gp_test_started; ··· 511 519 rcu_scale_free(wmbp); 512 520 } 513 521 522 + static void rcu_scale_cleanup(void); 523 + 514 524 /* 515 525 * RCU scale writer kthread. Repeatedly does a grace period. 516 526 */ ··· 616 622 b_rcu_gp_test_finished = 617 623 cur_ops->get_gp_seq(); 618 624 } 619 - if (shutdown) { 625 + if (shutdown_secs) { 626 + writer_tasks[me] = NULL; 620 627 smp_mb(); /* Assign before wake. */ 621 - wake_up(&shutdown_wq); 628 + rcu_scale_cleanup(); 629 + kernel_power_off(); 622 630 } 623 631 } 624 632 } ··· 664 668 rcu_scale_print_module_parms(struct rcu_scale_ops *cur_ops, const char *tag) 665 669 { 666 670 pr_alert("%s" SCALE_FLAG 667 - "--- %s: gp_async=%d gp_async_max=%d gp_exp=%d holdoff=%d minruntime=%d nreaders=%d nwriters=%d writer_holdoff=%d writer_holdoff_jiffies=%d verbose=%d shutdown=%d\n", 668 - scale_type, tag, gp_async, gp_async_max, gp_exp, holdoff, minruntime, nrealreaders, nrealwriters, writer_holdoff, writer_holdoff_jiffies, verbose, shutdown); 671 + "--- %s: gp_async=%d gp_async_max=%d gp_exp=%d holdoff=%d minruntime=%d nreaders=%d nwriters=%d writer_holdoff=%d writer_holdoff_jiffies=%d verbose=%d shutdown_secs=%d\n", 672 + scale_type, tag, gp_async, gp_async_max, gp_exp, holdoff, minruntime, nrealreaders, nrealwriters, writer_holdoff, writer_holdoff_jiffies, verbose, shutdown_secs); 669 673 } 670 674 671 675 /* ··· 717 721 718 722 kfree(obj); 719 723 } 724 + 725 + static void kfree_scale_cleanup(void); 720 726 721 727 static int 722 728 kfree_scale_thread(void *arg) ··· 789 791 rcuscale_seq_diff(b_rcu_gp_test_finished, b_rcu_gp_test_started), 790 792 PAGES_TO_MB(mem_begin - mem_during)); 791 793 792 - if (shutdown) { 794 + if (shutdown_secs) { 795 + kfree_reader_tasks[me] = NULL; 793 796 smp_mb(); /* Assign before wake. */ 794 - wake_up(&shutdown_wq); 797 + kfree_scale_cleanup(); 798 + kernel_power_off(); 795 799 } 796 800 } 797 801 ··· 818 818 } 819 819 820 820 torture_cleanup_end(); 821 - } 822 - 823 - /* 824 - * shutdown kthread. Just waits to be awakened, then shuts down system. 825 - */ 826 - static int 827 - kfree_scale_shutdown(void *arg) 828 - { 829 - wait_event_idle(shutdown_wq, 830 - atomic_read(&n_kfree_scale_thread_ended) >= kfree_nrealthreads); 831 - 832 - smp_mb(); /* Wake before output. */ 833 - 834 - kfree_scale_cleanup(); 835 - kernel_power_off(); 836 - return -EINVAL; 837 821 } 838 822 839 823 // Used if doing RCU-kfree'ing via call_rcu(). ··· 879 895 880 896 kfree_nrealthreads = compute_real(kfree_nthreads); 881 897 /* Start up the kthreads. */ 882 - if (shutdown) { 883 - init_waitqueue_head(&shutdown_wq); 884 - firsterr = torture_create_kthread(kfree_scale_shutdown, NULL, 885 - shutdown_task); 898 + if (shutdown_secs) { 899 + firsterr = torture_shutdown_init(shutdown_secs, kfree_scale_cleanup); 886 900 if (torture_init_error(firsterr)) 887 901 goto unwind; 888 - schedule_timeout_uninterruptible(1); 889 902 } 890 903 891 904 pr_alert("kfree object size=%zu, kfree_by_call_rcu=%d\n", ··· 1039 1058 torture_cleanup_end(); 1040 1059 } 1041 1060 1042 - /* 1043 - * RCU scalability shutdown kthread. Just waits to be awakened, then shuts 1044 - * down system. 1045 - */ 1046 - static int 1047 - rcu_scale_shutdown(void *arg) 1048 - { 1049 - wait_event_idle(shutdown_wq, atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters); 1050 - smp_mb(); /* Wake before output. */ 1051 - rcu_scale_cleanup(); 1052 - kernel_power_off(); 1053 - return -EINVAL; 1054 - } 1055 - 1056 1061 static int __init 1057 1062 rcu_scale_init(void) 1058 1063 { ··· 1088 1121 1089 1122 /* Start up the kthreads. */ 1090 1123 1091 - if (shutdown) { 1092 - init_waitqueue_head(&shutdown_wq); 1093 - firsterr = torture_create_kthread(rcu_scale_shutdown, NULL, 1094 - shutdown_task); 1124 + if (shutdown_secs) { 1125 + firsterr = torture_shutdown_init(shutdown_secs, rcu_scale_cleanup); 1095 1126 if (torture_init_error(firsterr)) 1096 1127 goto unwind; 1097 - schedule_timeout_uninterruptible(1); 1098 1128 } 1099 1129 reader_tasks = kzalloc_objs(reader_tasks[0], nrealreaders); 1100 1130 if (reader_tasks == NULL) { ··· 1165 1201 unwind: 1166 1202 torture_init_end(); 1167 1203 rcu_scale_cleanup(); 1168 - if (shutdown) { 1204 + if (shutdown_secs) { 1169 1205 WARN_ON(!IS_MODULE(CONFIG_RCU_SCALE_TEST)); 1170 1206 kernel_power_off(); 1171 1207 }
+63 -1
kernel/rcu/rcutorture.c
··· 842 842 843 843 static void srcu_torture_deferred_free(struct rcu_torture *rp) 844 844 { 845 + unsigned long flags; 846 + bool lockit = jiffies & 0x1; 847 + 848 + if (lockit) 849 + raw_spin_lock_irqsave(&current->pi_lock, flags); 845 850 call_srcu(srcu_ctlp, &rp->rtort_rcu, rcu_torture_cb); 851 + if (lockit) 852 + raw_spin_unlock_irqrestore(&current->pi_lock, flags); 846 853 } 847 854 848 855 static void srcu_torture_synchronize(void) ··· 1067 1060 .irq_capable = 1, 1068 1061 .name = "trivial" 1069 1062 }; 1063 + 1064 + #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 1065 + 1066 + /* 1067 + * Definitions for trivial CONFIG_PREEMPT=y torture testing. This 1068 + * implementation does not work well with large numbers of tasks or with 1069 + * long-term preemption. Either or both get you RCU CPU stall warnings. 1070 + */ 1071 + 1072 + static void rcu_sync_torture_init_trivial_preempt(void) 1073 + { 1074 + rcu_sync_torture_init(); 1075 + if (WARN_ONCE(onoff_interval || shuffle_interval, "%s: Non-zero onoff_interval (%d) or shuffle_interval (%d) breaks trivial RCU, resetting to zero", __func__, onoff_interval, shuffle_interval)) { 1076 + onoff_interval = 0; 1077 + shuffle_interval = 0; 1078 + } 1079 + } 1080 + 1081 + static int rcu_torture_read_lock_trivial_preempt(void) 1082 + { 1083 + struct task_struct *t = current; 1084 + 1085 + WRITE_ONCE(t->rcu_trivial_preempt_nesting, t->rcu_trivial_preempt_nesting + 1); 1086 + smp_mb(); 1087 + return 0; 1088 + } 1089 + 1090 + static void rcu_torture_read_unlock_trivial_preempt(int idx) 1091 + { 1092 + struct task_struct *t = current; 1093 + 1094 + smp_store_release(&t->rcu_trivial_preempt_nesting, t->rcu_trivial_preempt_nesting - 1); 1095 + } 1096 + 1097 + static struct rcu_torture_ops trivial_preempt_ops = { 1098 + .ttype = RCU_TRIVIAL_FLAVOR, 1099 + .init = rcu_sync_torture_init_trivial_preempt, 1100 + .readlock = rcu_torture_read_lock_trivial_preempt, 1101 + .read_delay = rcu_read_delay, // just reuse rcu's version. 1102 + .readunlock = rcu_torture_read_unlock_trivial_preempt, 1103 + .readlock_held = torture_readlock_not_held, 1104 + .get_gp_seq = rcu_no_completed, 1105 + .sync = synchronize_rcu_trivial_preempt, 1106 + .exp_sync = synchronize_rcu_trivial_preempt, 1107 + .irq_capable = 0, // In theory it should be, but let's keep it trivial. 1108 + .name = "trivial-preempt" 1109 + }; 1110 + 1111 + #define TRIVIAL_PREEMPT_OPS &trivial_preempt_ops, 1112 + 1113 + #else // #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 1114 + 1115 + #define TRIVIAL_PREEMPT_OPS 1116 + 1117 + #endif // #else // #ifdef CONFIG_TRIVIAL_PREEMPT_RCU 1070 1118 1071 1119 #ifdef CONFIG_TASKS_RCU 1072 1120 ··· 4511 4449 static struct rcu_torture_ops *torture_ops[] = { 4512 4450 &rcu_ops, &rcu_busted_ops, &srcu_ops, &srcud_ops, &busted_srcud_ops, 4513 4451 TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS 4514 - &trivial_ops, 4452 + &trivial_ops, TRIVIAL_PREEMPT_OPS 4515 4453 }; 4516 4454 4517 4455 if (!torture_init_begin(torture_type, verbose))
+14 -37
kernel/rcu/refscale.c
··· 92 92 torture_param(int, nruns, 30, "Number of experiments to run."); 93 93 // Reader delay in nanoseconds, 0 for no delay. 94 94 torture_param(int, readdelay, 0, "Read-side delay in nanoseconds."); 95 - 96 - #ifdef MODULE 97 - # define REFSCALE_SHUTDOWN 0 98 - #else 99 - # define REFSCALE_SHUTDOWN 1 100 - #endif 101 - 102 - torture_param(bool, shutdown, REFSCALE_SHUTDOWN, 103 - "Shutdown at end of scalability tests."); 95 + // Maximum shutdown delay in seconds, or zero for no shutdown. 96 + torture_param(int, shutdown_secs, !IS_MODULE(CONFIG_REPRO_TEST) * 300, 97 + "Shutdown at end of scalability tests or at specified timeout (s)."); 104 98 105 99 struct reader_task { 106 100 struct task_struct *task; ··· 103 109 u64 last_duration_ns; 104 110 }; 105 111 106 - static struct task_struct *shutdown_task; 107 - static wait_queue_head_t shutdown_wq; 108 - 109 112 static struct task_struct *main_task; 110 113 static wait_queue_head_t main_wq; 111 - static int shutdown_start; 112 114 113 115 static struct reader_task *reader_tasks; 114 116 ··· 1347 1357 return sum; 1348 1358 } 1349 1359 1360 + static void ref_scale_cleanup(void); 1361 + 1350 1362 // The main_func is the main orchestrator, it performs a bunch of 1351 1363 // experiments. For every experiment, it orders all the readers 1352 1364 // involved to start and waits for them to finish the experiment. It ··· 1435 1443 1436 1444 oom_exit: 1437 1445 // This will shutdown everything including us. 1438 - if (shutdown) { 1439 - shutdown_start = 1; 1440 - wake_up(&shutdown_wq); 1446 + if (shutdown_secs) { 1447 + main_task = NULL; // Avoid self-kill deadlock. 1448 + ref_scale_cleanup(); 1449 + kernel_power_off(); 1441 1450 } 1442 1451 1443 1452 // Wait for torture to stop us ··· 1456 1463 ref_scale_print_module_parms(const struct ref_scale_ops *cur_ops, const char *tag) 1457 1464 { 1458 1465 pr_alert("%s" SCALE_FLAG 1459 - "--- %s: verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%d nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag, 1460 - verbose, verbose_batched, shutdown, holdoff, lookup_instances, loops, nreaders, nruns, readdelay); 1466 + "--- %s: verbose=%d verbose_batched=%d shutdown_secs=%d holdoff=%d lookup_instances=%ld loops=%d nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag, 1467 + verbose, verbose_batched, shutdown_secs, holdoff, lookup_instances, loops, nreaders, nruns, readdelay); 1461 1468 } 1462 1469 1463 1470 static void ··· 1488 1495 cur_ops->cleanup(); 1489 1496 1490 1497 torture_cleanup_end(); 1491 - } 1492 - 1493 - // Shutdown kthread. Just waits to be awakened, then shuts down system. 1494 - static int 1495 - ref_scale_shutdown(void *arg) 1496 - { 1497 - wait_event_idle(shutdown_wq, shutdown_start); 1498 - 1499 - smp_mb(); // Wake before output. 1500 - ref_scale_cleanup(); 1501 - kernel_power_off(); 1502 - 1503 - return -EINVAL; 1504 1498 } 1505 1499 1506 1500 static int __init ··· 1533 1553 ref_scale_print_module_parms(cur_ops, "Start of test"); 1534 1554 1535 1555 // Shutdown task 1536 - if (shutdown) { 1537 - init_waitqueue_head(&shutdown_wq); 1538 - firsterr = torture_create_kthread(ref_scale_shutdown, NULL, 1539 - shutdown_task); 1556 + if (shutdown_secs) { 1557 + firsterr = torture_shutdown_init(shutdown_secs, ref_scale_cleanup); 1540 1558 if (torture_init_error(firsterr)) 1541 1559 goto unwind; 1542 - schedule_timeout_uninterruptible(1); 1543 1560 } 1544 1561 1545 1562 // Reader tasks (default to ~75% of online CPUs). ··· 1581 1604 unwind: 1582 1605 torture_init_end(); 1583 1606 ref_scale_cleanup(); 1584 - if (shutdown) { 1607 + if (shutdown_secs) { 1585 1608 WARN_ON(!IS_MODULE(CONFIG_RCU_REF_SCALE_TEST)); 1586 1609 kernel_power_off(); 1587 1610 }
+3 -3
kernel/rcu/tasks.h
··· 291 291 shift = ilog2(rcu_task_cpu_ids / lim); 292 292 if (((rcu_task_cpu_ids - 1) >> shift) >= lim) 293 293 shift++; 294 - WRITE_ONCE(rtp->percpu_enqueue_shift, shift); 295 - WRITE_ONCE(rtp->percpu_dequeue_lim, lim); 296 - smp_store_release(&rtp->percpu_enqueue_lim, lim); 294 + rtp->percpu_enqueue_shift = shift; 295 + rtp->percpu_dequeue_lim = lim; 296 + rtp->percpu_enqueue_lim = lim; 297 297 298 298 pr_info("%s: Setting shift to %d and lim to %d rcu_task_cb_adjust=%d rcu_task_cpu_ids=%d.\n", 299 299 rtp->name, data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim),
+72 -49
kernel/rcu/tree_nocb.h
··· 379 379 } 380 380 381 381 /* 382 + * Determine if the bypass queue needs to be flushed based on time and size. 383 + * For lazy-only bypass queues, use the lazy flush timeout; otherwise flush 384 + * based on jiffy advancement. The flush_faster controls flush aggressiveness. 385 + */ 386 + static bool nocb_bypass_needs_flush(struct rcu_data *rdp, long bypass_ncbs, 387 + long lazy_ncbs, unsigned long j, 388 + bool flush_faster) 389 + { 390 + bool bypass_is_lazy; 391 + unsigned long bypass_first; 392 + unsigned long flush_timeout; 393 + long qhimark_thresh; 394 + 395 + if (!bypass_ncbs) 396 + return false; 397 + 398 + qhimark_thresh = flush_faster ? qhimark : 2 * qhimark; 399 + if (bypass_ncbs >= qhimark_thresh) 400 + return true; 401 + 402 + bypass_first = READ_ONCE(rdp->nocb_bypass_first); 403 + bypass_is_lazy = (bypass_ncbs == lazy_ncbs); 404 + 405 + if (bypass_is_lazy) 406 + flush_timeout = rcu_get_jiffies_lazy_flush(); 407 + else 408 + flush_timeout = flush_faster ? 0 : 1; 409 + 410 + return time_after(j, bypass_first + flush_timeout); 411 + } 412 + 413 + /* 382 414 * See whether it is appropriate to use the ->nocb_bypass list in order 383 415 * to control contention on ->nocb_lock. A limited number of direct 384 416 * enqueues are permitted into ->cblist per jiffy. If ->nocb_bypass ··· 436 404 unsigned long cur_gp_seq; 437 405 unsigned long j = jiffies; 438 406 long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); 439 - bool bypass_is_lazy = (ncbs == READ_ONCE(rdp->lazy_len)); 407 + long lazy_len = READ_ONCE(rdp->lazy_len); 408 + bool bypass_is_lazy = (ncbs == lazy_len); 440 409 441 410 lockdep_assert_irqs_disabled(); 442 411 ··· 489 456 490 457 // If ->nocb_bypass has been used too long or is too full, 491 458 // flush ->nocb_bypass to ->cblist. 492 - if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || 493 - (ncbs && bypass_is_lazy && 494 - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + rcu_get_jiffies_lazy_flush()))) || 495 - ncbs >= qhimark) { 459 + if (nocb_bypass_needs_flush(rdp, ncbs, lazy_len, j, true)) { 496 460 rcu_nocb_lock(rdp); 497 461 *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); 498 462 ··· 703 673 bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); 704 674 lazy_ncbs = READ_ONCE(rdp->lazy_len); 705 675 706 - if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && 707 - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + rcu_get_jiffies_lazy_flush()) || 708 - bypass_ncbs > 2 * qhimark)) { 709 - flush_bypass = true; 710 - } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && 711 - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || 712 - bypass_ncbs > 2 * qhimark)) { 713 - flush_bypass = true; 714 - } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { 676 + flush_bypass = nocb_bypass_needs_flush(rdp, bypass_ncbs, lazy_ncbs, j, false); 677 + if (!flush_bypass && !bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { 715 678 rcu_nocb_unlock_irqrestore(rdp, flags); 716 679 continue; /* No callbacks here, try next. */ 717 680 } ··· 1104 1081 return 0; 1105 1082 } 1106 1083 1107 - int rcu_nocb_cpu_deoffload(int cpu) 1108 - { 1109 - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); 1110 - int ret = 0; 1111 - 1112 - cpus_read_lock(); 1113 - mutex_lock(&rcu_state.nocb_mutex); 1114 - if (rcu_rdp_is_offloaded(rdp)) { 1115 - if (!cpu_online(cpu)) { 1116 - ret = rcu_nocb_rdp_deoffload(rdp); 1117 - if (!ret) 1118 - cpumask_clear_cpu(cpu, rcu_nocb_mask); 1119 - } else { 1120 - pr_info("NOCB: Cannot CB-deoffload online CPU %d\n", rdp->cpu); 1121 - ret = -EINVAL; 1122 - } 1123 - } 1124 - mutex_unlock(&rcu_state.nocb_mutex); 1125 - cpus_read_unlock(); 1126 - 1127 - return ret; 1128 - } 1129 - EXPORT_SYMBOL_GPL(rcu_nocb_cpu_deoffload); 1130 - 1131 1084 static bool rcu_nocb_rdp_offload_wait_cond(struct rcu_data *rdp) 1132 1085 { 1133 1086 unsigned long flags; ··· 1148 1149 return 0; 1149 1150 } 1150 1151 1151 - int rcu_nocb_cpu_offload(int cpu) 1152 + /* Common helper for CPU offload/deoffload operations. */ 1153 + static int rcu_nocb_cpu_toggle_offload(int cpu, bool offload) 1152 1154 { 1153 1155 struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); 1154 1156 int ret = 0; 1155 1157 1156 1158 cpus_read_lock(); 1157 1159 mutex_lock(&rcu_state.nocb_mutex); 1158 - if (!rcu_rdp_is_offloaded(rdp)) { 1159 - if (!cpu_online(cpu)) { 1160 - ret = rcu_nocb_rdp_offload(rdp); 1161 - if (!ret) 1162 - cpumask_set_cpu(cpu, rcu_nocb_mask); 1163 - } else { 1164 - pr_info("NOCB: Cannot CB-offload online CPU %d\n", rdp->cpu); 1165 - ret = -EINVAL; 1166 - } 1160 + 1161 + /* Already in desired state, nothing to do. */ 1162 + if (rcu_rdp_is_offloaded(rdp) == offload) 1163 + goto out_unlock; 1164 + 1165 + if (cpu_online(cpu)) { 1166 + pr_info("NOCB: Cannot CB-%soffload online CPU %d\n", 1167 + offload ? "" : "de", rdp->cpu); 1168 + ret = -EINVAL; 1169 + goto out_unlock; 1167 1170 } 1171 + 1172 + if (offload) { 1173 + ret = rcu_nocb_rdp_offload(rdp); 1174 + if (!ret) 1175 + cpumask_set_cpu(cpu, rcu_nocb_mask); 1176 + } else { 1177 + ret = rcu_nocb_rdp_deoffload(rdp); 1178 + if (!ret) 1179 + cpumask_clear_cpu(cpu, rcu_nocb_mask); 1180 + } 1181 + 1182 + out_unlock: 1168 1183 mutex_unlock(&rcu_state.nocb_mutex); 1169 1184 cpus_read_unlock(); 1170 - 1171 1185 return ret; 1186 + } 1187 + 1188 + int rcu_nocb_cpu_deoffload(int cpu) 1189 + { 1190 + return rcu_nocb_cpu_toggle_offload(cpu, false /* de-offload */); 1191 + } 1192 + EXPORT_SYMBOL_GPL(rcu_nocb_cpu_deoffload); 1193 + 1194 + int rcu_nocb_cpu_offload(int cpu) 1195 + { 1196 + return rcu_nocb_cpu_toggle_offload(cpu, true /* offload */); 1172 1197 } 1173 1198 EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); 1174 1199
+22
kernel/rcu/update.c
··· 538 538 EXPORT_SYMBOL_GPL(torture_sched_setaffinity); 539 539 #endif 540 540 541 + #if IS_ENABLED(CONFIG_TRIVIAL_PREEMPT_RCU) 542 + // Trivial and stupid grace-period wait. Defined here so that lockdep 543 + // kernels can find tasklist_lock. 544 + void synchronize_rcu_trivial_preempt(void) 545 + { 546 + struct task_struct *g; 547 + struct task_struct *t; 548 + 549 + smp_mb(); // Order prior accesses before grace-period start. 550 + rcu_read_lock(); // Protect task list. 551 + for_each_process_thread(g, t) { 552 + if (t == current) 553 + continue; // Don't deadlock on ourselves! 554 + // Order later rcu_read_lock() on other tasks after QS. 555 + while (smp_load_acquire(&t->rcu_trivial_preempt_nesting)) 556 + continue; 557 + } 558 + rcu_read_unlock(); 559 + } 560 + EXPORT_SYMBOL_GPL(synchronize_rcu_trivial_preempt); 561 + #endif // #if IS_ENABLED(CONFIG_TRIVIAL_PREEMPT_RCU) 562 + 541 563 int rcu_cpu_stall_notifiers __read_mostly; // !0 = provide stall notifiers (rarely useful) 542 564 EXPORT_SYMBOL_GPL(rcu_cpu_stall_notifiers); 543 565
+1 -1
kernel/torture.c
··· 93 93 { 94 94 ktime_t hto = baset_ns; 95 95 96 - if (trsp) 96 + if (trsp && fuzzt_ns) 97 97 hto += torture_random(trsp) % fuzzt_ns; 98 98 set_current_state(TASK_IDLE); 99 99 return schedule_hrtimeout(&hto, mode);
-102
tools/testing/selftests/rcutorture/bin/kvm-check-branches.sh
··· 1 - #!/bin/sh 2 - # SPDX-License-Identifier: GPL-2.0+ 3 - # 4 - # Run a group of kvm.sh tests on the specified commits. This currently 5 - # unconditionally does three-minute runs on each scenario in CFLIST, 6 - # taking advantage of all available CPUs and trusting the "make" utility. 7 - # In the short term, adjustments can be made by editing this script and 8 - # CFLIST. If some adjustments appear to have ongoing value, this script 9 - # might grow some command-line arguments. 10 - # 11 - # Usage: kvm-check-branches.sh commit1 commit2..commit3 commit4 ... 12 - # 13 - # This script considers its arguments one at a time. If more elaborate 14 - # specification of commits is needed, please use "git rev-list" to 15 - # produce something that this simple script can understand. The reason 16 - # for retaining the simplicity is that it allows the user to more easily 17 - # see which commit came from which branch. 18 - # 19 - # This script creates a yyyy.mm.dd-hh.mm.ss-group entry in the "res" 20 - # directory. The calls to kvm.sh create the usual entries, but this script 21 - # moves them under the yyyy.mm.dd-hh.mm.ss-group entry, each in its own 22 - # directory numbered in run order, that is, "0001", "0002", and so on. 23 - # For successful runs, the large build artifacts are removed. Doing this 24 - # reduces the disk space required by about two orders of magnitude for 25 - # successful runs. 26 - # 27 - # Copyright (C) Facebook, 2020 28 - # 29 - # Authors: Paul E. McKenney <paulmck@kernel.org> 30 - 31 - if ! git status > /dev/null 2>&1 32 - then 33 - echo '!!!' This script needs to run in a git archive. 1>&2 34 - echo '!!!' Giving up. 1>&2 35 - exit 1 36 - fi 37 - 38 - # Remember where we started so that we can get back at the end. 39 - curcommit="`git status | head -1 | awk '{ print $NF }'`" 40 - 41 - nfail=0 42 - ntry=0 43 - resdir="tools/testing/selftests/rcutorture/res" 44 - ds="`date +%Y.%m.%d-%H.%M.%S`-group" 45 - if ! test -e $resdir 46 - then 47 - mkdir $resdir || : 48 - fi 49 - mkdir $resdir/$ds 50 - echo Results directory: $resdir/$ds 51 - 52 - RCUTORTURE="`pwd`/tools/testing/selftests/rcutorture"; export RCUTORTURE 53 - PATH=${RCUTORTURE}/bin:$PATH; export PATH 54 - . functions.sh 55 - echo Using all `identify_qemu_vcpus` CPUs. 56 - 57 - # Each pass through this loop does one command-line argument. 58 - for gitbr in $@ 59 - do 60 - echo ' --- git branch ' $gitbr 61 - 62 - # Each pass through this loop tests one commit. 63 - for i in `git rev-list "$gitbr"` 64 - do 65 - ntry=`expr $ntry + 1` 66 - idir=`awk -v ntry="$ntry" 'END { printf "%04d", ntry; }' < /dev/null` 67 - echo ' --- commit ' $i from branch $gitbr 68 - date 69 - mkdir $resdir/$ds/$idir 70 - echo $gitbr > $resdir/$ds/$idir/gitbr 71 - echo $i >> $resdir/$ds/$idir/gitbr 72 - 73 - # Test the specified commit. 74 - git checkout $i > $resdir/$ds/$idir/git-checkout.out 2>&1 75 - echo git checkout return code: $? "(Commit $ntry: $i)" 76 - kvm.sh --allcpus --duration 3 --trust-make --datestamp "$ds/$idir" > $resdir/$ds/$idir/kvm.sh.out 2>&1 77 - ret=$? 78 - echo kvm.sh return code $ret for commit $i from branch $gitbr 79 - echo Run results: $resdir/$ds/$idir 80 - if test "$ret" -ne 0 81 - then 82 - # Failure, so leave all evidence intact. 83 - nfail=`expr $nfail + 1` 84 - else 85 - # Success, so remove large files to save about 1GB. 86 - ( cd $resdir/$ds/$idir/$rrd; rm -f */vmlinux */bzImage */System.map */Module.symvers ) 87 - fi 88 - done 89 - done 90 - date 91 - 92 - # Go back to the original commit. 93 - git checkout "$curcommit" 94 - 95 - if test $nfail -ne 0 96 - then 97 - echo '!!! ' $nfail failures in $ntry 'runs!!!' 98 - exit 1 99 - else 100 - echo No failures in $ntry runs. 101 - exit 0 102 - fi
+7 -2
tools/testing/selftests/rcutorture/bin/kvm-recheck.sh
··· 39 39 X*) 40 40 ;; 41 41 *) 42 - kvm-recheck-${TORTURE_SUITE}.sh $i 42 + if test -f tools/testing/selftests/rcutorture/bin/kvm-recheck-${TORTURE_SUITE}.sh 43 + then 44 + kvm-recheck-${TORTURE_SUITE}.sh $i 45 + else 46 + echo No kvm-recheck-${TORTURE_SUITE}.sh, so no ${TORTURE_SUITE}-specific analysis. 47 + fi 43 48 esac 44 49 if test -f "$i/qemu-retval" && test "`cat $i/qemu-retval`" -ne 0 && test "`cat $i/qemu-retval`" -ne 137 45 50 then ··· 54 49 then 55 50 if test -f "$i/qemu-retval" && test "`cat $i/qemu-retval`" -eq 137 56 51 then 57 - echo QEMU killed 52 + echo "Summary: Potential hang (QEMU killed)" 58 53 fi 59 54 configcheck.sh $i/.config $i/ConfigFragment > $i/ConfigFragment.diags 2>&1 60 55 if grep -q '^CONFIG_KCSAN=y$' $i/ConfigFragment.input
-1
tools/testing/selftests/rcutorture/bin/parse-console.sh
··· 113 113 then 114 114 print_warning $title `cat $T.seq` 115 115 fi 116 - exit 2 117 116 fi 118 117 fi | tee -a $file.diags 119 118
+3 -3
tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh
··· 50 50 do 51 51 err= 52 52 val=$((d*1000+t*10+c)) 53 - tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs "SRCU-P" --kconfig "CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y" --bootargs "rcutorture.test_srcu_lockdep=$val rcutorture.reader_flavor=0x2" --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 53 + tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs "SRCU-P" --kconfig "CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y" --bootargs "rcutorture.test_srcu_lockdep=$val rcutorture.reader_flavor=0x4" --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 54 54 ret=$? 55 55 mv "$T/kvm.sh.out" "$RCUTORTURE/res/$ds/$val" 56 56 if ! grep -q '^CONFIG_PROVE_LOCKING=y' .config ··· 92 92 nerrs=$((nerrs+1)) 93 93 err=1 94 94 fi 95 - if test "$val" -eq 0xf && test "$ret" -eq 0 95 + if test "$val" = 0xf && test "$ret" -eq 0 96 96 then 97 97 err=1 98 98 echo -n Unexpected success for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err" 99 99 fi 100 - if test "$val" -eq 0x1 && test "$ret" -ne 0 100 + if test "$val" = 0x1 && test "$ret" -ne 0 101 101 then 102 102 err=1 103 103 echo -n Unexpected failure for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err"
+21
tools/testing/selftests/rcutorture/configs/rcu/NOCB01
··· 1 + CONFIG_SMP=y 2 + CONFIG_NR_CPUS=8 3 + CONFIG_PREEMPT_NONE=n 4 + CONFIG_PREEMPT_VOLUNTARY=n 5 + CONFIG_PREEMPT=y 6 + #CHECK#CONFIG_PREEMPT_RCU=y 7 + CONFIG_HZ_PERIODIC=n 8 + CONFIG_NO_HZ_IDLE=y 9 + CONFIG_NO_HZ_FULL=n 10 + CONFIG_RCU_TRACE=y 11 + CONFIG_HOTPLUG_CPU=y 12 + CONFIG_RCU_FANOUT=3 13 + CONFIG_RCU_FANOUT_LEAF=2 14 + CONFIG_RCU_NOCB_CPU=y 15 + CONFIG_DEBUG_LOCK_ALLOC=n 16 + CONFIG_PROVE_LOCKING=n 17 + CONFIG_RCU_BOOST=n 18 + CONFIG_DEBUG_OBJECTS_RCU_HEAD=n 19 + CONFIG_RCU_EXPERT=y 20 + CONFIG_RCU_EQS_DEBUG=y 21 + CONFIG_RCU_LAZY=y
+2
tools/testing/selftests/rcutorture/configs/rcu/NOCB01.boot
··· 1 + rcupdate.rcu_self_test=1 2 + rcu_nocbs=all
+20
tools/testing/selftests/rcutorture/configs/rcu/NOCB02
··· 1 + CONFIG_SMP=y 2 + CONFIG_NR_CPUS=8 3 + CONFIG_PREEMPT_NONE=n 4 + CONFIG_PREEMPT_VOLUNTARY=n 5 + CONFIG_PREEMPT=y 6 + #CHECK#CONFIG_PREEMPT_RCU=y 7 + CONFIG_HZ_PERIODIC=n 8 + CONFIG_NO_HZ_IDLE=y 9 + CONFIG_NO_HZ_FULL=n 10 + CONFIG_RCU_TRACE=y 11 + CONFIG_HOTPLUG_CPU=y 12 + CONFIG_RCU_FANOUT=3 13 + CONFIG_RCU_FANOUT_LEAF=2 14 + CONFIG_RCU_NOCB_CPU=y 15 + CONFIG_DEBUG_LOCK_ALLOC=n 16 + CONFIG_PROVE_LOCKING=n 17 + CONFIG_RCU_BOOST=n 18 + CONFIG_DEBUG_OBJECTS_RCU_HEAD=n 19 + CONFIG_RCU_EXPERT=y 20 + CONFIG_RCU_EQS_DEBUG=y
+3
tools/testing/selftests/rcutorture/configs/rcu/NOCB02.boot
··· 1 + rcupdate.rcu_self_test=1 2 + rcu_nocbs=all 3 + rcu_nocb_poll
+12
tools/testing/selftests/rcutorture/configs/rcu/TRIVIAL-PREEMPT
··· 1 + CONFIG_SMP=y 2 + CONFIG_NR_CPUS=8 3 + CONFIG_PREEMPT_NONE=n 4 + CONFIG_PREEMPT_VOLUNTARY=n 5 + CONFIG_PREEMPT=y 6 + CONFIG_HZ_PERIODIC=n 7 + CONFIG_NO_HZ_IDLE=y 8 + CONFIG_NO_HZ_FULL=n 9 + CONFIG_DEBUG_LOCK_ALLOC=n 10 + CONFIG_DEBUG_OBJECTS_RCU_HEAD=n 11 + CONFIG_RCU_EXPERT=y 12 + CONFIG_TRIVIAL_PREEMPT_RCU=y
+3
tools/testing/selftests/rcutorture/configs/rcu/TRIVIAL-PREEMPT.boot
··· 1 + rcutorture.torture_type=trivial-preempt 2 + rcutorture.onoff_interval=0 3 + rcutorture.shuffle_interval=0
+1 -1
tools/testing/selftests/rcutorture/configs/rcuscale/ver_functions.sh
··· 11 11 # 12 12 # Adds per-version torture-module parameters to kernels supporting them. 13 13 per_version_boot_params () { 14 - echo rcuscale.shutdown=1 \ 14 + echo rcuscale.shutdown_secs=$3 \ 15 15 rcuscale.verbose=0 \ 16 16 $1 17 17 }
+1 -1
tools/testing/selftests/rcutorture/configs/refscale/ver_functions.sh
··· 11 11 # 12 12 # Adds per-version torture-module parameters to kernels supporting them. 13 13 per_version_boot_params () { 14 - echo refscale.shutdown=1 \ 14 + echo refscale.shutdown_secs=$3 \ 15 15 refscale.verbose=0 \ 16 16 $1 17 17 }