Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'rcu.release.v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux
Pull RCU updates from Frederic Weisbecker:
"SRCU:
- Introduction of the new SRCU-lite flavour with a new pair of
srcu_read_[un]lock_lite() APIs. In practice the read side using
this flavour becomes lighter by removing a full memory barrier on
LOCK and a full memory barrier on UNLOCK. This comes at the expense
of a higher latency write side with two (in the best case of a
snaphot of unused read-sides) or more RCU grace periods on the
update side which now assumes by itself the whole full ordering
guarantee against the LOCK/UNLOCK counters on both indexes, along
with the accesses performed inside.
Uretprobes is a known potential user.
Note this doesn't replace the default normal flavour of SRCU which
still behaves the same as usual.
- Add testing of SRCU-lite through rcutorture and rcuscale
- Various cleanups on the way.
Fixes:
- Allow short-circuiting RCU-TASKS-RUDE grace periods on
architectures that have sane noinstr boundaries forbidding tracing
on low-level idle and kernel entry code. RCU-TASKS is enough on
such configurations because it involves an RCU grace period that
waits for all idle tasks to either schedule out voluntarily or
enter into RCU unwatched noinstr code.
- Allow and test start_poll_synchronize_rcu() with IRQs disabled.
- Mention rcuog kthreads in relevant documentation and Kconfig help
- Various fixes and consolidations
rcutorture:
- Add --no-affinity on tools to leave the affinity setting of guests
up to the user.
- Add guest_os_delay parameter to rcuscale for better warm-up
control.
- Fix and improve some rcuscale error handling.
- Various cleanups and fixes
stall:
- Remove dead code
- Stop dumping tasks if a stalled grace period eventually ended
midway as that only produces confusing output.
- Optimize detection of stalling CPUs and avoid useless node locking
otherwise.
NOCB:
- Fix rcu_barrier() hang due to a race against callbacks
deoffloading. This is not yet used, except by rcutorture, and waits
for its promised cpusets interface.
- Remove leftover function declaration"
* tag 'rcu.release.v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (42 commits)
rcuscale: Remove redundant WARN_ON_ONCE() splat
rcuscale: Do a proper cleanup if kfree_scale_init() fails
srcu: Unconditionally record srcu_read_lock_lite() in ->srcu_reader_flavor
srcu: Check for srcu_read_lock_lite() across all CPUs
srcu: Remove smp_mb() from srcu_read_unlock_lite()
rcutorture: Avoid printing cpu=-1 for no-fault RCU boost failure
rcuscale: Add guest_os_delay module parameter
refscale: Correct affinity check
torture: Add --no-affinity parameter to kvm.sh
rcu/nocb: Fix missed RCU barrier on deoffloading
rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu
rcu/srcutiny: don't return before reenabling preemption
rcu-tasks: Remove open-coded one-byte cmpxchg() emulation
doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
rcutorture: Test start-poll primitives with interrupts disabled
rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()
doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst
rcu: Add rcuog kthreads to RCU_NOCB_CPU help text
rcu: Use the BITS_PER_LONG macro
...
+474
-233
+1
-1
Documentation/RCU/stallwarn.rst
+1
-1
Documentation/RCU/stallwarn.rst
···
249
249
interrupts during the current stalled grace period.
250
250
251
251
The "idle=" portion of the message prints the dyntick-idle state.
252
-
The hex number before the first "/" is the low-order 12 bits of the
252
+
The hex number before the first "/" is the low-order 16 bits of the
253
253
dynticks counter, which will have an even-numbered value if the CPU
254
254
is in dyntick-idle mode and an odd-numbered value otherwise. The hex
255
255
number between the two "/"s is the value of the nesting, which will be
+8
-5
Documentation/admin-guide/kernel-parameters.txt
+8
-5
Documentation/admin-guide/kernel-parameters.txt
···
5415
5415
Set time (jiffies) between CPU-hotplug operations,
5416
5416
or zero to disable CPU-hotplug testing.
5417
5417
5418
-
rcutorture.read_exit= [KNL]
5419
-
Set the number of read-then-exit kthreads used
5420
-
to test the interaction of RCU updaters and
5421
-
task-exit processing.
5422
-
5423
5418
rcutorture.read_exit_burst= [KNL]
5424
5419
The number of times in a given read-then-exit
5425
5420
episode that a set of read-then-exit kthreads
···
5423
5428
rcutorture.read_exit_delay= [KNL]
5424
5429
The delay, in seconds, between successive
5425
5430
read-then-exit testing episodes.
5431
+
5432
+
rcutorture.reader_flavor= [KNL]
5433
+
A bit mask indicating which readers to use.
5434
+
If there is more than one bit set, the readers
5435
+
are entered from low-order bit up, and are
5436
+
exited in the opposite order. For SRCU, the
5437
+
0x1 bit is normal readers, 0x2 NMI-safe readers,
5438
+
and 0x4 light-weight readers.
5426
5439
5427
5440
rcutorture.shuffle_interval= [KNL]
5428
5441
Set task-shuffle interval (s). Shuffling tasks
+1
-1
Documentation/admin-guide/kernel-per-CPU-kthreads.rst
+1
-1
Documentation/admin-guide/kernel-per-CPU-kthreads.rst
-1
include/linux/rcutiny.h
-1
include/linux/rcutiny.h
···
165
165
static inline bool rcu_is_watching(void) { return true; }
166
166
static inline void rcu_momentary_eqs(void) { }
167
167
static inline void kfree_rcu_scheduler_running(void) { }
168
-
static inline bool rcu_gp_might_be_stalled(void) { return false; }
169
168
170
169
/* Avoid RCU read-side critical sections leaking across. */
171
170
static inline void rcu_all_qs(void) { barrier(); }
-1
include/linux/rcutree.h
-1
include/linux/rcutree.h
+69
-23
include/linux/srcu.h
+69
-23
include/linux/srcu.h
···
56
56
void cleanup_srcu_struct(struct srcu_struct *ssp);
57
57
int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp);
58
58
void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp);
59
+
#ifdef CONFIG_TINY_SRCU
60
+
#define __srcu_read_lock_lite __srcu_read_lock
61
+
#define __srcu_read_unlock_lite __srcu_read_unlock
62
+
#else // #ifdef CONFIG_TINY_SRCU
63
+
int __srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp);
64
+
void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) __releases(ssp);
65
+
#endif // #else // #ifdef CONFIG_TINY_SRCU
59
66
void synchronize_srcu(struct srcu_struct *ssp);
60
67
61
68
#define SRCU_GET_STATE_COMPLETED 0x1
···
183
176
184
177
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
185
178
186
-
#define SRCU_NMI_UNKNOWN 0x0
187
-
#define SRCU_NMI_UNSAFE 0x1
188
-
#define SRCU_NMI_SAFE 0x2
189
-
190
-
#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU)
191
-
void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe);
192
-
#else
193
-
static inline void srcu_check_nmi_safety(struct srcu_struct *ssp,
194
-
bool nmi_safe) { }
195
-
#endif
196
-
197
179
198
180
/**
199
181
* srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing
···
232
236
* a mutex that is held elsewhere while calling synchronize_srcu() or
233
237
* synchronize_srcu_expedited().
234
238
*
235
-
* Note that srcu_read_lock() and the matching srcu_read_unlock() must
236
-
* occur in the same context, for example, it is illegal to invoke
237
-
* srcu_read_unlock() in an irq handler if the matching srcu_read_lock()
238
-
* was invoked in process context.
239
+
* The return value from srcu_read_lock() must be passed unaltered
240
+
* to the matching srcu_read_unlock(). Note that srcu_read_lock() and
241
+
* the matching srcu_read_unlock() must occur in the same context, for
242
+
* example, it is illegal to invoke srcu_read_unlock() in an irq handler
243
+
* if the matching srcu_read_lock() was invoked in process context. Or,
244
+
* for that matter to invoke srcu_read_unlock() from one task and the
245
+
* matching srcu_read_lock() from another.
239
246
*/
240
247
static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp)
241
248
{
242
249
int retval;
243
250
244
-
srcu_check_nmi_safety(ssp, false);
251
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
245
252
retval = __srcu_read_lock(ssp);
246
253
srcu_lock_acquire(&ssp->dep_map);
254
+
return retval;
255
+
}
256
+
257
+
/**
258
+
* srcu_read_lock_lite - register a new reader for an SRCU-protected structure.
259
+
* @ssp: srcu_struct in which to register the new reader.
260
+
*
261
+
* Enter an SRCU read-side critical section, but for a light-weight
262
+
* smp_mb()-free reader. See srcu_read_lock() for more information.
263
+
*
264
+
* If srcu_read_lock_lite() is ever used on an srcu_struct structure,
265
+
* then none of the other flavors may be used, whether before, during,
266
+
* or after. Note that grace-period auto-expediting is disabled for _lite
267
+
* srcu_struct structures because auto-expedited grace periods invoke
268
+
* synchronize_rcu_expedited(), IPIs and all.
269
+
*
270
+
* Note that srcu_read_lock_lite() can be invoked only from those contexts
271
+
* where RCU is watching, that is, from contexts where it would be legal
272
+
* to invoke rcu_read_lock(). Otherwise, lockdep will complain.
273
+
*/
274
+
static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp)
275
+
{
276
+
int retval;
277
+
278
+
srcu_check_read_flavor_lite(ssp);
279
+
retval = __srcu_read_lock_lite(ssp);
280
+
rcu_try_lock_acquire(&ssp->dep_map);
247
281
return retval;
248
282
}
249
283
···
283
257
*
284
258
* Enter an SRCU read-side critical section, but in an NMI-safe manner.
285
259
* See srcu_read_lock() for more information.
260
+
*
261
+
* If srcu_read_lock_nmisafe() is ever used on an srcu_struct structure,
262
+
* then none of the other flavors may be used, whether before, during,
263
+
* or after.
286
264
*/
287
265
static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp)
288
266
{
289
267
int retval;
290
268
291
-
srcu_check_nmi_safety(ssp, true);
269
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI);
292
270
retval = __srcu_read_lock_nmisafe(ssp);
293
271
rcu_try_lock_acquire(&ssp->dep_map);
294
272
return retval;
···
304
274
{
305
275
int retval;
306
276
307
-
srcu_check_nmi_safety(ssp, false);
277
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
308
278
retval = __srcu_read_lock(ssp);
309
279
return retval;
310
280
}
···
333
303
static inline int srcu_down_read(struct srcu_struct *ssp) __acquires(ssp)
334
304
{
335
305
WARN_ON_ONCE(in_nmi());
336
-
srcu_check_nmi_safety(ssp, false);
306
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
337
307
return __srcu_read_lock(ssp);
338
308
}
339
309
···
348
318
__releases(ssp)
349
319
{
350
320
WARN_ON_ONCE(idx & ~0x1);
351
-
srcu_check_nmi_safety(ssp, false);
321
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
352
322
srcu_lock_release(&ssp->dep_map);
353
323
__srcu_read_unlock(ssp, idx);
324
+
}
325
+
326
+
/**
327
+
* srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure.
328
+
* @ssp: srcu_struct in which to unregister the old reader.
329
+
* @idx: return value from corresponding srcu_read_lock().
330
+
*
331
+
* Exit a light-weight SRCU read-side critical section.
332
+
*/
333
+
static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
334
+
__releases(ssp)
335
+
{
336
+
WARN_ON_ONCE(idx & ~0x1);
337
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE);
338
+
srcu_lock_release(&ssp->dep_map);
339
+
__srcu_read_unlock_lite(ssp, idx);
354
340
}
355
341
356
342
/**
···
380
334
__releases(ssp)
381
335
{
382
336
WARN_ON_ONCE(idx & ~0x1);
383
-
srcu_check_nmi_safety(ssp, true);
337
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI);
384
338
rcu_lock_release(&ssp->dep_map);
385
339
__srcu_read_unlock_nmisafe(ssp, idx);
386
340
}
···
389
343
static inline notrace void
390
344
srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp)
391
345
{
392
-
srcu_check_nmi_safety(ssp, false);
346
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
393
347
__srcu_read_unlock(ssp, idx);
394
348
}
395
349
···
406
360
{
407
361
WARN_ON_ONCE(idx & ~0x1);
408
362
WARN_ON_ONCE(in_nmi());
409
-
srcu_check_nmi_safety(ssp, false);
363
+
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
410
364
__srcu_read_unlock(ssp, idx);
411
365
}
412
366
+3
include/linux/srcutiny.h
+3
include/linux/srcutiny.h
···
81
81
synchronize_srcu(ssp);
82
82
}
83
83
84
+
#define srcu_check_read_flavor(ssp, read_flavor) do { } while (0)
85
+
#define srcu_check_read_flavor_lite(ssp) do { } while (0)
86
+
84
87
/* Defined here to avoid size increase for non-torture kernels. */
85
88
static inline void srcu_torture_stats_print(struct srcu_struct *ssp,
86
89
char *tt, char *tf)
+66
-1
include/linux/srcutree.h
+66
-1
include/linux/srcutree.h
···
25
25
/* Read-side state. */
26
26
atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */
27
27
atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */
28
-
int srcu_nmi_safety; /* NMI-safe srcu_struct structure? */
28
+
int srcu_reader_flavor; /* Reader flavor for srcu_struct structure? */
29
29
30
30
/* Update-side state. */
31
31
spinlock_t __private lock ____cacheline_internodealigned_in_smp;
···
42
42
int cpu;
43
43
struct srcu_struct *ssp;
44
44
};
45
+
46
+
/* Values for ->srcu_reader_flavor. */
47
+
#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().
48
+
#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe().
49
+
#define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite().
45
50
46
51
/*
47
52
* Node in SRCU combining tree, similar in function to rcu_data.
···
208
203
void synchronize_srcu_expedited(struct srcu_struct *ssp);
209
204
void srcu_barrier(struct srcu_struct *ssp);
210
205
void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf);
206
+
207
+
/*
208
+
* Counts the new reader in the appropriate per-CPU element of the
209
+
* srcu_struct. Returns an index that must be passed to the matching
210
+
* srcu_read_unlock_lite().
211
+
*
212
+
* Note that this_cpu_inc() is an RCU read-side critical section either
213
+
* because it disables interrupts, because it is a single instruction,
214
+
* or because it is a read-modify-write atomic operation, depending on
215
+
* the whims of the architecture.
216
+
*/
217
+
static inline int __srcu_read_lock_lite(struct srcu_struct *ssp)
218
+
{
219
+
int idx;
220
+
221
+
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite().");
222
+
idx = READ_ONCE(ssp->srcu_idx) & 0x1;
223
+
this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */
224
+
barrier(); /* Avoid leaking the critical section. */
225
+
return idx;
226
+
}
227
+
228
+
/*
229
+
* Removes the count for the old reader from the appropriate
230
+
* per-CPU element of the srcu_struct. Note that this may well be a
231
+
* different CPU than that which was incremented by the corresponding
232
+
* srcu_read_lock_lite(), but it must be within the same task.
233
+
*
234
+
* Note that this_cpu_inc() is an RCU read-side critical section either
235
+
* because it disables interrupts, because it is a single instruction,
236
+
* or because it is a read-modify-write atomic operation, depending on
237
+
* the whims of the architecture.
238
+
*/
239
+
static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
240
+
{
241
+
barrier(); /* Avoid leaking the critical section. */
242
+
this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */
243
+
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite().");
244
+
}
245
+
246
+
void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);
247
+
248
+
// Record _lite() usage even for CONFIG_PROVE_RCU=n kernels.
249
+
static inline void srcu_check_read_flavor_lite(struct srcu_struct *ssp)
250
+
{
251
+
struct srcu_data *sdp = raw_cpu_ptr(ssp->sda);
252
+
253
+
if (likely(READ_ONCE(sdp->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE))
254
+
return;
255
+
256
+
// Note that the cmpxchg() in srcu_check_read_flavor() is fully ordered.
257
+
__srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE);
258
+
}
259
+
260
+
// Record non-_lite() usage only for CONFIG_PROVE_RCU=y kernels.
261
+
static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
262
+
{
263
+
if (IS_ENABLED(CONFIG_PROVE_RCU))
264
+
__srcu_check_read_flavor(ssp, read_flavor);
265
+
}
211
266
212
267
#endif
+18
-10
kernel/rcu/Kconfig
+18
-10
kernel/rcu/Kconfig
···
249
249
workloads will incur significant increases in context-switch
250
250
rates.
251
251
252
-
This option offloads callback invocation from the set of CPUs
253
-
specified at boot time by the rcu_nocbs parameter. For each
254
-
such CPU, a kthread ("rcuox/N") will be created to invoke
255
-
callbacks, where the "N" is the CPU being offloaded, and where
256
-
the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
257
-
RCU-sched (!PREEMPTION kernels). Nothing prevents this kthread
258
-
from running on the specified CPUs, but (1) the kthreads may be
259
-
preempted between each callback, and (2) affinity or cgroups can
260
-
be used to force the kthreads to run on whatever set of CPUs is
261
-
desired.
252
+
This option offloads callback invocation from the set of
253
+
CPUs specified at boot time by the rcu_nocbs parameter.
254
+
For each such CPU, a kthread ("rcuox/N") will be created to
255
+
invoke callbacks, where the "N" is the CPU being offloaded,
256
+
and where the "x" is "p" for RCU-preempt (PREEMPTION kernels)
257
+
and "s" for RCU-sched (!PREEMPTION kernels). This option
258
+
also creates another kthread for each sqrt(nr_cpu_ids) CPUs
259
+
("rcuog/N", where N is the first CPU in that group to come
260
+
online), which handles grace periods for its group. Nothing
261
+
prevents these kthreads from running on the specified CPUs,
262
+
but (1) the kthreads may be preempted between each callback,
263
+
and (2) affinity or cgroups can be used to force the kthreads
264
+
to run on whatever set of CPUs is desired.
265
+
266
+
The sqrt(nr_cpu_ids) grouping may be overridden using the
267
+
rcutree.rcu_nocb_gp_stride kernel boot parameter. This can
268
+
be especially helpful for smaller numbers of CPUs, where
269
+
sqrt(nr_cpu_ids) can be a bit of a blunt instrument.
262
270
263
271
Say Y here if you need reduced OS jitter, despite added overhead.
264
272
Say N here if you are unsure.
-1
kernel/rcu/rcu_segcblist.h
-1
kernel/rcu/rcu_segcblist.h
···
120
120
void rcu_segcblist_add_len(struct rcu_segcblist *rsclp, long v);
121
121
void rcu_segcblist_init(struct rcu_segcblist *rsclp);
122
122
void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
123
-
void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
124
123
bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
125
124
bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
126
125
struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);
+4
-4
kernel/rcu/rcuscale.c
+4
-4
kernel/rcu/rcuscale.c
···
889
889
890
890
if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) {
891
891
pr_alert("ERROR: call_rcu() CBs are not being lazy as expected!\n");
892
-
WARN_ON_ONCE(1);
893
-
return -1;
892
+
firsterr = -1;
893
+
goto unwind;
894
894
}
895
895
896
896
if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) {
897
897
pr_alert("ERROR: call_rcu() CBs are being too lazy!\n");
898
-
WARN_ON_ONCE(1);
899
-
return -1;
898
+
firsterr = -1;
899
+
goto unwind;
900
900
}
901
901
}
902
902
+60
-24
kernel/rcu/rcutorture.c
+60
-24
kernel/rcu/rcutorture.c
···
57
57
58
58
/* Bits for ->extendables field, extendables param, and related definitions. */
59
59
#define RCUTORTURE_RDR_SHIFT_1 8 /* Put SRCU index in upper bits. */
60
-
#define RCUTORTURE_RDR_MASK_1 (1 << RCUTORTURE_RDR_SHIFT_1)
61
-
#define RCUTORTURE_RDR_SHIFT_2 9 /* Put SRCU index in upper bits. */
62
-
#define RCUTORTURE_RDR_MASK_2 (1 << RCUTORTURE_RDR_SHIFT_2)
60
+
#define RCUTORTURE_RDR_MASK_1 (0xff << RCUTORTURE_RDR_SHIFT_1)
61
+
#define RCUTORTURE_RDR_SHIFT_2 16 /* Put SRCU index in upper bits. */
62
+
#define RCUTORTURE_RDR_MASK_2 (0xff << RCUTORTURE_RDR_SHIFT_2)
63
63
#define RCUTORTURE_RDR_BH 0x01 /* Extend readers by disabling bh. */
64
64
#define RCUTORTURE_RDR_IRQ 0x02 /* ... disabling interrupts. */
65
65
#define RCUTORTURE_RDR_PREEMPT 0x04 /* ... disabling preemption. */
···
71
71
#define RCUTORTURE_MAX_EXTEND \
72
72
(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_IRQ | RCUTORTURE_RDR_PREEMPT | \
73
73
RCUTORTURE_RDR_RBH | RCUTORTURE_RDR_SCHED)
74
+
#define RCUTORTURE_RDR_ALLBITS \
75
+
(RCUTORTURE_MAX_EXTEND | RCUTORTURE_RDR_RCU_1 | RCUTORTURE_RDR_RCU_2 | \
76
+
RCUTORTURE_RDR_MASK_1 | RCUTORTURE_RDR_MASK_2)
74
77
#define RCUTORTURE_RDR_MAX_LOOPS 0x7 /* Maximum reader extensions. */
75
78
/* Must be power of two minus one. */
76
79
#define RCUTORTURE_RDR_MAX_SEGS (RCUTORTURE_RDR_MAX_LOOPS + 3)
···
111
108
torture_param(int, nocbs_toggle, 1000, "Time between toggling nocb state (ms)");
112
109
torture_param(int, read_exit_delay, 13, "Delay between read-then-exit episodes (s)");
113
110
torture_param(int, read_exit_burst, 16, "# of read-then-exit bursts per episode, zero to disable");
111
+
torture_param(int, reader_flavor, 0x1, "Reader flavors to use, one per bit.");
114
112
torture_param(int, shuffle_interval, 3, "Number of seconds between shuffles");
115
113
torture_param(int, shutdown_secs, 0, "Shutdown time (s), <= zero to disable.");
116
114
torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable.");
···
397
393
int slow_gps;
398
394
int no_pi_lock;
399
395
int debug_objects;
396
+
int start_poll_irqsoff;
400
397
const char *name;
401
398
};
402
399
···
586
581
.can_boost = IS_ENABLED(CONFIG_RCU_BOOST),
587
582
.extendables = RCUTORTURE_MAX_EXTEND,
588
583
.debug_objects = 1,
584
+
.start_poll_irqsoff = 1,
589
585
.name = "rcu"
590
586
};
591
587
···
647
641
648
642
static int srcu_torture_read_lock(void)
649
643
{
650
-
if (cur_ops == &srcud_ops)
651
-
return srcu_read_lock_nmisafe(srcu_ctlp);
652
-
else
653
-
return srcu_read_lock(srcu_ctlp);
644
+
int idx;
645
+
int ret = 0;
646
+
647
+
if ((reader_flavor & 0x1) || !(reader_flavor & 0x7)) {
648
+
idx = srcu_read_lock(srcu_ctlp);
649
+
WARN_ON_ONCE(idx & ~0x1);
650
+
ret += idx;
651
+
}
652
+
if (reader_flavor & 0x2) {
653
+
idx = srcu_read_lock_nmisafe(srcu_ctlp);
654
+
WARN_ON_ONCE(idx & ~0x1);
655
+
ret += idx << 1;
656
+
}
657
+
if (reader_flavor & 0x4) {
658
+
idx = srcu_read_lock_lite(srcu_ctlp);
659
+
WARN_ON_ONCE(idx & ~0x1);
660
+
ret += idx << 2;
661
+
}
662
+
return ret;
654
663
}
655
664
656
665
static void
···
689
668
690
669
static void srcu_torture_read_unlock(int idx)
691
670
{
692
-
if (cur_ops == &srcud_ops)
693
-
srcu_read_unlock_nmisafe(srcu_ctlp, idx);
694
-
else
695
-
srcu_read_unlock(srcu_ctlp, idx);
671
+
WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
672
+
if (reader_flavor & 0x4)
673
+
srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2);
674
+
if (reader_flavor & 0x2)
675
+
srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1);
676
+
if ((reader_flavor & 0x1) || !(reader_flavor & 0x7))
677
+
srcu_read_unlock(srcu_ctlp, idx & 0x1);
696
678
}
697
679
698
680
static int torture_srcu_read_lock_held(void)
···
1083
1059
// At most one persisted message per boost test.
1084
1060
j = jiffies;
1085
1061
lp = READ_ONCE(last_persist);
1086
-
if (time_after(j, lp + mininterval) && cmpxchg(&last_persist, lp, j) == lp)
1087
-
pr_info("Boost inversion persisted: No QS from CPU %d\n", cpu);
1062
+
if (time_after(j, lp + mininterval) &&
1063
+
cmpxchg(&last_persist, lp, j) == lp) {
1064
+
if (cpu < 0)
1065
+
pr_info("Boost inversion persisted: QS from all CPUs\n");
1066
+
else
1067
+
pr_info("Boost inversion persisted: No QS from CPU %d\n", cpu);
1068
+
}
1088
1069
return false; // passed on a technicality
1089
1070
}
1090
1071
VERBOSE_TOROUT_STRING("rcu_torture_boost boosting failed");
···
1724
1695
cur_ops->cond_sync_exp_full(&gp_snap_full);
1725
1696
break;
1726
1697
case RTWS_POLL_GET:
1698
+
if (cur_ops->start_poll_irqsoff)
1699
+
local_irq_disable();
1727
1700
gp_snap = cur_ops->start_gp_poll();
1701
+
if (cur_ops->start_poll_irqsoff)
1702
+
local_irq_enable();
1728
1703
while (!cur_ops->poll_gp_state(gp_snap)) {
1729
1704
torture_hrtimeout_jiffies(torture_random(&rand) % 16,
1730
1705
&rand);
1731
1706
}
1732
1707
break;
1733
1708
case RTWS_POLL_GET_FULL:
1709
+
if (cur_ops->start_poll_irqsoff)
1710
+
local_irq_disable();
1734
1711
cur_ops->start_gp_poll_full(&gp_snap_full);
1712
+
if (cur_ops->start_poll_irqsoff)
1713
+
local_irq_enable();
1735
1714
while (!cur_ops->poll_gp_state_full(&gp_snap_full)) {
1736
1715
torture_hrtimeout_jiffies(torture_random(&rand) % 16,
1737
1716
&rand);
···
1857
1820
int statesold = *readstate & ~newstate;
1858
1821
1859
1822
WARN_ON_ONCE(idxold2 < 0);
1860
-
WARN_ON_ONCE((idxold2 >> RCUTORTURE_RDR_SHIFT_2) > 1);
1823
+
WARN_ON_ONCE(idxold2 & ~RCUTORTURE_RDR_ALLBITS);
1861
1824
rtrsp->rt_readstate = newstate;
1862
1825
1863
1826
/* First, put new protection in place to avoid critical-section gap. */
···
1872
1835
if (statesnew & RCUTORTURE_RDR_SCHED)
1873
1836
rcu_read_lock_sched();
1874
1837
if (statesnew & RCUTORTURE_RDR_RCU_1)
1875
-
idxnew1 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_1;
1838
+
idxnew1 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_1) & RCUTORTURE_RDR_MASK_1;
1876
1839
if (statesnew & RCUTORTURE_RDR_RCU_2)
1877
-
idxnew2 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_2;
1840
+
idxnew2 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_2) & RCUTORTURE_RDR_MASK_2;
1878
1841
1879
1842
/*
1880
1843
* Next, remove old protection, in decreasing order of strength
···
1894
1857
if (statesold & RCUTORTURE_RDR_RBH)
1895
1858
rcu_read_unlock_bh();
1896
1859
if (statesold & RCUTORTURE_RDR_RCU_2) {
1897
-
cur_ops->readunlock((idxold2 >> RCUTORTURE_RDR_SHIFT_2) & 0x1);
1860
+
cur_ops->readunlock((idxold2 & RCUTORTURE_RDR_MASK_2) >> RCUTORTURE_RDR_SHIFT_2);
1898
1861
WARN_ON_ONCE(idxnew2 != -1);
1899
1862
idxold2 = 0;
1900
1863
}
···
1904
1867
lockit = !cur_ops->no_pi_lock && !statesnew && !(torture_random(trsp) & 0xffff);
1905
1868
if (lockit)
1906
1869
raw_spin_lock_irqsave(¤t->pi_lock, flags);
1907
-
cur_ops->readunlock((idxold1 >> RCUTORTURE_RDR_SHIFT_1) & 0x1);
1870
+
cur_ops->readunlock((idxold1 & RCUTORTURE_RDR_MASK_1) >> RCUTORTURE_RDR_SHIFT_1);
1908
1871
WARN_ON_ONCE(idxnew1 != -1);
1909
1872
idxold1 = 0;
1910
1873
if (lockit)
···
1919
1882
if (idxnew1 == -1)
1920
1883
idxnew1 = idxold1 & RCUTORTURE_RDR_MASK_1;
1921
1884
WARN_ON_ONCE(idxnew1 < 0);
1922
-
if (WARN_ON_ONCE((idxnew1 >> RCUTORTURE_RDR_SHIFT_1) > 1))
1923
-
pr_info("Unexpected idxnew1 value of %#x\n", idxnew1);
1924
1885
if (idxnew2 == -1)
1925
1886
idxnew2 = idxold2 & RCUTORTURE_RDR_MASK_2;
1926
1887
WARN_ON_ONCE(idxnew2 < 0);
1927
-
WARN_ON_ONCE((idxnew2 >> RCUTORTURE_RDR_SHIFT_2) > 1);
1928
1888
*readstate = idxnew1 | idxnew2 | newstate;
1929
1889
WARN_ON_ONCE(*readstate < 0);
1930
-
if (WARN_ON_ONCE((*readstate >> RCUTORTURE_RDR_SHIFT_2) > 1))
1931
-
pr_info("Unexpected idxnew2 value of %#x\n", idxnew2);
1890
+
if (WARN_ON_ONCE(*readstate & ~RCUTORTURE_RDR_ALLBITS))
1891
+
pr_info("Unexpected readstate value of %#x\n", *readstate);
1932
1892
}
1933
1893
1934
1894
/* Return the biggest extendables mask given current RCU and boot parameters. */
···
1950
1916
unsigned long preempts_irq = preempts | RCUTORTURE_RDR_IRQ;
1951
1917
unsigned long bhs = RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH;
1952
1918
1953
-
WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT_1);
1919
+
WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT_1); // Can't have reader idx bits.
1954
1920
/* Mostly only one bit (need preemption!), sometimes lots of bits. */
1955
1921
if (!(randmask1 & 0x7))
1956
1922
mask = mask & randmask2;
···
2423
2389
"n_barrier_cbs=%d "
2424
2390
"onoff_interval=%d onoff_holdoff=%d "
2425
2391
"read_exit_delay=%d read_exit_burst=%d "
2392
+
"reader_flavor=%x "
2426
2393
"nocbs_nthreads=%d nocbs_toggle=%d "
2427
2394
"test_nmis=%d\n",
2428
2395
torture_type, tag, nrealreaders, nfakewriters,
···
2436
2401
n_barrier_cbs,
2437
2402
onoff_interval, onoff_holdoff,
2438
2403
read_exit_delay, read_exit_burst,
2404
+
reader_flavor,
2439
2405
nocbs_nthreads, nocbs_toggle,
2440
2406
test_nmis);
2441
2407
}
+52
-4
kernel/rcu/refscale.c
+52
-4
kernel/rcu/refscale.c
···
75
75
torture_param(int, verbose, 0, "Enable verbose debugging printk()s");
76
76
torture_param(int, verbose_batched, 0, "Batch verbose debugging printk()s");
77
77
78
+
// Number of seconds to extend warm-up and cool-down for multiple guest OSes
79
+
torture_param(long, guest_os_delay, 0,
80
+
"Number of seconds to extend warm-up/cool-down for multiple guest OSes.");
78
81
// Wait until there are multiple CPUs before starting test.
79
82
torture_param(int, holdoff, IS_BUILTIN(CONFIG_RCU_REF_SCALE_TEST) ? 10 : 0,
80
83
"Holdoff time before test start (s)");
···
213
210
.readsection = srcu_ref_scale_read_section,
214
211
.delaysection = srcu_ref_scale_delay_section,
215
212
.name = "srcu"
213
+
};
214
+
215
+
static void srcu_lite_ref_scale_read_section(const int nloops)
216
+
{
217
+
int i;
218
+
int idx;
219
+
220
+
for (i = nloops; i >= 0; i--) {
221
+
idx = srcu_read_lock_lite(srcu_ctlp);
222
+
srcu_read_unlock_lite(srcu_ctlp, idx);
223
+
}
224
+
}
225
+
226
+
static void srcu_lite_ref_scale_delay_section(const int nloops, const int udl, const int ndl)
227
+
{
228
+
int i;
229
+
int idx;
230
+
231
+
for (i = nloops; i >= 0; i--) {
232
+
idx = srcu_read_lock_lite(srcu_ctlp);
233
+
un_delay(udl, ndl);
234
+
srcu_read_unlock_lite(srcu_ctlp, idx);
235
+
}
236
+
}
237
+
238
+
static const struct ref_scale_ops srcu_lite_ops = {
239
+
.init = rcu_sync_scale_init,
240
+
.readsection = srcu_lite_ref_scale_read_section,
241
+
.delaysection = srcu_lite_ref_scale_delay_section,
242
+
.name = "srcu-lite"
216
243
};
217
244
218
245
#ifdef CONFIG_TASKS_RCU
···
834
801
cur_ops->delaysection(loops, readdelay / 1000, readdelay % 1000);
835
802
}
836
803
804
+
// Warm up cache, or, if needed run a series of rcu_scale_one_reader()
805
+
// to allow multiple rcuscale guest OSes to collect mutually valid data.
806
+
static void rcu_scale_warm_cool(void)
807
+
{
808
+
unsigned long jdone = jiffies + (guest_os_delay > 0 ? guest_os_delay * HZ : -1);
809
+
810
+
do {
811
+
rcu_scale_one_reader();
812
+
cond_resched();
813
+
} while (time_before(jiffies, jdone));
814
+
}
815
+
837
816
// Reader kthread. Repeatedly does empty RCU read-side
838
817
// critical section, minimizing update-side interference.
839
818
static int
···
874
829
goto end;
875
830
876
831
// Make sure that the CPU is affinitized appropriately during testing.
877
-
WARN_ON_ONCE(raw_smp_processor_id() != me);
832
+
WARN_ON_ONCE(raw_smp_processor_id() != me % nr_cpu_ids);
878
833
879
834
WRITE_ONCE(rt->start_reader, 0);
880
835
if (!atomic_dec_return(&n_started))
···
1002
957
schedule_timeout_uninterruptible(1);
1003
958
1004
959
// Start exp readers up per experiment
960
+
rcu_scale_warm_cool();
1005
961
for (exp = 0; exp < nruns && !torture_must_stop(); exp++) {
1006
962
if (torture_must_stop())
1007
963
goto end;
···
1033
987
1034
988
result_avg[exp] = div_u64(1000 * process_durations(nreaders), nreaders * loops);
1035
989
}
990
+
rcu_scale_warm_cool();
1036
991
1037
992
// Print the average of all experiments
1038
993
SCALEOUT("END OF TEST. Calculating average duration per loop (nanoseconds)...\n");
···
1129
1082
long i;
1130
1083
int firsterr = 0;
1131
1084
static const struct ref_scale_ops *scale_ops[] = {
1132
-
&rcu_ops, &srcu_ops, RCU_TRACE_OPS RCU_TASKS_OPS &refcnt_ops, &rwlock_ops,
1133
-
&rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &clock_ops, &jiffies_ops,
1134
-
&typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops,
1085
+
&rcu_ops, &srcu_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS
1086
+
&refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops,
1087
+
&clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops,
1088
+
&typesafe_seqlock_ops,
1135
1089
};
1136
1090
1137
1091
if (!torture_init_begin(scale_type, verbose))
+1
-1
kernel/rcu/srcutiny.c
+1
-1
kernel/rcu/srcutiny.c
···
122
122
ssp = container_of(wp, struct srcu_struct, srcu_work);
123
123
preempt_disable(); // Needed for PREEMPT_AUTO
124
124
if (ssp->srcu_gp_running || ULONG_CMP_GE(ssp->srcu_idx, READ_ONCE(ssp->srcu_idx_max))) {
125
-
return; /* Already running or nothing to do. */
126
125
preempt_enable();
126
+
return; /* Already running or nothing to do. */
127
127
}
128
128
129
129
/* Remove recently arrived callbacks and wait for readers. */
+89
-56
kernel/rcu/srcutree.c
+89
-56
kernel/rcu/srcutree.c
···
128
128
* Initialize the per-CPU srcu_data array, which feeds into the
129
129
* leaves of the srcu_node tree.
130
130
*/
131
-
WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) !=
131
+
BUILD_BUG_ON(ARRAY_SIZE(sdp->srcu_lock_count) !=
132
132
ARRAY_SIZE(sdp->srcu_unlock_count));
133
133
for_each_possible_cpu(cpu) {
134
134
sdp = per_cpu_ptr(ssp->sda, cpu);
···
187
187
/* Each pass through this loop initializes one srcu_node structure. */
188
188
srcu_for_each_node_breadth_first(ssp, snp) {
189
189
spin_lock_init(&ACCESS_PRIVATE(snp, lock));
190
-
WARN_ON_ONCE(ARRAY_SIZE(snp->srcu_have_cbs) !=
190
+
BUILD_BUG_ON(ARRAY_SIZE(snp->srcu_have_cbs) !=
191
191
ARRAY_SIZE(snp->srcu_data_have_cbs));
192
192
for (i = 0; i < ARRAY_SIZE(snp->srcu_have_cbs); i++) {
193
193
snp->srcu_have_cbs[i] = SRCU_SNP_INIT_SEQ;
···
419
419
}
420
420
421
421
/*
422
-
* Returns approximate total of the readers' ->srcu_lock_count[] values
423
-
* for the rank of per-CPU counters specified by idx.
422
+
* Is the current or any upcoming grace period to be expedited?
424
423
*/
425
-
static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx)
424
+
static bool srcu_gp_is_expedited(struct srcu_struct *ssp)
426
425
{
427
-
int cpu;
428
-
unsigned long sum = 0;
426
+
struct srcu_usage *sup = ssp->srcu_sup;
429
427
430
-
for_each_possible_cpu(cpu) {
431
-
struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu);
432
-
433
-
sum += atomic_long_read(&cpuc->srcu_lock_count[idx]);
434
-
}
435
-
return sum;
428
+
return ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp));
436
429
}
437
430
438
431
/*
439
-
* Returns approximate total of the readers' ->srcu_unlock_count[] values
440
-
* for the rank of per-CPU counters specified by idx.
432
+
* Computes approximate total of the readers' ->srcu_lock_count[] values
433
+
* for the rank of per-CPU counters specified by idx, and returns true if
434
+
* the caller did the proper barrier (gp), and if the count of the locks
435
+
* matches that of the unlocks passed in.
441
436
*/
442
-
static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx)
437
+
static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, unsigned long unlocks)
443
438
{
444
439
int cpu;
445
440
unsigned long mask = 0;
446
441
unsigned long sum = 0;
447
442
448
443
for_each_possible_cpu(cpu) {
449
-
struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu);
444
+
struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu);
450
445
451
-
sum += atomic_long_read(&cpuc->srcu_unlock_count[idx]);
446
+
sum += atomic_long_read(&sdp->srcu_lock_count[idx]);
452
447
if (IS_ENABLED(CONFIG_PROVE_RCU))
453
-
mask = mask | READ_ONCE(cpuc->srcu_nmi_safety);
448
+
mask = mask | READ_ONCE(sdp->srcu_reader_flavor);
454
449
}
455
-
WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask >> 1)),
456
-
"Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp);
450
+
WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)),
451
+
"Mixed reader flavors for srcu_struct at %ps.\n", ssp);
452
+
if (mask & SRCU_READ_FLAVOR_LITE && !gp)
453
+
return false;
454
+
return sum == unlocks;
455
+
}
456
+
457
+
/*
458
+
* Returns approximate total of the readers' ->srcu_unlock_count[] values
459
+
* for the rank of per-CPU counters specified by idx.
460
+
*/
461
+
static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, unsigned long *rdm)
462
+
{
463
+
int cpu;
464
+
unsigned long mask = 0;
465
+
unsigned long sum = 0;
466
+
467
+
for_each_possible_cpu(cpu) {
468
+
struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu);
469
+
470
+
sum += atomic_long_read(&sdp->srcu_unlock_count[idx]);
471
+
mask = mask | READ_ONCE(sdp->srcu_reader_flavor);
472
+
}
473
+
WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)),
474
+
"Mixed reader flavors for srcu_struct at %ps.\n", ssp);
475
+
*rdm = mask;
457
476
return sum;
458
477
}
459
478
···
482
463
*/
483
464
static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx)
484
465
{
466
+
bool did_gp;
467
+
unsigned long rdm;
485
468
unsigned long unlocks;
486
469
487
-
unlocks = srcu_readers_unlock_idx(ssp, idx);
470
+
unlocks = srcu_readers_unlock_idx(ssp, idx, &rdm);
471
+
did_gp = !!(rdm & SRCU_READ_FLAVOR_LITE);
488
472
489
473
/*
490
474
* Make sure that a lock is always counted if the corresponding
491
475
* unlock is counted. Needs to be a smp_mb() as the read side may
492
476
* contain a read from a variable that is written to before the
493
477
* synchronize_srcu() in the write side. In this case smp_mb()s
494
-
* A and B act like the store buffering pattern.
478
+
* A and B (or X and Y) act like the store buffering pattern.
495
479
*
496
-
* This smp_mb() also pairs with smp_mb() C to prevent accesses
497
-
* after the synchronize_srcu() from being executed before the
498
-
* grace period ends.
480
+
* This smp_mb() also pairs with smp_mb() C (or, in the case of X,
481
+
* Z) to prevent accesses after the synchronize_srcu() from being
482
+
* executed before the grace period ends.
499
483
*/
500
-
smp_mb(); /* A */
484
+
if (!did_gp)
485
+
smp_mb(); /* A */
486
+
else
487
+
synchronize_rcu(); /* X */
501
488
502
489
/*
503
490
* If the locks are the same as the unlocks, then there must have
···
561
536
* which are unlikely to be configured with an address space fully
562
537
* populated with memory, at least not anytime soon.
563
538
*/
564
-
return srcu_readers_lock_idx(ssp, idx) == unlocks;
539
+
return srcu_readers_lock_idx(ssp, idx, did_gp, unlocks);
565
540
}
566
541
567
542
/**
···
579
554
unsigned long sum = 0;
580
555
581
556
for_each_possible_cpu(cpu) {
582
-
struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu);
557
+
struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu);
583
558
584
-
sum += atomic_long_read(&cpuc->srcu_lock_count[0]);
585
-
sum += atomic_long_read(&cpuc->srcu_lock_count[1]);
586
-
sum -= atomic_long_read(&cpuc->srcu_unlock_count[0]);
587
-
sum -= atomic_long_read(&cpuc->srcu_unlock_count[1]);
559
+
sum += atomic_long_read(&sdp->srcu_lock_count[0]);
560
+
sum += atomic_long_read(&sdp->srcu_lock_count[1]);
561
+
sum -= atomic_long_read(&sdp->srcu_unlock_count[0]);
562
+
sum -= atomic_long_read(&sdp->srcu_unlock_count[1]);
588
563
}
589
564
return sum;
590
565
}
···
647
622
unsigned long jbase = SRCU_INTERVAL;
648
623
struct srcu_usage *sup = ssp->srcu_sup;
649
624
650
-
if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp)))
625
+
if (srcu_gp_is_expedited(ssp))
651
626
jbase = 0;
652
627
if (rcu_seq_state(READ_ONCE(sup->srcu_gp_seq))) {
653
628
j = jiffies - 1;
···
712
687
}
713
688
EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
714
689
715
-
#ifdef CONFIG_PROVE_RCU
716
690
/*
717
-
* Check for consistent NMI safety.
691
+
* Check for consistent reader flavor.
718
692
*/
719
-
void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
693
+
void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
720
694
{
721
-
int nmi_safe_mask = 1 << nmi_safe;
722
-
int old_nmi_safe_mask;
695
+
int old_read_flavor;
723
696
struct srcu_data *sdp;
724
697
725
-
/* NMI-unsafe use in NMI is a bad sign */
726
-
WARN_ON_ONCE(!nmi_safe && in_nmi());
698
+
/* NMI-unsafe use in NMI is a bad sign, as is multi-bit read_flavor values. */
699
+
WARN_ON_ONCE((read_flavor != SRCU_READ_FLAVOR_NMI) && in_nmi());
700
+
WARN_ON_ONCE(read_flavor & (read_flavor - 1));
701
+
727
702
sdp = raw_cpu_ptr(ssp->sda);
728
-
old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety);
729
-
if (!old_nmi_safe_mask) {
730
-
WRITE_ONCE(sdp->srcu_nmi_safety, nmi_safe_mask);
731
-
return;
703
+
old_read_flavor = READ_ONCE(sdp->srcu_reader_flavor);
704
+
if (!old_read_flavor) {
705
+
old_read_flavor = cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor);
706
+
if (!old_read_flavor)
707
+
return;
732
708
}
733
-
WARN_ONCE(old_nmi_safe_mask != nmi_safe_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_nmi_safe_mask, nmi_safe_mask);
709
+
WARN_ONCE(old_read_flavor != read_flavor, "CPU %d old state %d new state %d\n", sdp->cpu, old_read_flavor, read_flavor);
734
710
}
735
-
EXPORT_SYMBOL_GPL(srcu_check_nmi_safety);
736
-
#endif /* CONFIG_PROVE_RCU */
711
+
EXPORT_SYMBOL_GPL(__srcu_check_read_flavor);
737
712
738
713
/*
739
714
* Counts the new reader in the appropriate per-CPU element of the
···
892
867
spin_lock_irq_rcu_node(sup);
893
868
idx = rcu_seq_state(sup->srcu_gp_seq);
894
869
WARN_ON_ONCE(idx != SRCU_STATE_SCAN2);
895
-
if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp)))
870
+
if (srcu_gp_is_expedited(ssp))
896
871
cbdelay = 0;
897
872
898
873
WRITE_ONCE(sup->srcu_last_gp_end, ktime_get_mono_fast_ns());
···
1147
1122
* it stays until either (1) Compilers learn about this sort of
1148
1123
* control dependency or (2) Some production workload running on
1149
1124
* a production system is unduly delayed by this slowpath smp_mb().
1125
+
* Except for _lite() readers, where it is inoperative, which
1126
+
* means that it is a good thing that it is redundant.
1150
1127
*/
1151
1128
smp_mb(); /* E */ /* Pairs with B and C. */
1152
1129
···
1166
1139
}
1167
1140
1168
1141
/*
1169
-
* If SRCU is likely idle, return true, otherwise return false.
1142
+
* If SRCU is likely idle, in other words, the next SRCU grace period
1143
+
* should be expedited, return true, otherwise return false. Except that
1144
+
* in the presence of _lite() readers, always return false.
1170
1145
*
1171
1146
* Note that it is OK for several current from-idle requests for a new
1172
1147
* grace period from idle to specify expediting because they will all end
···
1188
1159
* negligible when amortized over that time period, and the extra latency
1189
1160
* of a needlessly non-expedited grace period is similarly negligible.
1190
1161
*/
1191
-
static bool srcu_might_be_idle(struct srcu_struct *ssp)
1162
+
static bool srcu_should_expedite(struct srcu_struct *ssp)
1192
1163
{
1193
1164
unsigned long curseq;
1194
1165
unsigned long flags;
···
1197
1168
unsigned long tlast;
1198
1169
1199
1170
check_init_srcu_struct(ssp);
1171
+
/* If _lite() readers, don't do unsolicited expediting. */
1172
+
if (this_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE)
1173
+
return false;
1200
1174
/* If the local srcu_data structure has callbacks, not idle. */
1201
1175
sdp = raw_cpu_ptr(ssp->sda);
1202
1176
spin_lock_irqsave_rcu_node(sdp, flags);
···
1501
1469
* Implementation of these memory-ordering guarantees is similar to
1502
1470
* that of synchronize_rcu().
1503
1471
*
1504
-
* If SRCU is likely idle, expedite the first request. This semantic
1505
-
* was provided by Classic SRCU, and is relied upon by its users, so TREE
1506
-
* SRCU must also provide it. Note that detecting idleness is heuristic
1507
-
* and subject to both false positives and negatives.
1472
+
* If SRCU is likely idle as determined by srcu_should_expedite(),
1473
+
* expedite the first request. This semantic was provided by Classic SRCU,
1474
+
* and is relied upon by its users, so TREE SRCU must also provide it.
1475
+
* Note that detecting idleness is heuristic and subject to both false
1476
+
* positives and negatives.
1508
1477
*/
1509
1478
void synchronize_srcu(struct srcu_struct *ssp)
1510
1479
{
1511
-
if (srcu_might_be_idle(ssp) || rcu_gp_is_expedited())
1480
+
if (srcu_should_expedite(ssp) || rcu_gp_is_expedited())
1512
1481
synchronize_srcu_expedited(ssp);
1513
1482
else
1514
1483
__synchronize_srcu(ssp, true);
+3
-17
kernel/rcu/tasks.h
+3
-17
kernel/rcu/tasks.h
···
1407
1407
*/
1408
1408
void synchronize_rcu_tasks_rude(void)
1409
1409
{
1410
-
synchronize_rcu_tasks_generic(&rcu_tasks_rude);
1410
+
if (!IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR) || IS_ENABLED(CONFIG_FORCE_TASKS_RUDE_RCU))
1411
+
synchronize_rcu_tasks_generic(&rcu_tasks_rude);
1411
1412
}
1412
1413
EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_rude);
1413
1414
···
1550
1549
*/
1551
1550
u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new)
1552
1551
{
1553
-
union rcu_special ret;
1554
-
union rcu_special trs_old = READ_ONCE(t->trc_reader_special);
1555
-
union rcu_special trs_new = trs_old;
1556
-
1557
-
if (trs_old.b.need_qs != old)
1558
-
return trs_old.b.need_qs;
1559
-
trs_new.b.need_qs = new;
1560
-
1561
-
// Although cmpxchg() appears to KCSAN to update all four bytes,
1562
-
// only the .b.need_qs byte actually changes.
1563
-
instrument_atomic_read_write(&t->trc_reader_special.b.need_qs,
1564
-
sizeof(t->trc_reader_special.b.need_qs));
1565
-
// Avoid false-positive KCSAN failures.
1566
-
ret.s = data_race(cmpxchg(&t->trc_reader_special.s, trs_old.s, trs_new.s));
1567
-
1568
-
return ret.b.need_qs;
1552
+
return cmpxchg(&t->trc_reader_special.b.need_qs, old, new);
1569
1553
}
1570
1554
EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs);
1571
1555
+13
-11
kernel/rcu/tree.c
+13
-11
kernel/rcu/tree.c
···
3511
3511
}
3512
3512
3513
3513
static void
3514
-
schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
3514
+
__schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
3515
3515
{
3516
3516
long delay, delay_left;
3517
3517
···
3523
3523
return;
3524
3524
}
3525
3525
queue_delayed_work(system_unbound_wq, &krcp->monitor_work, delay);
3526
+
}
3527
+
3528
+
static void
3529
+
schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
3530
+
{
3531
+
unsigned long flags;
3532
+
3533
+
raw_spin_lock_irqsave(&krcp->lock, flags);
3534
+
__schedule_delayed_monitor_work(krcp);
3535
+
raw_spin_unlock_irqrestore(&krcp->lock, flags);
3526
3536
}
3527
3537
3528
3538
static void
···
3846
3836
3847
3837
// Set timer to drain after KFREE_DRAIN_JIFFIES.
3848
3838
if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
3849
-
schedule_delayed_monitor_work(krcp);
3839
+
__schedule_delayed_monitor_work(krcp);
3850
3840
3851
3841
unlock_return:
3852
3842
krc_this_cpu_unlock(krcp, flags);
···
4204
4194
struct rcu_data *rdp;
4205
4195
struct rcu_node *rnp;
4206
4196
4207
-
lockdep_assert_irqs_enabled();
4208
4197
local_irq_save(flags);
4209
4198
rdp = this_cpu_ptr(&rcu_data);
4210
4199
rnp = rdp->mynode;
···
4228
4219
* grace period has elapsed in the meantime. If the needed grace period
4229
4220
* is not already slated to start, notifies RCU core of the need for that
4230
4221
* grace period.
4231
-
*
4232
-
* Interrupts must be enabled for the case where it is necessary to awaken
4233
-
* the grace-period kthread.
4234
4222
*/
4235
4223
unsigned long start_poll_synchronize_rcu(void)
4236
4224
{
···
4248
4242
* grace period (whether normal or expedited) has elapsed in the meantime.
4249
4243
* If the needed grace period is not already slated to start, notifies
4250
4244
* RCU core of the need for that grace period.
4251
-
*
4252
-
* Interrupts must be enabled for the case where it is necessary to awaken
4253
-
* the grace-period kthread.
4254
4245
*/
4255
4246
void start_poll_synchronize_rcu_full(struct rcu_gp_oldstate *rgosp)
4256
4247
{
···
5583
5580
* Complain and fall back to the compile-time values if this
5584
5581
* limit is exceeded.
5585
5582
*/
5586
-
if (rcu_fanout_leaf < 2 ||
5587
-
rcu_fanout_leaf > sizeof(unsigned long) * 8) {
5583
+
if (rcu_fanout_leaf < 2 || rcu_fanout_leaf > BITS_PER_LONG) {
5588
5584
rcu_fanout_leaf = RCU_FANOUT_LEAF;
5589
5585
WARN_ON(1);
5590
5586
return;
+12
-1
kernel/rcu/tree_nocb.h
+12
-1
kernel/rcu/tree_nocb.h
···
891
891
swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
892
892
nocb_cb_wait_cond(rdp));
893
893
if (kthread_should_park()) {
894
-
kthread_parkme();
894
+
/*
895
+
* kthread_park() must be preceded by an rcu_barrier().
896
+
* But yet another rcu_barrier() might have sneaked in between
897
+
* the barrier callback execution and the callbacks counter
898
+
* decrement.
899
+
*/
900
+
if (rdp->nocb_cb_sleep) {
901
+
rcu_nocb_lock_irqsave(rdp, flags);
902
+
WARN_ON_ONCE(rcu_segcblist_n_cbs(&rdp->cblist));
903
+
rcu_nocb_unlock_irqrestore(rdp, flags);
904
+
kthread_parkme();
905
+
}
895
906
} else if (READ_ONCE(rdp->nocb_cb_sleep)) {
896
907
WARN_ON(signal_pending(current));
897
908
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
+11
-11
kernel/rcu/tree_plugin.h
+11
-11
kernel/rcu/tree_plugin.h
···
183
183
switch (blkd_state) {
184
184
case 0:
185
185
case RCU_EXP_TASKS:
186
-
case RCU_EXP_TASKS + RCU_GP_BLKD:
186
+
case RCU_EXP_TASKS | RCU_GP_BLKD:
187
187
case RCU_GP_TASKS:
188
-
case RCU_GP_TASKS + RCU_EXP_TASKS:
188
+
case RCU_GP_TASKS | RCU_EXP_TASKS:
189
189
190
190
/*
191
191
* Blocking neither GP, or first task blocking the normal
···
198
198
199
199
case RCU_EXP_BLKD:
200
200
case RCU_GP_BLKD:
201
-
case RCU_GP_BLKD + RCU_EXP_BLKD:
202
-
case RCU_GP_TASKS + RCU_EXP_BLKD:
203
-
case RCU_GP_TASKS + RCU_GP_BLKD + RCU_EXP_BLKD:
204
-
case RCU_GP_TASKS + RCU_EXP_TASKS + RCU_GP_BLKD + RCU_EXP_BLKD:
201
+
case RCU_GP_BLKD | RCU_EXP_BLKD:
202
+
case RCU_GP_TASKS | RCU_EXP_BLKD:
203
+
case RCU_GP_TASKS | RCU_GP_BLKD | RCU_EXP_BLKD:
204
+
case RCU_GP_TASKS | RCU_EXP_TASKS | RCU_GP_BLKD | RCU_EXP_BLKD:
205
205
206
206
/*
207
207
* First task arriving that blocks either GP, or first task
···
214
214
list_add_tail(&t->rcu_node_entry, &rnp->blkd_tasks);
215
215
break;
216
216
217
-
case RCU_EXP_TASKS + RCU_EXP_BLKD:
218
-
case RCU_EXP_TASKS + RCU_GP_BLKD + RCU_EXP_BLKD:
219
-
case RCU_GP_TASKS + RCU_EXP_TASKS + RCU_EXP_BLKD:
217
+
case RCU_EXP_TASKS | RCU_EXP_BLKD:
218
+
case RCU_EXP_TASKS | RCU_GP_BLKD | RCU_EXP_BLKD:
219
+
case RCU_GP_TASKS | RCU_EXP_TASKS | RCU_EXP_BLKD:
220
220
221
221
/*
222
222
* Second or subsequent task blocking the expedited GP.
···
227
227
list_add(&t->rcu_node_entry, rnp->exp_tasks);
228
228
break;
229
229
230
-
case RCU_GP_TASKS + RCU_GP_BLKD:
231
-
case RCU_GP_TASKS + RCU_EXP_TASKS + RCU_GP_BLKD:
230
+
case RCU_GP_TASKS | RCU_GP_BLKD:
231
+
case RCU_GP_TASKS | RCU_EXP_TASKS | RCU_GP_BLKD:
232
232
233
233
/*
234
234
* Second or subsequent task blocking the normal GP.
+18
-39
kernel/rcu/tree_stall.h
+18
-39
kernel/rcu/tree_stall.h
···
76
76
}
77
77
EXPORT_SYMBOL_GPL(rcu_jiffies_till_stall_check);
78
78
79
-
/**
80
-
* rcu_gp_might_be_stalled - Is it likely that the grace period is stalled?
81
-
*
82
-
* Returns @true if the current grace period is sufficiently old that
83
-
* it is reasonable to assume that it might be stalled. This can be
84
-
* useful when deciding whether to allocate memory to enable RCU-mediated
85
-
* freeing on the one hand or just invoking synchronize_rcu() on the other.
86
-
* The latter is preferable when the grace period is stalled.
87
-
*
88
-
* Note that sampling of the .gp_start and .gp_seq fields must be done
89
-
* carefully to avoid false positives at the beginnings and ends of
90
-
* grace periods.
91
-
*/
92
-
bool rcu_gp_might_be_stalled(void)
93
-
{
94
-
unsigned long d = rcu_jiffies_till_stall_check() / RCU_STALL_MIGHT_DIV;
95
-
unsigned long j = jiffies;
96
-
97
-
if (d < RCU_STALL_MIGHT_MIN)
98
-
d = RCU_STALL_MIGHT_MIN;
99
-
smp_mb(); // jiffies before .gp_seq to avoid false positives.
100
-
if (!rcu_gp_in_progress())
101
-
return false;
102
-
// Long delays at this point avoids false positive, but a delay
103
-
// of ULONG_MAX/4 jiffies voids your no-false-positive warranty.
104
-
smp_mb(); // .gp_seq before second .gp_start
105
-
// And ditto here.
106
-
return !time_before(j, READ_ONCE(rcu_state.gp_start) + d);
107
-
}
108
-
109
79
/* Don't do RCU CPU stall warnings during long sysrq printouts. */
110
80
void rcu_sysrq_start(void)
111
81
{
···
335
365
* that don't support NMI-based stack dumps. The NMI-triggered stack
336
366
* traces are more accurate because they are printed by the target CPU.
337
367
*/
338
-
static void rcu_dump_cpu_stacks(void)
368
+
static void rcu_dump_cpu_stacks(unsigned long gp_seq)
339
369
{
340
370
int cpu;
341
371
unsigned long flags;
···
343
373
344
374
rcu_for_each_leaf_node(rnp) {
345
375
printk_deferred_enter();
346
-
raw_spin_lock_irqsave_rcu_node(rnp, flags);
347
-
for_each_leaf_node_possible_cpu(rnp, cpu)
376
+
for_each_leaf_node_possible_cpu(rnp, cpu) {
377
+
if (gp_seq != data_race(rcu_state.gp_seq)) {
378
+
printk_deferred_exit();
379
+
pr_err("INFO: Stall ended during stack backtracing.\n");
380
+
return;
381
+
}
382
+
if (!(data_race(rnp->qsmask) & leaf_node_cpu_bit(rnp, cpu)))
383
+
continue;
384
+
raw_spin_lock_irqsave_rcu_node(rnp, flags);
348
385
if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu)) {
349
386
if (cpu_is_offline(cpu))
350
387
pr_err("Offline CPU %d blocking current GP.\n", cpu);
351
388
else
352
389
dump_cpu_task(cpu);
353
390
}
354
-
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
391
+
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
392
+
}
355
393
printk_deferred_exit();
356
394
}
357
395
}
···
616
638
(long)rcu_seq_current(&rcu_state.gp_seq), totqlen,
617
639
data_race(rcu_state.n_online_cpus)); // Diagnostic read
618
640
if (ndetected) {
619
-
rcu_dump_cpu_stacks();
641
+
rcu_dump_cpu_stacks(gp_seq);
620
642
621
643
/* Complain about tasks blocking the grace period. */
622
644
rcu_for_each_leaf_node(rnp)
···
648
670
rcu_force_quiescent_state(); /* Kick them all. */
649
671
}
650
672
651
-
static void print_cpu_stall(unsigned long gps)
673
+
static void print_cpu_stall(unsigned long gp_seq, unsigned long gps)
652
674
{
653
675
int cpu;
654
676
unsigned long flags;
···
685
707
rcu_check_gp_kthread_expired_fqs_timer();
686
708
rcu_check_gp_kthread_starvation();
687
709
688
-
rcu_dump_cpu_stacks();
710
+
rcu_dump_cpu_stacks(gp_seq);
689
711
690
712
raw_spin_lock_irqsave_rcu_node(rnp, flags);
691
713
/* Rewrite if needed in case of slow consoles. */
···
767
789
gs2 = READ_ONCE(rcu_state.gp_seq);
768
790
if (gs1 != gs2 ||
769
791
ULONG_CMP_LT(j, js) ||
770
-
ULONG_CMP_GE(gps, js))
792
+
ULONG_CMP_GE(gps, js) ||
793
+
!rcu_seq_state(gs2))
771
794
return; /* No stall or GP completed since entering function. */
772
795
rnp = rdp->mynode;
773
796
jn = jiffies + ULONG_MAX / 2;
···
789
810
pr_err("INFO: %s detected stall, but suppressed full report due to a stuck CSD-lock.\n", rcu_state.name);
790
811
} else if (self_detected) {
791
812
/* We haven't checked in, so go dump stack. */
792
-
print_cpu_stall(gps);
813
+
print_cpu_stall(gs2, gps);
793
814
} else {
794
815
/* They had a few time units to dump stack, so complain. */
795
816
print_other_cpu_stall(gs2, gps);
+23
-20
tools/testing/selftests/rcutorture/bin/kvm-test-1-run-batch.sh
+23
-20
tools/testing/selftests/rcutorture/bin/kvm-test-1-run-batch.sh
···
56
56
echo > $i/kvm-test-1-run-qemu.sh.out
57
57
export TORTURE_AFFINITY=
58
58
kvm-get-cpus-script.sh $T/cpuarray.awk $T/cpubatches.awk $T/cpustate
59
-
cat << ' ___EOF___' >> $T/cpubatches.awk
60
-
END {
61
-
affinitylist = "";
62
-
if (!gotcpus()) {
63
-
print "echo No CPU-affinity information, so no taskset command.";
64
-
} else if (cpu_count !~ /^[0-9][0-9]*$/) {
65
-
print "echo " scenario ": Bogus number of CPUs (old qemu-cmd?), so no taskset command.";
66
-
} else {
67
-
affinitylist = nextcpus(cpu_count);
68
-
if (!(affinitylist ~ /^[0-9,-][0-9,-]*$/))
69
-
print "echo " scenario ": Bogus CPU-affinity information, so no taskset command.";
70
-
else if (!dumpcpustate())
71
-
print "echo " scenario ": Could not dump state, so no taskset command.";
72
-
else
73
-
print "export TORTURE_AFFINITY=" affinitylist;
59
+
if test -z "${TORTURE_NO_AFFINITY}"
60
+
then
61
+
cat << ' ___EOF___' >> $T/cpubatches.awk
62
+
END {
63
+
affinitylist = "";
64
+
if (!gotcpus()) {
65
+
print "echo No CPU-affinity information, so no taskset command.";
66
+
} else if (cpu_count !~ /^[0-9][0-9]*$/) {
67
+
print "echo " scenario ": Bogus number of CPUs (old qemu-cmd?), so no taskset command.";
68
+
} else {
69
+
affinitylist = nextcpus(cpu_count);
70
+
if (!(affinitylist ~ /^[0-9,-][0-9,-]*$/))
71
+
print "echo " scenario ": Bogus CPU-affinity information, so no taskset command.";
72
+
else if (!dumpcpustate())
73
+
print "echo " scenario ": Could not dump state, so no taskset command.";
74
+
else
75
+
print "export TORTURE_AFFINITY=" affinitylist;
76
+
}
74
77
}
75
-
}
76
-
___EOF___
77
-
cpu_count="`grep '# TORTURE_CPU_COUNT=' $i/qemu-cmd | sed -e 's/^.*=//'`"
78
-
affinity_export="`awk -f $T/cpubatches.awk -v cpu_count="$cpu_count" -v scenario=$i < /dev/null`"
79
-
$affinity_export
78
+
___EOF___
79
+
cpu_count="`grep '# TORTURE_CPU_COUNT=' $i/qemu-cmd | sed -e 's/^.*=//'`"
80
+
affinity_export="`awk -f $T/cpubatches.awk -v cpu_count="$cpu_count" -v scenario=$i < /dev/null`"
81
+
$affinity_export
82
+
fi
80
83
kvm-test-1-run-qemu.sh $i >> $i/kvm-test-1-run-qemu.sh.out 2>&1 &
81
84
done
82
85
for i in $runfiles
+6
tools/testing/selftests/rcutorture/bin/kvm.sh
+6
tools/testing/selftests/rcutorture/bin/kvm.sh
···
42
42
TORTURE_KCONFIG_KASAN_ARG=""
43
43
TORTURE_KCONFIG_KCSAN_ARG=""
44
44
TORTURE_KMAKE_ARG=""
45
+
TORTURE_NO_AFFINITY=""
45
46
TORTURE_QEMU_MEM=512
46
47
torture_qemu_mem_default=1
47
48
TORTURE_REMOTE=
···
83
82
echo " --kmake-arg kernel-make-arguments"
84
83
echo " --mac nn:nn:nn:nn:nn:nn"
85
84
echo " --memory megabytes|nnnG"
85
+
echo " --no-affinity"
86
86
echo " --no-initrd"
87
87
echo " --qemu-args qemu-arguments"
88
88
echo " --qemu-cmd qemu-system-..."
···
221
219
TORTURE_QEMU_MEM=$2
222
220
torture_qemu_mem_default=
223
221
shift
222
+
;;
223
+
--no-affinity)
224
+
TORTURE_NO_AFFINITY="no-affinity"
224
225
;;
225
226
--no-initrd)
226
227
TORTURE_INITRD=""; export TORTURE_INITRD
···
422
417
TORTURE_KCONFIG_KCSAN_ARG="$TORTURE_KCONFIG_KCSAN_ARG"; export TORTURE_KCONFIG_KCSAN_ARG
423
418
TORTURE_KMAKE_ARG="$TORTURE_KMAKE_ARG"; export TORTURE_KMAKE_ARG
424
419
TORTURE_MOD="$TORTURE_MOD"; export TORTURE_MOD
420
+
TORTURE_NO_AFFINITY="$TORTURE_NO_AFFINITY"; export TORTURE_NO_AFFINITY
425
421
TORTURE_QEMU_CMD="$TORTURE_QEMU_CMD"; export TORTURE_QEMU_CMD
426
422
TORTURE_QEMU_INTERACTIVE="$TORTURE_QEMU_INTERACTIVE"; export TORTURE_QEMU_INTERACTIVE
427
423
TORTURE_QEMU_MAC="$TORTURE_QEMU_MAC"; export TORTURE_QEMU_MAC
+1
tools/testing/selftests/rcutorture/configs/rcu/CFLIST
+1
tools/testing/selftests/rcutorture/configs/rcu/CFLIST
+10
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L
+10
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L
+3
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot
+3
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot
+1
tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot
+1
tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot