Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge branch 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
* 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
rcu: Make RCU lockdep check the lockdep_recursion variable
rcu: Update docs for rcu_access_pointer and rcu_dereference_protected
rcu: Better explain the condition parameter of rcu_dereference_check()
rcu: Add rcu_access_pointer and rcu_dereference_protected
+120
-30
+21
-16
Documentation/RCU/NMI-RCU.txt
+21
-16
Documentation/RCU/NMI-RCU.txt
···
34
34
cpu = smp_processor_id();
35
35
++nmi_count(cpu);
36
36
37
-
if (!rcu_dereference(nmi_callback)(regs, cpu))
37
+
if (!rcu_dereference_sched(nmi_callback)(regs, cpu))
38
38
default_do_nmi(regs);
39
39
40
40
nmi_exit();
···
47
47
default_do_nmi() function to handle a machine-specific NMI. Finally,
48
48
preemption is restored.
49
49
50
-
Strictly speaking, rcu_dereference() is not needed, since this code runs
51
-
only on i386, which does not need rcu_dereference() anyway. However,
52
-
it is a good documentation aid, particularly for anyone attempting to
53
-
do something similar on Alpha.
50
+
In theory, rcu_dereference_sched() is not needed, since this code runs
51
+
only on i386, which in theory does not need rcu_dereference_sched()
52
+
anyway. However, in practice it is a good documentation aid, particularly
53
+
for anyone attempting to do something similar on Alpha or on systems
54
+
with aggressive optimizing compilers.
54
55
55
-
Quick Quiz: Why might the rcu_dereference() be necessary on Alpha,
56
+
Quick Quiz: Why might the rcu_dereference_sched() be necessary on Alpha,
56
57
given that the code referenced by the pointer is read-only?
57
58
58
59
···
100
99
101
100
Answer to Quick Quiz
102
101
103
-
Why might the rcu_dereference() be necessary on Alpha, given
102
+
Why might the rcu_dereference_sched() be necessary on Alpha, given
104
103
that the code referenced by the pointer is read-only?
105
104
106
105
Answer: The caller to set_nmi_callback() might well have
107
-
initialized some data that is to be used by the
108
-
new NMI handler. In this case, the rcu_dereference()
109
-
would be needed, because otherwise a CPU that received
110
-
an NMI just after the new handler was set might see
111
-
the pointer to the new NMI handler, but the old
112
-
pre-initialized version of the handler's data.
106
+
initialized some data that is to be used by the new NMI
107
+
handler. In this case, the rcu_dereference_sched() would
108
+
be needed, because otherwise a CPU that received an NMI
109
+
just after the new handler was set might see the pointer
110
+
to the new NMI handler, but the old pre-initialized
111
+
version of the handler's data.
113
112
114
-
More important, the rcu_dereference() makes it clear
115
-
to someone reading the code that the pointer is being
116
-
protected by RCU.
113
+
This same sad story can happen on other CPUs when using
114
+
a compiler with aggressive pointer-value speculation
115
+
optimizations.
116
+
117
+
More important, the rcu_dereference_sched() makes it
118
+
clear to someone reading the code that the pointer is
119
+
being protected by RCU-sched.
+4
-3
Documentation/RCU/checklist.txt
+4
-3
Documentation/RCU/checklist.txt
···
260
260
The reason that it is permissible to use RCU list-traversal
261
261
primitives when the update-side lock is held is that doing so
262
262
can be quite helpful in reducing code bloat when common code is
263
-
shared between readers and updaters.
263
+
shared between readers and updaters. Additional primitives
264
+
are provided for this case, as discussed in lockdep.txt.
264
265
265
266
10. Conversely, if you are in an RCU read-side critical section,
266
267
and you don't hold the appropriate update-side lock, you -must-
···
345
344
requiring SRCU's read-side deadlock immunity or low read-side
346
345
realtime latency.
347
346
348
-
Note that, rcu_assign_pointer() and rcu_dereference() relate to
349
-
SRCU just as they do to other forms of RCU.
347
+
Note that, rcu_assign_pointer() relates to SRCU just as they do
348
+
to other forms of RCU.
350
349
351
350
15. The whole point of call_rcu(), synchronize_rcu(), and friends
352
351
is to wait until all pre-existing readers have finished before
+26
-2
Documentation/RCU/lockdep.txt
+26
-2
Documentation/RCU/lockdep.txt
···
32
32
srcu_dereference(p, sp):
33
33
Check for SRCU read-side critical section.
34
34
rcu_dereference_check(p, c):
35
-
Use explicit check expression "c".
35
+
Use explicit check expression "c". This is useful in
36
+
code that is invoked by both readers and updaters.
36
37
rcu_dereference_raw(p)
37
38
Don't check. (Use sparingly, if at all.)
39
+
rcu_dereference_protected(p, c):
40
+
Use explicit check expression "c", and omit all barriers
41
+
and compiler constraints. This is useful when the data
42
+
structure cannot change, for example, in code that is
43
+
invoked only by updaters.
44
+
rcu_access_pointer(p):
45
+
Return the value of the pointer and omit all barriers,
46
+
but retain the compiler constraints that prevent duplicating
47
+
or coalescsing. This is useful when when testing the
48
+
value of the pointer itself, for example, against NULL.
38
49
39
50
The rcu_dereference_check() check expression can be any boolean
40
51
expression, but would normally include one of the rcu_read_lock_held()
···
70
59
RCU read-side critical sections, in case (2) the ->file_lock prevents
71
60
any change from taking place, and finally, in case (3) the current task
72
61
is the only task accessing the file_struct, again preventing any change
73
-
from taking place.
62
+
from taking place. If the above statement was invoked only from updater
63
+
code, it could instead be written as follows:
64
+
65
+
file = rcu_dereference_protected(fdt->fd[fd],
66
+
lockdep_is_held(&files->file_lock) ||
67
+
atomic_read(&files->count) == 1);
68
+
69
+
This would verify cases #2 and #3 above, and furthermore lockdep would
70
+
complain if this was used in an RCU read-side critical section unless one
71
+
of these two cases held. Because rcu_dereference_protected() omits all
72
+
barriers and compiler constraints, it generates better code than do the
73
+
other flavors of rcu_dereference(). On the other hand, it is illegal
74
+
to use rcu_dereference_protected() if either the RCU-protected pointer
75
+
or the RCU-protected data that it points to can change concurrently.
74
76
75
77
There are currently only "universal" versions of the rcu_assign_pointer()
76
78
and RCU list-/tree-traversal primitives, which do not (yet) check for
+6
Documentation/RCU/whatisRCU.txt
+6
Documentation/RCU/whatisRCU.txt
···
840
840
init_srcu_struct
841
841
cleanup_srcu_struct
842
842
843
+
All: lockdep-checked RCU-protected pointer access
844
+
845
+
rcu_dereference_check
846
+
rcu_dereference_protected
847
+
rcu_access_pointer
848
+
843
849
See the comment headers in the source code (or the docbook generated
844
850
from them) for more information.
845
851
+56
-9
include/linux/rcupdate.h
+56
-9
include/linux/rcupdate.h
···
101
101
# define rcu_read_release_sched() \
102
102
lock_release(&rcu_sched_lock_map, 1, _THIS_IP_)
103
103
104
-
static inline int debug_lockdep_rcu_enabled(void)
105
-
{
106
-
return likely(rcu_scheduler_active && debug_locks);
107
-
}
104
+
extern int debug_lockdep_rcu_enabled(void);
108
105
109
106
/**
110
107
* rcu_read_lock_held - might we be in RCU read-side critical section?
···
192
195
193
196
/**
194
197
* rcu_dereference_check - rcu_dereference with debug checking
198
+
* @p: The pointer to read, prior to dereferencing
199
+
* @c: The conditions under which the dereference will take place
195
200
*
196
-
* Do an rcu_dereference(), but check that the context is correct.
197
-
* For example, rcu_dereference_check(gp, rcu_read_lock_held()) to
198
-
* ensure that the rcu_dereference_check() executes within an RCU
199
-
* read-side critical section. It is also possible to check for
200
-
* locks being held, for example, by using lockdep_is_held().
201
+
* Do an rcu_dereference(), but check that the conditions under which the
202
+
* dereference will take place are correct. Typically the conditions indicate
203
+
* the various locking conditions that should be held at that point. The check
204
+
* should return true if the conditions are satisfied.
205
+
*
206
+
* For example:
207
+
*
208
+
* bar = rcu_dereference_check(foo->bar, rcu_read_lock_held() ||
209
+
* lockdep_is_held(&foo->lock));
210
+
*
211
+
* could be used to indicate to lockdep that foo->bar may only be dereferenced
212
+
* if either the RCU read lock is held, or that the lock required to replace
213
+
* the bar struct at foo->bar is held.
214
+
*
215
+
* Note that the list of conditions may also include indications of when a lock
216
+
* need not be held, for example during initialisation or destruction of the
217
+
* target struct:
218
+
*
219
+
* bar = rcu_dereference_check(foo->bar, rcu_read_lock_held() ||
220
+
* lockdep_is_held(&foo->lock) ||
221
+
* atomic_read(&foo->usage) == 0);
201
222
*/
202
223
#define rcu_dereference_check(p, c) \
203
224
({ \
···
224
209
rcu_dereference_raw(p); \
225
210
})
226
211
212
+
/**
213
+
* rcu_dereference_protected - fetch RCU pointer when updates prevented
214
+
*
215
+
* Return the value of the specified RCU-protected pointer, but omit
216
+
* both the smp_read_barrier_depends() and the ACCESS_ONCE(). This
217
+
* is useful in cases where update-side locks prevent the value of the
218
+
* pointer from changing. Please note that this primitive does -not-
219
+
* prevent the compiler from repeating this reference or combining it
220
+
* with other references, so it should not be used without protection
221
+
* of appropriate locks.
222
+
*/
223
+
#define rcu_dereference_protected(p, c) \
224
+
({ \
225
+
if (debug_lockdep_rcu_enabled() && !(c)) \
226
+
lockdep_rcu_dereference(__FILE__, __LINE__); \
227
+
(p); \
228
+
})
229
+
227
230
#else /* #ifdef CONFIG_PROVE_RCU */
228
231
229
232
#define rcu_dereference_check(p, c) rcu_dereference_raw(p)
233
+
#define rcu_dereference_protected(p, c) (p)
230
234
231
235
#endif /* #else #ifdef CONFIG_PROVE_RCU */
236
+
237
+
/**
238
+
* rcu_access_pointer - fetch RCU pointer with no dereferencing
239
+
*
240
+
* Return the value of the specified RCU-protected pointer, but omit the
241
+
* smp_read_barrier_depends() and keep the ACCESS_ONCE(). This is useful
242
+
* when the value of this pointer is accessed, but the pointer is not
243
+
* dereferenced, for example, when testing an RCU-protected pointer against
244
+
* NULL. This may also be used in cases where update-side locks prevent
245
+
* the value of the pointer from changing, but rcu_dereference_protected()
246
+
* is a lighter-weight primitive for this use case.
247
+
*/
248
+
#define rcu_access_pointer(p) ACCESS_ONCE(p)
232
249
233
250
/**
234
251
* rcu_read_lock - mark the beginning of an RCU read-side critical section.
+7
kernel/rcupdate.c
+7
kernel/rcupdate.c
···
69
69
70
70
#ifdef CONFIG_DEBUG_LOCK_ALLOC
71
71
72
+
int debug_lockdep_rcu_enabled(void)
73
+
{
74
+
return rcu_scheduler_active && debug_locks &&
75
+
current->lockdep_recursion == 0;
76
+
}
77
+
EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled);
78
+
72
79
/**
73
80
* rcu_read_lock_bh_held - might we be in RCU-bh read-side critical section?
74
81
*