Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
doc: Update and wordsmith rculist_nulls.rst
Do some wordsmithing and breaking up of RCU readers.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
+57
-58
+57
-58
Documentation/RCU/rculist_nulls.rst
+57
-58
Documentation/RCU/rculist_nulls.rst
···
14
14
=============
15
15
16
16
Using special makers (called 'nulls') is a convenient way
17
-
to solve following problem :
17
+
to solve following problem.
18
18
19
-
A typical RCU linked list managing objects which are
20
-
allocated with SLAB_TYPESAFE_BY_RCU kmem_cache can
21
-
use following algos :
19
+
Without 'nulls', a typical RCU linked list managing objects which are
20
+
allocated with SLAB_TYPESAFE_BY_RCU kmem_cache can use the following
21
+
algorithms:
22
22
23
-
1) Lookup algo
24
-
--------------
23
+
1) Lookup algorithm
24
+
-------------------
25
25
26
26
::
27
27
28
-
rcu_read_lock()
29
28
begin:
29
+
rcu_read_lock()
30
30
obj = lockless_lookup(key);
31
31
if (obj) {
32
32
if (!try_get_ref(obj)) // might fail for free objects
···
38
38
*/
39
39
if (obj->key != key) { // not the object we expected
40
40
put_ref(obj);
41
+
rcu_read_unlock();
41
42
goto begin;
42
43
}
43
44
}
···
53
52
{
54
53
struct hlist_node *node, *next;
55
54
for (pos = rcu_dereference((head)->first);
56
-
pos && ({ next = pos->next; smp_rmb(); prefetch(next); 1; }) &&
57
-
({ tpos = hlist_entry(pos, typeof(*tpos), member); 1; });
58
-
pos = rcu_dereference(next))
55
+
pos && ({ next = pos->next; smp_rmb(); prefetch(next); 1; }) &&
56
+
({ tpos = hlist_entry(pos, typeof(*tpos), member); 1; });
57
+
pos = rcu_dereference(next))
59
58
if (obj->key == key)
60
59
return obj;
61
60
return NULL;
···
65
64
66
65
struct hlist_node *node;
67
66
for (pos = rcu_dereference((head)->first);
68
-
pos && ({ prefetch(pos->next); 1; }) &&
69
-
({ tpos = hlist_entry(pos, typeof(*tpos), member); 1; });
70
-
pos = rcu_dereference(pos->next))
67
+
pos && ({ prefetch(pos->next); 1; }) &&
68
+
({ tpos = hlist_entry(pos, typeof(*tpos), member); 1; });
69
+
pos = rcu_dereference(pos->next))
71
70
if (obj->key == key)
72
71
return obj;
73
72
return NULL;
···
83
82
solved by pre-fetching the "next" field (with proper barriers) before
84
83
checking the key."
85
84
86
-
2) Insert algo
87
-
--------------
85
+
2) Insertion algorithm
86
+
----------------------
88
87
89
88
We need to make sure a reader cannot read the new 'obj->obj_next' value
90
-
and previous value of 'obj->key'. Or else, an item could be deleted
89
+
and previous value of 'obj->key'. Otherwise, an item could be deleted
91
90
from a chain, and inserted into another chain. If new chain was empty
92
-
before the move, 'next' pointer is NULL, and lockless reader can
93
-
not detect it missed following items in original chain.
91
+
before the move, 'next' pointer is NULL, and lockless reader can not
92
+
detect the fact that it missed following items in original chain.
94
93
95
94
::
96
95
97
96
/*
98
-
* Please note that new inserts are done at the head of list,
99
-
* not in the middle or end.
100
-
*/
97
+
* Please note that new inserts are done at the head of list,
98
+
* not in the middle or end.
99
+
*/
101
100
obj = kmem_cache_alloc(...);
102
101
lock_chain(); // typically a spin_lock()
103
102
obj->key = key;
104
-
/*
105
-
* we need to make sure obj->key is updated before obj->next
106
-
* or obj->refcnt
107
-
*/
108
-
smp_wmb();
109
-
atomic_set(&obj->refcnt, 1);
103
+
atomic_set_release(&obj->refcnt, 1); // key before refcnt
110
104
hlist_add_head_rcu(&obj->obj_node, list);
111
105
unlock_chain(); // typically a spin_unlock()
112
106
113
107
114
-
3) Remove algo
115
-
--------------
108
+
3) Removal algorithm
109
+
--------------------
110
+
116
111
Nothing special here, we can use a standard RCU hlist deletion.
117
112
But thanks to SLAB_TYPESAFE_BY_RCU, beware a deleted object can be reused
118
113
very very fast (before the end of RCU grace period)
···
130
133
========================
131
134
132
135
With hlist_nulls we can avoid extra smp_rmb() in lockless_lookup()
133
-
and extra smp_wmb() in insert function.
136
+
and extra _release() in insert function.
134
137
135
138
For example, if we choose to store the slot number as the 'nulls'
136
139
end-of-list marker for each slot of the hash table, we can detect
···
139
142
the lookup met the end of chain. If final 'nulls' value
140
143
is not the slot number, then we must restart the lookup at
141
144
the beginning. If the object was moved to the same chain,
142
-
then the reader doesn't care : It might eventually
145
+
then the reader doesn't care: It might occasionally
143
146
scan the list again without harm.
144
147
145
148
146
-
1) lookup algo
147
-
--------------
149
+
1) lookup algorithm
150
+
-------------------
148
151
149
152
::
150
153
151
154
head = &table[slot];
152
-
rcu_read_lock();
153
155
begin:
156
+
rcu_read_lock();
154
157
hlist_nulls_for_each_entry_rcu(obj, node, head, member) {
155
158
if (obj->key == key) {
156
-
if (!try_get_ref(obj)) // might fail for free objects
157
-
goto begin;
158
-
if (obj->key != key) { // not the object we expected
159
-
put_ref(obj);
159
+
if (!try_get_ref(obj)) { // might fail for free objects
160
+
rcu_read_unlock();
160
161
goto begin;
161
162
}
162
-
goto out;
163
+
if (obj->key != key) { // not the object we expected
164
+
put_ref(obj);
165
+
rcu_read_unlock();
166
+
goto begin;
167
+
}
168
+
goto out;
169
+
}
163
170
}
164
-
/*
165
-
* if the nulls value we got at the end of this lookup is
166
-
* not the expected one, we must restart lookup.
167
-
* We probably met an item that was moved to another chain.
168
-
*/
169
-
if (get_nulls_value(node) != slot)
170
-
goto begin;
171
+
172
+
// If the nulls value we got at the end of this lookup is
173
+
// not the expected one, we must restart lookup.
174
+
// We probably met an item that was moved to another chain.
175
+
if (get_nulls_value(node) != slot) {
176
+
put_ref(obj);
177
+
rcu_read_unlock();
178
+
goto begin;
179
+
}
171
180
obj = NULL;
172
181
173
182
out:
174
183
rcu_read_unlock();
175
184
176
-
2) Insert function
177
-
------------------
185
+
2) Insert algorithm
186
+
-------------------
178
187
179
188
::
180
189
181
190
/*
182
-
* Please note that new inserts are done at the head of list,
183
-
* not in the middle or end.
184
-
*/
191
+
* Please note that new inserts are done at the head of list,
192
+
* not in the middle or end.
193
+
*/
185
194
obj = kmem_cache_alloc(cachep);
186
195
lock_chain(); // typically a spin_lock()
187
196
obj->key = key;
197
+
atomic_set_release(&obj->refcnt, 1); // key before refcnt
188
198
/*
189
-
* changes to obj->key must be visible before refcnt one
190
-
*/
191
-
smp_wmb();
192
-
atomic_set(&obj->refcnt, 1);
193
-
/*
194
-
* insert obj in RCU way (readers might be traversing chain)
195
-
*/
199
+
* insert obj in RCU way (readers might be traversing chain)
200
+
*/
196
201
hlist_nulls_add_head_rcu(&obj->obj_node, list);
197
202
unlock_chain(); // typically a spin_unlock()