Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'lkmm.2025.03.21a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu
Pull kernel memory model updates from Paul McKenney:
"Add more atomic operations, rework tags, and update documentation:
- Add additional atomic operations (Puranjay Mohan)
- Make better use of herd7 tags (Jonas Oberhauser)
- Update documentation (Akira Yokosawa)
These changes require v7.58 of the herd7 and klitmus tools, up from
v7.52"
* tag 'lkmm.2025.03.21a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu:
tools/memory-model: glossary.txt: Fix indents
tools/memory-model/README: Fix typo
tools/memory-model: Distinguish between syntactic and semantic tags
tools/memory-model: Switch to softcoded herd7 tags
tools/memory-model: Define effect of Mb tags on RMWs in tools/...
tools/memory-model: Define applicable tags on operation in tools/...
tools/memory-model: Legitimize current use of tags in LKMM macros
tools/memory-model: Add atomic_andnot() with its variants
tools/memory-model: Add atomic_and()/or()/xor() and add_negative
+172
-114
+16
-16
tools/memory-model/Documentation/glossary.txt
+16
-16
tools/memory-model/Documentation/glossary.txt
···
15
15
3 do_something(p->a);
16
16
4 rcu_read_unlock();
17
17
18
-
In this case, because the address of "p->a" on line 3 is computed
19
-
from the value returned by the rcu_dereference() on line 2, the
20
-
address dependency extends from that rcu_dereference() to that
21
-
"p->a". In rare cases, optimizing compilers can destroy address
22
-
dependencies. Please see Documentation/RCU/rcu_dereference.rst
23
-
for more information.
18
+
In this case, because the address of "p->a" on line 3 is computed
19
+
from the value returned by the rcu_dereference() on line 2, the
20
+
address dependency extends from that rcu_dereference() to that
21
+
"p->a". In rare cases, optimizing compilers can destroy address
22
+
dependencies. Please see Documentation/RCU/rcu_dereference.rst
23
+
for more information.
24
24
25
-
See also "Control Dependency" and "Data Dependency".
25
+
See also "Control Dependency" and "Data Dependency".
26
26
27
27
Acquire: With respect to a lock, acquiring that lock, for example,
28
28
using spin_lock(). With respect to a non-lock shared variable,
···
59
59
1 if (READ_ONCE(x))
60
60
2 WRITE_ONCE(y, 1);
61
61
62
-
Here, the control dependency extends from the READ_ONCE() on
63
-
line 1 to the WRITE_ONCE() on line 2. Control dependencies are
64
-
fragile, and can be easily destroyed by optimizing compilers.
65
-
Please see control-dependencies.txt for more information.
62
+
Here, the control dependency extends from the READ_ONCE() on
63
+
line 1 to the WRITE_ONCE() on line 2. Control dependencies are
64
+
fragile, and can be easily destroyed by optimizing compilers.
65
+
Please see control-dependencies.txt for more information.
66
66
67
-
See also "Address Dependency" and "Data Dependency".
67
+
See also "Address Dependency" and "Data Dependency".
68
68
69
69
Cycle: Memory-barrier pairing is restricted to a pair of CPUs, as the
70
70
name suggests. And in a great many cases, a pair of CPUs is all
···
72
72
extended to additional CPUs, and the result is called a "cycle".
73
73
In a cycle, each CPU's ordering interacts with that of the next:
74
74
75
-
CPU 0 CPU 1 CPU 2
76
-
WRITE_ONCE(x, 1); WRITE_ONCE(y, 1); WRITE_ONCE(z, 1);
77
-
smp_mb(); smp_mb(); smp_mb();
78
-
r0 = READ_ONCE(y); r1 = READ_ONCE(z); r2 = READ_ONCE(x);
75
+
CPU 0 CPU 1 CPU 2
76
+
WRITE_ONCE(x, 1); WRITE_ONCE(y, 1); WRITE_ONCE(z, 1);
77
+
smp_mb(); smp_mb(); smp_mb();
78
+
r0 = READ_ONCE(y); r1 = READ_ONCE(z); r2 = READ_ONCE(x);
79
79
80
80
CPU 0's smp_mb() interacts with that of CPU 1, which interacts
81
81
with that of CPU 2, which in turn interacts with that of CPU 0
+26
-23
tools/memory-model/Documentation/herd-representation.txt
+26
-23
tools/memory-model/Documentation/herd-representation.txt
···
18
18
#
19
19
# By convention, a blank line in a cell means "same as the preceding line".
20
20
#
21
+
# Note that the syntactic representation does not always match the sets and
22
+
# relations in linux-kernel.cat, due to redefinitions in linux-kernel.bell and
23
+
# lock.cat. For example, the po link between LKR and LKW is upgraded to an rmw
24
+
# link, and W[ACQUIRE] are not included in the Acquire set.
25
+
#
21
26
# Disclaimer. The table includes representations of "add" and "and" operations;
22
27
# corresponding/identical representations of "sub", "inc", "dec" and "or", "xor",
23
28
# "andnot" operations are omitted.
···
32
27
------------------------------------------------------------------------------
33
28
| Non-RMW ops | |
34
29
------------------------------------------------------------------------------
35
-
| READ_ONCE | R[once] |
30
+
| READ_ONCE | R[ONCE] |
36
31
| atomic_read | |
37
-
| WRITE_ONCE | W[once] |
32
+
| WRITE_ONCE | W[ONCE] |
38
33
| atomic_set | |
39
-
| smp_load_acquire | R[acquire] |
34
+
| smp_load_acquire | R[ACQUIRE] |
40
35
| atomic_read_acquire | |
41
-
| smp_store_release | W[release] |
36
+
| smp_store_release | W[RELEASE] |
42
37
| atomic_set_release | |
43
-
| smp_store_mb | W[once] ->po F[mb] |
44
-
| smp_mb | F[mb] |
38
+
| smp_store_mb | W[ONCE] ->po F[MB] |
39
+
| smp_mb | F[MB] |
45
40
| smp_rmb | F[rmb] |
46
41
| smp_wmb | F[wmb] |
47
42
| smp_mb__before_atomic | F[before-atomic] |
···
54
49
| rcu_read_lock | F[rcu-lock] |
55
50
| rcu_read_unlock | F[rcu-unlock] |
56
51
| synchronize_rcu | F[sync-rcu] |
57
-
| rcu_dereference | R[once] |
58
-
| rcu_assign_pointer | W[release] |
52
+
| rcu_dereference | R[ONCE] |
53
+
| rcu_assign_pointer | W[RELEASE] |
59
54
| srcu_read_lock | R[srcu-lock] |
60
55
| srcu_down_read | |
61
56
| srcu_read_unlock | W[srcu-unlock] |
···
65
60
------------------------------------------------------------------------------
66
61
| RMW ops w/o return value | |
67
62
------------------------------------------------------------------------------
68
-
| atomic_add | R*[noreturn] ->rmw W*[once] |
63
+
| atomic_add | R*[NORETURN] ->rmw W*[NORETURN] |
69
64
| atomic_and | |
70
65
| spin_lock | LKR ->po LKW |
71
66
------------------------------------------------------------------------------
72
67
| RMW ops w/ return value | |
73
68
------------------------------------------------------------------------------
74
-
| atomic_add_return | F[mb] ->po R*[once] |
75
-
| | ->rmw W*[once] ->po F[mb] |
69
+
| atomic_add_return | R*[MB] ->rmw W*[MB] |
76
70
| atomic_fetch_add | |
77
71
| atomic_fetch_and | |
78
72
| atomic_xchg | |
79
73
| xchg | |
80
74
| atomic_add_negative | |
81
-
| atomic_add_return_relaxed | R*[once] ->rmw W*[once] |
75
+
| atomic_add_return_relaxed | R*[ONCE] ->rmw W*[ONCE] |
82
76
| atomic_fetch_add_relaxed | |
83
77
| atomic_fetch_and_relaxed | |
84
78
| atomic_xchg_relaxed | |
85
79
| xchg_relaxed | |
86
80
| atomic_add_negative_relaxed | |
87
-
| atomic_add_return_acquire | R*[acquire] ->rmw W*[once] |
81
+
| atomic_add_return_acquire | R*[ACQUIRE] ->rmw W*[ACQUIRE] |
88
82
| atomic_fetch_add_acquire | |
89
83
| atomic_fetch_and_acquire | |
90
84
| atomic_xchg_acquire | |
91
85
| xchg_acquire | |
92
86
| atomic_add_negative_acquire | |
93
-
| atomic_add_return_release | R*[once] ->rmw W*[release] |
87
+
| atomic_add_return_release | R*[RELEASE] ->rmw W*[RELEASE] |
94
88
| atomic_fetch_add_release | |
95
89
| atomic_fetch_and_release | |
96
90
| atomic_xchg_release | |
···
98
94
------------------------------------------------------------------------------
99
95
| Conditional RMW ops | |
100
96
------------------------------------------------------------------------------
101
-
| atomic_cmpxchg | On success: F[mb] ->po R*[once] |
102
-
| | ->rmw W*[once] ->po F[mb] |
103
-
| | On failure: R*[once] |
97
+
| atomic_cmpxchg | On success: R*[MB] ->rmw W*[MB] |
98
+
| | On failure: R*[MB] |
104
99
| cmpxchg | |
105
100
| atomic_add_unless | |
106
-
| atomic_cmpxchg_relaxed | On success: R*[once] ->rmw W*[once] |
107
-
| | On failure: R*[once] |
108
-
| atomic_cmpxchg_acquire | On success: R*[acquire] ->rmw W*[once] |
109
-
| | On failure: R*[once] |
110
-
| atomic_cmpxchg_release | On success: R*[once] ->rmw W*[release] |
111
-
| | On failure: R*[once] |
101
+
| atomic_cmpxchg_relaxed | On success: R*[ONCE] ->rmw W*[ONCE] |
102
+
| | On failure: R*[ONCE] |
103
+
| atomic_cmpxchg_acquire | On success: R*[ACQUIRE] ->rmw W*[ACQUIRE] |
104
+
| | On failure: R*[ACQUIRE] |
105
+
| atomic_cmpxchg_release | On success: R*[RELEASE] ->rmw W*[RELEASE] |
106
+
| | On failure: R*[RELEASE] |
112
107
| spin_trylock | On success: LKR ->po LKW |
113
108
| | On failure: LF |
114
109
------------------------------------------------------------------------------
+2
-2
tools/memory-model/README
+2
-2
tools/memory-model/README
···
20
20
REQUIREMENTS
21
21
============
22
22
23
-
Version 7.52 or higher of the "herd7" and "klitmus7" tools must be
23
+
Version 7.58 or higher of the "herd7" and "klitmus7" tools must be
24
24
downloaded separately:
25
25
26
26
https://github.com/herd/herdtools7
···
79
79
https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git/tree/CodeSamples/formal/herd
80
80
https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git/tree/CodeSamples/formal/litmus
81
81
82
-
Documentation describing litmus tests and now to use them may be found
82
+
Documentation describing litmus tests and how to use them may be found
83
83
here:
84
84
85
85
tools/memory-model/Documentation/litmus-tests.txt
+24
-9
tools/memory-model/linux-kernel.bell
+24
-9
tools/memory-model/linux-kernel.bell
···
13
13
14
14
"Linux-kernel memory consistency model"
15
15
16
-
enum Accesses = 'once (*READ_ONCE,WRITE_ONCE*) ||
17
-
'release (*smp_store_release*) ||
18
-
'acquire (*smp_load_acquire*) ||
19
-
'noreturn (* R of non-return RMW *)
20
-
instructions R[{'once,'acquire,'noreturn}]
21
-
instructions W[{'once,'release}]
22
-
instructions RMW[{'once,'acquire,'release}]
16
+
enum Accesses = 'ONCE (*READ_ONCE,WRITE_ONCE*) ||
17
+
'RELEASE (*smp_store_release*) ||
18
+
'ACQUIRE (*smp_load_acquire*) ||
19
+
'NORETURN (* R of non-return RMW *) ||
20
+
'MB (*xchg(),cmpxchg(),...*)
21
+
instructions R[Accesses]
22
+
instructions W[Accesses]
23
+
instructions RMW[Accesses]
23
24
24
25
enum Barriers = 'wmb (*smp_wmb*) ||
25
26
'rmb (*smp_rmb*) ||
26
-
'mb (*smp_mb*) ||
27
+
'MB (*smp_mb*) ||
27
28
'barrier (*barrier*) ||
28
29
'rcu-lock (*rcu_read_lock*) ||
29
30
'rcu-unlock (*rcu_read_unlock*) ||
···
35
34
'after-unlock-lock (*smp_mb__after_unlock_lock*) ||
36
35
'after-srcu-read-unlock (*smp_mb__after_srcu_read_unlock*)
37
36
instructions F[Barriers]
37
+
38
+
39
+
(*
40
+
* Filter out syntactic annotations that do not provide the corresponding
41
+
* semantic ordering, such as Acquire on a store or Mb on a failed RMW.
42
+
*)
43
+
let FailedRMW = RMW \ (domain(rmw) | range(rmw))
44
+
let Acquire = ACQUIRE \ W \ FailedRMW
45
+
let Release = RELEASE \ R \ FailedRMW
46
+
let Mb = MB \ FailedRMW
47
+
let Noreturn = NORETURN \ W
38
48
39
49
(* SRCU *)
40
50
enum SRCU = 'srcu-lock || 'srcu-unlock || 'sync-srcu
···
85
73
flag ~empty different-values(srcu-rscs) as srcu-bad-value-match
86
74
87
75
(* Compute marked and plain memory accesses *)
88
-
let Marked = (~M) | IW | Once | Release | Acquire | domain(rmw) | range(rmw) |
76
+
let Marked = (~M) | IW | ONCE | RELEASE | ACQUIRE | MB | RMW |
89
77
LKR | LKW | UL | LF | RL | RU | Srcu-lock | Srcu-unlock
90
78
let Plain = M \ Marked
91
79
···
94
82
let addr = carry-dep ; addr
95
83
let ctrl = carry-dep ; ctrl
96
84
let data = carry-dep ; data
85
+
86
+
flag ~empty (if "lkmmv2" then 0 else _)
87
+
as this-model-requires-variant-higher-than-lkmmv1
+10
tools/memory-model/linux-kernel.cat
+10
tools/memory-model/linux-kernel.cat
···
34
34
let rmb = [R4rmb] ; fencerel(Rmb) ; [R4rmb]
35
35
let wmb = [W] ; fencerel(Wmb) ; [W]
36
36
let mb = ([M] ; fencerel(Mb) ; [M]) |
37
+
(*
38
+
* full-barrier RMWs (successful cmpxchg(), xchg(), etc.) act as
39
+
* though there were enclosed by smp_mb().
40
+
* The effect of these virtual smp_mb() is formalized by adding
41
+
* Mb tags to the read and write of the operation, and providing
42
+
* the same ordering as though there were additional po edges
43
+
* between the Mb tag and the read resp. write.
44
+
*)
45
+
([M] ; po ; [Mb & R]) |
46
+
([Mb & W] ; po ; [M]) |
37
47
([M] ; fencerel(Before-atomic) ; [RMW] ; po? ; [M]) |
38
48
([M] ; po? ; [RMW] ; fencerel(After-atomic) ; [M]) |
39
49
([M] ; po? ; [LKW] ; fencerel(After-spinlock) ; [M]) |
+1
tools/memory-model/linux-kernel.cfg
+1
tools/memory-model/linux-kernel.cfg
+93
-64
tools/memory-model/linux-kernel.def
+93
-64
tools/memory-model/linux-kernel.def
···
6
6
// which appeared in ASPLOS 2018.
7
7
8
8
// ONCE
9
-
READ_ONCE(X) __load{once}(X)
10
-
WRITE_ONCE(X,V) { __store{once}(X,V); }
9
+
READ_ONCE(X) __load{ONCE}(X)
10
+
WRITE_ONCE(X,V) { __store{ONCE}(X,V); }
11
11
12
12
// Release Acquire and friends
13
-
smp_store_release(X,V) { __store{release}(*X,V); }
14
-
smp_load_acquire(X) __load{acquire}(*X)
15
-
rcu_assign_pointer(X,V) { __store{release}(X,V); }
16
-
rcu_dereference(X) __load{once}(X)
17
-
smp_store_mb(X,V) { __store{once}(X,V); __fence{mb}; }
13
+
smp_store_release(X,V) { __store{RELEASE}(*X,V); }
14
+
smp_load_acquire(X) __load{ACQUIRE}(*X)
15
+
rcu_assign_pointer(X,V) { __store{RELEASE}(X,V); }
16
+
rcu_dereference(X) __load{ONCE}(X)
17
+
smp_store_mb(X,V) { __store{ONCE}(X,V); __fence{MB}; }
18
18
19
19
// Fences
20
-
smp_mb() { __fence{mb}; }
20
+
smp_mb() { __fence{MB}; }
21
21
smp_rmb() { __fence{rmb}; }
22
22
smp_wmb() { __fence{wmb}; }
23
23
smp_mb__before_atomic() { __fence{before-atomic}; }
···
28
28
barrier() { __fence{barrier}; }
29
29
30
30
// Exchange
31
-
xchg(X,V) __xchg{mb}(X,V)
32
-
xchg_relaxed(X,V) __xchg{once}(X,V)
33
-
xchg_release(X,V) __xchg{release}(X,V)
34
-
xchg_acquire(X,V) __xchg{acquire}(X,V)
35
-
cmpxchg(X,V,W) __cmpxchg{mb}(X,V,W)
36
-
cmpxchg_relaxed(X,V,W) __cmpxchg{once}(X,V,W)
37
-
cmpxchg_acquire(X,V,W) __cmpxchg{acquire}(X,V,W)
38
-
cmpxchg_release(X,V,W) __cmpxchg{release}(X,V,W)
31
+
xchg(X,V) __xchg{MB}(X,V)
32
+
xchg_relaxed(X,V) __xchg{ONCE}(X,V)
33
+
xchg_release(X,V) __xchg{RELEASE}(X,V)
34
+
xchg_acquire(X,V) __xchg{ACQUIRE}(X,V)
35
+
cmpxchg(X,V,W) __cmpxchg{MB}(X,V,W)
36
+
cmpxchg_relaxed(X,V,W) __cmpxchg{ONCE}(X,V,W)
37
+
cmpxchg_acquire(X,V,W) __cmpxchg{ACQUIRE}(X,V,W)
38
+
cmpxchg_release(X,V,W) __cmpxchg{RELEASE}(X,V,W)
39
39
40
40
// Spinlocks
41
41
spin_lock(X) { __lock(X); }
···
63
63
atomic_read_acquire(X) smp_load_acquire(X)
64
64
atomic_set_release(X,V) { smp_store_release(X,V); }
65
65
66
-
atomic_add(V,X) { __atomic_op(X,+,V); }
67
-
atomic_sub(V,X) { __atomic_op(X,-,V); }
68
-
atomic_inc(X) { __atomic_op(X,+,1); }
69
-
atomic_dec(X) { __atomic_op(X,-,1); }
66
+
atomic_add(V,X) { __atomic_op{NORETURN}(X,+,V); }
67
+
atomic_sub(V,X) { __atomic_op{NORETURN}(X,-,V); }
68
+
atomic_and(V,X) { __atomic_op{NORETURN}(X,&,V); }
69
+
atomic_or(V,X) { __atomic_op{NORETURN}(X,|,V); }
70
+
atomic_xor(V,X) { __atomic_op{NORETURN}(X,^,V); }
71
+
atomic_inc(X) { __atomic_op{NORETURN}(X,+,1); }
72
+
atomic_dec(X) { __atomic_op{NORETURN}(X,-,1); }
73
+
atomic_andnot(V,X) { __atomic_op{NORETURN}(X,&~,V); }
70
74
71
-
atomic_add_return(V,X) __atomic_op_return{mb}(X,+,V)
72
-
atomic_add_return_relaxed(V,X) __atomic_op_return{once}(X,+,V)
73
-
atomic_add_return_acquire(V,X) __atomic_op_return{acquire}(X,+,V)
74
-
atomic_add_return_release(V,X) __atomic_op_return{release}(X,+,V)
75
-
atomic_fetch_add(V,X) __atomic_fetch_op{mb}(X,+,V)
76
-
atomic_fetch_add_relaxed(V,X) __atomic_fetch_op{once}(X,+,V)
77
-
atomic_fetch_add_acquire(V,X) __atomic_fetch_op{acquire}(X,+,V)
78
-
atomic_fetch_add_release(V,X) __atomic_fetch_op{release}(X,+,V)
75
+
atomic_add_return(V,X) __atomic_op_return{MB}(X,+,V)
76
+
atomic_add_return_relaxed(V,X) __atomic_op_return{ONCE}(X,+,V)
77
+
atomic_add_return_acquire(V,X) __atomic_op_return{ACQUIRE}(X,+,V)
78
+
atomic_add_return_release(V,X) __atomic_op_return{RELEASE}(X,+,V)
79
+
atomic_fetch_add(V,X) __atomic_fetch_op{MB}(X,+,V)
80
+
atomic_fetch_add_relaxed(V,X) __atomic_fetch_op{ONCE}(X,+,V)
81
+
atomic_fetch_add_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,+,V)
82
+
atomic_fetch_add_release(V,X) __atomic_fetch_op{RELEASE}(X,+,V)
79
83
80
-
atomic_inc_return(X) __atomic_op_return{mb}(X,+,1)
81
-
atomic_inc_return_relaxed(X) __atomic_op_return{once}(X,+,1)
82
-
atomic_inc_return_acquire(X) __atomic_op_return{acquire}(X,+,1)
83
-
atomic_inc_return_release(X) __atomic_op_return{release}(X,+,1)
84
-
atomic_fetch_inc(X) __atomic_fetch_op{mb}(X,+,1)
85
-
atomic_fetch_inc_relaxed(X) __atomic_fetch_op{once}(X,+,1)
86
-
atomic_fetch_inc_acquire(X) __atomic_fetch_op{acquire}(X,+,1)
87
-
atomic_fetch_inc_release(X) __atomic_fetch_op{release}(X,+,1)
84
+
atomic_fetch_and(V,X) __atomic_fetch_op{MB}(X,&,V)
85
+
atomic_fetch_and_relaxed(V,X) __atomic_fetch_op{ONCE}(X,&,V)
86
+
atomic_fetch_and_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,&,V)
87
+
atomic_fetch_and_release(V,X) __atomic_fetch_op{RELEASE}(X,&,V)
88
88
89
-
atomic_sub_return(V,X) __atomic_op_return{mb}(X,-,V)
90
-
atomic_sub_return_relaxed(V,X) __atomic_op_return{once}(X,-,V)
91
-
atomic_sub_return_acquire(V,X) __atomic_op_return{acquire}(X,-,V)
92
-
atomic_sub_return_release(V,X) __atomic_op_return{release}(X,-,V)
93
-
atomic_fetch_sub(V,X) __atomic_fetch_op{mb}(X,-,V)
94
-
atomic_fetch_sub_relaxed(V,X) __atomic_fetch_op{once}(X,-,V)
95
-
atomic_fetch_sub_acquire(V,X) __atomic_fetch_op{acquire}(X,-,V)
96
-
atomic_fetch_sub_release(V,X) __atomic_fetch_op{release}(X,-,V)
89
+
atomic_fetch_or(V,X) __atomic_fetch_op{MB}(X,|,V)
90
+
atomic_fetch_or_relaxed(V,X) __atomic_fetch_op{ONCE}(X,|,V)
91
+
atomic_fetch_or_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,|,V)
92
+
atomic_fetch_or_release(V,X) __atomic_fetch_op{RELEASE}(X,|,V)
97
93
98
-
atomic_dec_return(X) __atomic_op_return{mb}(X,-,1)
99
-
atomic_dec_return_relaxed(X) __atomic_op_return{once}(X,-,1)
100
-
atomic_dec_return_acquire(X) __atomic_op_return{acquire}(X,-,1)
101
-
atomic_dec_return_release(X) __atomic_op_return{release}(X,-,1)
102
-
atomic_fetch_dec(X) __atomic_fetch_op{mb}(X,-,1)
103
-
atomic_fetch_dec_relaxed(X) __atomic_fetch_op{once}(X,-,1)
104
-
atomic_fetch_dec_acquire(X) __atomic_fetch_op{acquire}(X,-,1)
105
-
atomic_fetch_dec_release(X) __atomic_fetch_op{release}(X,-,1)
94
+
atomic_fetch_xor(V,X) __atomic_fetch_op{MB}(X,^,V)
95
+
atomic_fetch_xor_relaxed(V,X) __atomic_fetch_op{ONCE}(X,^,V)
96
+
atomic_fetch_xor_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,^,V)
97
+
atomic_fetch_xor_release(V,X) __atomic_fetch_op{RELEASE}(X,^,V)
106
98
107
-
atomic_xchg(X,V) __xchg{mb}(X,V)
108
-
atomic_xchg_relaxed(X,V) __xchg{once}(X,V)
109
-
atomic_xchg_release(X,V) __xchg{release}(X,V)
110
-
atomic_xchg_acquire(X,V) __xchg{acquire}(X,V)
111
-
atomic_cmpxchg(X,V,W) __cmpxchg{mb}(X,V,W)
112
-
atomic_cmpxchg_relaxed(X,V,W) __cmpxchg{once}(X,V,W)
113
-
atomic_cmpxchg_acquire(X,V,W) __cmpxchg{acquire}(X,V,W)
114
-
atomic_cmpxchg_release(X,V,W) __cmpxchg{release}(X,V,W)
99
+
atomic_inc_return(X) __atomic_op_return{MB}(X,+,1)
100
+
atomic_inc_return_relaxed(X) __atomic_op_return{ONCE}(X,+,1)
101
+
atomic_inc_return_acquire(X) __atomic_op_return{ACQUIRE}(X,+,1)
102
+
atomic_inc_return_release(X) __atomic_op_return{RELEASE}(X,+,1)
103
+
atomic_fetch_inc(X) __atomic_fetch_op{MB}(X,+,1)
104
+
atomic_fetch_inc_relaxed(X) __atomic_fetch_op{ONCE}(X,+,1)
105
+
atomic_fetch_inc_acquire(X) __atomic_fetch_op{ACQUIRE}(X,+,1)
106
+
atomic_fetch_inc_release(X) __atomic_fetch_op{RELEASE}(X,+,1)
115
107
116
-
atomic_sub_and_test(V,X) __atomic_op_return{mb}(X,-,V) == 0
117
-
atomic_dec_and_test(X) __atomic_op_return{mb}(X,-,1) == 0
118
-
atomic_inc_and_test(X) __atomic_op_return{mb}(X,+,1) == 0
119
-
atomic_add_negative(V,X) __atomic_op_return{mb}(X,+,V) < 0
108
+
atomic_sub_return(V,X) __atomic_op_return{MB}(X,-,V)
109
+
atomic_sub_return_relaxed(V,X) __atomic_op_return{ONCE}(X,-,V)
110
+
atomic_sub_return_acquire(V,X) __atomic_op_return{ACQUIRE}(X,-,V)
111
+
atomic_sub_return_release(V,X) __atomic_op_return{RELEASE}(X,-,V)
112
+
atomic_fetch_sub(V,X) __atomic_fetch_op{MB}(X,-,V)
113
+
atomic_fetch_sub_relaxed(V,X) __atomic_fetch_op{ONCE}(X,-,V)
114
+
atomic_fetch_sub_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,-,V)
115
+
atomic_fetch_sub_release(V,X) __atomic_fetch_op{RELEASE}(X,-,V)
116
+
117
+
atomic_dec_return(X) __atomic_op_return{MB}(X,-,1)
118
+
atomic_dec_return_relaxed(X) __atomic_op_return{ONCE}(X,-,1)
119
+
atomic_dec_return_acquire(X) __atomic_op_return{ACQUIRE}(X,-,1)
120
+
atomic_dec_return_release(X) __atomic_op_return{RELEASE}(X,-,1)
121
+
atomic_fetch_dec(X) __atomic_fetch_op{MB}(X,-,1)
122
+
atomic_fetch_dec_relaxed(X) __atomic_fetch_op{ONCE}(X,-,1)
123
+
atomic_fetch_dec_acquire(X) __atomic_fetch_op{ACQUIRE}(X,-,1)
124
+
atomic_fetch_dec_release(X) __atomic_fetch_op{RELEASE}(X,-,1)
125
+
126
+
atomic_xchg(X,V) __xchg{MB}(X,V)
127
+
atomic_xchg_relaxed(X,V) __xchg{ONCE}(X,V)
128
+
atomic_xchg_release(X,V) __xchg{RELEASE}(X,V)
129
+
atomic_xchg_acquire(X,V) __xchg{ACQUIRE}(X,V)
130
+
atomic_cmpxchg(X,V,W) __cmpxchg{MB}(X,V,W)
131
+
atomic_cmpxchg_relaxed(X,V,W) __cmpxchg{ONCE}(X,V,W)
132
+
atomic_cmpxchg_acquire(X,V,W) __cmpxchg{ACQUIRE}(X,V,W)
133
+
atomic_cmpxchg_release(X,V,W) __cmpxchg{RELEASE}(X,V,W)
134
+
135
+
atomic_sub_and_test(V,X) __atomic_op_return{MB}(X,-,V) == 0
136
+
atomic_dec_and_test(X) __atomic_op_return{MB}(X,-,1) == 0
137
+
atomic_inc_and_test(X) __atomic_op_return{MB}(X,+,1) == 0
138
+
atomic_add_negative(V,X) __atomic_op_return{MB}(X,+,V) < 0
139
+
atomic_add_negative_relaxed(V,X) __atomic_op_return{ONCE}(X,+,V) < 0
140
+
atomic_add_negative_acquire(V,X) __atomic_op_return{ACQUIRE}(X,+,V) < 0
141
+
atomic_add_negative_release(V,X) __atomic_op_return{RELEASE}(X,+,V) < 0
142
+
143
+
atomic_fetch_andnot(V,X) __atomic_fetch_op{MB}(X,&~,V)
144
+
atomic_fetch_andnot_acquire(V,X) __atomic_fetch_op{ACQUIRE}(X,&~,V)
145
+
atomic_fetch_andnot_release(V,X) __atomic_fetch_op{RELEASE}(X,&~,V)
146
+
atomic_fetch_andnot_relaxed(V,X) __atomic_fetch_op{ONCE}(X,&~,V)
147
+
148
+
atomic_add_unless(X,V,W) __atomic_add_unless{MB}(X,V,W)