Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'sched-core-2023-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
- Improve the scalability of the CFS bandwidth unthrottling logic with
large number of CPUs.
- Fix & rework various cpuidle routines, simplify interaction with the
generic scheduler code. Add __cpuidle methods as noinstr to objtool's
noinstr detection and fix boatloads of cpuidle bugs & quirks.
- Add new ABI: introduce MEMBARRIER_CMD_GET_REGISTRATIONS, to query
previously issued registrations.
- Limit scheduler slice duration to the sysctl_sched_latency period, to
improve scheduling granularity with a large number of SCHED_IDLE
tasks.
- Debuggability enhancement on sys_exit(): warn about disabled IRQs,
but also enable them to prevent a cascade of followup problems and
repeat warnings.
- Fix the rescheduling logic in prio_changed_dl().
- Micro-optimize cpufreq and sched-util methods.
- Micro-optimize ttwu_runnable()
- Micro-optimize the idle-scanning in update_numa_stats(),
select_idle_capacity() and steal_cookie_task().
- Update the RSEQ code & self-tests
- Constify various scheduler methods
- Remove unused methods
- Refine __init tags
- Documentation updates
- Misc other cleanups, fixes
* tag 'sched-core-2023-02-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (110 commits)
sched/rt: pick_next_rt_entity(): check list_entry
sched/deadline: Add more reschedule cases to prio_changed_dl()
sched/fair: sanitize vruntime of entity being placed
sched/fair: Remove capacity inversion detection
sched/fair: unlink misfit task from cpu overutilized
objtool: mem*() are not uaccess safe
cpuidle: Fix poll_idle() noinstr annotation
sched/clock: Make local_clock() noinstr
sched/clock/x86: Mark sched_clock() noinstr
x86/pvclock: Improve atomic update of last_value in pvclock_clocksource_read()
x86/atomics: Always inline arch_atomic64*()
cpuidle: tracing, preempt: Squash _rcuidle tracing
cpuidle: tracing: Warn about !rcu_is_watching()
cpuidle: lib/bug: Disable rcu_is_watching() during WARN/BUG
cpuidle: drivers: firmware: psci: Dont instrument suspend code
KVM: selftests: Fix build of rseq test
exit: Detect and fix irq disabled state in oops
cpuidle, arm64: Fix the ARM64 cpuidle logic
cpuidle: mvebu: Fix duplicate flags assignment
sched/fair: Limit sched slice duration
...
+6774
-5496
+3
Documentation/admin-guide/cgroup-v2.rst
+3
Documentation/admin-guide/cgroup-v2.rst
···
619
619
and is an example of this type.
620
620
621
621
622
+
.. _cgroupv2-limits-distributor:
623
+
622
624
Limits
623
625
------
624
626
···
637
635
"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume
638
636
on an IO device and is an example of this type.
639
637
638
+
.. _cgroupv2-protections-distributor:
640
639
641
640
Protections
642
641
-----------
+1
Documentation/scheduler/index.rst
+1
Documentation/scheduler/index.rst
+741
Documentation/scheduler/sched-util-clamp.rst
+741
Documentation/scheduler/sched-util-clamp.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
====================
4
+
Utilization Clamping
5
+
====================
6
+
7
+
1. Introduction
8
+
===============
9
+
10
+
Utilization clamping, also known as util clamp or uclamp, is a scheduler
11
+
feature that allows user space to help in managing the performance requirement
12
+
of tasks. It was introduced in v5.3 release. The CGroup support was merged in
13
+
v5.4.
14
+
15
+
Uclamp is a hinting mechanism that allows the scheduler to understand the
16
+
performance requirements and restrictions of the tasks, thus it helps the
17
+
scheduler to make a better decision. And when schedutil cpufreq governor is
18
+
used, util clamp will influence the CPU frequency selection as well.
19
+
20
+
Since the scheduler and schedutil are both driven by PELT (util_avg) signals,
21
+
util clamp acts on that to achieve its goal by clamping the signal to a certain
22
+
point; hence the name. That is, by clamping utilization we are making the
23
+
system run at a certain performance point.
24
+
25
+
The right way to view util clamp is as a mechanism to make request or hint on
26
+
performance constraints. It consists of two tunables:
27
+
28
+
* UCLAMP_MIN, which sets the lower bound.
29
+
* UCLAMP_MAX, which sets the upper bound.
30
+
31
+
These two bounds will ensure a task will operate within this performance range
32
+
of the system. UCLAMP_MIN implies boosting a task, while UCLAMP_MAX implies
33
+
capping a task.
34
+
35
+
One can tell the system (scheduler) that some tasks require a minimum
36
+
performance point to operate at to deliver the desired user experience. Or one
37
+
can tell the system that some tasks should be restricted from consuming too
38
+
much resources and should not go above a specific performance point. Viewing
39
+
the uclamp values as performance points rather than utilization is a better
40
+
abstraction from user space point of view.
41
+
42
+
As an example, a game can use util clamp to form a feedback loop with its
43
+
perceived Frames Per Second (FPS). It can dynamically increase the minimum
44
+
performance point required by its display pipeline to ensure no frame is
45
+
dropped. It can also dynamically 'prime' up these tasks if it knows in the
46
+
coming few hundred milliseconds a computationally intensive scene is about to
47
+
happen.
48
+
49
+
On mobile hardware where the capability of the devices varies a lot, this
50
+
dynamic feedback loop offers a great flexibility to ensure best user experience
51
+
given the capabilities of any system.
52
+
53
+
Of course a static configuration is possible too. The exact usage will depend
54
+
on the system, application and the desired outcome.
55
+
56
+
Another example is in Android where tasks are classified as background,
57
+
foreground, top-app, etc. Util clamp can be used to constrain how much
58
+
resources background tasks are consuming by capping the performance point they
59
+
can run at. This constraint helps reserve resources for important tasks, like
60
+
the ones belonging to the currently active app (top-app group). Beside this
61
+
helps in limiting how much power they consume. This can be more obvious in
62
+
heterogeneous systems (e.g. Arm big.LITTLE); the constraint will help bias the
63
+
background tasks to stay on the little cores which will ensure that:
64
+
65
+
1. The big cores are free to run top-app tasks immediately. top-app
66
+
tasks are the tasks the user is currently interacting with, hence
67
+
the most important tasks in the system.
68
+
2. They don't run on a power hungry core and drain battery even if they
69
+
are CPU intensive tasks.
70
+
71
+
.. note::
72
+
**little cores**:
73
+
CPUs with capacity < 1024
74
+
75
+
**big cores**:
76
+
CPUs with capacity = 1024
77
+
78
+
By making these uclamp performance requests, or rather hints, user space can
79
+
ensure system resources are used optimally to deliver the best possible user
80
+
experience.
81
+
82
+
Another use case is to help with **overcoming the ramp up latency inherit in
83
+
how scheduler utilization signal is calculated**.
84
+
85
+
On the other hand, a busy task for instance that requires to run at maximum
86
+
performance point will suffer a delay of ~200ms (PELT HALFIFE = 32ms) for the
87
+
scheduler to realize that. This is known to affect workloads like gaming on
88
+
mobile devices where frames will drop due to slow response time to select the
89
+
higher frequency required for the tasks to finish their work in time. Setting
90
+
UCLAMP_MIN=1024 will ensure such tasks will always see the highest performance
91
+
level when they start running.
92
+
93
+
The overall visible effect goes beyond better perceived user
94
+
experience/performance and stretches to help achieve a better overall
95
+
performance/watt if used effectively.
96
+
97
+
User space can form a feedback loop with the thermal subsystem too to ensure
98
+
the device doesn't heat up to the point where it will throttle.
99
+
100
+
Both SCHED_NORMAL/OTHER and SCHED_FIFO/RR honour uclamp requests/hints.
101
+
102
+
In the SCHED_FIFO/RR case, uclamp gives the option to run RT tasks at any
103
+
performance point rather than being tied to MAX frequency all the time. Which
104
+
can be useful on general purpose systems that run on battery powered devices.
105
+
106
+
Note that by design RT tasks don't have per-task PELT signal and must always
107
+
run at a constant frequency to combat undeterministic DVFS rampup delays.
108
+
109
+
Note that using schedutil always implies a single delay to modify the frequency
110
+
when an RT task wakes up. This cost is unchanged by using uclamp. Uclamp only
111
+
helps picking what frequency to request instead of schedutil always requesting
112
+
MAX for all RT tasks.
113
+
114
+
See :ref:`section 3.4 <uclamp-default-values>` for default values and
115
+
:ref:`3.4.1 <sched-util-clamp-min-rt-default>` on how to change RT tasks
116
+
default value.
117
+
118
+
2. Design
119
+
=========
120
+
121
+
Util clamp is a property of every task in the system. It sets the boundaries of
122
+
its utilization signal; acting as a bias mechanism that influences certain
123
+
decisions within the scheduler.
124
+
125
+
The actual utilization signal of a task is never clamped in reality. If you
126
+
inspect PELT signals at any point of time you should continue to see them as
127
+
they are intact. Clamping happens only when needed, e.g: when a task wakes up
128
+
and the scheduler needs to select a suitable CPU for it to run on.
129
+
130
+
Since the goal of util clamp is to allow requesting a minimum and maximum
131
+
performance point for a task to run on, it must be able to influence the
132
+
frequency selection as well as task placement to be most effective. Both of
133
+
which have implications on the utilization value at CPU runqueue (rq for short)
134
+
level, which brings us to the main design challenge.
135
+
136
+
When a task wakes up on an rq, the utilization signal of the rq will be
137
+
affected by the uclamp settings of all the tasks enqueued on it. For example if
138
+
a task requests to run at UTIL_MIN = 512, then the util signal of the rq needs
139
+
to respect to this request as well as all other requests from all of the
140
+
enqueued tasks.
141
+
142
+
To be able to aggregate the util clamp value of all the tasks attached to the
143
+
rq, uclamp must do some housekeeping at every enqueue/dequeue, which is the
144
+
scheduler hot path. Hence care must be taken since any slow down will have
145
+
significant impact on a lot of use cases and could hinder its usability in
146
+
practice.
147
+
148
+
The way this is handled is by dividing the utilization range into buckets
149
+
(struct uclamp_bucket) which allows us to reduce the search space from every
150
+
task on the rq to only a subset of tasks on the top-most bucket.
151
+
152
+
When a task is enqueued, the counter in the matching bucket is incremented,
153
+
and on dequeue it is decremented. This makes keeping track of the effective
154
+
uclamp value at rq level a lot easier.
155
+
156
+
As tasks are enqueued and dequeued, we keep track of the current effective
157
+
uclamp value of the rq. See :ref:`section 2.1 <uclamp-buckets>` for details on
158
+
how this works.
159
+
160
+
Later at any path that wants to identify the effective uclamp value of the rq,
161
+
it will simply need to read this effective uclamp value of the rq at that exact
162
+
moment of time it needs to take a decision.
163
+
164
+
For task placement case, only Energy Aware and Capacity Aware Scheduling
165
+
(EAS/CAS) make use of uclamp for now, which implies that it is applied on
166
+
heterogeneous systems only.
167
+
When a task wakes up, the scheduler will look at the current effective uclamp
168
+
value of every rq and compare it with the potential new value if the task were
169
+
to be enqueued there. Favoring the rq that will end up with the most energy
170
+
efficient combination.
171
+
172
+
Similarly in schedutil, when it needs to make a frequency update it will look
173
+
at the current effective uclamp value of the rq which is influenced by the set
174
+
of tasks currently enqueued there and select the appropriate frequency that
175
+
will satisfy constraints from requests.
176
+
177
+
Other paths like setting overutilization state (which effectively disables EAS)
178
+
make use of uclamp as well. Such cases are considered necessary housekeeping to
179
+
allow the 2 main use cases above and will not be covered in detail here as they
180
+
could change with implementation details.
181
+
182
+
.. _uclamp-buckets:
183
+
184
+
2.1. Buckets
185
+
------------
186
+
187
+
::
188
+
189
+
[struct rq]
190
+
191
+
(bottom) (top)
192
+
193
+
0 1024
194
+
| |
195
+
+-----------+-----------+-----------+---- ----+-----------+
196
+
| Bucket 0 | Bucket 1 | Bucket 2 | ... | Bucket N |
197
+
+-----------+-----------+-----------+---- ----+-----------+
198
+
: : :
199
+
+- p0 +- p3 +- p4
200
+
: :
201
+
+- p1 +- p5
202
+
:
203
+
+- p2
204
+
205
+
206
+
.. note::
207
+
The diagram above is an illustration rather than a true depiction of the
208
+
internal data structure.
209
+
210
+
To reduce the search space when trying to decide the effective uclamp value of
211
+
an rq as tasks are enqueued/dequeued, the whole utilization range is divided
212
+
into N buckets where N is configured at compile time by setting
213
+
CONFIG_UCLAMP_BUCKETS_COUNT. By default it is set to 5.
214
+
215
+
The rq has a bucket for each uclamp_id tunables: [UCLAMP_MIN, UCLAMP_MAX].
216
+
217
+
The range of each bucket is 1024/N. For example, for the default value of
218
+
5 there will be 5 buckets, each of which will cover the following range:
219
+
220
+
::
221
+
222
+
DELTA = round_closest(1024/5) = 204.8 = 205
223
+
224
+
Bucket 0: [0:204]
225
+
Bucket 1: [205:409]
226
+
Bucket 2: [410:614]
227
+
Bucket 3: [615:819]
228
+
Bucket 4: [820:1024]
229
+
230
+
When a task p with following tunable parameters
231
+
232
+
::
233
+
234
+
p->uclamp[UCLAMP_MIN] = 300
235
+
p->uclamp[UCLAMP_MAX] = 1024
236
+
237
+
is enqueued into the rq, bucket 1 will be incremented for UCLAMP_MIN and bucket
238
+
4 will be incremented for UCLAMP_MAX to reflect the fact the rq has a task in
239
+
this range.
240
+
241
+
The rq then keeps track of its current effective uclamp value for each
242
+
uclamp_id.
243
+
244
+
When a task p is enqueued, the rq value changes to:
245
+
246
+
::
247
+
248
+
// update bucket logic goes here
249
+
rq->uclamp[UCLAMP_MIN] = max(rq->uclamp[UCLAMP_MIN], p->uclamp[UCLAMP_MIN])
250
+
// repeat for UCLAMP_MAX
251
+
252
+
Similarly, when p is dequeued the rq value changes to:
253
+
254
+
::
255
+
256
+
// update bucket logic goes here
257
+
rq->uclamp[UCLAMP_MIN] = search_top_bucket_for_highest_value()
258
+
// repeat for UCLAMP_MAX
259
+
260
+
When all buckets are empty, the rq uclamp values are reset to system defaults.
261
+
See :ref:`section 3.4 <uclamp-default-values>` for details on default values.
262
+
263
+
264
+
2.2. Max aggregation
265
+
--------------------
266
+
267
+
Util clamp is tuned to honour the request for the task that requires the
268
+
highest performance point.
269
+
270
+
When multiple tasks are attached to the same rq, then util clamp must make sure
271
+
the task that needs the highest performance point gets it even if there's
272
+
another task that doesn't need it or is disallowed from reaching this point.
273
+
274
+
For example, if there are multiple tasks attached to an rq with the following
275
+
values:
276
+
277
+
::
278
+
279
+
p0->uclamp[UCLAMP_MIN] = 300
280
+
p0->uclamp[UCLAMP_MAX] = 900
281
+
282
+
p1->uclamp[UCLAMP_MIN] = 500
283
+
p1->uclamp[UCLAMP_MAX] = 500
284
+
285
+
then assuming both p0 and p1 are enqueued to the same rq, both UCLAMP_MIN
286
+
and UCLAMP_MAX become:
287
+
288
+
::
289
+
290
+
rq->uclamp[UCLAMP_MIN] = max(300, 500) = 500
291
+
rq->uclamp[UCLAMP_MAX] = max(900, 500) = 900
292
+
293
+
As we shall see in :ref:`section 5.1 <uclamp-capping-fail>`, this max
294
+
aggregation is the cause of one of limitations when using util clamp, in
295
+
particular for UCLAMP_MAX hint when user space would like to save power.
296
+
297
+
2.3. Hierarchical aggregation
298
+
-----------------------------
299
+
300
+
As stated earlier, util clamp is a property of every task in the system. But
301
+
the actual applied (effective) value can be influenced by more than just the
302
+
request made by the task or another actor on its behalf (middleware library).
303
+
304
+
The effective util clamp value of any task is restricted as follows:
305
+
306
+
1. By the uclamp settings defined by the cgroup CPU controller it is attached
307
+
to, if any.
308
+
2. The restricted value in (1) is then further restricted by the system wide
309
+
uclamp settings.
310
+
311
+
:ref:`Section 3 <uclamp-interfaces>` discusses the interfaces and will expand
312
+
further on that.
313
+
314
+
For now suffice to say that if a task makes a request, its actual effective
315
+
value will have to adhere to some restrictions imposed by cgroup and system
316
+
wide settings.
317
+
318
+
The system will still accept the request even if effectively will be beyond the
319
+
constraints, but as soon as the task moves to a different cgroup or a sysadmin
320
+
modifies the system settings, the request will be satisfied only if it is
321
+
within new constraints.
322
+
323
+
In other words, this aggregation will not cause an error when a task changes
324
+
its uclamp values, but rather the system may not be able to satisfy requests
325
+
based on those factors.
326
+
327
+
2.4. Range
328
+
----------
329
+
330
+
Uclamp performance request has the range of 0 to 1024 inclusive.
331
+
332
+
For cgroup interface percentage is used (that is 0 to 100 inclusive).
333
+
Just like other cgroup interfaces, you can use 'max' instead of 100.
334
+
335
+
.. _uclamp-interfaces:
336
+
337
+
3. Interfaces
338
+
=============
339
+
340
+
3.1. Per task interface
341
+
-----------------------
342
+
343
+
sched_setattr() syscall was extended to accept two new fields:
344
+
345
+
* sched_util_min: requests the minimum performance point the system should run
346
+
at when this task is running. Or lower performance bound.
347
+
* sched_util_max: requests the maximum performance point the system should run
348
+
at when this task is running. Or upper performance bound.
349
+
350
+
For example, the following scenario have 40% to 80% utilization constraints:
351
+
352
+
::
353
+
354
+
attr->sched_util_min = 40% * 1024;
355
+
attr->sched_util_max = 80% * 1024;
356
+
357
+
When task @p is running, **the scheduler should try its best to ensure it
358
+
starts at 40% performance level**. If the task runs for a long enough time so
359
+
that its actual utilization goes above 80%, the utilization, or performance
360
+
level, will be capped.
361
+
362
+
The special value -1 is used to reset the uclamp settings to the system
363
+
default.
364
+
365
+
Note that resetting the uclamp value to system default using -1 is not the same
366
+
as manually setting uclamp value to system default. This distinction is
367
+
important because as we shall see in system interfaces, the default value for
368
+
RT could be changed. SCHED_NORMAL/OTHER might gain similar knobs too in the
369
+
future.
370
+
371
+
3.2. cgroup interface
372
+
---------------------
373
+
374
+
There are two uclamp related values in the CPU cgroup controller:
375
+
376
+
* cpu.uclamp.min
377
+
* cpu.uclamp.max
378
+
379
+
When a task is attached to a CPU controller, its uclamp values will be impacted
380
+
as follows:
381
+
382
+
* cpu.uclamp.min is a protection as described in :ref:`section 3-3 of cgroup
383
+
v2 documentation <cgroupv2-protections-distributor>`.
384
+
385
+
If a task uclamp_min value is lower than cpu.uclamp.min, then the task will
386
+
inherit the cgroup cpu.uclamp.min value.
387
+
388
+
In a cgroup hierarchy, effective cpu.uclamp.min is the max of (child,
389
+
parent).
390
+
391
+
* cpu.uclamp.max is a limit as described in :ref:`section 3-2 of cgroup v2
392
+
documentation <cgroupv2-limits-distributor>`.
393
+
394
+
If a task uclamp_max value is higher than cpu.uclamp.max, then the task will
395
+
inherit the cgroup cpu.uclamp.max value.
396
+
397
+
In a cgroup hierarchy, effective cpu.uclamp.max is the min of (child,
398
+
parent).
399
+
400
+
For example, given following parameters:
401
+
402
+
::
403
+
404
+
p0->uclamp[UCLAMP_MIN] = // system default;
405
+
p0->uclamp[UCLAMP_MAX] = // system default;
406
+
407
+
p1->uclamp[UCLAMP_MIN] = 40% * 1024;
408
+
p1->uclamp[UCLAMP_MAX] = 50% * 1024;
409
+
410
+
cgroup0->cpu.uclamp.min = 20% * 1024;
411
+
cgroup0->cpu.uclamp.max = 60% * 1024;
412
+
413
+
cgroup1->cpu.uclamp.min = 60% * 1024;
414
+
cgroup1->cpu.uclamp.max = 100% * 1024;
415
+
416
+
when p0 and p1 are attached to cgroup0, the values become:
417
+
418
+
::
419
+
420
+
p0->uclamp[UCLAMP_MIN] = cgroup0->cpu.uclamp.min = 20% * 1024;
421
+
p0->uclamp[UCLAMP_MAX] = cgroup0->cpu.uclamp.max = 60% * 1024;
422
+
423
+
p1->uclamp[UCLAMP_MIN] = 40% * 1024; // intact
424
+
p1->uclamp[UCLAMP_MAX] = 50% * 1024; // intact
425
+
426
+
when p0 and p1 are attached to cgroup1, these instead become:
427
+
428
+
::
429
+
430
+
p0->uclamp[UCLAMP_MIN] = cgroup1->cpu.uclamp.min = 60% * 1024;
431
+
p0->uclamp[UCLAMP_MAX] = cgroup1->cpu.uclamp.max = 100% * 1024;
432
+
433
+
p1->uclamp[UCLAMP_MIN] = cgroup1->cpu.uclamp.min = 60% * 1024;
434
+
p1->uclamp[UCLAMP_MAX] = 50% * 1024; // intact
435
+
436
+
Note that cgroup interfaces allows cpu.uclamp.max value to be lower than
437
+
cpu.uclamp.min. Other interfaces don't allow that.
438
+
439
+
3.3. System interface
440
+
---------------------
441
+
442
+
3.3.1 sched_util_clamp_min
443
+
--------------------------
444
+
445
+
System wide limit of allowed UCLAMP_MIN range. By default it is set to 1024,
446
+
which means that permitted effective UCLAMP_MIN range for tasks is [0:1024].
447
+
By changing it to 512 for example the range reduces to [0:512]. This is useful
448
+
to restrict how much boosting tasks are allowed to acquire.
449
+
450
+
Requests from tasks to go above this knob value will still succeed, but
451
+
they won't be satisfied until it is more than p->uclamp[UCLAMP_MIN].
452
+
453
+
The value must be smaller than or equal to sched_util_clamp_max.
454
+
455
+
3.3.2 sched_util_clamp_max
456
+
--------------------------
457
+
458
+
System wide limit of allowed UCLAMP_MAX range. By default it is set to 1024,
459
+
which means that permitted effective UCLAMP_MAX range for tasks is [0:1024].
460
+
461
+
By changing it to 512 for example the effective allowed range reduces to
462
+
[0:512]. This means is that no task can run above 512, which implies that all
463
+
rqs are restricted too. IOW, the whole system is capped to half its performance
464
+
capacity.
465
+
466
+
This is useful to restrict the overall maximum performance point of the system.
467
+
For example, it can be handy to limit performance when running low on battery
468
+
or when the system wants to limit access to more energy hungry performance
469
+
levels when it's in idle state or screen is off.
470
+
471
+
Requests from tasks to go above this knob value will still succeed, but they
472
+
won't be satisfied until it is more than p->uclamp[UCLAMP_MAX].
473
+
474
+
The value must be greater than or equal to sched_util_clamp_min.
475
+
476
+
.. _uclamp-default-values:
477
+
478
+
3.4. Default values
479
+
-------------------
480
+
481
+
By default all SCHED_NORMAL/SCHED_OTHER tasks are initialized to:
482
+
483
+
::
484
+
485
+
p_fair->uclamp[UCLAMP_MIN] = 0
486
+
p_fair->uclamp[UCLAMP_MAX] = 1024
487
+
488
+
That is, by default they're boosted to run at the maximum performance point of
489
+
changed at boot or runtime. No argument was made yet as to why we should
490
+
provide this, but can be added in the future.
491
+
492
+
For SCHED_FIFO/SCHED_RR tasks:
493
+
494
+
::
495
+
496
+
p_rt->uclamp[UCLAMP_MIN] = 1024
497
+
p_rt->uclamp[UCLAMP_MAX] = 1024
498
+
499
+
That is by default they're boosted to run at the maximum performance point of
500
+
the system which retains the historical behavior of the RT tasks.
501
+
502
+
RT tasks default uclamp_min value can be modified at boot or runtime via
503
+
sysctl. See below section.
504
+
505
+
.. _sched-util-clamp-min-rt-default:
506
+
507
+
3.4.1 sched_util_clamp_min_rt_default
508
+
-------------------------------------
509
+
510
+
Running RT tasks at maximum performance point is expensive on battery powered
511
+
devices and not necessary. To allow system developer to offer good performance
512
+
guarantees for these tasks without pushing it all the way to maximum
513
+
performance point, this sysctl knob allows tuning the best boost value to
514
+
address the system requirement without burning power running at maximum
515
+
performance point all the time.
516
+
517
+
Application developer are encouraged to use the per task util clamp interface
518
+
to ensure they are performance and power aware. Ideally this knob should be set
519
+
to 0 by system designers and leave the task of managing performance
520
+
requirements to the apps.
521
+
522
+
4. How to use util clamp
523
+
========================
524
+
525
+
Util clamp promotes the concept of user space assisted power and performance
526
+
management. At the scheduler level there is no info required to make the best
527
+
decision. However, with util clamp user space can hint to the scheduler to make
528
+
better decision about task placement and frequency selection.
529
+
530
+
Best results are achieved by not making any assumptions about the system the
531
+
application is running on and to use it in conjunction with a feedback loop to
532
+
dynamically monitor and adjust. Ultimately this will allow for a better user
533
+
experience at a better perf/watt.
534
+
535
+
For some systems and use cases, static setup will help to achieve good results.
536
+
Portability will be a problem in this case. How much work one can do at 100,
537
+
200 or 1024 is different for each system. Unless there's a specific target
538
+
system, static setup should be avoided.
539
+
540
+
There are enough possibilities to create a whole framework based on util clamp
541
+
or self contained app that makes use of it directly.
542
+
543
+
4.1. Boost important and DVFS-latency-sensitive tasks
544
+
-----------------------------------------------------
545
+
546
+
A GUI task might not be busy to warrant driving the frequency high when it
547
+
wakes up. However, it requires to finish its work within a specific time window
548
+
to deliver the desired user experience. The right frequency it requires at
549
+
wakeup will be system dependent. On some underpowered systems it will be high,
550
+
on other overpowered ones it will be low or 0.
551
+
552
+
This task can increase its UCLAMP_MIN value every time it misses the deadline
553
+
to ensure on next wake up it runs at a higher performance point. It should try
554
+
to approach the lowest UCLAMP_MIN value that allows to meet its deadline on any
555
+
particular system to achieve the best possible perf/watt for that system.
556
+
557
+
On heterogeneous systems, it might be important for this task to run on
558
+
a faster CPU.
559
+
560
+
**Generally it is advised to perceive the input as performance level or point
561
+
which will imply both task placement and frequency selection**.
562
+
563
+
4.2. Cap background tasks
564
+
-------------------------
565
+
566
+
Like explained for Android case in the introduction. Any app can lower
567
+
UCLAMP_MAX for some background tasks that don't care about performance but
568
+
could end up being busy and consume unnecessary system resources on the system.
569
+
570
+
4.3. Powersave mode
571
+
-------------------
572
+
573
+
sched_util_clamp_max system wide interface can be used to limit all tasks from
574
+
operating at the higher performance points which are usually energy
575
+
inefficient.
576
+
577
+
This is not unique to uclamp as one can achieve the same by reducing max
578
+
frequency of the cpufreq governor. It can be considered a more convenient
579
+
alternative interface.
580
+
581
+
4.4. Per-app performance restriction
582
+
------------------------------------
583
+
584
+
Middleware/Utility can provide the user an option to set UCLAMP_MIN/MAX for an
585
+
app every time it is executed to guarantee a minimum performance point and/or
586
+
limit it from draining system power at the cost of reduced performance for
587
+
these apps.
588
+
589
+
If you want to prevent your laptop from heating up while on the go from
590
+
compiling the kernel and happy to sacrifice performance to save power, but
591
+
still would like to keep your browser performance intact, uclamp makes it
592
+
possible.
593
+
594
+
5. Limitations
595
+
==============
596
+
597
+
.. _uclamp-capping-fail:
598
+
599
+
5.1. Capping frequency with uclamp_max fails under certain conditions
600
+
---------------------------------------------------------------------
601
+
602
+
If task p0 is capped to run at 512:
603
+
604
+
::
605
+
606
+
p0->uclamp[UCLAMP_MAX] = 512
607
+
608
+
and it shares the rq with p1 which is free to run at any performance point:
609
+
610
+
::
611
+
612
+
p1->uclamp[UCLAMP_MAX] = 1024
613
+
614
+
then due to max aggregation the rq will be allowed to reach max performance
615
+
point:
616
+
617
+
::
618
+
619
+
rq->uclamp[UCLAMP_MAX] = max(512, 1024) = 1024
620
+
621
+
Assuming both p0 and p1 have UCLAMP_MIN = 0, then the frequency selection for
622
+
the rq will depend on the actual utilization value of the tasks.
623
+
624
+
If p1 is a small task but p0 is a CPU intensive task, then due to the fact that
625
+
both are running at the same rq, p1 will cause the frequency capping to be left
626
+
from the rq although p1, which is allowed to run at any performance point,
627
+
doesn't actually need to run at that frequency.
628
+
629
+
5.2. UCLAMP_MAX can break PELT (util_avg) signal
630
+
------------------------------------------------
631
+
632
+
PELT assumes that frequency will always increase as the signals grow to ensure
633
+
there's always some idle time on the CPU. But with UCLAMP_MAX, this frequency
634
+
increase will be prevented which can lead to no idle time in some
635
+
circumstances. When there's no idle time, a task will stuck in a busy loop,
636
+
which would result in util_avg being 1024.
637
+
638
+
Combing with issue described below, this can lead to unwanted frequency spikes
639
+
when severely capped tasks share the rq with a small non capped task.
640
+
641
+
As an example if task p, which have:
642
+
643
+
::
644
+
645
+
p0->util_avg = 300
646
+
p0->uclamp[UCLAMP_MAX] = 0
647
+
648
+
wakes up on an idle CPU, then it will run at min frequency (Fmin) this
649
+
CPU is capable of. The max CPU frequency (Fmax) matters here as well,
650
+
since it designates the shortest computational time to finish the task's
651
+
work on this CPU.
652
+
653
+
::
654
+
655
+
rq->uclamp[UCLAMP_MAX] = 0
656
+
657
+
If the ratio of Fmax/Fmin is 3, then maximum value will be:
658
+
659
+
::
660
+
661
+
300 * (Fmax/Fmin) = 900
662
+
663
+
which indicates the CPU will still see idle time since 900 is < 1024. The
664
+
_actual_ util_avg will not be 900 though, but somewhere between 300 and 900. As
665
+
long as there's idle time, p->util_avg updates will be off by a some margin,
666
+
but not proportional to Fmax/Fmin.
667
+
668
+
::
669
+
670
+
p0->util_avg = 300 + small_error
671
+
672
+
Now if the ratio of Fmax/Fmin is 4, the maximum value becomes:
673
+
674
+
::
675
+
676
+
300 * (Fmax/Fmin) = 1200
677
+
678
+
which is higher than 1024 and indicates that the CPU has no idle time. When
679
+
this happens, then the _actual_ util_avg will become:
680
+
681
+
::
682
+
683
+
p0->util_avg = 1024
684
+
685
+
If task p1 wakes up on this CPU, which have:
686
+
687
+
::
688
+
689
+
p1->util_avg = 200
690
+
p1->uclamp[UCLAMP_MAX] = 1024
691
+
692
+
then the effective UCLAMP_MAX for the CPU will be 1024 according to max
693
+
aggregation rule. But since the capped p0 task was running and throttled
694
+
severely, then the rq->util_avg will be:
695
+
696
+
::
697
+
698
+
p0->util_avg = 1024
699
+
p1->util_avg = 200
700
+
701
+
rq->util_avg = 1024
702
+
rq->uclamp[UCLAMP_MAX] = 1024
703
+
704
+
Hence lead to a frequency spike since if p0 wasn't throttled we should get:
705
+
706
+
::
707
+
708
+
p0->util_avg = 300
709
+
p1->util_avg = 200
710
+
711
+
rq->util_avg = 500
712
+
713
+
and run somewhere near mid performance point of that CPU, not the Fmax we get.
714
+
715
+
5.3. Schedutil response time issues
716
+
-----------------------------------
717
+
718
+
schedutil has three limitations:
719
+
720
+
1. Hardware takes non-zero time to respond to any frequency change
721
+
request. On some platforms can be in the order of few ms.
722
+
2. Non fast-switch systems require a worker deadline thread to wake up
723
+
and perform the frequency change, which adds measurable overhead.
724
+
3. schedutil rate_limit_us drops any requests during this rate_limit_us
725
+
window.
726
+
727
+
If a relatively small task is doing critical job and requires a certain
728
+
performance point when it wakes up and starts running, then all these
729
+
limitations will prevent it from getting what it wants in the time scale it
730
+
expects.
731
+
732
+
This limitation is not only impactful when using uclamp, but will be more
733
+
prevalent as we no longer gradually ramp up or down. We could easily be
734
+
jumping between frequencies depending on the order tasks wake up, and their
735
+
respective uclamp values.
736
+
737
+
We regard that as a limitation of the capabilities of the underlying system
738
+
itself.
739
+
740
+
There is room to improve the behavior of schedutil rate_limit_us, but not much
741
+
to be done for 1 or 2. They are considered hard limitations of the system.
-1
arch/alpha/kernel/process.c
-1
arch/alpha/kernel/process.c
-1
arch/alpha/kernel/vmlinux.lds.S
-1
arch/alpha/kernel/vmlinux.lds.S
+3
arch/arc/kernel/process.c
+3
arch/arc/kernel/process.c
···
114
114
"sleep %0 \n"
115
115
:
116
116
:"I"(arg)); /* can't be "r" has to be embedded const */
117
+
118
+
raw_local_irq_disable();
117
119
}
118
120
119
121
#else /* ARC700 */
···
124
122
{
125
123
/* sleep, but enable both set E1/E2 (levels of interrupts) before committing */
126
124
__asm__ __volatile__("sleep 0x3 \n");
125
+
raw_local_irq_disable();
127
126
}
128
127
129
128
#endif
-1
arch/arc/kernel/vmlinux.lds.S
-1
arch/arc/kernel/vmlinux.lds.S
-1
arch/arm/include/asm/vmlinux.lds.h
-1
arch/arm/include/asm/vmlinux.lds.h
+2
-2
arch/arm/kernel/cpuidle.c
+2
-2
arch/arm/kernel/cpuidle.c
···
26
26
*
27
27
* Returns the index passed as parameter
28
28
*/
29
-
int arm_cpuidle_simple_enter(struct cpuidle_device *dev,
30
-
struct cpuidle_driver *drv, int index)
29
+
__cpuidle int arm_cpuidle_simple_enter(struct cpuidle_device *dev, struct
30
+
cpuidle_driver *drv, int index)
31
31
{
32
32
cpu_do_idle();
33
33
-1
arch/arm/kernel/process.c
-1
arch/arm/kernel/process.c
+3
-3
arch/arm/kernel/smp.c
+3
-3
arch/arm/kernel/smp.c
···
638
638
unsigned int cpu = smp_processor_id();
639
639
640
640
if ((unsigned)ipinr < NR_IPI)
641
-
trace_ipi_entry_rcuidle(ipi_types[ipinr]);
641
+
trace_ipi_entry(ipi_types[ipinr]);
642
642
643
643
switch (ipinr) {
644
644
case IPI_WAKEUP:
···
685
685
}
686
686
687
687
if ((unsigned)ipinr < NR_IPI)
688
-
trace_ipi_exit_rcuidle(ipi_types[ipinr]);
688
+
trace_ipi_exit(ipi_types[ipinr]);
689
689
}
690
690
691
691
/* Legacy version, should go away once all irqchips have been converted */
···
708
708
709
709
static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
710
710
{
711
-
trace_ipi_raise_rcuidle(target, ipi_types[ipinr]);
711
+
trace_ipi_raise(target, ipi_types[ipinr]);
712
712
__ipi_send_mask(ipi_desc[ipinr], target);
713
713
}
714
714
+2
-2
arch/arm/mach-davinci/cpuidle.c
+2
-2
arch/arm/mach-davinci/cpuidle.c
···
44
44
}
45
45
46
46
/* Actual code that puts the SoC in different idle states */
47
-
static int davinci_enter_idle(struct cpuidle_device *dev,
48
-
struct cpuidle_driver *drv, int index)
47
+
static __cpuidle int davinci_enter_idle(struct cpuidle_device *dev,
48
+
struct cpuidle_driver *drv, int index)
49
49
{
50
50
davinci_save_ddr_power(1, ddr2_pdown);
51
51
cpu_do_idle();
+2
-1
arch/arm/mach-gemini/board-dt.c
+2
-1
arch/arm/mach-gemini/board-dt.c
+2
-2
arch/arm/mach-imx/cpuidle-imx5.c
+2
-2
arch/arm/mach-imx/cpuidle-imx5.c
···
8
8
#include <asm/system_misc.h>
9
9
#include "cpuidle.h"
10
10
11
-
static int imx5_cpuidle_enter(struct cpuidle_device *dev,
12
-
struct cpuidle_driver *drv, int index)
11
+
static __cpuidle int imx5_cpuidle_enter(struct cpuidle_device *dev,
12
+
struct cpuidle_driver *drv, int index)
13
13
{
14
14
arm_pm_idle();
15
15
return index;
+4
-4
arch/arm/mach-imx/cpuidle-imx6q.c
+4
-4
arch/arm/mach-imx/cpuidle-imx6q.c
···
17
17
static int num_idle_cpus = 0;
18
18
static DEFINE_RAW_SPINLOCK(cpuidle_lock);
19
19
20
-
static int imx6q_enter_wait(struct cpuidle_device *dev,
21
-
struct cpuidle_driver *drv, int index)
20
+
static __cpuidle int imx6q_enter_wait(struct cpuidle_device *dev,
21
+
struct cpuidle_driver *drv, int index)
22
22
{
23
23
raw_spin_lock(&cpuidle_lock);
24
24
if (++num_idle_cpus == num_online_cpus())
25
25
imx6_set_lpm(WAIT_UNCLOCKED);
26
26
raw_spin_unlock(&cpuidle_lock);
27
27
28
-
ct_idle_enter();
28
+
ct_cpuidle_enter();
29
29
cpu_do_idle();
30
-
ct_idle_exit();
30
+
ct_cpuidle_exit();
31
31
32
32
raw_spin_lock(&cpuidle_lock);
33
33
if (num_idle_cpus-- == num_online_cpus())
+2
-2
arch/arm/mach-imx/cpuidle-imx6sl.c
+2
-2
arch/arm/mach-imx/cpuidle-imx6sl.c
···
11
11
#include "common.h"
12
12
#include "cpuidle.h"
13
13
14
-
static int imx6sl_enter_wait(struct cpuidle_device *dev,
15
-
struct cpuidle_driver *drv, int index)
14
+
static __cpuidle int imx6sl_enter_wait(struct cpuidle_device *dev,
15
+
struct cpuidle_driver *drv, int index)
16
16
{
17
17
imx6_set_lpm(WAIT_UNCLOCKED);
18
18
/*
+6
-3
arch/arm/mach-imx/cpuidle-imx6sx.c
+6
-3
arch/arm/mach-imx/cpuidle-imx6sx.c
···
30
30
return 0;
31
31
}
32
32
33
-
static int imx6sx_enter_wait(struct cpuidle_device *dev,
34
-
struct cpuidle_driver *drv, int index)
33
+
static __cpuidle int imx6sx_enter_wait(struct cpuidle_device *dev,
34
+
struct cpuidle_driver *drv, int index)
35
35
{
36
36
imx6_set_lpm(WAIT_UNCLOCKED);
37
37
···
47
47
cpu_pm_enter();
48
48
cpu_cluster_pm_enter();
49
49
50
+
ct_cpuidle_enter();
50
51
cpu_suspend(0, imx6sx_idle_finish);
52
+
ct_cpuidle_exit();
51
53
52
54
cpu_cluster_pm_exit();
53
55
cpu_pm_exit();
···
89
87
*/
90
88
.exit_latency = 300,
91
89
.target_residency = 500,
92
-
.flags = CPUIDLE_FLAG_TIMER_STOP,
90
+
.flags = CPUIDLE_FLAG_TIMER_STOP |
91
+
CPUIDLE_FLAG_RCU_IDLE,
93
92
.enter = imx6sx_enter_wait,
94
93
.name = "LOW-POWER-IDLE",
95
94
.desc = "ARM power off",
+2
-2
arch/arm/mach-imx/cpuidle-imx7ulp.c
+2
-2
arch/arm/mach-imx/cpuidle-imx7ulp.c
···
12
12
#include "common.h"
13
13
#include "cpuidle.h"
14
14
15
-
static int imx7ulp_enter_wait(struct cpuidle_device *dev,
16
-
struct cpuidle_driver *drv, int index)
15
+
static __cpuidle int imx7ulp_enter_wait(struct cpuidle_device *dev,
16
+
struct cpuidle_driver *drv, int index)
17
17
{
18
18
if (index == 1)
19
19
imx7ulp_set_lpm(ULP_PM_WAIT);
+4
-2
arch/arm/mach-omap2/common.h
+4
-2
arch/arm/mach-omap2/common.h
···
256
256
257
257
#if defined(CONFIG_SMP) && defined(CONFIG_PM)
258
258
extern int omap4_mpuss_init(void);
259
-
extern int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state);
259
+
extern int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state,
260
+
bool rcuidle);
260
261
extern int omap4_hotplug_cpu(unsigned int cpu, unsigned int power_state);
261
262
#else
262
263
static inline int omap4_enter_lowpower(unsigned int cpu,
263
-
unsigned int power_state)
264
+
unsigned int power_state,
265
+
bool rcuidle)
264
266
{
265
267
cpu_do_idle();
266
268
return 0;
+15
-1
arch/arm/mach-omap2/cpuidle34xx.c
+15
-1
arch/arm/mach-omap2/cpuidle34xx.c
···
133
133
}
134
134
135
135
/* Execute ARM wfi */
136
-
omap_sram_idle();
136
+
omap_sram_idle(true);
137
137
138
138
/*
139
139
* Call idle CPU PM enter notifier chain to restore
···
265
265
.owner = THIS_MODULE,
266
266
.states = {
267
267
{
268
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
268
269
.enter = omap3_enter_idle_bm,
269
270
.exit_latency = 2 + 2,
270
271
.target_residency = 5,
···
273
272
.desc = "MPU ON + CORE ON",
274
273
},
275
274
{
275
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
276
276
.enter = omap3_enter_idle_bm,
277
277
.exit_latency = 10 + 10,
278
278
.target_residency = 30,
···
281
279
.desc = "MPU ON + CORE ON",
282
280
},
283
281
{
282
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
284
283
.enter = omap3_enter_idle_bm,
285
284
.exit_latency = 50 + 50,
286
285
.target_residency = 300,
···
289
286
.desc = "MPU RET + CORE ON",
290
287
},
291
288
{
289
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
292
290
.enter = omap3_enter_idle_bm,
293
291
.exit_latency = 1500 + 1800,
294
292
.target_residency = 4000,
···
297
293
.desc = "MPU OFF + CORE ON",
298
294
},
299
295
{
296
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
300
297
.enter = omap3_enter_idle_bm,
301
298
.exit_latency = 2500 + 7500,
302
299
.target_residency = 12000,
···
305
300
.desc = "MPU RET + CORE RET",
306
301
},
307
302
{
303
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
308
304
.enter = omap3_enter_idle_bm,
309
305
.exit_latency = 3000 + 8500,
310
306
.target_residency = 15000,
···
313
307
.desc = "MPU OFF + CORE RET",
314
308
},
315
309
{
310
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
316
311
.enter = omap3_enter_idle_bm,
317
312
.exit_latency = 10000 + 30000,
318
313
.target_residency = 30000,
···
335
328
.owner = THIS_MODULE,
336
329
.states = {
337
330
{
331
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
338
332
.enter = omap3_enter_idle_bm,
339
333
.exit_latency = 110 + 162,
340
334
.target_residency = 5,
···
343
335
.desc = "MPU ON + CORE ON",
344
336
},
345
337
{
338
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
346
339
.enter = omap3_enter_idle_bm,
347
340
.exit_latency = 106 + 180,
348
341
.target_residency = 309,
···
351
342
.desc = "MPU ON + CORE ON",
352
343
},
353
344
{
345
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
354
346
.enter = omap3_enter_idle_bm,
355
347
.exit_latency = 107 + 410,
356
348
.target_residency = 46057,
···
359
349
.desc = "MPU RET + CORE ON",
360
350
},
361
351
{
352
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
362
353
.enter = omap3_enter_idle_bm,
363
354
.exit_latency = 121 + 3374,
364
355
.target_residency = 46057,
···
367
356
.desc = "MPU OFF + CORE ON",
368
357
},
369
358
{
359
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
370
360
.enter = omap3_enter_idle_bm,
371
361
.exit_latency = 855 + 1146,
372
362
.target_residency = 46057,
···
375
363
.desc = "MPU RET + CORE RET",
376
364
},
377
365
{
366
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
378
367
.enter = omap3_enter_idle_bm,
379
368
.exit_latency = 7580 + 4134,
380
369
.target_residency = 484329,
···
383
370
.desc = "MPU OFF + CORE RET",
384
371
},
385
372
{
373
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
386
374
.enter = omap3_enter_idle_bm,
387
375
.exit_latency = 7505 + 15274,
388
376
.target_residency = 484329,
+16
-13
arch/arm/mach-omap2/cpuidle44xx.c
+16
-13
arch/arm/mach-omap2/cpuidle44xx.c
···
105
105
}
106
106
raw_spin_unlock_irqrestore(&mpu_lock, flag);
107
107
108
-
omap4_enter_lowpower(dev->cpu, cx->cpu_state);
108
+
omap4_enter_lowpower(dev->cpu, cx->cpu_state, true);
109
109
110
110
raw_spin_lock_irqsave(&mpu_lock, flag);
111
111
if (cx->mpu_state_vote == num_online_cpus())
···
151
151
(cx->mpu_logic_state == PWRDM_POWER_OFF);
152
152
153
153
/* Enter broadcast mode for periodic timers */
154
-
RCU_NONIDLE(tick_broadcast_enable());
154
+
tick_broadcast_enable();
155
155
156
156
/* Enter broadcast mode for one-shot timers */
157
-
RCU_NONIDLE(tick_broadcast_enter());
157
+
tick_broadcast_enter();
158
158
159
159
/*
160
160
* Call idle CPU PM enter notifier chain so that
···
166
166
167
167
if (dev->cpu == 0) {
168
168
pwrdm_set_logic_retst(mpu_pd, cx->mpu_logic_state);
169
-
RCU_NONIDLE(omap_set_pwrdm_state(mpu_pd, cx->mpu_state));
169
+
omap_set_pwrdm_state(mpu_pd, cx->mpu_state);
170
170
171
171
/*
172
172
* Call idle CPU cluster PM enter notifier chain
···
178
178
index = 0;
179
179
cx = state_ptr + index;
180
180
pwrdm_set_logic_retst(mpu_pd, cx->mpu_logic_state);
181
-
RCU_NONIDLE(omap_set_pwrdm_state(mpu_pd, cx->mpu_state));
181
+
omap_set_pwrdm_state(mpu_pd, cx->mpu_state);
182
182
mpuss_can_lose_context = 0;
183
183
}
184
184
}
185
185
}
186
186
187
-
omap4_enter_lowpower(dev->cpu, cx->cpu_state);
187
+
omap4_enter_lowpower(dev->cpu, cx->cpu_state, true);
188
188
cpu_done[dev->cpu] = true;
189
189
190
190
/* Wakeup CPU1 only if it is not offlined */
···
194
194
mpuss_can_lose_context)
195
195
gic_dist_disable();
196
196
197
-
RCU_NONIDLE(clkdm_deny_idle(cpu_clkdm[1]));
198
-
RCU_NONIDLE(omap_set_pwrdm_state(cpu_pd[1], PWRDM_POWER_ON));
199
-
RCU_NONIDLE(clkdm_allow_idle(cpu_clkdm[1]));
197
+
clkdm_deny_idle(cpu_clkdm[1]);
198
+
omap_set_pwrdm_state(cpu_pd[1], PWRDM_POWER_ON);
199
+
clkdm_allow_idle(cpu_clkdm[1]);
200
200
201
201
if (IS_PM44XX_ERRATUM(PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD) &&
202
202
mpuss_can_lose_context) {
···
222
222
cpu_pm_exit();
223
223
224
224
cpu_pm_out:
225
-
RCU_NONIDLE(tick_broadcast_exit());
225
+
tick_broadcast_exit();
226
226
227
227
fail:
228
228
cpuidle_coupled_parallel_barrier(dev, &abort_barrier);
···
247
247
/* C2 - CPU0 OFF + CPU1 OFF + MPU CSWR */
248
248
.exit_latency = 328 + 440,
249
249
.target_residency = 960,
250
-
.flags = CPUIDLE_FLAG_COUPLED,
250
+
.flags = CPUIDLE_FLAG_COUPLED |
251
+
CPUIDLE_FLAG_RCU_IDLE,
251
252
.enter = omap_enter_idle_coupled,
252
253
.name = "C2",
253
254
.desc = "CPUx OFF, MPUSS CSWR",
···
257
256
/* C3 - CPU0 OFF + CPU1 OFF + MPU OSWR */
258
257
.exit_latency = 460 + 518,
259
258
.target_residency = 1100,
260
-
.flags = CPUIDLE_FLAG_COUPLED,
259
+
.flags = CPUIDLE_FLAG_COUPLED |
260
+
CPUIDLE_FLAG_RCU_IDLE,
261
261
.enter = omap_enter_idle_coupled,
262
262
.name = "C3",
263
263
.desc = "CPUx OFF, MPUSS OSWR",
···
284
282
/* C2 - CPU0 RET + CPU1 RET + MPU CSWR */
285
283
.exit_latency = 48 + 60,
286
284
.target_residency = 100,
287
-
.flags = CPUIDLE_FLAG_TIMER_STOP,
285
+
.flags = CPUIDLE_FLAG_TIMER_STOP |
286
+
CPUIDLE_FLAG_RCU_IDLE,
288
287
.enter = omap_enter_idle_smp,
289
288
.name = "C2",
290
289
.desc = "CPUx CSWR, MPUSS CSWR",
+11
-1
arch/arm/mach-omap2/omap-mpuss-lowpower.c
+11
-1
arch/arm/mach-omap2/omap-mpuss-lowpower.c
···
33
33
* and first to wake-up when MPUSS low power states are excercised
34
34
*/
35
35
36
+
#include <linux/cpuidle.h>
36
37
#include <linux/kernel.h>
37
38
#include <linux/io.h>
38
39
#include <linux/errno.h>
···
215
214
* of OMAP4 MPUSS subsystem
216
215
* @cpu : CPU ID
217
216
* @power_state: Low power state.
217
+
* @rcuidle: RCU needs to be idled
218
218
*
219
219
* MPUSS states for the context save:
220
220
* save_state =
···
224
222
* 2 - CPUx L1 and logic lost + GIC lost: MPUSS OSWR
225
223
* 3 - CPUx L1 and logic lost + GIC + L2 lost: DEVICE OFF
226
224
*/
227
-
int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state)
225
+
__cpuidle int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state,
226
+
bool rcuidle)
228
227
{
229
228
struct omap4_cpu_pm_info *pm_info = &per_cpu(omap4_pm_info, cpu);
230
229
unsigned int save_state = 0, cpu_logic_state = PWRDM_POWER_RET;
···
271
268
cpu_clear_prev_logic_pwrst(cpu);
272
269
pwrdm_set_next_pwrst(pm_info->pwrdm, power_state);
273
270
pwrdm_set_logic_retst(pm_info->pwrdm, cpu_logic_state);
271
+
272
+
if (rcuidle)
273
+
ct_cpuidle_enter();
274
+
274
275
set_cpu_wakeup_addr(cpu, __pa_symbol(omap_pm_ops.resume));
275
276
omap_pm_ops.scu_prepare(cpu, power_state);
276
277
l2x0_pwrst_prepare(cpu, save_state);
···
289
282
290
283
if (IS_PM44XX_ERRATUM(PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD) && cpu)
291
284
gic_dist_enable();
285
+
286
+
if (rcuidle)
287
+
ct_cpuidle_exit();
292
288
293
289
/*
294
290
* Restore the CPUx power state to ON otherwise CPUx
+1
-1
arch/arm/mach-omap2/pm.h
+1
-1
arch/arm/mach-omap2/pm.h
···
29
29
30
30
extern void *omap3_secure_ram_storage;
31
31
extern void omap3_pm_off_mode_enable(int);
32
-
extern void omap_sram_idle(void);
32
+
extern void omap_sram_idle(bool rcuidle);
33
33
extern int omap_pm_clkdms_setup(struct clockdomain *clkdm, void *unused);
34
34
35
35
extern int omap3_pm_get_suspend_state(struct powerdomain *pwrdm);
+11
-3
arch/arm/mach-omap2/pm34xx.c
+11
-3
arch/arm/mach-omap2/pm34xx.c
···
26
26
#include <linux/delay.h>
27
27
#include <linux/slab.h>
28
28
#include <linux/of.h>
29
+
#include <linux/cpuidle.h>
29
30
30
31
#include <trace/events/power.h>
31
32
···
175
174
return 0;
176
175
}
177
176
178
-
void omap_sram_idle(void)
177
+
__cpuidle void omap_sram_idle(bool rcuidle)
179
178
{
180
179
/* Variable to tell what needs to be saved and restored
181
180
* in omap_sram_idle*/
···
255
254
*/
256
255
if (save_state)
257
256
omap34xx_save_context(omap3_arm_context);
257
+
258
+
if (rcuidle)
259
+
ct_cpuidle_enter();
260
+
258
261
if (save_state == 1 || save_state == 3)
259
262
cpu_suspend(save_state, omap34xx_do_sram_idle);
260
263
else
261
264
omap34xx_do_sram_idle(save_state);
265
+
266
+
if (rcuidle)
267
+
ct_cpuidle_exit();
262
268
263
269
/* Restore normal SDRC POWER settings */
264
270
if (cpu_is_omap3430() && omap_rev() >= OMAP3430_REV_ES3_0 &&
···
302
294
if (omap_irq_pending())
303
295
return;
304
296
305
-
omap_sram_idle();
297
+
omap3_do_wfi();
306
298
}
307
299
308
300
#ifdef CONFIG_SUSPEND
···
324
316
325
317
omap3_intc_suspend();
326
318
327
-
omap_sram_idle();
319
+
omap_sram_idle(false);
328
320
329
321
restore:
330
322
/* Restore next_pwrsts */
+1
-1
arch/arm/mach-omap2/pm44xx.c
+1
-1
arch/arm/mach-omap2/pm44xx.c
···
76
76
* domain CSWR is not supported by hardware.
77
77
* More details can be found in OMAP4430 TRM section 4.3.4.2.
78
78
*/
79
-
omap4_enter_lowpower(cpu_id, cpu_suspend_state);
79
+
omap4_enter_lowpower(cpu_id, cpu_suspend_state, false);
80
80
81
81
/* Restore next powerdomain state */
82
82
list_for_each_entry(pwrst, &pwrst_list, node) {
+5
-5
arch/arm/mach-omap2/powerdomain.c
+5
-5
arch/arm/mach-omap2/powerdomain.c
···
187
187
trace_state = (PWRDM_TRACE_STATES_FLAG |
188
188
((next & OMAP_POWERSTATE_MASK) << 8) |
189
189
((prev & OMAP_POWERSTATE_MASK) << 0));
190
-
trace_power_domain_target_rcuidle(pwrdm->name,
191
-
trace_state,
192
-
raw_smp_processor_id());
190
+
trace_power_domain_target(pwrdm->name,
191
+
trace_state,
192
+
raw_smp_processor_id());
193
193
}
194
194
break;
195
195
default:
···
541
541
542
542
if (arch_pwrdm && arch_pwrdm->pwrdm_set_next_pwrst) {
543
543
/* Trace the pwrdm desired target state */
544
-
trace_power_domain_target_rcuidle(pwrdm->name, pwrst,
545
-
raw_smp_processor_id());
544
+
trace_power_domain_target(pwrdm->name, pwrst,
545
+
raw_smp_processor_id());
546
546
/* Program the pwrdm desired target state */
547
547
ret = arch_pwrdm->pwrdm_set_next_pwrst(pwrdm, pwrst);
548
548
}
+2
-3
arch/arm/mach-s3c/cpuidle-s3c64xx.c
+2
-3
arch/arm/mach-s3c/cpuidle-s3c64xx.c
···
19
19
#include "regs-sys-s3c64xx.h"
20
20
#include "regs-syscon-power-s3c64xx.h"
21
21
22
-
static int s3c64xx_enter_idle(struct cpuidle_device *dev,
23
-
struct cpuidle_driver *drv,
24
-
int index)
22
+
static __cpuidle int s3c64xx_enter_idle(struct cpuidle_device *dev,
23
+
struct cpuidle_driver *drv, int index)
25
24
{
26
25
unsigned long tmp;
27
26
+3
-3
arch/arm64/kernel/cpuidle.c
+3
-3
arch/arm64/kernel/cpuidle.c
···
62
62
return psci_acpi_cpu_init_idle(cpu);
63
63
}
64
64
65
-
int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi)
65
+
__cpuidle int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi)
66
66
{
67
67
u32 state = lpi->address;
68
68
69
69
if (ARM64_LPI_IS_RETENTION_STATE(lpi->arch_flags))
70
-
return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(psci_cpu_suspend_enter,
70
+
return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(psci_cpu_suspend_enter,
71
71
lpi->index, state);
72
72
else
73
-
return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter,
73
+
return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter,
74
74
lpi->index, state);
75
75
}
76
76
#endif
-1
arch/arm64/kernel/idle.c
-1
arch/arm64/kernel/idle.c
+2
-2
arch/arm64/kernel/smp.c
+2
-2
arch/arm64/kernel/smp.c
···
865
865
unsigned int cpu = smp_processor_id();
866
866
867
867
if ((unsigned)ipinr < NR_IPI)
868
-
trace_ipi_entry_rcuidle(ipi_types[ipinr]);
868
+
trace_ipi_entry(ipi_types[ipinr]);
869
869
870
870
switch (ipinr) {
871
871
case IPI_RESCHEDULE:
···
914
914
}
915
915
916
916
if ((unsigned)ipinr < NR_IPI)
917
-
trace_ipi_exit_rcuidle(ipi_types[ipinr]);
917
+
trace_ipi_exit(ipi_types[ipinr]);
918
918
}
919
919
920
920
static irqreturn_t ipi_handler(int irq, void *data)
+11
-1
arch/arm64/kernel/suspend.c
+11
-1
arch/arm64/kernel/suspend.c
···
4
4
#include <linux/slab.h>
5
5
#include <linux/uaccess.h>
6
6
#include <linux/pgtable.h>
7
+
#include <linux/cpuidle.h>
7
8
#include <asm/alternative.h>
8
9
#include <asm/cacheflush.h>
9
10
#include <asm/cpufeature.h>
···
105
104
* From this point debug exceptions are disabled to prevent
106
105
* updates to mdscr register (saved and restored along with
107
106
* general purpose registers) from kernel debuggers.
107
+
*
108
+
* Strictly speaking the trace_hardirqs_off() here is superfluous,
109
+
* hardirqs should be firmly off by now. This really ought to use
110
+
* something like raw_local_daif_save().
108
111
*/
109
112
flags = local_daif_save();
110
113
···
125
120
*/
126
121
arm_cpuidle_save_irq_context(&context);
127
122
123
+
ct_cpuidle_enter();
124
+
128
125
if (__cpu_suspend_enter(&state)) {
129
126
/* Call the suspend finisher */
130
127
ret = fn(arg);
···
140
133
*/
141
134
if (!ret)
142
135
ret = -EOPNOTSUPP;
136
+
137
+
ct_cpuidle_exit();
143
138
} else {
144
-
RCU_NONIDLE(__cpu_suspend_exit());
139
+
ct_cpuidle_exit();
140
+
__cpu_suspend_exit();
145
141
}
146
142
147
143
arm_cpuidle_restore_irq_context(&context);
-1
arch/arm64/kernel/vmlinux.lds.S
-1
arch/arm64/kernel/vmlinux.lds.S
-1
arch/csky/kernel/process.c
-1
arch/csky/kernel/process.c
+1
-1
arch/csky/kernel/smp.c
+1
-1
arch/csky/kernel/smp.c
-1
arch/csky/kernel/vmlinux.lds.S
-1
arch/csky/kernel/vmlinux.lds.S
-1
arch/hexagon/kernel/process.c
-1
arch/hexagon/kernel/process.c
-1
arch/hexagon/kernel/vmlinux.lds.S
-1
arch/hexagon/kernel/vmlinux.lds.S
+1
arch/ia64/kernel/process.c
+1
arch/ia64/kernel/process.c
+1
arch/ia64/kernel/time.c
+1
arch/ia64/kernel/time.c
-1
arch/ia64/kernel/vmlinux.lds.S
-1
arch/ia64/kernel/vmlinux.lds.S
+1
arch/loongarch/kernel/idle.c
+1
arch/loongarch/kernel/idle.c
-1
arch/loongarch/kernel/vmlinux.lds.S
-1
arch/loongarch/kernel/vmlinux.lds.S
-1
arch/m68k/kernel/vmlinux-nommu.lds
-1
arch/m68k/kernel/vmlinux-nommu.lds
-1
arch/m68k/kernel/vmlinux-std.lds
-1
arch/m68k/kernel/vmlinux-std.lds
-1
arch/m68k/kernel/vmlinux-sun3.lds
-1
arch/m68k/kernel/vmlinux-sun3.lds
-1
arch/microblaze/kernel/process.c
-1
arch/microblaze/kernel/process.c
-1
arch/microblaze/kernel/vmlinux.lds.S
-1
arch/microblaze/kernel/vmlinux.lds.S
+6
-8
arch/mips/kernel/idle.c
+6
-8
arch/mips/kernel/idle.c
···
33
33
{
34
34
unsigned long cfg = read_c0_conf();
35
35
write_c0_conf(cfg | R30XX_CONF_HALT);
36
-
raw_local_irq_enable();
37
36
}
38
37
39
38
void __cpuidle r4k_wait(void)
40
39
{
41
40
raw_local_irq_enable();
42
41
__r4k_wait();
42
+
raw_local_irq_disable();
43
43
}
44
44
45
45
/*
···
57
57
" .set arch=r4000 \n"
58
58
" wait \n"
59
59
" .set pop \n");
60
-
raw_local_irq_enable();
61
60
}
62
61
63
62
/*
···
76
77
" wait \n"
77
78
" mtc0 $1, $12 # stalls until W stage \n"
78
79
" .set pop \n");
79
-
raw_local_irq_enable();
80
80
}
81
81
82
82
/*
···
101
103
" nop \n"
102
104
" .set pop \n"
103
105
: : "r" (au1k_wait), "r" (c0status));
106
+
107
+
raw_local_irq_disable();
104
108
}
105
109
106
110
static int __initdata nowait;
···
241
241
}
242
242
}
243
243
244
-
void arch_cpu_idle(void)
244
+
__cpuidle void arch_cpu_idle(void)
245
245
{
246
246
if (cpu_wait)
247
247
cpu_wait();
248
-
else
249
-
raw_local_irq_enable();
250
248
}
251
249
252
250
#ifdef CONFIG_CPU_IDLE
253
251
254
-
int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
255
-
struct cpuidle_driver *drv, int index)
252
+
__cpuidle int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
253
+
struct cpuidle_driver *drv, int index)
256
254
{
257
255
arch_cpu_idle();
258
256
return index;
-1
arch/mips/kernel/vmlinux.lds.S
-1
arch/mips/kernel/vmlinux.lds.S
-1
arch/nios2/kernel/process.c
-1
arch/nios2/kernel/process.c
-1
arch/nios2/kernel/vmlinux.lds.S
-1
arch/nios2/kernel/vmlinux.lds.S
+1
arch/openrisc/kernel/process.c
+1
arch/openrisc/kernel/process.c
-1
arch/openrisc/kernel/vmlinux.lds.S
-1
arch/openrisc/kernel/vmlinux.lds.S
-2
arch/parisc/kernel/process.c
-2
arch/parisc/kernel/process.c
-1
arch/parisc/kernel/vmlinux.lds.S
-1
arch/parisc/kernel/vmlinux.lds.S
+2
-3
arch/powerpc/kernel/idle.c
+2
-3
arch/powerpc/kernel/idle.c
···
51
51
* Some power_save functions return with
52
52
* interrupts enabled, some don't.
53
53
*/
54
-
if (irqs_disabled())
55
-
raw_local_irq_enable();
54
+
if (!irqs_disabled())
55
+
raw_local_irq_disable();
56
56
} else {
57
-
raw_local_irq_enable();
58
57
/*
59
58
* Go into low thread priority and possibly
60
59
* low power mode.
-1
arch/powerpc/kernel/vmlinux.lds.S
-1
arch/powerpc/kernel/vmlinux.lds.S
-1
arch/riscv/kernel/process.c
-1
arch/riscv/kernel/process.c
-1
arch/riscv/kernel/vmlinux-xip.lds.S
-1
arch/riscv/kernel/vmlinux-xip.lds.S
-1
arch/riscv/kernel/vmlinux.lds.S
-1
arch/riscv/kernel/vmlinux.lds.S
+1
-2
arch/s390/kernel/idle.c
+1
-2
arch/s390/kernel/idle.c
···
12
12
#include <linux/notifier.h>
13
13
#include <linux/init.h>
14
14
#include <linux/cpu.h>
15
-
#include <linux/sched/cputime.h>
16
15
#include <trace/events/power.h>
17
16
#include <asm/cpu_mf.h>
17
+
#include <asm/cputime.h>
18
18
#include <asm/nmi.h>
19
19
#include <asm/smp.h>
20
20
#include "entry.h"
···
66
66
idle->idle_count++;
67
67
account_idle_time(cputime_to_nsecs(idle_time));
68
68
raw_write_seqcount_end(&idle->seqcount);
69
-
raw_local_irq_enable();
70
69
}
71
70
72
71
static ssize_t show_idle_count(struct device *dev,
-1
arch/s390/kernel/vmlinux.lds.S
-1
arch/s390/kernel/vmlinux.lds.S
+1
-1
arch/s390/kernel/vtime.c
+1
-1
arch/s390/kernel/vtime.c
···
7
7
*/
8
8
9
9
#include <linux/kernel_stat.h>
10
-
#include <linux/sched/cputime.h>
11
10
#include <linux/export.h>
12
11
#include <linux/kernel.h>
13
12
#include <linux/timex.h>
14
13
#include <linux/types.h>
15
14
#include <linux/time.h>
16
15
#include <asm/alternative.h>
16
+
#include <asm/cputime.h>
17
17
#include <asm/vtimer.h>
18
18
#include <asm/vtime.h>
19
19
#include <asm/cpu_mf.h>
+1
arch/sh/kernel/idle.c
+1
arch/sh/kernel/idle.c
-1
arch/sh/kernel/vmlinux.lds.S
-1
arch/sh/kernel/vmlinux.lds.S
+4
arch/sparc/kernel/leon_pmc.c
+4
arch/sparc/kernel/leon_pmc.c
···
57
57
"lda [%0] %1, %%g0\n"
58
58
:
59
59
: "r"(address), "i"(ASI_LEON_BYPASS));
60
+
61
+
raw_local_irq_disable();
60
62
}
61
63
62
64
/*
···
72
70
73
71
/* For systems without power-down, this will be no-op */
74
72
__asm__ __volatile__ ("wr %g0, %asr19\n\t");
73
+
74
+
raw_local_irq_disable();
75
75
}
76
76
77
77
/* Install LEON Power Down function */
-1
arch/sparc/kernel/process_32.c
-1
arch/sparc/kernel/process_32.c
+2
-1
arch/sparc/kernel/process_64.c
+2
-1
arch/sparc/kernel/process_64.c
-1
arch/sparc/kernel/vmlinux.lds.S
-1
arch/sparc/kernel/vmlinux.lds.S
-1
arch/um/kernel/dyn.lds.S
-1
arch/um/kernel/dyn.lds.S
-1
arch/um/kernel/process.c
-1
arch/um/kernel/process.c
-1
arch/um/kernel/uml.lds.S
-1
arch/um/kernel/uml.lds.S
+1
arch/x86/boot/compressed/vmlinux.lds.S
+1
arch/x86/boot/compressed/vmlinux.lds.S
+2
-13
arch/x86/coco/tdx/tdcall.S
+2
-13
arch/x86/coco/tdx/tdcall.S
···
31
31
TDX_R12 | TDX_R13 | \
32
32
TDX_R14 | TDX_R15 )
33
33
34
+
.section .noinstr.text, "ax"
35
+
34
36
/*
35
37
* __tdx_module_call() - Used by TDX guests to request services from
36
38
* the TDX module (does not include VMM services) using TDCALL instruction.
···
141
139
142
140
movl $TDVMCALL_EXPOSE_REGS_MASK, %ecx
143
141
144
-
/*
145
-
* For the idle loop STI needs to be called directly before the TDCALL
146
-
* that enters idle (EXIT_REASON_HLT case). STI instruction enables
147
-
* interrupts only one instruction later. If there is a window between
148
-
* STI and the instruction that emulates the HALT state, there is a
149
-
* chance for interrupts to happen in this window, which can delay the
150
-
* HLT operation indefinitely. Since this is the not the desired
151
-
* result, conditionally call STI before TDCALL.
152
-
*/
153
-
testq $TDX_HCALL_ISSUE_STI, %rsi
154
-
jz .Lskip_sti
155
-
sti
156
-
.Lskip_sti:
157
142
tdcall
158
143
159
144
/*
+7
-18
arch/x86/coco/tdx/tdx.c
+7
-18
arch/x86/coco/tdx/tdx.c
···
64
64
}
65
65
66
66
/* Called from __tdx_hypercall() for unrecoverable failure */
67
-
void __tdx_hypercall_failed(void)
67
+
noinstr void __tdx_hypercall_failed(void)
68
68
{
69
+
instrumentation_begin();
69
70
panic("TDVMCALL failed. TDX module bug?");
70
71
}
71
72
···
76
75
* Reusing the KVM EXIT_REASON macros makes it easier to connect the host and
77
76
* guest sides of these calls.
78
77
*/
79
-
static u64 hcall_func(u64 exit_reason)
78
+
static __always_inline u64 hcall_func(u64 exit_reason)
80
79
{
81
80
return exit_reason;
82
81
}
···
221
220
}
222
221
}
223
222
224
-
static u64 __cpuidle __halt(const bool irq_disabled, const bool do_sti)
223
+
static u64 __cpuidle __halt(const bool irq_disabled)
225
224
{
226
225
struct tdx_hypercall_args args = {
227
226
.r10 = TDX_HYPERCALL_STANDARD,
···
241
240
* can keep the vCPU in virtual HLT, even if an IRQ is
242
241
* pending, without hanging/breaking the guest.
243
242
*/
244
-
return __tdx_hypercall(&args, do_sti ? TDX_HCALL_ISSUE_STI : 0);
243
+
return __tdx_hypercall(&args, 0);
245
244
}
246
245
247
246
static int handle_halt(struct ve_info *ve)
248
247
{
249
-
/*
250
-
* Since non safe halt is mainly used in CPU offlining
251
-
* and the guest will always stay in the halt state, don't
252
-
* call the STI instruction (set do_sti as false).
253
-
*/
254
248
const bool irq_disabled = irqs_disabled();
255
-
const bool do_sti = false;
256
249
257
-
if (__halt(irq_disabled, do_sti))
250
+
if (__halt(irq_disabled))
258
251
return -EIO;
259
252
260
253
return ve_instr_len(ve);
···
256
261
257
262
void __cpuidle tdx_safe_halt(void)
258
263
{
259
-
/*
260
-
* For do_sti=true case, __tdx_hypercall() function enables
261
-
* interrupts using the STI instruction before the TDCALL. So
262
-
* set irq_disabled as false.
263
-
*/
264
264
const bool irq_disabled = false;
265
-
const bool do_sti = true;
266
265
267
266
/*
268
267
* Use WARN_ONCE() to report the failure.
269
268
*/
270
-
if (__halt(irq_disabled, do_sti))
269
+
if (__halt(irq_disabled))
271
270
WARN_ONCE(1, "HLT instruction emulation failed\n");
272
271
}
273
272
+5
-8
arch/x86/events/amd/brs.c
+5
-8
arch/x86/events/amd/brs.c
···
41
41
return MSR_AMD_SAMP_BR_FROM + 2 * idx + 1;
42
42
}
43
43
44
-
static inline void set_debug_extn_cfg(u64 val)
44
+
static __always_inline void set_debug_extn_cfg(u64 val)
45
45
{
46
46
/* bits[4:3] must always be set to 11b */
47
-
wrmsrl(MSR_AMD_DBG_EXTN_CFG, val | 3ULL << 3);
47
+
__wrmsr(MSR_AMD_DBG_EXTN_CFG, val | 3ULL << 3, val >> 32);
48
48
}
49
49
50
-
static inline u64 get_debug_extn_cfg(void)
50
+
static __always_inline u64 get_debug_extn_cfg(void)
51
51
{
52
-
u64 val;
53
-
54
-
rdmsrl(MSR_AMD_DBG_EXTN_CFG, val);
55
-
return val;
52
+
return __rdmsr(MSR_AMD_DBG_EXTN_CFG);
56
53
}
57
54
58
55
static bool __init amd_brs_detect(void)
···
402
405
* called from ACPI processor_idle.c or acpi_pad.c
403
406
* with interrupts disabled
404
407
*/
405
-
void perf_amd_brs_lopwr_cb(bool lopwr_in)
408
+
void noinstr perf_amd_brs_lopwr_cb(bool lopwr_in)
406
409
{
407
410
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
408
411
union amd_debug_extn_cfg cfg;
+22
-22
arch/x86/include/asm/atomic64_32.h
+22
-22
arch/x86/include/asm/atomic64_32.h
···
71
71
* the old value.
72
72
*/
73
73
74
-
static inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 o, s64 n)
74
+
static __always_inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 o, s64 n)
75
75
{
76
76
return arch_cmpxchg64(&v->counter, o, n);
77
77
}
···
85
85
* Atomically xchgs the value of @v to @n and returns
86
86
* the old value.
87
87
*/
88
-
static inline s64 arch_atomic64_xchg(atomic64_t *v, s64 n)
88
+
static __always_inline s64 arch_atomic64_xchg(atomic64_t *v, s64 n)
89
89
{
90
90
s64 o;
91
91
unsigned high = (unsigned)(n >> 32);
···
104
104
*
105
105
* Atomically sets the value of @v to @n.
106
106
*/
107
-
static inline void arch_atomic64_set(atomic64_t *v, s64 i)
107
+
static __always_inline void arch_atomic64_set(atomic64_t *v, s64 i)
108
108
{
109
109
unsigned high = (unsigned)(i >> 32);
110
110
unsigned low = (unsigned)i;
···
119
119
*
120
120
* Atomically reads the value of @v and returns it.
121
121
*/
122
-
static inline s64 arch_atomic64_read(const atomic64_t *v)
122
+
static __always_inline s64 arch_atomic64_read(const atomic64_t *v)
123
123
{
124
124
s64 r;
125
125
alternative_atomic64(read, "=&A" (r), "c" (v) : "memory");
···
133
133
*
134
134
* Atomically adds @i to @v and returns @i + *@v
135
135
*/
136
-
static inline s64 arch_atomic64_add_return(s64 i, atomic64_t *v)
136
+
static __always_inline s64 arch_atomic64_add_return(s64 i, atomic64_t *v)
137
137
{
138
138
alternative_atomic64(add_return,
139
139
ASM_OUTPUT2("+A" (i), "+c" (v)),
···
145
145
/*
146
146
* Other variants with different arithmetic operators:
147
147
*/
148
-
static inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v)
148
+
static __always_inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v)
149
149
{
150
150
alternative_atomic64(sub_return,
151
151
ASM_OUTPUT2("+A" (i), "+c" (v)),
···
154
154
}
155
155
#define arch_atomic64_sub_return arch_atomic64_sub_return
156
156
157
-
static inline s64 arch_atomic64_inc_return(atomic64_t *v)
157
+
static __always_inline s64 arch_atomic64_inc_return(atomic64_t *v)
158
158
{
159
159
s64 a;
160
160
alternative_atomic64(inc_return, "=&A" (a),
···
163
163
}
164
164
#define arch_atomic64_inc_return arch_atomic64_inc_return
165
165
166
-
static inline s64 arch_atomic64_dec_return(atomic64_t *v)
166
+
static __always_inline s64 arch_atomic64_dec_return(atomic64_t *v)
167
167
{
168
168
s64 a;
169
169
alternative_atomic64(dec_return, "=&A" (a),
···
179
179
*
180
180
* Atomically adds @i to @v.
181
181
*/
182
-
static inline s64 arch_atomic64_add(s64 i, atomic64_t *v)
182
+
static __always_inline s64 arch_atomic64_add(s64 i, atomic64_t *v)
183
183
{
184
184
__alternative_atomic64(add, add_return,
185
185
ASM_OUTPUT2("+A" (i), "+c" (v)),
···
194
194
*
195
195
* Atomically subtracts @i from @v.
196
196
*/
197
-
static inline s64 arch_atomic64_sub(s64 i, atomic64_t *v)
197
+
static __always_inline s64 arch_atomic64_sub(s64 i, atomic64_t *v)
198
198
{
199
199
__alternative_atomic64(sub, sub_return,
200
200
ASM_OUTPUT2("+A" (i), "+c" (v)),
···
208
208
*
209
209
* Atomically increments @v by 1.
210
210
*/
211
-
static inline void arch_atomic64_inc(atomic64_t *v)
211
+
static __always_inline void arch_atomic64_inc(atomic64_t *v)
212
212
{
213
213
__alternative_atomic64(inc, inc_return, /* no output */,
214
214
"S" (v) : "memory", "eax", "ecx", "edx");
···
221
221
*
222
222
* Atomically decrements @v by 1.
223
223
*/
224
-
static inline void arch_atomic64_dec(atomic64_t *v)
224
+
static __always_inline void arch_atomic64_dec(atomic64_t *v)
225
225
{
226
226
__alternative_atomic64(dec, dec_return, /* no output */,
227
227
"S" (v) : "memory", "eax", "ecx", "edx");
···
237
237
* Atomically adds @a to @v, so long as it was not @u.
238
238
* Returns non-zero if the add was done, zero otherwise.
239
239
*/
240
-
static inline int arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
240
+
static __always_inline int arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
241
241
{
242
242
unsigned low = (unsigned)u;
243
243
unsigned high = (unsigned)(u >> 32);
···
248
248
}
249
249
#define arch_atomic64_add_unless arch_atomic64_add_unless
250
250
251
-
static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
251
+
static __always_inline int arch_atomic64_inc_not_zero(atomic64_t *v)
252
252
{
253
253
int r;
254
254
alternative_atomic64(inc_not_zero, "=&a" (r),
···
257
257
}
258
258
#define arch_atomic64_inc_not_zero arch_atomic64_inc_not_zero
259
259
260
-
static inline s64 arch_atomic64_dec_if_positive(atomic64_t *v)
260
+
static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v)
261
261
{
262
262
s64 r;
263
263
alternative_atomic64(dec_if_positive, "=&A" (r),
···
269
269
#undef alternative_atomic64
270
270
#undef __alternative_atomic64
271
271
272
-
static inline void arch_atomic64_and(s64 i, atomic64_t *v)
272
+
static __always_inline void arch_atomic64_and(s64 i, atomic64_t *v)
273
273
{
274
274
s64 old, c = 0;
275
275
···
277
277
c = old;
278
278
}
279
279
280
-
static inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v)
280
+
static __always_inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v)
281
281
{
282
282
s64 old, c = 0;
283
283
···
288
288
}
289
289
#define arch_atomic64_fetch_and arch_atomic64_fetch_and
290
290
291
-
static inline void arch_atomic64_or(s64 i, atomic64_t *v)
291
+
static __always_inline void arch_atomic64_or(s64 i, atomic64_t *v)
292
292
{
293
293
s64 old, c = 0;
294
294
···
296
296
c = old;
297
297
}
298
298
299
-
static inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v)
299
+
static __always_inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v)
300
300
{
301
301
s64 old, c = 0;
302
302
···
307
307
}
308
308
#define arch_atomic64_fetch_or arch_atomic64_fetch_or
309
309
310
-
static inline void arch_atomic64_xor(s64 i, atomic64_t *v)
310
+
static __always_inline void arch_atomic64_xor(s64 i, atomic64_t *v)
311
311
{
312
312
s64 old, c = 0;
313
313
···
315
315
c = old;
316
316
}
317
317
318
-
static inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v)
318
+
static __always_inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v)
319
319
{
320
320
s64 old, c = 0;
321
321
···
326
326
}
327
327
#define arch_atomic64_fetch_xor arch_atomic64_fetch_xor
328
328
329
-
static inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v)
329
+
static __always_inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v)
330
330
{
331
331
s64 old, c = 0;
332
332
+18
-18
arch/x86/include/asm/atomic64_64.h
+18
-18
arch/x86/include/asm/atomic64_64.h
···
17
17
* Atomically reads the value of @v.
18
18
* Doesn't imply a read memory barrier.
19
19
*/
20
-
static inline s64 arch_atomic64_read(const atomic64_t *v)
20
+
static __always_inline s64 arch_atomic64_read(const atomic64_t *v)
21
21
{
22
22
return __READ_ONCE((v)->counter);
23
23
}
···
29
29
*
30
30
* Atomically sets the value of @v to @i.
31
31
*/
32
-
static inline void arch_atomic64_set(atomic64_t *v, s64 i)
32
+
static __always_inline void arch_atomic64_set(atomic64_t *v, s64 i)
33
33
{
34
34
__WRITE_ONCE(v->counter, i);
35
35
}
···
55
55
*
56
56
* Atomically subtracts @i from @v.
57
57
*/
58
-
static inline void arch_atomic64_sub(s64 i, atomic64_t *v)
58
+
static __always_inline void arch_atomic64_sub(s64 i, atomic64_t *v)
59
59
{
60
60
asm volatile(LOCK_PREFIX "subq %1,%0"
61
61
: "=m" (v->counter)
···
71
71
* true if the result is zero, or false for all
72
72
* other cases.
73
73
*/
74
-
static inline bool arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
74
+
static __always_inline bool arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
75
75
{
76
76
return GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, e, "er", i);
77
77
}
···
113
113
* returns true if the result is 0, or false for all other
114
114
* cases.
115
115
*/
116
-
static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
116
+
static __always_inline bool arch_atomic64_dec_and_test(atomic64_t *v)
117
117
{
118
118
return GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, e);
119
119
}
···
127
127
* and returns true if the result is zero, or false for all
128
128
* other cases.
129
129
*/
130
-
static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
130
+
static __always_inline bool arch_atomic64_inc_and_test(atomic64_t *v)
131
131
{
132
132
return GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, e);
133
133
}
···
142
142
* if the result is negative, or false when
143
143
* result is greater than or equal to zero.
144
144
*/
145
-
static inline bool arch_atomic64_add_negative(s64 i, atomic64_t *v)
145
+
static __always_inline bool arch_atomic64_add_negative(s64 i, atomic64_t *v)
146
146
{
147
147
return GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, s, "er", i);
148
148
}
···
161
161
}
162
162
#define arch_atomic64_add_return arch_atomic64_add_return
163
163
164
-
static inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v)
164
+
static __always_inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v)
165
165
{
166
166
return arch_atomic64_add_return(-i, v);
167
167
}
168
168
#define arch_atomic64_sub_return arch_atomic64_sub_return
169
169
170
-
static inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v)
170
+
static __always_inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v)
171
171
{
172
172
return xadd(&v->counter, i);
173
173
}
174
174
#define arch_atomic64_fetch_add arch_atomic64_fetch_add
175
175
176
-
static inline s64 arch_atomic64_fetch_sub(s64 i, atomic64_t *v)
176
+
static __always_inline s64 arch_atomic64_fetch_sub(s64 i, atomic64_t *v)
177
177
{
178
178
return xadd(&v->counter, -i);
179
179
}
180
180
#define arch_atomic64_fetch_sub arch_atomic64_fetch_sub
181
181
182
-
static inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
182
+
static __always_inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
183
183
{
184
184
return arch_cmpxchg(&v->counter, old, new);
185
185
}
···
191
191
}
192
192
#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg
193
193
194
-
static inline s64 arch_atomic64_xchg(atomic64_t *v, s64 new)
194
+
static __always_inline s64 arch_atomic64_xchg(atomic64_t *v, s64 new)
195
195
{
196
196
return arch_xchg(&v->counter, new);
197
197
}
198
198
#define arch_atomic64_xchg arch_atomic64_xchg
199
199
200
-
static inline void arch_atomic64_and(s64 i, atomic64_t *v)
200
+
static __always_inline void arch_atomic64_and(s64 i, atomic64_t *v)
201
201
{
202
202
asm volatile(LOCK_PREFIX "andq %1,%0"
203
203
: "+m" (v->counter)
···
205
205
: "memory");
206
206
}
207
207
208
-
static inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v)
208
+
static __always_inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v)
209
209
{
210
210
s64 val = arch_atomic64_read(v);
211
211
···
215
215
}
216
216
#define arch_atomic64_fetch_and arch_atomic64_fetch_and
217
217
218
-
static inline void arch_atomic64_or(s64 i, atomic64_t *v)
218
+
static __always_inline void arch_atomic64_or(s64 i, atomic64_t *v)
219
219
{
220
220
asm volatile(LOCK_PREFIX "orq %1,%0"
221
221
: "+m" (v->counter)
···
223
223
: "memory");
224
224
}
225
225
226
-
static inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v)
226
+
static __always_inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v)
227
227
{
228
228
s64 val = arch_atomic64_read(v);
229
229
···
233
233
}
234
234
#define arch_atomic64_fetch_or arch_atomic64_fetch_or
235
235
236
-
static inline void arch_atomic64_xor(s64 i, atomic64_t *v)
236
+
static __always_inline void arch_atomic64_xor(s64 i, atomic64_t *v)
237
237
{
238
238
asm volatile(LOCK_PREFIX "xorq %1,%0"
239
239
: "+m" (v->counter)
···
241
241
: "memory");
242
242
}
243
243
244
-
static inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v)
244
+
static __always_inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v)
245
245
{
246
246
s64 val = arch_atomic64_read(v);
247
247
+2
-2
arch/x86/include/asm/fpu/xcr.h
+2
-2
arch/x86/include/asm/fpu/xcr.h
···
5
5
#define XCR_XFEATURE_ENABLED_MASK 0x00000000
6
6
#define XCR_XFEATURE_IN_USE_MASK 0x00000001
7
7
8
-
static inline u64 xgetbv(u32 index)
8
+
static __always_inline u64 xgetbv(u32 index)
9
9
{
10
10
u32 eax, edx;
11
11
···
27
27
*
28
28
* Callers should check X86_FEATURE_XGETBV1.
29
29
*/
30
-
static inline u64 xfeatures_in_use(void)
30
+
static __always_inline u64 xfeatures_in_use(void)
31
31
{
32
32
return xgetbv(XCR_XFEATURE_IN_USE_MASK);
33
33
}
+4
-7
arch/x86/include/asm/irqflags.h
+4
-7
arch/x86/include/asm/irqflags.h
···
8
8
9
9
#include <asm/nospec-branch.h>
10
10
11
-
/* Provide __cpuidle; we can't safely include <linux/cpu.h> */
12
-
#define __cpuidle __section(".cpuidle.text")
13
-
14
11
/*
15
12
* Interrupt control:
16
13
*/
···
42
45
asm volatile("sti": : :"memory");
43
46
}
44
47
45
-
static inline __cpuidle void native_safe_halt(void)
48
+
static __always_inline void native_safe_halt(void)
46
49
{
47
50
mds_idle_clear_cpu_buffers();
48
51
asm volatile("sti; hlt": : :"memory");
49
52
}
50
53
51
-
static inline __cpuidle void native_halt(void)
54
+
static __always_inline void native_halt(void)
52
55
{
53
56
mds_idle_clear_cpu_buffers();
54
57
asm volatile("hlt": : :"memory");
···
81
84
* Used in the idle loop; sti takes one instruction cycle
82
85
* to complete:
83
86
*/
84
-
static inline __cpuidle void arch_safe_halt(void)
87
+
static __always_inline void arch_safe_halt(void)
85
88
{
86
89
native_safe_halt();
87
90
}
···
90
93
* Used when interrupts are already enabled or to
91
94
* shutdown the processor:
92
95
*/
93
-
static inline __cpuidle void halt(void)
96
+
static __always_inline void halt(void)
94
97
{
95
98
native_halt();
96
99
}
+1
-1
arch/x86/include/asm/kvmclock.h
+1
-1
arch/x86/include/asm/kvmclock.h
···
8
8
9
9
DECLARE_PER_CPU(struct pvclock_vsyscall_time_info *, hv_clock_per_cpu);
10
10
11
-
static inline struct pvclock_vcpu_time_info *this_cpu_pvti(void)
11
+
static __always_inline struct pvclock_vcpu_time_info *this_cpu_pvti(void)
12
12
{
13
13
return &this_cpu_read(hv_clock_per_cpu)->pvti;
14
14
}
+7
-7
arch/x86/include/asm/mwait.h
+7
-7
arch/x86/include/asm/mwait.h
···
26
26
#define TPAUSE_C01_STATE 1
27
27
#define TPAUSE_C02_STATE 0
28
28
29
-
static inline void __monitor(const void *eax, unsigned long ecx,
29
+
static __always_inline void __monitor(const void *eax, unsigned long ecx,
30
30
unsigned long edx)
31
31
{
32
32
/* "monitor %eax, %ecx, %edx;" */
···
34
34
:: "a" (eax), "c" (ecx), "d"(edx));
35
35
}
36
36
37
-
static inline void __monitorx(const void *eax, unsigned long ecx,
37
+
static __always_inline void __monitorx(const void *eax, unsigned long ecx,
38
38
unsigned long edx)
39
39
{
40
40
/* "monitorx %eax, %ecx, %edx;" */
···
42
42
:: "a" (eax), "c" (ecx), "d"(edx));
43
43
}
44
44
45
-
static inline void __mwait(unsigned long eax, unsigned long ecx)
45
+
static __always_inline void __mwait(unsigned long eax, unsigned long ecx)
46
46
{
47
47
mds_idle_clear_cpu_buffers();
48
48
···
77
77
* EAX (logical) address to monitor
78
78
* ECX #GP if not zero
79
79
*/
80
-
static inline void __mwaitx(unsigned long eax, unsigned long ebx,
81
-
unsigned long ecx)
80
+
static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx,
81
+
unsigned long ecx)
82
82
{
83
83
/* No MDS buffer clear as this is AMD/HYGON only */
84
84
···
87
87
:: "a" (eax), "b" (ebx), "c" (ecx));
88
88
}
89
89
90
-
static inline void __sti_mwait(unsigned long eax, unsigned long ecx)
90
+
static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx)
91
91
{
92
92
mds_idle_clear_cpu_buffers();
93
93
/* "mwait %eax, %ecx;" */
···
105
105
* New with Core Duo processors, MWAIT can take some hints based on CPU
106
106
* capability.
107
107
*/
108
-
static inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx)
108
+
static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx)
109
109
{
110
110
if (static_cpu_has_bug(X86_BUG_MONITOR) || !current_set_polling_and_test()) {
111
111
if (static_cpu_has_bug(X86_BUG_CLFLUSH_MONITOR)) {
+1
-1
arch/x86/include/asm/nospec-branch.h
+1
-1
arch/x86/include/asm/nospec-branch.h
···
564
564
*
565
565
* Clear CPU buffers if the corresponding static key is enabled
566
566
*/
567
-
static inline void mds_idle_clear_cpu_buffers(void)
567
+
static __always_inline void mds_idle_clear_cpu_buffers(void)
568
568
{
569
569
if (static_branch_likely(&mds_idle_clear))
570
570
mds_clear_cpu_buffers();
+5
-3
arch/x86/include/asm/paravirt.h
+5
-3
arch/x86/include/asm/paravirt.h
···
26
26
27
27
void paravirt_set_sched_clock(u64 (*func)(void));
28
28
29
-
static inline u64 paravirt_sched_clock(void)
29
+
static __always_inline u64 paravirt_sched_clock(void)
30
30
{
31
31
return static_call(pv_sched_clock)();
32
32
}
···
168
168
PVOP_VCALL1(cpu.write_cr4, x);
169
169
}
170
170
171
-
static inline void arch_safe_halt(void)
171
+
static __always_inline void arch_safe_halt(void)
172
172
{
173
173
PVOP_VCALL0(irq.safe_halt);
174
174
}
···
178
178
PVOP_VCALL0(irq.halt);
179
179
}
180
180
181
-
static inline void wbinvd(void)
181
+
extern noinstr void pv_native_wbinvd(void);
182
+
183
+
static __always_inline void wbinvd(void)
182
184
{
183
185
PVOP_ALT_VCALL0(cpu.wbinvd, "wbinvd", ALT_NOT(X86_FEATURE_XENPV));
184
186
}
+1
-1
arch/x86/include/asm/perf_event.h
+1
-1
arch/x86/include/asm/perf_event.h
+2
-1
arch/x86/include/asm/pvclock.h
+2
-1
arch/x86/include/asm/pvclock.h
···
7
7
8
8
/* some helper functions for xen and kvm pv clock sources */
9
9
u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src);
10
+
u64 pvclock_clocksource_read_nowd(struct pvclock_vcpu_time_info *src);
10
11
u8 pvclock_read_flags(struct pvclock_vcpu_time_info *src);
11
12
void pvclock_set_flags(u8 flags);
12
13
unsigned long pvclock_tsc_khz(struct pvclock_vcpu_time_info *src);
···
40
39
* Scale a 64-bit delta by scaling and multiplying by a 32-bit fraction,
41
40
* yielding a 64-bit result.
42
41
*/
43
-
static inline u64 pvclock_scale_delta(u64 delta, u32 mul_frac, int shift)
42
+
static __always_inline u64 pvclock_scale_delta(u64 delta, u32 mul_frac, int shift)
44
43
{
45
44
u64 product;
46
45
#ifdef __i386__
+4
-4
arch/x86/include/asm/special_insns.h
+4
-4
arch/x86/include/asm/special_insns.h
···
115
115
}
116
116
#endif
117
117
118
-
static inline void native_wbinvd(void)
118
+
static __always_inline void native_wbinvd(void)
119
119
{
120
120
asm volatile("wbinvd": : :"memory");
121
121
}
···
179
179
native_write_cr4(x);
180
180
}
181
181
182
-
static inline void wbinvd(void)
182
+
static __always_inline void wbinvd(void)
183
183
{
184
184
native_wbinvd();
185
185
}
···
196
196
197
197
#endif /* CONFIG_PARAVIRT_XXL */
198
198
199
-
static inline void clflush(volatile void *__p)
199
+
static __always_inline void clflush(volatile void *__p)
200
200
{
201
201
asm volatile("clflush %0" : "+m" (*(volatile char __force *)__p));
202
202
}
···
295
295
return 0;
296
296
}
297
297
298
-
static inline void tile_release(void)
298
+
static __always_inline void tile_release(void)
299
299
{
300
300
/*
301
301
* Instruction opcode for TILERELEASE; supported in binutils
+1
-1
arch/x86/include/asm/xen/hypercall.h
+1
-1
arch/x86/include/asm/xen/hypercall.h
+1
-1
arch/x86/kernel/cpu/bugs.c
+1
-1
arch/x86/kernel/cpu/bugs.c
+1
-1
arch/x86/kernel/cpu/vmware.c
+1
-1
arch/x86/kernel/cpu/vmware.c
+2
-2
arch/x86/kernel/fpu/core.c
+2
-2
arch/x86/kernel/fpu/core.c
···
853
853
* Initialize register state that may prevent from entering low-power idle.
854
854
* This function will be invoked from the cpuidle driver only when needed.
855
855
*/
856
-
void fpu_idle_fpregs(void)
856
+
noinstr void fpu_idle_fpregs(void)
857
857
{
858
858
/* Note: AMX_TILE being enabled implies XGETBV1 support */
859
859
if (cpu_feature_enabled(X86_FEATURE_AMX_TILE) &&
860
860
(xfeatures_in_use() & XFEATURE_MASK_XTILE)) {
861
861
tile_release();
862
-
fpregs_deactivate(¤t->thread.fpu);
862
+
__this_cpu_write(fpu_fpregs_owner_ctx, NULL);
863
863
}
864
864
}
+3
-3
arch/x86/kernel/kvmclock.c
+3
-3
arch/x86/kernel/kvmclock.c
···
71
71
return -ENODEV;
72
72
}
73
73
74
-
static u64 kvm_clock_read(void)
74
+
static noinstr u64 kvm_clock_read(void)
75
75
{
76
76
u64 ret;
77
77
78
78
preempt_disable_notrace();
79
-
ret = pvclock_clocksource_read(this_cpu_pvti());
79
+
ret = pvclock_clocksource_read_nowd(this_cpu_pvti());
80
80
preempt_enable_notrace();
81
81
return ret;
82
82
}
···
86
86
return kvm_clock_read();
87
87
}
88
88
89
-
static u64 kvm_sched_clock_read(void)
89
+
static noinstr u64 kvm_sched_clock_read(void)
90
90
{
91
91
return kvm_clock_read() - kvm_sched_clock_offset;
92
92
}
+12
-2
arch/x86/kernel/paravirt.c
+12
-2
arch/x86/kernel/paravirt.c
···
216
216
native_set_debugreg(regno, val);
217
217
}
218
218
219
+
noinstr void pv_native_wbinvd(void)
220
+
{
221
+
native_wbinvd();
222
+
}
223
+
219
224
static noinstr void pv_native_irq_enable(void)
220
225
{
221
226
native_irq_enable();
···
229
224
static noinstr void pv_native_irq_disable(void)
230
225
{
231
226
native_irq_disable();
227
+
}
228
+
229
+
static noinstr void pv_native_safe_halt(void)
230
+
{
231
+
native_safe_halt();
232
232
}
233
233
#endif
234
234
···
266
256
.cpu.read_cr0 = native_read_cr0,
267
257
.cpu.write_cr0 = native_write_cr0,
268
258
.cpu.write_cr4 = native_write_cr4,
269
-
.cpu.wbinvd = native_wbinvd,
259
+
.cpu.wbinvd = pv_native_wbinvd,
270
260
.cpu.read_msr = native_read_msr,
271
261
.cpu.write_msr = native_write_msr,
272
262
.cpu.read_msr_safe = native_read_msr_safe,
···
300
290
.irq.save_fl = __PV_IS_CALLEE_SAVE(native_save_fl),
301
291
.irq.irq_disable = __PV_IS_CALLEE_SAVE(pv_native_irq_disable),
302
292
.irq.irq_enable = __PV_IS_CALLEE_SAVE(pv_native_irq_enable),
303
-
.irq.safe_halt = native_safe_halt,
293
+
.irq.safe_halt = pv_native_safe_halt,
304
294
.irq.halt = native_halt,
305
295
#endif /* CONFIG_PARAVIRT_XXL */
306
296
+32
-33
arch/x86/kernel/process.c
+32
-33
arch/x86/kernel/process.c
···
24
24
#include <linux/cpuidle.h>
25
25
#include <linux/acpi.h>
26
26
#include <linux/elf-randomize.h>
27
+
#include <linux/static_call.h>
27
28
#include <trace/events/power.h>
28
29
#include <linux/hw_breakpoint.h>
29
30
#include <asm/cpu.h>
···
695
694
unsigned long boot_option_idle_override = IDLE_NO_OVERRIDE;
696
695
EXPORT_SYMBOL(boot_option_idle_override);
697
696
698
-
static void (*x86_idle)(void);
697
+
/*
698
+
* We use this if we don't have any better idle routine..
699
+
*/
700
+
void __cpuidle default_idle(void)
701
+
{
702
+
raw_safe_halt();
703
+
raw_local_irq_disable();
704
+
}
705
+
#if defined(CONFIG_APM_MODULE) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE)
706
+
EXPORT_SYMBOL(default_idle);
707
+
#endif
708
+
709
+
DEFINE_STATIC_CALL_NULL(x86_idle, default_idle);
710
+
711
+
static bool x86_idle_set(void)
712
+
{
713
+
return !!static_call_query(x86_idle);
714
+
}
699
715
700
716
#ifndef CONFIG_SMP
701
717
static inline void play_dead(void)
···
735
717
/*
736
718
* Called from the generic idle code.
737
719
*/
738
-
void arch_cpu_idle(void)
720
+
void __cpuidle arch_cpu_idle(void)
739
721
{
740
-
x86_idle();
722
+
static_call(x86_idle)();
741
723
}
742
-
743
-
/*
744
-
* We use this if we don't have any better idle routine..
745
-
*/
746
-
void __cpuidle default_idle(void)
747
-
{
748
-
raw_safe_halt();
749
-
}
750
-
#if defined(CONFIG_APM_MODULE) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE)
751
-
EXPORT_SYMBOL(default_idle);
752
-
#endif
753
724
754
725
#ifdef CONFIG_XEN
755
726
bool xen_set_default_idle(void)
756
727
{
757
-
bool ret = !!x86_idle;
728
+
bool ret = x86_idle_set();
758
729
759
-
x86_idle = default_idle;
730
+
static_call_update(x86_idle, default_idle);
760
731
761
732
return ret;
762
733
}
···
807
800
808
801
default_idle();
809
802
810
-
/*
811
-
* The switch back from broadcast mode needs to be called with
812
-
* interrupts disabled.
813
-
*/
814
-
raw_local_irq_disable();
815
803
tick_broadcast_exit();
816
-
raw_local_irq_enable();
817
804
}
818
805
819
806
/*
···
865
864
}
866
865
867
866
__monitor((void *)¤t_thread_info()->flags, 0, 0);
868
-
if (!need_resched())
867
+
if (!need_resched()) {
869
868
__sti_mwait(0, 0);
870
-
else
871
-
raw_local_irq_enable();
872
-
} else {
873
-
raw_local_irq_enable();
869
+
raw_local_irq_disable();
870
+
}
874
871
}
875
872
__current_clr_polling();
876
873
}
···
879
880
if (boot_option_idle_override == IDLE_POLL && smp_num_siblings > 1)
880
881
pr_warn_once("WARNING: polling idle and HT enabled, performance may degrade\n");
881
882
#endif
882
-
if (x86_idle || boot_option_idle_override == IDLE_POLL)
883
+
if (x86_idle_set() || boot_option_idle_override == IDLE_POLL)
883
884
return;
884
885
885
886
if (boot_cpu_has_bug(X86_BUG_AMD_E400)) {
886
887
pr_info("using AMD E400 aware idle routine\n");
887
-
x86_idle = amd_e400_idle;
888
+
static_call_update(x86_idle, amd_e400_idle);
888
889
} else if (prefer_mwait_c1_over_halt(c)) {
889
890
pr_info("using mwait in idle threads\n");
890
-
x86_idle = mwait_idle;
891
+
static_call_update(x86_idle, mwait_idle);
891
892
} else if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) {
892
893
pr_info("using TDX aware idle routine\n");
893
-
x86_idle = tdx_safe_halt;
894
+
static_call_update(x86_idle, tdx_safe_halt);
894
895
} else
895
-
x86_idle = default_idle;
896
+
static_call_update(x86_idle, default_idle);
896
897
}
897
898
898
899
void amd_e400_c1e_apic_setup(void)
···
945
946
* To continue to load the CPU idle driver, don't touch
946
947
* the boot_option_idle_override.
947
948
*/
948
-
x86_idle = default_idle;
949
+
static_call_update(x86_idle, default_idle);
949
950
boot_option_idle_override = IDLE_HALT;
950
951
} else if (!strcmp(str, "nomwait")) {
951
952
/*
+16
-6
arch/x86/kernel/pvclock.c
+16
-6
arch/x86/kernel/pvclock.c
···
64
64
return flags & valid_flags;
65
65
}
66
66
67
-
u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src)
67
+
static __always_inline
68
+
u64 __pvclock_clocksource_read(struct pvclock_vcpu_time_info *src, bool dowd)
68
69
{
69
70
unsigned version;
70
71
u64 ret;
···
78
77
flags = src->flags;
79
78
} while (pvclock_read_retry(src, version));
80
79
81
-
if (unlikely((flags & PVCLOCK_GUEST_STOPPED) != 0)) {
80
+
if (dowd && unlikely((flags & PVCLOCK_GUEST_STOPPED) != 0)) {
82
81
src->flags &= ~PVCLOCK_GUEST_STOPPED;
83
82
pvclock_touch_watchdogs();
84
83
}
···
101
100
* updating at the same time, and one of them could be slightly behind,
102
101
* making the assumption that last_value always go forward fail to hold.
103
102
*/
104
-
last = atomic64_read(&last_value);
103
+
last = arch_atomic64_read(&last_value);
105
104
do {
106
-
if (ret < last)
105
+
if (ret <= last)
107
106
return last;
108
-
last = atomic64_cmpxchg(&last_value, last, ret);
109
-
} while (unlikely(last != ret));
107
+
} while (!arch_atomic64_try_cmpxchg(&last_value, &last, ret));
110
108
111
109
return ret;
110
+
}
111
+
112
+
u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src)
113
+
{
114
+
return __pvclock_clocksource_read(src, true);
115
+
}
116
+
117
+
noinstr u64 pvclock_clocksource_read_nowd(struct pvclock_vcpu_time_info *src)
118
+
{
119
+
return __pvclock_clocksource_read(src, false);
112
120
}
113
121
114
122
void pvclock_read_wallclock(struct pvclock_wall_clock *wall_clock,
+3
-4
arch/x86/kernel/tsc.c
+3
-4
arch/x86/kernel/tsc.c
···
215
215
/*
216
216
* Scheduler clock - returns current time in nanosec units.
217
217
*/
218
-
u64 native_sched_clock(void)
218
+
noinstr u64 native_sched_clock(void)
219
219
{
220
220
if (static_branch_likely(&__use_tsc)) {
221
221
u64 tsc_now = rdtsc();
···
248
248
/* We need to define a real function for sched_clock, to override the
249
249
weak default version */
250
250
#ifdef CONFIG_PARAVIRT
251
-
unsigned long long sched_clock(void)
251
+
noinstr u64 sched_clock(void)
252
252
{
253
253
return paravirt_sched_clock();
254
254
}
···
258
258
return static_call_query(pv_sched_clock) == native_sched_clock;
259
259
}
260
260
#else
261
-
unsigned long long
262
-
sched_clock(void) __attribute__((alias("native_sched_clock")));
261
+
u64 sched_clock(void) __attribute__((alias("native_sched_clock")));
263
262
264
263
bool using_native_sched_clock(void) { return true; }
265
264
#endif
-1
arch/x86/kernel/vmlinux.lds.S
-1
arch/x86/kernel/vmlinux.lds.S
+2
-3
arch/x86/lib/memcpy_64.S
+2
-3
arch/x86/lib/memcpy_64.S
···
8
8
#include <asm/alternative.h>
9
9
#include <asm/export.h>
10
10
11
-
.pushsection .noinstr.text, "ax"
11
+
.section .noinstr.text, "ax"
12
12
13
13
/*
14
14
* We build a jump to memcpy_orig by default which gets NOPped out on
···
43
43
SYM_FUNC_END(__memcpy)
44
44
EXPORT_SYMBOL(__memcpy)
45
45
46
-
SYM_FUNC_ALIAS_WEAK(memcpy, __memcpy)
46
+
SYM_FUNC_ALIAS(memcpy, __memcpy)
47
47
EXPORT_SYMBOL(memcpy)
48
48
49
49
/*
···
184
184
RET
185
185
SYM_FUNC_END(memcpy_orig)
186
186
187
-
.popsection
+3
-1
arch/x86/lib/memmove_64.S
+3
-1
arch/x86/lib/memmove_64.S
···
13
13
14
14
#undef memmove
15
15
16
+
.section .noinstr.text, "ax"
17
+
16
18
/*
17
19
* Implement memmove(). This can handle overlap between src and dst.
18
20
*
···
215
213
SYM_FUNC_END(__memmove)
216
214
EXPORT_SYMBOL(__memmove)
217
215
218
-
SYM_FUNC_ALIAS_WEAK(memmove, __memmove)
216
+
SYM_FUNC_ALIAS(memmove, __memmove)
219
217
EXPORT_SYMBOL(memmove)
+3
-1
arch/x86/lib/memset_64.S
+3
-1
arch/x86/lib/memset_64.S
···
6
6
#include <asm/alternative.h>
7
7
#include <asm/export.h>
8
8
9
+
.section .noinstr.text, "ax"
10
+
9
11
/*
10
12
* ISO C memset - set a memory block to a byte value. This function uses fast
11
13
* string to get better performance than the original function. The code is
···
45
43
SYM_FUNC_END(__memset)
46
44
EXPORT_SYMBOL(__memset)
47
45
48
-
SYM_FUNC_ALIAS_WEAK(memset, __memset)
46
+
SYM_FUNC_ALIAS(memset, __memset)
49
47
EXPORT_SYMBOL(memset)
50
48
51
49
/*
+1
-1
arch/x86/xen/enlighten_pv.c
+1
-1
arch/x86/xen/enlighten_pv.c
+1
-1
arch/x86/xen/irq.c
+1
-1
arch/x86/xen/irq.c
+10
-2
arch/x86/xen/time.c
+10
-2
arch/x86/xen/time.c
···
60
60
return xen_clocksource_read();
61
61
}
62
62
63
-
static u64 xen_sched_clock(void)
63
+
static noinstr u64 xen_sched_clock(void)
64
64
{
65
-
return xen_clocksource_read() - xen_sched_clock_offset;
65
+
struct pvclock_vcpu_time_info *src;
66
+
u64 ret;
67
+
68
+
preempt_disable_notrace();
69
+
src = &__this_cpu_read(xen_vcpu)->time;
70
+
ret = pvclock_clocksource_read_nowd(src);
71
+
ret -= xen_sched_clock_offset;
72
+
preempt_enable_notrace();
73
+
return ret;
66
74
}
67
75
68
76
static void xen_read_wallclock(struct timespec64 *ts)
+1
arch/xtensa/kernel/process.c
+1
arch/xtensa/kernel/process.c
-1
arch/xtensa/kernel/vmlinux.lds.S
-1
arch/xtensa/kernel/vmlinux.lds.S
+17
-11
drivers/acpi/processor_idle.c
+17
-11
drivers/acpi/processor_idle.c
···
109
109
static void __cpuidle acpi_safe_halt(void)
110
110
{
111
111
if (!tif_need_resched()) {
112
-
safe_halt();
113
-
local_irq_disable();
112
+
raw_safe_halt();
113
+
raw_local_irq_disable();
114
114
}
115
115
}
116
116
···
523
523
return bm_status;
524
524
}
525
525
526
-
static void wait_for_freeze(void)
526
+
static __cpuidle void io_idle(unsigned long addr)
527
527
{
528
+
/* IO port based C-state */
529
+
inb(addr);
530
+
528
531
#ifdef CONFIG_X86
529
532
/* No delay is needed if we are in guest */
530
533
if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
···
572
569
} else if (cx->entry_method == ACPI_CSTATE_HALT) {
573
570
acpi_safe_halt();
574
571
} else {
575
-
/* IO port based C-state */
576
-
inb(cx->address);
577
-
wait_for_freeze();
572
+
io_idle(cx->address);
578
573
}
579
574
580
575
perf_lopwr_cb(false);
···
594
593
if (cx->entry_method == ACPI_CSTATE_HALT)
595
594
safe_halt();
596
595
else if (cx->entry_method == ACPI_CSTATE_SYSTEMIO) {
597
-
inb(cx->address);
598
-
wait_for_freeze();
596
+
io_idle(cx->address);
599
597
} else
600
598
return -ENODEV;
601
599
···
607
607
return 0;
608
608
}
609
609
610
-
static bool acpi_idle_fallback_to_c1(struct acpi_processor *pr)
610
+
static __always_inline bool acpi_idle_fallback_to_c1(struct acpi_processor *pr)
611
611
{
612
612
return IS_ENABLED(CONFIG_HOTPLUG_CPU) && !pr->flags.has_cst &&
613
613
!(acpi_gbl_FADT.flags & ACPI_FADT_C2_MP_SUPPORTED);
···
642
642
*/
643
643
bool dis_bm = pr->flags.bm_control;
644
644
645
+
instrumentation_begin();
646
+
645
647
/* If we can skip BM, demote to a safe state. */
646
648
if (!cx->bm_sts_skip && acpi_idle_bm_check()) {
647
649
dis_bm = false;
···
665
663
raw_spin_unlock(&c3_lock);
666
664
}
667
665
668
-
ct_idle_enter();
666
+
ct_cpuidle_enter();
669
667
670
668
acpi_idle_do_entry(cx);
671
669
672
-
ct_idle_exit();
670
+
ct_cpuidle_exit();
673
671
674
672
/* Re-enable bus master arbitration */
675
673
if (dis_bm) {
···
678
676
c3_cpu_count--;
679
677
raw_spin_unlock(&c3_lock);
680
678
}
679
+
680
+
instrumentation_end();
681
681
682
682
return index;
683
683
}
···
1223
1219
state->target_residency = lpi->min_residency;
1224
1220
if (lpi->arch_flags)
1225
1221
state->flags |= CPUIDLE_FLAG_TIMER_STOP;
1222
+
if (i != 0 && lpi->entry_method == ACPI_CSTATE_FFH)
1223
+
state->flags |= CPUIDLE_FLAG_RCU_IDLE;
1226
1224
state->enter = acpi_idle_lpi_enter;
1227
1225
drv->safe_state_index = i;
1228
1226
}
+12
-12
drivers/base/power/runtime.c
+12
-12
drivers/base/power/runtime.c
···
468
468
int (*callback)(struct device *);
469
469
int retval;
470
470
471
-
trace_rpm_idle_rcuidle(dev, rpmflags);
471
+
trace_rpm_idle(dev, rpmflags);
472
472
retval = rpm_check_suspend_allowed(dev);
473
473
if (retval < 0)
474
474
; /* Conditions are wrong. */
···
508
508
dev->power.request_pending = true;
509
509
queue_work(pm_wq, &dev->power.work);
510
510
}
511
-
trace_rpm_return_int_rcuidle(dev, _THIS_IP_, 0);
511
+
trace_rpm_return_int(dev, _THIS_IP_, 0);
512
512
return 0;
513
513
}
514
514
···
530
530
wake_up_all(&dev->power.wait_queue);
531
531
532
532
out:
533
-
trace_rpm_return_int_rcuidle(dev, _THIS_IP_, retval);
533
+
trace_rpm_return_int(dev, _THIS_IP_, retval);
534
534
return retval ? retval : rpm_suspend(dev, rpmflags | RPM_AUTO);
535
535
}
536
536
···
562
562
struct device *parent = NULL;
563
563
int retval;
564
564
565
-
trace_rpm_suspend_rcuidle(dev, rpmflags);
565
+
trace_rpm_suspend(dev, rpmflags);
566
566
567
567
repeat:
568
568
retval = rpm_check_suspend_allowed(dev);
···
713
713
}
714
714
715
715
out:
716
-
trace_rpm_return_int_rcuidle(dev, _THIS_IP_, retval);
716
+
trace_rpm_return_int(dev, _THIS_IP_, retval);
717
717
718
718
return retval;
719
719
···
765
765
struct device *parent = NULL;
766
766
int retval = 0;
767
767
768
-
trace_rpm_resume_rcuidle(dev, rpmflags);
768
+
trace_rpm_resume(dev, rpmflags);
769
769
770
770
repeat:
771
771
if (dev->power.runtime_error) {
···
935
935
spin_lock_irq(&dev->power.lock);
936
936
}
937
937
938
-
trace_rpm_return_int_rcuidle(dev, _THIS_IP_, retval);
938
+
trace_rpm_return_int(dev, _THIS_IP_, retval);
939
939
940
940
return retval;
941
941
}
···
1091
1091
if (retval < 0) {
1092
1092
return retval;
1093
1093
} else if (retval > 0) {
1094
-
trace_rpm_usage_rcuidle(dev, rpmflags);
1094
+
trace_rpm_usage(dev, rpmflags);
1095
1095
return 0;
1096
1096
}
1097
1097
}
···
1129
1129
if (retval < 0) {
1130
1130
return retval;
1131
1131
} else if (retval > 0) {
1132
-
trace_rpm_usage_rcuidle(dev, rpmflags);
1132
+
trace_rpm_usage(dev, rpmflags);
1133
1133
return 0;
1134
1134
}
1135
1135
}
···
1212
1212
} else {
1213
1213
retval = atomic_inc_not_zero(&dev->power.usage_count);
1214
1214
}
1215
-
trace_rpm_usage_rcuidle(dev, 0);
1215
+
trace_rpm_usage(dev, 0);
1216
1216
spin_unlock_irqrestore(&dev->power.lock, flags);
1217
1217
1218
1218
return retval;
···
1576
1576
if (ret == 0)
1577
1577
rpm_idle(dev, RPM_AUTO | RPM_ASYNC);
1578
1578
else if (ret > 0)
1579
-
trace_rpm_usage_rcuidle(dev, RPM_AUTO | RPM_ASYNC);
1579
+
trace_rpm_usage(dev, RPM_AUTO | RPM_ASYNC);
1580
1580
1581
1581
out:
1582
1582
spin_unlock_irq(&dev->power.lock);
···
1646
1646
atomic_inc(&dev->power.usage_count);
1647
1647
rpm_resume(dev, 0);
1648
1648
} else {
1649
-
trace_rpm_usage_rcuidle(dev, 0);
1649
+
trace_rpm_usage(dev, 0);
1650
1650
}
1651
1651
}
1652
1652
+4
-4
drivers/clk/clk.c
+4
-4
drivers/clk/clk.c
···
1055
1055
if (--core->enable_count > 0)
1056
1056
return;
1057
1057
1058
-
trace_clk_disable_rcuidle(core);
1058
+
trace_clk_disable(core);
1059
1059
1060
1060
if (core->ops->disable)
1061
1061
core->ops->disable(core->hw);
1062
1062
1063
-
trace_clk_disable_complete_rcuidle(core);
1063
+
trace_clk_disable_complete(core);
1064
1064
1065
1065
clk_core_disable(core->parent);
1066
1066
}
···
1114
1114
if (ret)
1115
1115
return ret;
1116
1116
1117
-
trace_clk_enable_rcuidle(core);
1117
+
trace_clk_enable(core);
1118
1118
1119
1119
if (core->ops->enable)
1120
1120
ret = core->ops->enable(core->hw);
1121
1121
1122
-
trace_clk_enable_complete_rcuidle(core);
1122
+
trace_clk_enable_complete(core);
1123
1123
1124
1124
if (ret) {
1125
1125
clk_core_disable(core->parent);
+2
-2
drivers/cpuidle/cpuidle-arm.c
+2
-2
drivers/cpuidle/cpuidle-arm.c
···
31
31
* Called from the CPUidle framework to program the device to the
32
32
* specified target state selected by the governor.
33
33
*/
34
-
static int arm_enter_idle_state(struct cpuidle_device *dev,
35
-
struct cpuidle_driver *drv, int idx)
34
+
static __cpuidle int arm_enter_idle_state(struct cpuidle_device *dev,
35
+
struct cpuidle_driver *drv, int idx)
36
36
{
37
37
/*
38
38
* Pass idle state index to arm_cpuidle_suspend which in turn
+8
-4
drivers/cpuidle/cpuidle-big_little.c
+8
-4
drivers/cpuidle/cpuidle-big_little.c
···
64
64
.enter = bl_enter_powerdown,
65
65
.exit_latency = 700,
66
66
.target_residency = 2500,
67
-
.flags = CPUIDLE_FLAG_TIMER_STOP,
67
+
.flags = CPUIDLE_FLAG_TIMER_STOP |
68
+
CPUIDLE_FLAG_RCU_IDLE,
68
69
.name = "C1",
69
70
.desc = "ARM little-cluster power down",
70
71
},
···
86
85
.enter = bl_enter_powerdown,
87
86
.exit_latency = 500,
88
87
.target_residency = 2000,
89
-
.flags = CPUIDLE_FLAG_TIMER_STOP,
88
+
.flags = CPUIDLE_FLAG_TIMER_STOP |
89
+
CPUIDLE_FLAG_RCU_IDLE,
90
90
.name = "C1",
91
91
.desc = "ARM big-cluster power down",
92
92
},
···
122
120
* Called from the CPUidle framework to program the device to the
123
121
* specified target state selected by the governor.
124
122
*/
125
-
static int bl_enter_powerdown(struct cpuidle_device *dev,
126
-
struct cpuidle_driver *drv, int idx)
123
+
static __cpuidle int bl_enter_powerdown(struct cpuidle_device *dev,
124
+
struct cpuidle_driver *drv, int idx)
127
125
{
128
126
cpu_pm_enter();
127
+
ct_cpuidle_enter();
129
128
130
129
cpu_suspend(0, bl_powerdown_finisher);
131
130
132
131
/* signals the MCPM core that CPU is out of low power state */
133
132
mcpm_cpu_powered_up();
133
+
ct_cpuidle_exit();
134
134
135
135
cpu_pm_exit();
136
136
+10
-5
drivers/cpuidle/cpuidle-mvebu-v7.c
+10
-5
drivers/cpuidle/cpuidle-mvebu-v7.c
···
25
25
26
26
static int (*mvebu_v7_cpu_suspend)(int);
27
27
28
-
static int mvebu_v7_enter_idle(struct cpuidle_device *dev,
29
-
struct cpuidle_driver *drv,
30
-
int index)
28
+
static __cpuidle int mvebu_v7_enter_idle(struct cpuidle_device *dev,
29
+
struct cpuidle_driver *drv,
30
+
int index)
31
31
{
32
32
int ret;
33
33
bool deepidle = false;
···
36
36
if (drv->states[index].flags & MVEBU_V7_FLAG_DEEP_IDLE)
37
37
deepidle = true;
38
38
39
+
ct_cpuidle_enter();
39
40
ret = mvebu_v7_cpu_suspend(deepidle);
41
+
ct_cpuidle_exit();
42
+
40
43
cpu_pm_exit();
41
44
42
45
if (ret)
···
56
53
.exit_latency = 100,
57
54
.power_usage = 50,
58
55
.target_residency = 1000,
56
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
59
57
.name = "MV CPU IDLE",
60
58
.desc = "CPU power down",
61
59
},
···
65
61
.exit_latency = 1000,
66
62
.power_usage = 5,
67
63
.target_residency = 10000,
68
-
.flags = MVEBU_V7_FLAG_DEEP_IDLE,
64
+
.flags = MVEBU_V7_FLAG_DEEP_IDLE | CPUIDLE_FLAG_RCU_IDLE,
69
65
.name = "MV CPU DEEP IDLE",
70
66
.desc = "CPU and L2 Fabric power down",
71
67
},
···
80
76
.exit_latency = 100,
81
77
.power_usage = 5,
82
78
.target_residency = 1000,
83
-
.flags = MVEBU_V7_FLAG_DEEP_IDLE,
79
+
.flags = MVEBU_V7_FLAG_DEEP_IDLE | CPUIDLE_FLAG_RCU_IDLE,
84
80
.name = "Deep Idle",
85
81
.desc = "CPU and L2 Fabric power down",
86
82
},
···
95
91
.exit_latency = 10,
96
92
.power_usage = 5,
97
93
.target_residency = 100,
94
+
.flags = CPUIDLE_FLAG_RCU_IDLE,
98
95
.name = "Idle",
99
96
.desc = "CPU and SCU power down",
100
97
},
+7
-15
drivers/cpuidle/cpuidle-psci.c
+7
-15
drivers/cpuidle/cpuidle-psci.c
···
49
49
return __this_cpu_read(domain_state);
50
50
}
51
51
52
-
static inline int psci_enter_state(int idx, u32 state)
53
-
{
54
-
return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter, idx, state);
55
-
}
56
-
57
-
static int __psci_enter_domain_idle_state(struct cpuidle_device *dev,
58
-
struct cpuidle_driver *drv, int idx,
59
-
bool s2idle)
52
+
static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev,
53
+
struct cpuidle_driver *drv, int idx,
54
+
bool s2idle)
60
55
{
61
56
struct psci_cpuidle_data *data = this_cpu_ptr(&psci_cpuidle_data);
62
57
u32 *states = data->psci_states;
···
64
69
return -1;
65
70
66
71
/* Do runtime PM to manage a hierarchical CPU toplogy. */
67
-
ct_irq_enter_irqson();
68
72
if (s2idle)
69
73
dev_pm_genpd_suspend(pd_dev);
70
74
else
71
75
pm_runtime_put_sync_suspend(pd_dev);
72
-
ct_irq_exit_irqson();
73
76
74
77
state = psci_get_domain_state();
75
78
if (!state)
···
75
82
76
83
ret = psci_cpu_suspend_enter(state) ? -1 : idx;
77
84
78
-
ct_irq_enter_irqson();
79
85
if (s2idle)
80
86
dev_pm_genpd_resume(pd_dev);
81
87
else
82
88
pm_runtime_get_sync(pd_dev);
83
-
ct_irq_exit_irqson();
84
89
85
90
cpu_pm_exit();
86
91
···
183
192
pr_warn("Failed %d while setup cpuhp state\n", err);
184
193
}
185
194
186
-
static int psci_enter_idle_state(struct cpuidle_device *dev,
187
-
struct cpuidle_driver *drv, int idx)
195
+
static __cpuidle int psci_enter_idle_state(struct cpuidle_device *dev,
196
+
struct cpuidle_driver *drv, int idx)
188
197
{
189
198
u32 *state = __this_cpu_read(psci_cpuidle_data.psci_states);
190
199
191
-
return psci_enter_state(idx, state[idx]);
200
+
return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter, idx, state[idx]);
192
201
}
193
202
194
203
static const struct of_device_id psci_idle_state_match[] = {
···
231
240
* of a shared state for the domain, assumes the domain states are all
232
241
* deeper states.
233
242
*/
243
+
drv->states[state_count - 1].flags |= CPUIDLE_FLAG_RCU_IDLE;
234
244
drv->states[state_count - 1].enter = psci_enter_domain_idle_state;
235
245
drv->states[state_count - 1].enter_s2idle = psci_enter_s2idle_domain_idle_state;
236
246
psci_cpuidle_use_cpuhp = true;
+2
-2
drivers/cpuidle/cpuidle-qcom-spm.c
+2
-2
drivers/cpuidle/cpuidle-qcom-spm.c
···
58
58
return ret;
59
59
}
60
60
61
-
static int spm_enter_idle_state(struct cpuidle_device *dev,
62
-
struct cpuidle_driver *drv, int idx)
61
+
static __cpuidle int spm_enter_idle_state(struct cpuidle_device *dev,
62
+
struct cpuidle_driver *drv, int idx)
63
63
{
64
64
struct cpuidle_qcom_spm_data *data = container_of(drv, struct cpuidle_qcom_spm_data,
65
65
cpuidle_driver);
+10
-9
drivers/cpuidle/cpuidle-riscv-sbi.c
+10
-9
drivers/cpuidle/cpuidle-riscv-sbi.c
···
93
93
return sbi_suspend_finisher(state, 0, 0);
94
94
}
95
95
96
-
static int sbi_cpuidle_enter_state(struct cpuidle_device *dev,
97
-
struct cpuidle_driver *drv, int idx)
96
+
static __cpuidle int sbi_cpuidle_enter_state(struct cpuidle_device *dev,
97
+
struct cpuidle_driver *drv, int idx)
98
98
{
99
99
u32 *states = __this_cpu_read(sbi_cpuidle_data.states);
100
100
u32 state = states[idx];
···
106
106
idx, state);
107
107
}
108
108
109
-
static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev,
110
-
struct cpuidle_driver *drv, int idx,
111
-
bool s2idle)
109
+
static __cpuidle int __sbi_enter_domain_idle_state(struct cpuidle_device *dev,
110
+
struct cpuidle_driver *drv, int idx,
111
+
bool s2idle)
112
112
{
113
113
struct sbi_cpuidle_data *data = this_cpu_ptr(&sbi_cpuidle_data);
114
114
u32 *states = data->states;
···
121
121
return -1;
122
122
123
123
/* Do runtime PM to manage a hierarchical CPU toplogy. */
124
-
ct_irq_enter_irqson();
125
124
if (s2idle)
126
125
dev_pm_genpd_suspend(pd_dev);
127
126
else
128
127
pm_runtime_put_sync_suspend(pd_dev);
129
-
ct_irq_exit_irqson();
128
+
129
+
ct_cpuidle_enter();
130
130
131
131
if (sbi_is_domain_state_available())
132
132
state = sbi_get_domain_state();
···
135
135
136
136
ret = sbi_suspend(state) ? -1 : idx;
137
137
138
-
ct_irq_enter_irqson();
138
+
ct_cpuidle_exit();
139
+
139
140
if (s2idle)
140
141
dev_pm_genpd_resume(pd_dev);
141
142
else
142
143
pm_runtime_get_sync(pd_dev);
143
-
ct_irq_exit_irqson();
144
144
145
145
cpu_pm_exit();
146
146
···
251
251
* of a shared state for the domain, assumes the domain states are all
252
252
* deeper states.
253
253
*/
254
+
drv->states[state_count - 1].flags |= CPUIDLE_FLAG_RCU_IDLE;
254
255
drv->states[state_count - 1].enter = sbi_enter_domain_idle_state;
255
256
drv->states[state_count - 1].enter_s2idle =
256
257
sbi_enter_s2idle_domain_idle_state;
+21
-10
drivers/cpuidle/cpuidle-tegra.c
+21
-10
drivers/cpuidle/cpuidle-tegra.c
···
160
160
return 0;
161
161
}
162
162
163
-
static int tegra_cpuidle_state_enter(struct cpuidle_device *dev,
164
-
int index, unsigned int cpu)
163
+
static __cpuidle int tegra_cpuidle_state_enter(struct cpuidle_device *dev,
164
+
int index, unsigned int cpu)
165
165
{
166
166
int err;
167
167
···
180
180
}
181
181
182
182
local_fiq_disable();
183
-
RCU_NONIDLE(tegra_pm_set_cpu_in_lp2());
183
+
tegra_pm_set_cpu_in_lp2();
184
184
cpu_pm_enter();
185
+
186
+
ct_cpuidle_enter();
185
187
186
188
switch (index) {
187
189
case TEGRA_C7:
···
199
197
break;
200
198
}
201
199
200
+
ct_cpuidle_exit();
201
+
202
202
cpu_pm_exit();
203
-
RCU_NONIDLE(tegra_pm_clear_cpu_in_lp2());
203
+
tegra_pm_clear_cpu_in_lp2();
204
204
local_fiq_enable();
205
205
206
206
return err ?: index;
···
226
222
return index;
227
223
}
228
224
229
-
static int tegra_cpuidle_enter(struct cpuidle_device *dev,
230
-
struct cpuidle_driver *drv,
231
-
int index)
225
+
static __cpuidle int tegra_cpuidle_enter(struct cpuidle_device *dev,
226
+
struct cpuidle_driver *drv,
227
+
int index)
232
228
{
229
+
bool do_rcu = drv->states[index].flags & CPUIDLE_FLAG_RCU_IDLE;
233
230
unsigned int cpu = cpu_logical_map(dev->cpu);
234
231
int ret;
235
232
···
238
233
if (dev->states_usage[index].disable)
239
234
return -1;
240
235
241
-
if (index == TEGRA_C1)
236
+
if (index == TEGRA_C1) {
237
+
if (do_rcu)
238
+
ct_cpuidle_enter();
242
239
ret = arm_cpuidle_simple_enter(dev, drv, index);
243
-
else
240
+
if (do_rcu)
241
+
ct_cpuidle_exit();
242
+
} else
244
243
ret = tegra_cpuidle_state_enter(dev, index, cpu);
245
244
246
245
if (ret < 0) {
···
294
285
.exit_latency = 2000,
295
286
.target_residency = 2200,
296
287
.power_usage = 100,
297
-
.flags = CPUIDLE_FLAG_TIMER_STOP,
288
+
.flags = CPUIDLE_FLAG_TIMER_STOP |
289
+
CPUIDLE_FLAG_RCU_IDLE,
298
290
.name = "C7",
299
291
.desc = "CPU core powered off",
300
292
},
···
305
295
.target_residency = 10000,
306
296
.power_usage = 0,
307
297
.flags = CPUIDLE_FLAG_TIMER_STOP |
298
+
CPUIDLE_FLAG_RCU_IDLE |
308
299
CPUIDLE_FLAG_COUPLED,
309
300
.name = "CC6",
310
301
.desc = "CPU cluster powered off",
+52
-20
drivers/cpuidle/cpuidle.c
+52
-20
drivers/cpuidle/cpuidle.c
···
14
14
#include <linux/mutex.h>
15
15
#include <linux/sched.h>
16
16
#include <linux/sched/clock.h>
17
+
#include <linux/sched/idle.h>
17
18
#include <linux/notifier.h>
18
19
#include <linux/pm_qos.h>
19
20
#include <linux/cpu.h>
···
137
136
}
138
137
139
138
#ifdef CONFIG_SUSPEND
140
-
static void enter_s2idle_proper(struct cpuidle_driver *drv,
141
-
struct cpuidle_device *dev, int index)
139
+
static noinstr void enter_s2idle_proper(struct cpuidle_driver *drv,
140
+
struct cpuidle_device *dev, int index)
142
141
{
143
-
ktime_t time_start, time_end;
144
142
struct cpuidle_state *target_state = &drv->states[index];
143
+
ktime_t time_start, time_end;
144
+
145
+
instrumentation_begin();
145
146
146
147
time_start = ns_to_ktime(local_clock());
147
148
···
154
151
* suspended is generally unsafe.
155
152
*/
156
153
stop_critical_timings();
157
-
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
158
-
ct_idle_enter();
154
+
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) {
155
+
ct_cpuidle_enter();
156
+
/* Annotate away the indirect call */
157
+
instrumentation_begin();
158
+
}
159
159
target_state->enter_s2idle(dev, drv, index);
160
160
if (WARN_ON_ONCE(!irqs_disabled()))
161
-
local_irq_disable();
162
-
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
163
-
ct_idle_exit();
161
+
raw_local_irq_disable();
162
+
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) {
163
+
instrumentation_end();
164
+
ct_cpuidle_exit();
165
+
}
164
166
tick_unfreeze();
165
167
start_critical_timings();
166
168
···
173
165
174
166
dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
175
167
dev->states_usage[index].s2idle_usage++;
168
+
instrumentation_end();
176
169
}
177
170
178
171
/**
···
208
199
* @drv: cpuidle driver for this cpu
209
200
* @index: index into the states table in @drv of the state to enter
210
201
*/
211
-
int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
212
-
int index)
202
+
noinstr int cpuidle_enter_state(struct cpuidle_device *dev,
203
+
struct cpuidle_driver *drv,
204
+
int index)
213
205
{
214
206
int entered_state;
215
207
216
208
struct cpuidle_state *target_state = &drv->states[index];
217
209
bool broadcast = !!(target_state->flags & CPUIDLE_FLAG_TIMER_STOP);
218
210
ktime_t time_start, time_end;
211
+
212
+
instrumentation_begin();
219
213
220
214
/*
221
215
* Tell the time framework to switch to a broadcast timer because our
···
246
234
time_start = ns_to_ktime(local_clock());
247
235
248
236
stop_critical_timings();
249
-
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
250
-
ct_idle_enter();
237
+
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) {
238
+
ct_cpuidle_enter();
239
+
/* Annotate away the indirect call */
240
+
instrumentation_begin();
241
+
}
242
+
243
+
/*
244
+
* NOTE!!
245
+
*
246
+
* For cpuidle_state::enter() methods that do *NOT* set
247
+
* CPUIDLE_FLAG_RCU_IDLE RCU will be disabled here and these functions
248
+
* must be marked either noinstr or __cpuidle.
249
+
*
250
+
* For cpuidle_state::enter() methods that *DO* set
251
+
* CPUIDLE_FLAG_RCU_IDLE this isn't required, but they must mark the
252
+
* function calling ct_cpuidle_enter() as noinstr/__cpuidle and all
253
+
* functions called within the RCU-idle region.
254
+
*/
251
255
entered_state = target_state->enter(dev, drv, index);
252
-
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
253
-
ct_idle_exit();
256
+
257
+
if (WARN_ONCE(!irqs_disabled(), "%ps leaked IRQ state", target_state->enter))
258
+
raw_local_irq_disable();
259
+
260
+
if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) {
261
+
instrumentation_end();
262
+
ct_cpuidle_exit();
263
+
}
254
264
start_critical_timings();
255
265
256
266
sched_clock_idle_wakeup_event();
···
282
248
/* The cpu is no longer idle or about to enter idle. */
283
249
sched_idle_set_state(NULL);
284
250
285
-
if (broadcast) {
286
-
if (WARN_ON_ONCE(!irqs_disabled()))
287
-
local_irq_disable();
288
-
251
+
if (broadcast)
289
252
tick_broadcast_exit();
290
-
}
291
253
292
254
if (!cpuidle_state_is_coupled(drv, index))
293
255
local_irq_enable();
···
334
304
dev->last_residency_ns = 0;
335
305
dev->states_usage[index].rejected++;
336
306
}
307
+
308
+
instrumentation_end();
337
309
338
310
return entered_state;
339
311
}
···
426
394
* @dev: the cpuidle device
427
395
*
428
396
*/
429
-
u64 cpuidle_poll_time(struct cpuidle_driver *drv,
397
+
__cpuidle u64 cpuidle_poll_time(struct cpuidle_driver *drv,
430
398
struct cpuidle_device *dev)
431
399
{
432
400
int i;
+1
-1
drivers/cpuidle/dt_idle_states.c
+1
-1
drivers/cpuidle/dt_idle_states.c
+6
-2
drivers/cpuidle/poll_state.c
+6
-2
drivers/cpuidle/poll_state.c
···
13
13
static int __cpuidle poll_idle(struct cpuidle_device *dev,
14
14
struct cpuidle_driver *drv, int index)
15
15
{
16
-
u64 time_start = local_clock();
16
+
u64 time_start;
17
+
18
+
time_start = local_clock();
17
19
18
20
dev->poll_time_limit = false;
19
21
20
-
local_irq_enable();
22
+
raw_local_irq_enable();
21
23
if (!current_set_polling_and_test()) {
22
24
unsigned int loop_count = 0;
23
25
u64 limit;
···
38
36
}
39
37
}
40
38
}
39
+
raw_local_irq_disable();
40
+
41
41
current_clr_polling();
42
42
43
43
return index;
+30
-12
drivers/firmware/psci/psci.c
+30
-12
drivers/firmware/psci/psci.c
···
108
108
return !(state & ~valid_mask);
109
109
}
110
110
111
-
static unsigned long __invoke_psci_fn_hvc(unsigned long function_id,
112
-
unsigned long arg0, unsigned long arg1,
113
-
unsigned long arg2)
111
+
static __always_inline unsigned long
112
+
__invoke_psci_fn_hvc(unsigned long function_id,
113
+
unsigned long arg0, unsigned long arg1,
114
+
unsigned long arg2)
114
115
{
115
116
struct arm_smccc_res res;
116
117
···
119
118
return res.a0;
120
119
}
121
120
122
-
static unsigned long __invoke_psci_fn_smc(unsigned long function_id,
123
-
unsigned long arg0, unsigned long arg1,
124
-
unsigned long arg2)
121
+
static __always_inline unsigned long
122
+
__invoke_psci_fn_smc(unsigned long function_id,
123
+
unsigned long arg0, unsigned long arg1,
124
+
unsigned long arg2)
125
125
{
126
126
struct arm_smccc_res res;
127
127
···
130
128
return res.a0;
131
129
}
132
130
133
-
static int psci_to_linux_errno(int errno)
131
+
static __always_inline int psci_to_linux_errno(int errno)
134
132
{
135
133
switch (errno) {
136
134
case PSCI_RET_SUCCESS:
···
171
169
return psci_to_linux_errno(err);
172
170
}
173
171
174
-
static int __psci_cpu_suspend(u32 fn, u32 state, unsigned long entry_point)
172
+
static __always_inline int
173
+
__psci_cpu_suspend(u32 fn, u32 state, unsigned long entry_point)
175
174
{
176
175
int err;
177
176
···
180
177
return psci_to_linux_errno(err);
181
178
}
182
179
183
-
static int psci_0_1_cpu_suspend(u32 state, unsigned long entry_point)
180
+
static __always_inline int
181
+
psci_0_1_cpu_suspend(u32 state, unsigned long entry_point)
184
182
{
185
183
return __psci_cpu_suspend(psci_0_1_function_ids.cpu_suspend,
186
184
state, entry_point);
187
185
}
188
186
189
-
static int psci_0_2_cpu_suspend(u32 state, unsigned long entry_point)
187
+
static __always_inline int
188
+
psci_0_2_cpu_suspend(u32 state, unsigned long entry_point)
190
189
{
191
190
return __psci_cpu_suspend(PSCI_FN_NATIVE(0_2, CPU_SUSPEND),
192
191
state, entry_point);
···
455
450
#endif
456
451
457
452
#ifdef CONFIG_CPU_IDLE
458
-
static int psci_suspend_finisher(unsigned long state)
453
+
static noinstr int psci_suspend_finisher(unsigned long state)
459
454
{
460
455
u32 power_state = state;
461
-
phys_addr_t pa_cpu_resume = __pa_symbol(cpu_resume);
456
+
phys_addr_t pa_cpu_resume;
457
+
458
+
pa_cpu_resume = __pa_symbol_nodebug((unsigned long)cpu_resume);
462
459
463
460
return psci_ops.cpu_suspend(power_state, pa_cpu_resume);
464
461
}
···
472
465
if (!psci_power_state_loses_context(state)) {
473
466
struct arm_cpuidle_irq_context context;
474
467
468
+
ct_cpuidle_enter();
475
469
arm_cpuidle_save_irq_context(&context);
476
470
ret = psci_ops.cpu_suspend(state, 0);
477
471
arm_cpuidle_restore_irq_context(&context);
472
+
ct_cpuidle_exit();
478
473
} else {
474
+
/*
475
+
* ARM64 cpu_suspend() wants to do ct_cpuidle_*() itself.
476
+
*/
477
+
if (!IS_ENABLED(CONFIG_ARM64))
478
+
ct_cpuidle_enter();
479
+
479
480
ret = cpu_suspend(state, psci_suspend_finisher);
481
+
482
+
if (!IS_ENABLED(CONFIG_ARM64))
483
+
ct_cpuidle_exit();
480
484
}
481
485
482
486
return ret;
+9
-10
drivers/idle/intel_idle.c
+9
-10
drivers/idle/intel_idle.c
···
168
168
169
169
raw_local_irq_enable();
170
170
ret = __intel_idle(dev, drv, index);
171
-
172
-
/*
173
-
* The lockdep hardirqs state may be changed to 'on' with timer
174
-
* tick interrupt followed by __do_softirq(). Use local_irq_disable()
175
-
* to keep the hardirqs state correct.
176
-
*/
177
-
local_irq_disable();
171
+
raw_local_irq_disable();
178
172
179
173
return ret;
180
174
}
···
181
187
int ret;
182
188
183
189
if (smt_active)
184
-
wrmsrl(MSR_IA32_SPEC_CTRL, 0);
190
+
native_wrmsrl(MSR_IA32_SPEC_CTRL, 0);
185
191
186
192
ret = __intel_idle(dev, drv, index);
187
193
188
194
if (smt_active)
189
-
wrmsrl(MSR_IA32_SPEC_CTRL, spec_ctrl);
195
+
native_wrmsrl(MSR_IA32_SPEC_CTRL, spec_ctrl);
190
196
191
197
return ret;
192
198
}
···
1837
1843
return true;
1838
1844
}
1839
1845
1846
+
static bool force_irq_on __read_mostly;
1847
+
module_param(force_irq_on, bool, 0444);
1848
+
1840
1849
static void __init intel_idle_init_cstates_icpu(struct cpuidle_driver *drv)
1841
1850
{
1842
1851
int cstate;
···
1892
1895
/* Structure copy. */
1893
1896
drv->states[drv->state_count] = cpuidle_state_table[cstate];
1894
1897
1895
-
if (cpuidle_state_table[cstate].flags & CPUIDLE_FLAG_IRQ_ENABLE)
1898
+
if ((cpuidle_state_table[cstate].flags & CPUIDLE_FLAG_IRQ_ENABLE) || force_irq_on) {
1899
+
printk("intel_idle: forced intel_idle_irq for state %d\n", cstate);
1896
1900
drv->states[drv->state_count].enter = intel_idle_irq;
1901
+
}
1897
1902
1898
1903
if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) &&
1899
1904
cpuidle_state_table[cstate].flags & CPUIDLE_FLAG_IBRS) {
+1
-10
drivers/perf/arm_pmu.c
+1
-10
drivers/perf/arm_pmu.c
···
752
752
case CPU_PM_ENTER_FAILED:
753
753
/*
754
754
* Restore and enable the counter.
755
-
* armpmu_start() indirectly calls
756
-
*
757
-
* perf_event_update_userpage()
758
-
*
759
-
* that requires RCU read locking to be functional,
760
-
* wrap the call within RCU_NONIDLE to make the
761
-
* RCU subsystem aware this cpu is not idle from
762
-
* an RCU perspective for the armpmu_start() call
763
-
* duration.
764
755
*/
765
-
RCU_NONIDLE(armpmu_start(event, PERF_EF_RELOAD));
756
+
armpmu_start(event, PERF_EF_RELOAD);
766
757
break;
767
758
default:
768
759
break;
+1
-7
drivers/perf/riscv_pmu_sbi.c
+1
-7
drivers/perf/riscv_pmu_sbi.c
···
771
771
case CPU_PM_ENTER_FAILED:
772
772
/*
773
773
* Restore and enable the counter.
774
-
*
775
-
* Requires RCU read locking to be functional,
776
-
* wrap the call within RCU_NONIDLE to make the
777
-
* RCU subsystem aware this cpu is not idle from
778
-
* an RCU perspective for the riscv_pmu_start() call
779
-
* duration.
780
774
*/
781
-
RCU_NONIDLE(riscv_pmu_start(event, PERF_EF_RELOAD));
775
+
riscv_pmu_start(event, PERF_EF_RELOAD);
782
776
break;
783
777
default:
784
778
break;
+5
fs/binfmt_elf.c
+5
fs/binfmt_elf.c
···
46
46
#include <linux/cred.h>
47
47
#include <linux/dax.h>
48
48
#include <linux/uaccess.h>
49
+
#include <linux/rseq.h>
49
50
#include <asm/param.h>
50
51
#include <asm/page.h>
51
52
···
289
288
if (bprm->have_execfd) {
290
289
NEW_AUX_ENT(AT_EXECFD, bprm->execfd);
291
290
}
291
+
#ifdef CONFIG_RSEQ
292
+
NEW_AUX_ENT(AT_RSEQ_FEATURE_SIZE, offsetof(struct rseq, end));
293
+
NEW_AUX_ENT(AT_RSEQ_ALIGN, __alignof__(struct rseq));
294
+
#endif
292
295
#undef NEW_AUX_ENT
293
296
/* AT_NULL is zero; clear the rest too */
294
297
memset(elf_info, 0, (char *)mm->saved_auxv +
+4
fs/exec.c
+4
fs/exec.c
···
1010
1010
active_mm = tsk->active_mm;
1011
1011
tsk->active_mm = mm;
1012
1012
tsk->mm = mm;
1013
+
mm_init_cid(mm);
1013
1014
/*
1014
1015
* This prevents preemption while active_mm is being loaded and
1015
1016
* it and mm are being updated, which could cause problems for
···
1823
1822
*/
1824
1823
check_unsafe_exec(bprm);
1825
1824
current->in_execve = 1;
1825
+
sched_mm_cid_before_execve(current);
1826
1826
1827
1827
file = do_open_execat(fd, filename, flags);
1828
1828
retval = PTR_ERR(file);
···
1854
1852
if (retval < 0)
1855
1853
goto out;
1856
1854
1855
+
sched_mm_cid_after_execve(current);
1857
1856
/* execve succeeded */
1858
1857
current->fs->in_exec = 0;
1859
1858
current->in_execve = 0;
···
1874
1871
force_fatal_sig(SIGSEGV);
1875
1872
1876
1873
out_unmark:
1874
+
sched_mm_cid_after_execve(current);
1877
1875
current->fs->in_exec = 0;
1878
1876
current->in_execve = 0;
1879
1877
+3
-6
include/asm-generic/vmlinux.lds.h
+3
-6
include/asm-generic/vmlinux.lds.h
···
558
558
ALIGN_FUNCTION(); \
559
559
__noinstr_text_start = .; \
560
560
*(.noinstr.text) \
561
+
__cpuidle_text_start = .; \
562
+
*(.cpuidle.text) \
563
+
__cpuidle_text_end = .; \
561
564
__noinstr_text_end = .;
562
565
563
566
/*
···
600
597
__lock_text_start = .; \
601
598
*(.spinlock.text) \
602
599
__lock_text_end = .;
603
-
604
-
#define CPUIDLE_TEXT \
605
-
ALIGN_FUNCTION(); \
606
-
__cpuidle_text_start = .; \
607
-
*(.cpuidle.text) \
608
-
__cpuidle_text_end = .;
609
600
610
601
#define KPROBES_TEXT \
611
602
ALIGN_FUNCTION(); \
+1
-1
include/linux/auxvec.h
+1
-1
include/linux/auxvec.h
···
4
4
5
5
#include <uapi/linux/auxvec.h>
6
6
7
-
#define AT_VECTOR_SIZE_BASE 20 /* NEW_AUX_ENT entries in auxiliary table */
7
+
#define AT_VECTOR_SIZE_BASE 22 /* NEW_AUX_ENT entries in auxiliary table */
8
8
/* number of "#define AT_.*" above, minus {AT_NULL, AT_IGNORE, AT_NOTELF} */
9
9
#endif /* _LINUX_AUXVEC_H */
+2
-2
include/linux/clockchips.h
+2
-2
include/linux/clockchips.h
···
211
211
extern void tick_setup_hrtimer_broadcast(void);
212
212
extern int tick_check_broadcast_expired(void);
213
213
# else
214
-
static inline int tick_check_broadcast_expired(void) { return 0; }
214
+
static __always_inline int tick_check_broadcast_expired(void) { return 0; }
215
215
static inline void tick_setup_hrtimer_broadcast(void) { }
216
216
# endif
217
217
···
219
219
220
220
static inline void clockevents_suspend(void) { }
221
221
static inline void clockevents_resume(void) { }
222
-
static inline int tick_check_broadcast_expired(void) { return 0; }
222
+
static __always_inline int tick_check_broadcast_expired(void) { return 0; }
223
223
static inline void tick_setup_hrtimer_broadcast(void) { }
224
224
225
225
#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
+16
-2
include/linux/compiler_types.h
+16
-2
include/linux/compiler_types.h
···
232
232
#endif
233
233
234
234
/* Section for code which can't be instrumented at all */
235
-
#define noinstr \
236
-
noinline notrace __attribute((__section__(".noinstr.text"))) \
235
+
#define __noinstr_section(section) \
236
+
noinline notrace __attribute((__section__(section))) \
237
237
__no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \
238
238
__no_sanitize_memory
239
+
240
+
#define noinstr __noinstr_section(".noinstr.text")
241
+
242
+
/*
243
+
* The __cpuidle section is used twofold:
244
+
*
245
+
* 1) the original use -- identifying if a CPU is 'stuck' in idle state based
246
+
* on it's instruction pointer. See cpu_in_idle().
247
+
*
248
+
* 2) supressing instrumentation around where cpuidle disables RCU; where the
249
+
* function isn't strictly required for #1, this is interchangeable with
250
+
* noinstr.
251
+
*/
252
+
#define __cpuidle __noinstr_section(".cpuidle.text")
239
253
240
254
#endif /* __KERNEL__ */
241
255
+27
include/linux/context_tracking.h
+27
include/linux/context_tracking.h
···
130
130
return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
131
131
}
132
132
133
+
static __always_inline bool warn_rcu_enter(void)
134
+
{
135
+
bool ret = false;
136
+
137
+
/*
138
+
* Horrible hack to shut up recursive RCU isn't watching fail since
139
+
* lots of the actual reporting also relies on RCU.
140
+
*/
141
+
preempt_disable_notrace();
142
+
if (rcu_dynticks_curr_cpu_in_eqs()) {
143
+
ret = true;
144
+
ct_state_inc(RCU_DYNTICKS_IDX);
145
+
}
146
+
147
+
return ret;
148
+
}
149
+
150
+
static __always_inline void warn_rcu_exit(bool rcu)
151
+
{
152
+
if (rcu)
153
+
ct_state_inc(RCU_DYNTICKS_IDX);
154
+
preempt_enable_notrace();
155
+
}
156
+
133
157
#else
134
158
static inline void ct_idle_enter(void) { }
135
159
static inline void ct_idle_exit(void) { }
160
+
161
+
static __always_inline bool warn_rcu_enter(void) { return false; }
162
+
static __always_inline void warn_rcu_exit(bool rcu) { }
136
163
#endif /* !CONFIG_CONTEXT_TRACKING_IDLE */
137
164
138
165
#endif
-3
include/linux/cpu.h
-3
include/linux/cpu.h
+45
-5
include/linux/cpuidle.h
+45
-5
include/linux/cpuidle.h
···
14
14
#include <linux/percpu.h>
15
15
#include <linux/list.h>
16
16
#include <linux/hrtimer.h>
17
+
#include <linux/context_tracking.h>
17
18
18
19
#define CPUIDLE_STATE_MAX 10
19
20
#define CPUIDLE_NAME_LEN 16
···
115
114
116
115
DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
117
116
DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
117
+
118
+
static __always_inline void ct_cpuidle_enter(void)
119
+
{
120
+
lockdep_assert_irqs_disabled();
121
+
/*
122
+
* Idle is allowed to (temporary) enable IRQs. It
123
+
* will return with IRQs disabled.
124
+
*
125
+
* Trace IRQs enable here, then switch off RCU, and have
126
+
* arch_cpu_idle() use raw_local_irq_enable(). Note that
127
+
* ct_idle_enter() relies on lockdep IRQ state, so switch that
128
+
* last -- this is very similar to the entry code.
129
+
*/
130
+
trace_hardirqs_on_prepare();
131
+
lockdep_hardirqs_on_prepare();
132
+
instrumentation_end();
133
+
ct_idle_enter();
134
+
lockdep_hardirqs_on(_RET_IP_);
135
+
}
136
+
137
+
static __always_inline void ct_cpuidle_exit(void)
138
+
{
139
+
/*
140
+
* Carefully undo the above.
141
+
*/
142
+
lockdep_hardirqs_off(_RET_IP_);
143
+
ct_idle_exit();
144
+
instrumentation_begin();
145
+
}
118
146
119
147
/****************************
120
148
* CPUIDLE DRIVER INTERFACE *
···
307
277
#define __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, \
308
278
idx, \
309
279
state, \
310
-
is_retention) \
280
+
is_retention, is_rcu) \
311
281
({ \
312
282
int __ret = 0; \
313
283
\
···
319
289
if (!is_retention) \
320
290
__ret = cpu_pm_enter(); \
321
291
if (!__ret) { \
292
+
if (!is_rcu) \
293
+
ct_cpuidle_enter(); \
322
294
__ret = low_level_idle_enter(state); \
295
+
if (!is_rcu) \
296
+
ct_cpuidle_exit(); \
323
297
if (!is_retention) \
324
298
cpu_pm_exit(); \
325
299
} \
···
332
298
})
333
299
334
300
#define CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx) \
335
-
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0)
301
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0, 0)
336
302
337
303
#define CPU_PM_CPU_IDLE_ENTER_RETENTION(low_level_idle_enter, idx) \
338
-
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1)
304
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1, 0)
339
305
340
306
#define CPU_PM_CPU_IDLE_ENTER_PARAM(low_level_idle_enter, idx, state) \
341
-
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0)
307
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 0)
308
+
309
+
#define CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(low_level_idle_enter, idx, state) \
310
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 1)
342
311
343
312
#define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(low_level_idle_enter, idx, state) \
344
-
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1)
313
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 0)
314
+
315
+
#define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(low_level_idle_enter, idx, state) \
316
+
__CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 1)
345
317
346
318
#endif /* _LINUX_CPUIDLE_H */
+2
-2
include/linux/cpumask.h
+2
-2
include/linux/cpumask.h
···
1017
1017
* concurrent CPU hotplug operations unless invoked from a cpuhp_lock held
1018
1018
* region.
1019
1019
*/
1020
-
static inline unsigned int num_online_cpus(void)
1020
+
static __always_inline unsigned int num_online_cpus(void)
1021
1021
{
1022
-
return atomic_read(&__num_online_cpus);
1022
+
return arch_atomic_read(&__num_online_cpus);
1023
1023
}
1024
1024
#define num_possible_cpus() cpumask_weight(cpu_possible_mask)
1025
1025
#define num_present_cpus() cpumask_weight(cpu_present_mask)
+2
-2
include/linux/math64.h
+2
-2
include/linux/math64.h
···
161
161
#if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__)
162
162
163
163
#ifndef mul_u64_u32_shr
164
-
static inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift)
164
+
static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift)
165
165
{
166
166
return (u64)(((unsigned __int128)a * mul) >> shift);
167
167
}
···
177
177
#else
178
178
179
179
#ifndef mul_u64_u32_shr
180
-
static inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift)
180
+
static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift)
181
181
{
182
182
u32 ah, al;
183
183
u64 ret;
+25
include/linux/mm.h
+25
include/linux/mm.h
···
1982
1982
/* Set in unmap_vmas() to indicate a final unmap call. Only used by hugetlb */
1983
1983
#define ZAP_FLAG_UNMAP ((__force zap_flags_t) BIT(1))
1984
1984
1985
+
#ifdef CONFIG_SCHED_MM_CID
1986
+
void sched_mm_cid_before_execve(struct task_struct *t);
1987
+
void sched_mm_cid_after_execve(struct task_struct *t);
1988
+
void sched_mm_cid_fork(struct task_struct *t);
1989
+
void sched_mm_cid_exit_signals(struct task_struct *t);
1990
+
static inline int task_mm_cid(struct task_struct *t)
1991
+
{
1992
+
return t->mm_cid;
1993
+
}
1994
+
#else
1995
+
static inline void sched_mm_cid_before_execve(struct task_struct *t) { }
1996
+
static inline void sched_mm_cid_after_execve(struct task_struct *t) { }
1997
+
static inline void sched_mm_cid_fork(struct task_struct *t) { }
1998
+
static inline void sched_mm_cid_exit_signals(struct task_struct *t) { }
1999
+
static inline int task_mm_cid(struct task_struct *t)
2000
+
{
2001
+
/*
2002
+
* Use the processor id as a fall-back when the mm cid feature is
2003
+
* disabled. This provides functional per-cpu data structure accesses
2004
+
* in user-space, althrough it won't provide the memory usage benefits.
2005
+
*/
2006
+
return raw_smp_processor_id();
2007
+
}
2008
+
#endif
2009
+
1985
2010
#ifdef CONFIG_MMU
1986
2011
extern bool can_do_mlock(void);
1987
2012
#else
+42
-1
include/linux/mm_types.h
+42
-1
include/linux/mm_types.h
···
645
645
* &struct mm_struct is freed.
646
646
*/
647
647
atomic_t mm_count;
648
-
648
+
#ifdef CONFIG_SCHED_MM_CID
649
+
/**
650
+
* @cid_lock: Protect cid bitmap updates vs lookups.
651
+
*
652
+
* Prevent situations where updates to the cid bitmap happen
653
+
* concurrently with lookups. Those can lead to situations
654
+
* where a lookup cannot find a free bit simply because it was
655
+
* unlucky enough to load, non-atomically, bitmap words as they
656
+
* were being concurrently updated by the updaters.
657
+
*/
658
+
raw_spinlock_t cid_lock;
659
+
#endif
649
660
#ifdef CONFIG_MMU
650
661
atomic_long_t pgtables_bytes; /* PTE page table pages */
651
662
#endif
···
919
908
vmi->mas.index = addr;
920
909
vmi->mas.node = MAS_START;
921
910
}
911
+
912
+
#ifdef CONFIG_SCHED_MM_CID
913
+
/* Accessor for struct mm_struct's cidmask. */
914
+
static inline cpumask_t *mm_cidmask(struct mm_struct *mm)
915
+
{
916
+
unsigned long cid_bitmap = (unsigned long)mm;
917
+
918
+
cid_bitmap += offsetof(struct mm_struct, cpu_bitmap);
919
+
/* Skip cpu_bitmap */
920
+
cid_bitmap += cpumask_size();
921
+
return (struct cpumask *)cid_bitmap;
922
+
}
923
+
924
+
static inline void mm_init_cid(struct mm_struct *mm)
925
+
{
926
+
raw_spin_lock_init(&mm->cid_lock);
927
+
cpumask_clear(mm_cidmask(mm));
928
+
}
929
+
930
+
static inline unsigned int mm_cid_size(void)
931
+
{
932
+
return cpumask_size();
933
+
}
934
+
#else /* CONFIG_SCHED_MM_CID */
935
+
static inline void mm_init_cid(struct mm_struct *mm) { }
936
+
static inline unsigned int mm_cid_size(void)
937
+
{
938
+
return 0;
939
+
}
940
+
#endif /* CONFIG_SCHED_MM_CID */
922
941
923
942
struct mmu_gather;
924
943
extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
+1
-1
include/linux/percpu-defs.h
+1
-1
include/linux/percpu-defs.h
···
310
310
#ifdef CONFIG_DEBUG_PREEMPT
311
311
extern void __this_cpu_preempt_check(const char *op);
312
312
#else
313
-
static inline void __this_cpu_preempt_check(const char *op) { }
313
+
static __always_inline void __this_cpu_preempt_check(const char *op) { }
314
314
#endif
315
315
316
316
#define __pcpu_size_call_return(stem, variable) \
+9
include/linux/sched.h
+9
include/linux/sched.h
···
1302
1302
1303
1303
#ifdef CONFIG_RSEQ
1304
1304
struct rseq __user *rseq;
1305
+
u32 rseq_len;
1305
1306
u32 rseq_sig;
1306
1307
/*
1307
1308
* RmW on rseq_event_mask must be performed atomically
1308
1309
* with respect to preemption.
1309
1310
*/
1310
1311
unsigned long rseq_event_mask;
1312
+
#endif
1313
+
1314
+
#ifdef CONFIG_SCHED_MM_CID
1315
+
int mm_cid; /* Current cid in mm */
1316
+
int mm_cid_active; /* Whether cid bitmap is active */
1311
1317
#endif
1312
1318
1313
1319
struct tlbflush_unmap_batch tlb_ubc;
···
2358
2352
{
2359
2353
if (clone_flags & CLONE_VM) {
2360
2354
t->rseq = NULL;
2355
+
t->rseq_len = 0;
2361
2356
t->rseq_sig = 0;
2362
2357
t->rseq_event_mask = 0;
2363
2358
} else {
2364
2359
t->rseq = current->rseq;
2360
+
t->rseq_len = current->rseq_len;
2365
2361
t->rseq_sig = current->rseq_sig;
2366
2362
t->rseq_event_mask = current->rseq_event_mask;
2367
2363
}
···
2372
2364
static inline void rseq_execve(struct task_struct *t)
2373
2365
{
2374
2366
t->rseq = NULL;
2367
+
t->rseq_len = 0;
2375
2368
t->rseq_sig = 0;
2376
2369
t->rseq_event_mask = 0;
2377
2370
}
+3
-5
include/linux/sched/clock.h
+3
-5
include/linux/sched/clock.h
···
45
45
return sched_clock();
46
46
}
47
47
48
-
static inline u64 local_clock(void)
48
+
static __always_inline u64 local_clock(void)
49
49
{
50
50
return sched_clock();
51
51
}
···
79
79
return sched_clock_cpu(cpu);
80
80
}
81
81
82
-
static inline u64 local_clock(void)
83
-
{
84
-
return sched_clock_cpu(raw_smp_processor_id());
85
-
}
82
+
extern u64 local_clock(void);
83
+
86
84
#endif
87
85
88
86
#ifdef CONFIG_IRQ_TIME_ACCOUNTING
-9
include/linux/sched/cputime.h
-9
include/linux/sched/cputime.h
···
8
8
* cputime accounting APIs:
9
9
*/
10
10
11
-
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
12
-
#include <asm/cputime.h>
13
-
14
-
#ifndef cputime_to_nsecs
15
-
# define cputime_to_nsecs(__ct) \
16
-
(cputime_to_usecs(__ct) * NSEC_PER_USEC)
17
-
#endif
18
-
#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */
19
-
20
11
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
21
12
extern bool task_cputime(struct task_struct *t,
22
13
u64 *utime, u64 *stime);
+30
-10
include/linux/sched/idle.h
+30
-10
include/linux/sched/idle.h
···
23
23
*/
24
24
#ifdef TIF_POLLING_NRFLAG
25
25
26
-
static inline void __current_set_polling(void)
26
+
#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H
27
+
28
+
static __always_inline void __current_set_polling(void)
27
29
{
28
-
set_thread_flag(TIF_POLLING_NRFLAG);
30
+
arch_set_bit(TIF_POLLING_NRFLAG,
31
+
(unsigned long *)(¤t_thread_info()->flags));
29
32
}
30
33
31
-
static inline bool __must_check current_set_polling_and_test(void)
34
+
static __always_inline void __current_clr_polling(void)
35
+
{
36
+
arch_clear_bit(TIF_POLLING_NRFLAG,
37
+
(unsigned long *)(¤t_thread_info()->flags));
38
+
}
39
+
40
+
#else
41
+
42
+
static __always_inline void __current_set_polling(void)
43
+
{
44
+
set_bit(TIF_POLLING_NRFLAG,
45
+
(unsigned long *)(¤t_thread_info()->flags));
46
+
}
47
+
48
+
static __always_inline void __current_clr_polling(void)
49
+
{
50
+
clear_bit(TIF_POLLING_NRFLAG,
51
+
(unsigned long *)(¤t_thread_info()->flags));
52
+
}
53
+
54
+
#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H */
55
+
56
+
static __always_inline bool __must_check current_set_polling_and_test(void)
32
57
{
33
58
__current_set_polling();
34
59
···
66
41
return unlikely(tif_need_resched());
67
42
}
68
43
69
-
static inline void __current_clr_polling(void)
70
-
{
71
-
clear_thread_flag(TIF_POLLING_NRFLAG);
72
-
}
73
-
74
-
static inline bool __must_check current_clr_polling_and_test(void)
44
+
static __always_inline bool __must_check current_clr_polling_and_test(void)
75
45
{
76
46
__current_clr_polling();
77
47
···
93
73
}
94
74
#endif
95
75
96
-
static inline void current_clr_polling(void)
76
+
static __always_inline void current_clr_polling(void)
97
77
{
98
78
__current_clr_polling();
99
79
+17
-1
include/linux/thread_info.h
+17
-1
include/linux/thread_info.h
···
177
177
clear_ti_thread_flag(task_thread_info(t), TIF_##fl)
178
178
#endif /* !CONFIG_GENERIC_ENTRY */
179
179
180
-
#define tif_need_resched() test_thread_flag(TIF_NEED_RESCHED)
180
+
#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H
181
+
182
+
static __always_inline bool tif_need_resched(void)
183
+
{
184
+
return arch_test_bit(TIF_NEED_RESCHED,
185
+
(unsigned long *)(¤t_thread_info()->flags));
186
+
}
187
+
188
+
#else
189
+
190
+
static __always_inline bool tif_need_resched(void)
191
+
{
192
+
return test_bit(TIF_NEED_RESCHED,
193
+
(unsigned long *)(¤t_thread_info()->flags));
194
+
}
195
+
196
+
#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H */
181
197
182
198
#ifndef CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES
183
199
static inline int arch_within_stack_frames(const void * const stack,
+18
include/linux/trace_recursion.h
+18
include/linux/trace_recursion.h
···
135
135
# define do_ftrace_record_recursion(ip, pip) do { } while (0)
136
136
#endif
137
137
138
+
#ifdef CONFIG_ARCH_WANTS_NO_INSTR
139
+
# define trace_warn_on_no_rcu(ip) \
140
+
({ \
141
+
bool __ret = !rcu_is_watching(); \
142
+
if (__ret && !trace_recursion_test(TRACE_RECORD_RECURSION_BIT)) { \
143
+
trace_recursion_set(TRACE_RECORD_RECURSION_BIT); \
144
+
WARN_ONCE(true, "RCU not on for: %pS\n", (void *)ip); \
145
+
trace_recursion_clear(TRACE_RECORD_RECURSION_BIT); \
146
+
} \
147
+
__ret; \
148
+
})
149
+
#else
150
+
# define trace_warn_on_no_rcu(ip) false
151
+
#endif
152
+
138
153
/*
139
154
* Preemption is promised to be disabled when return bit >= 0.
140
155
*/
···
158
143
{
159
144
unsigned int val = READ_ONCE(current->trace_recursion);
160
145
int bit;
146
+
147
+
if (trace_warn_on_no_rcu(ip))
148
+
return -1;
161
149
162
150
bit = trace_get_context_bit() + start;
163
151
if (unlikely(val & (1 << bit))) {
+13
-2
include/linux/tracepoint.h
+13
-2
include/linux/tracepoint.h
···
178
178
#endif /* CONFIG_HAVE_STATIC_CALL */
179
179
180
180
/*
181
+
* ARCH_WANTS_NO_INSTR archs are expected to have sanitized entry and idle
182
+
* code that disallow any/all tracing/instrumentation when RCU isn't watching.
183
+
*/
184
+
#ifdef CONFIG_ARCH_WANTS_NO_INSTR
185
+
#define RCUIDLE_COND(rcuidle) (rcuidle)
186
+
#else
187
+
/* srcu can't be used from NMI */
188
+
#define RCUIDLE_COND(rcuidle) (rcuidle && in_nmi())
189
+
#endif
190
+
191
+
/*
181
192
* it_func[0] is never NULL because there is at least one element in the array
182
193
* when the array itself is non NULL.
183
194
*/
···
199
188
if (!(cond)) \
200
189
return; \
201
190
\
202
-
/* srcu can't be used from NMI */ \
203
-
WARN_ON_ONCE(rcuidle && in_nmi()); \
191
+
if (WARN_ON_ONCE(RCUIDLE_COND(rcuidle))) \
192
+
return; \
204
193
\
205
194
/* keep srcu and sched-rcu usage consistent */ \
206
195
preempt_disable_notrace(); \
+6
-1
include/trace/events/rseq.h
+6
-1
include/trace/events/rseq.h
···
16
16
17
17
TP_STRUCT__entry(
18
18
__field(s32, cpu_id)
19
+
__field(s32, node_id)
20
+
__field(s32, mm_cid)
19
21
),
20
22
21
23
TP_fast_assign(
22
24
__entry->cpu_id = raw_smp_processor_id();
25
+
__entry->node_id = cpu_to_node(__entry->cpu_id);
26
+
__entry->mm_cid = task_mm_cid(t);
23
27
),
24
28
25
-
TP_printk("cpu_id=%d", __entry->cpu_id)
29
+
TP_printk("cpu_id=%d node_id=%d mm_cid=%d", __entry->cpu_id,
30
+
__entry->node_id, __entry->mm_cid)
26
31
);
27
32
28
33
TRACE_EVENT(rseq_ip_fixup,
+2
include/uapi/linux/auxvec.h
+2
include/uapi/linux/auxvec.h
···
30
30
* differ from AT_PLATFORM. */
31
31
#define AT_RANDOM 25 /* address of 16 random bytes */
32
32
#define AT_HWCAP2 26 /* extension of AT_HWCAP */
33
+
#define AT_RSEQ_FEATURE_SIZE 27 /* rseq supported feature size */
34
+
#define AT_RSEQ_ALIGN 28 /* rseq allocation alignment */
33
35
34
36
#define AT_EXECFN 31 /* filename of program */
35
37
+4
include/uapi/linux/membarrier.h
+4
include/uapi/linux/membarrier.h
···
137
137
* @MEMBARRIER_CMD_SHARED:
138
138
* Alias to MEMBARRIER_CMD_GLOBAL. Provided for
139
139
* header backward compatibility.
140
+
* @MEMBARRIER_CMD_GET_REGISTRATIONS:
141
+
* Returns a bitmask of previously issued
142
+
* registration commands.
140
143
*
141
144
* Command to be passed to the membarrier system call. The commands need to
142
145
* be a single bit each, except for MEMBARRIER_CMD_QUERY which is assigned to
···
156
153
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE = (1 << 6),
157
154
MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ = (1 << 7),
158
155
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ = (1 << 8),
156
+
MEMBARRIER_CMD_GET_REGISTRATIONS = (1 << 9),
159
157
160
158
/* Alias for header backward compatibility. */
161
159
MEMBARRIER_CMD_SHARED = MEMBARRIER_CMD_GLOBAL,
+22
include/uapi/linux/rseq.h
+22
include/uapi/linux/rseq.h
···
130
130
* this thread.
131
131
*/
132
132
__u32 flags;
133
+
134
+
/*
135
+
* Restartable sequences node_id field. Updated by the kernel. Read by
136
+
* user-space with single-copy atomicity semantics. This field should
137
+
* only be read by the thread which registered this data structure.
138
+
* Aligned on 32-bit. Contains the current NUMA node ID.
139
+
*/
140
+
__u32 node_id;
141
+
142
+
/*
143
+
* Restartable sequences mm_cid field. Updated by the kernel. Read by
144
+
* user-space with single-copy atomicity semantics. This field should
145
+
* only be read by the thread which registered this data structure.
146
+
* Aligned on 32-bit. Contains the current thread's concurrency ID
147
+
* (allocated uniquely within a memory map).
148
+
*/
149
+
__u32 mm_cid;
150
+
151
+
/*
152
+
* Flexible array member at end of structure, after last feature field.
153
+
*/
154
+
char end[];
133
155
} __attribute__((aligned(4 * sizeof(__u64))));
134
156
135
157
#endif /* _UAPI_LINUX_RSEQ_H */
+4
init/Kconfig
+4
init/Kconfig
+6
-6
kernel/context_tracking.c
+6
-6
kernel/context_tracking.c
···
510
510
* In this we case we don't care about any concurrency/ordering.
511
511
*/
512
512
if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
513
-
atomic_set(&ct->state, state);
513
+
arch_atomic_set(&ct->state, state);
514
514
} else {
515
515
/*
516
516
* Even if context tracking is disabled on this CPU, because it's outside
···
527
527
*/
528
528
if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
529
529
/* Tracking for vtime only, no concurrent RCU EQS accounting */
530
-
atomic_set(&ct->state, state);
530
+
arch_atomic_set(&ct->state, state);
531
531
} else {
532
532
/*
533
533
* Tracking for vtime and RCU EQS. Make sure we don't race
···
535
535
* RCU only requires RCU_DYNTICKS_IDX increments to be fully
536
536
* ordered.
537
537
*/
538
-
atomic_add(state, &ct->state);
538
+
arch_atomic_add(state, &ct->state);
539
539
}
540
540
}
541
541
}
···
630
630
* In this we case we don't care about any concurrency/ordering.
631
631
*/
632
632
if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
633
-
atomic_set(&ct->state, CONTEXT_KERNEL);
633
+
arch_atomic_set(&ct->state, CONTEXT_KERNEL);
634
634
635
635
} else {
636
636
if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
637
637
/* Tracking for vtime only, no concurrent RCU EQS accounting */
638
-
atomic_set(&ct->state, CONTEXT_KERNEL);
638
+
arch_atomic_set(&ct->state, CONTEXT_KERNEL);
639
639
} else {
640
640
/*
641
641
* Tracking for vtime and RCU EQS. Make sure we don't race
···
643
643
* RCU only requires RCU_DYNTICKS_IDX increments to be fully
644
644
* ordered.
645
645
*/
646
-
atomic_sub(state, &ct->state);
646
+
arch_atomic_sub(state, &ct->state);
647
647
}
648
648
}
649
649
}
-9
kernel/cpu_pm.c
-9
kernel/cpu_pm.c
···
30
30
{
31
31
int ret;
32
32
33
-
/*
34
-
* This introduces a RCU read critical section, which could be
35
-
* disfunctional in cpu idle. Copy RCU_NONIDLE code to let RCU know
36
-
* this.
37
-
*/
38
-
ct_irq_enter_irqson();
39
33
rcu_read_lock();
40
34
ret = raw_notifier_call_chain(&cpu_pm_notifier.chain, event, NULL);
41
35
rcu_read_unlock();
42
-
ct_irq_exit_irqson();
43
36
44
37
return notifier_to_errno(ret);
45
38
}
···
42
49
unsigned long flags;
43
50
int ret;
44
51
45
-
ct_irq_enter_irqson();
46
52
raw_spin_lock_irqsave(&cpu_pm_notifier.lock, flags);
47
53
ret = raw_notifier_call_chain_robust(&cpu_pm_notifier.chain, event_up, event_down, NULL);
48
54
raw_spin_unlock_irqrestore(&cpu_pm_notifier.lock, flags);
49
-
ct_irq_exit_irqson();
50
55
51
56
return notifier_to_errno(ret);
52
57
}
+7
kernel/exit.c
+7
kernel/exit.c
···
807
807
struct task_struct *tsk = current;
808
808
int group_dead;
809
809
810
+
WARN_ON(irqs_disabled());
811
+
810
812
synchronize_group_exit(tsk, code);
811
813
812
814
WARN_ON(tsk->plug);
···
940
938
if (unlikely(!tsk->pid))
941
939
panic("Attempted to kill the idle task!");
942
940
941
+
if (unlikely(irqs_disabled())) {
942
+
pr_info("note: %s[%d] exited with irqs disabled\n",
943
+
current->comm, task_pid_nr(current));
944
+
local_irq_enable();
945
+
}
943
946
if (unlikely(in_atomic())) {
944
947
pr_info("note: %s[%d] exited with preempt_count %d\n",
945
948
current->comm, task_pid_nr(current),
+7
-1
kernel/fork.c
+7
-1
kernel/fork.c
···
1060
1060
tsk->reported_split_lock = 0;
1061
1061
#endif
1062
1062
1063
+
#ifdef CONFIG_SCHED_MM_CID
1064
+
tsk->mm_cid = -1;
1065
+
tsk->mm_cid_active = 0;
1066
+
#endif
1063
1067
return tsk;
1064
1068
1065
1069
free_stack:
···
1173
1169
1174
1170
mm->user_ns = get_user_ns(user_ns);
1175
1171
lru_gen_init_mm(mm);
1172
+
mm_init_cid(mm);
1176
1173
return mm;
1177
1174
1178
1175
fail_pcpu:
···
1606
1601
1607
1602
tsk->mm = mm;
1608
1603
tsk->active_mm = mm;
1604
+
sched_mm_cid_fork(tsk);
1609
1605
return 0;
1610
1606
}
1611
1607
···
3040
3034
* dynamically sized based on the maximum CPU number this system
3041
3035
* can have, taking hotplug into account (nr_cpu_ids).
3042
3036
*/
3043
-
mm_size = sizeof(struct mm_struct) + cpumask_size();
3037
+
mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();
3044
3038
3045
3039
mm_cachep = kmem_cache_create_usercopy("mm_struct",
3046
3040
mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
+3
kernel/locking/lockdep.c
+3
kernel/locking/lockdep.c
···
55
55
#include <linux/rcupdate.h>
56
56
#include <linux/kprobes.h>
57
57
#include <linux/lockdep.h>
58
+
#include <linux/context_tracking.h>
58
59
59
60
#include <asm/sections.h>
60
61
···
6556
6555
{
6557
6556
struct task_struct *curr = current;
6558
6557
int dl = READ_ONCE(debug_locks);
6558
+
bool rcu = warn_rcu_enter();
6559
6559
6560
6560
/* Note: the following can be executed concurrently, so be careful. */
6561
6561
pr_warn("\n");
···
6597
6595
lockdep_print_held_locks(curr);
6598
6596
pr_warn("\nstack backtrace:\n");
6599
6597
dump_stack();
6598
+
warn_rcu_exit(rcu);
6600
6599
}
6601
6600
EXPORT_SYMBOL_GPL(lockdep_rcu_suspicious);
+5
kernel/panic.c
+5
kernel/panic.c
···
34
34
#include <linux/ratelimit.h>
35
35
#include <linux/debugfs.h>
36
36
#include <linux/sysfs.h>
37
+
#include <linux/context_tracking.h>
37
38
#include <trace/events/error_report.h>
38
39
#include <asm/sections.h>
39
40
···
680
679
void warn_slowpath_fmt(const char *file, int line, unsigned taint,
681
680
const char *fmt, ...)
682
681
{
682
+
bool rcu = warn_rcu_enter();
683
683
struct warn_args args;
684
684
685
685
pr_warn(CUT_HERE);
···
695
693
va_start(args.args, fmt);
696
694
__warn(file, line, __builtin_return_address(0), taint, NULL, &args);
697
695
va_end(args.args);
696
+
warn_rcu_exit(rcu);
698
697
}
699
698
EXPORT_SYMBOL(warn_slowpath_fmt);
700
699
#else
701
700
void __warn_printk(const char *fmt, ...)
702
701
{
702
+
bool rcu = warn_rcu_enter();
703
703
va_list args;
704
704
705
705
pr_warn(CUT_HERE);
···
709
705
va_start(args, fmt);
710
706
vprintk(fmt, args);
711
707
va_end(args);
708
+
warn_rcu_exit(rcu);
712
709
}
713
710
EXPORT_SYMBOL(__warn_printk);
714
711
#endif
+1
-1
kernel/printk/printk.c
+1
-1
kernel/printk/printk.c
+1
-1
kernel/ptrace.c
+1
-1
kernel/ptrace.c
+52
-13
kernel/rseq.c
+52
-13
kernel/rseq.c
···
18
18
#define CREATE_TRACE_POINTS
19
19
#include <trace/events/rseq.h>
20
20
21
+
/* The original rseq structure size (including padding) is 32 bytes. */
22
+
#define ORIG_RSEQ_SIZE 32
23
+
21
24
#define RSEQ_CS_NO_RESTART_FLAGS (RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT | \
22
25
RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL | \
23
26
RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE)
···
85
82
* F1. <failure>
86
83
*/
87
84
88
-
static int rseq_update_cpu_id(struct task_struct *t)
85
+
static int rseq_update_cpu_node_id(struct task_struct *t)
89
86
{
90
-
u32 cpu_id = raw_smp_processor_id();
91
87
struct rseq __user *rseq = t->rseq;
88
+
u32 cpu_id = raw_smp_processor_id();
89
+
u32 node_id = cpu_to_node(cpu_id);
90
+
u32 mm_cid = task_mm_cid(t);
92
91
93
-
if (!user_write_access_begin(rseq, sizeof(*rseq)))
92
+
WARN_ON_ONCE((int) mm_cid < 0);
93
+
if (!user_write_access_begin(rseq, t->rseq_len))
94
94
goto efault;
95
95
unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end);
96
96
unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end);
97
+
unsafe_put_user(node_id, &rseq->node_id, efault_end);
98
+
unsafe_put_user(mm_cid, &rseq->mm_cid, efault_end);
99
+
/*
100
+
* Additional feature fields added after ORIG_RSEQ_SIZE
101
+
* need to be conditionally updated only if
102
+
* t->rseq_len != ORIG_RSEQ_SIZE.
103
+
*/
97
104
user_write_access_end();
98
105
trace_rseq_update(t);
99
106
return 0;
···
114
101
return -EFAULT;
115
102
}
116
103
117
-
static int rseq_reset_rseq_cpu_id(struct task_struct *t)
104
+
static int rseq_reset_rseq_cpu_node_id(struct task_struct *t)
118
105
{
119
-
u32 cpu_id_start = 0, cpu_id = RSEQ_CPU_ID_UNINITIALIZED;
106
+
u32 cpu_id_start = 0, cpu_id = RSEQ_CPU_ID_UNINITIALIZED, node_id = 0,
107
+
mm_cid = 0;
120
108
121
109
/*
122
110
* Reset cpu_id_start to its initial state (0).
···
131
117
*/
132
118
if (put_user(cpu_id, &t->rseq->cpu_id))
133
119
return -EFAULT;
120
+
/*
121
+
* Reset node_id to its initial state (0).
122
+
*/
123
+
if (put_user(node_id, &t->rseq->node_id))
124
+
return -EFAULT;
125
+
/*
126
+
* Reset mm_cid to its initial state (0).
127
+
*/
128
+
if (put_user(mm_cid, &t->rseq->mm_cid))
129
+
return -EFAULT;
130
+
/*
131
+
* Additional feature fields added after ORIG_RSEQ_SIZE
132
+
* need to be conditionally reset only if
133
+
* t->rseq_len != ORIG_RSEQ_SIZE.
134
+
*/
134
135
return 0;
135
136
}
136
137
···
330
301
if (unlikely(ret < 0))
331
302
goto error;
332
303
}
333
-
if (unlikely(rseq_update_cpu_id(t)))
304
+
if (unlikely(rseq_update_cpu_node_id(t)))
334
305
goto error;
335
306
return;
336
307
···
373
344
/* Unregister rseq for current thread. */
374
345
if (current->rseq != rseq || !current->rseq)
375
346
return -EINVAL;
376
-
if (rseq_len != sizeof(*rseq))
347
+
if (rseq_len != current->rseq_len)
377
348
return -EINVAL;
378
349
if (current->rseq_sig != sig)
379
350
return -EPERM;
380
-
ret = rseq_reset_rseq_cpu_id(current);
351
+
ret = rseq_reset_rseq_cpu_node_id(current);
381
352
if (ret)
382
353
return ret;
383
354
current->rseq = NULL;
384
355
current->rseq_sig = 0;
356
+
current->rseq_len = 0;
385
357
return 0;
386
358
}
387
359
···
395
365
* the provided address differs from the prior
396
366
* one.
397
367
*/
398
-
if (current->rseq != rseq || rseq_len != sizeof(*rseq))
368
+
if (current->rseq != rseq || rseq_len != current->rseq_len)
399
369
return -EINVAL;
400
370
if (current->rseq_sig != sig)
401
371
return -EPERM;
···
404
374
}
405
375
406
376
/*
407
-
* If there was no rseq previously registered,
408
-
* ensure the provided rseq is properly aligned and valid.
377
+
* If there was no rseq previously registered, ensure the provided rseq
378
+
* is properly aligned, as communcated to user-space through the ELF
379
+
* auxiliary vector AT_RSEQ_ALIGN. If rseq_len is the original rseq
380
+
* size, the required alignment is the original struct rseq alignment.
381
+
*
382
+
* In order to be valid, rseq_len is either the original rseq size, or
383
+
* large enough to contain all supported fields, as communicated to
384
+
* user-space through the ELF auxiliary vector AT_RSEQ_FEATURE_SIZE.
409
385
*/
410
-
if (!IS_ALIGNED((unsigned long)rseq, __alignof__(*rseq)) ||
411
-
rseq_len != sizeof(*rseq))
386
+
if (rseq_len < ORIG_RSEQ_SIZE ||
387
+
(rseq_len == ORIG_RSEQ_SIZE && !IS_ALIGNED((unsigned long)rseq, ORIG_RSEQ_SIZE)) ||
388
+
(rseq_len != ORIG_RSEQ_SIZE && (!IS_ALIGNED((unsigned long)rseq, __alignof__(*rseq)) ||
389
+
rseq_len < offsetof(struct rseq, end))))
412
390
return -EINVAL;
413
391
if (!access_ok(rseq, rseq_len))
414
392
return -EFAULT;
415
393
current->rseq = rseq;
394
+
current->rseq_len = rseq_len;
416
395
current->rseq_sig = sig;
417
396
/*
418
397
* If rseq was previously inactive, and has just been
+21
-6
kernel/sched/clock.c
+21
-6
kernel/sched/clock.c
···
93
93
94
94
static DEFINE_PER_CPU_SHARED_ALIGNED(struct sched_clock_data, sched_clock_data);
95
95
96
-
notrace static inline struct sched_clock_data *this_scd(void)
96
+
static __always_inline struct sched_clock_data *this_scd(void)
97
97
{
98
98
return this_cpu_ptr(&sched_clock_data);
99
99
}
···
244
244
* min, max except they take wrapping into account
245
245
*/
246
246
247
-
notrace static inline u64 wrap_min(u64 x, u64 y)
247
+
static __always_inline u64 wrap_min(u64 x, u64 y)
248
248
{
249
249
return (s64)(x - y) < 0 ? x : y;
250
250
}
251
251
252
-
notrace static inline u64 wrap_max(u64 x, u64 y)
252
+
static __always_inline u64 wrap_max(u64 x, u64 y)
253
253
{
254
254
return (s64)(x - y) > 0 ? x : y;
255
255
}
···
260
260
* - filter out backward motion
261
261
* - use the GTOD tick value to create a window to filter crazy TSC values
262
262
*/
263
-
notrace static u64 sched_clock_local(struct sched_clock_data *scd)
263
+
static __always_inline u64 sched_clock_local(struct sched_clock_data *scd)
264
264
{
265
265
u64 now, clock, old_clock, min_clock, max_clock, gtod;
266
266
s64 delta;
···
287
287
clock = wrap_max(clock, min_clock);
288
288
clock = wrap_min(clock, max_clock);
289
289
290
-
if (!try_cmpxchg64(&scd->clock, &old_clock, clock))
290
+
if (!arch_try_cmpxchg64(&scd->clock, &old_clock, clock))
291
291
goto again;
292
292
293
293
return clock;
294
294
}
295
295
296
-
notrace static u64 sched_clock_remote(struct sched_clock_data *scd)
296
+
noinstr u64 local_clock(void)
297
+
{
298
+
u64 clock;
299
+
300
+
if (static_branch_likely(&__sched_clock_stable))
301
+
return sched_clock() + __sched_clock_offset;
302
+
303
+
preempt_disable_notrace();
304
+
clock = sched_clock_local(this_scd());
305
+
preempt_enable_notrace();
306
+
307
+
return clock;
308
+
}
309
+
EXPORT_SYMBOL_GPL(local_clock);
310
+
311
+
static notrace u64 sched_clock_remote(struct sched_clock_data *scd)
297
312
{
298
313
struct sched_clock_data *my_scd = this_scd();
299
314
u64 this_clock, remote_clock;
+96
-38
kernel/sched/core.c
+96
-38
kernel/sched/core.c
···
152
152
DEFINE_STATIC_KEY_FALSE(__sched_core_enabled);
153
153
154
154
/* kernel prio, less is more */
155
-
static inline int __task_prio(struct task_struct *p)
155
+
static inline int __task_prio(const struct task_struct *p)
156
156
{
157
157
if (p->sched_class == &stop_sched_class) /* trumps deadline */
158
158
return -2;
···
174
174
*/
175
175
176
176
/* real prio, less is less */
177
-
static inline bool prio_less(struct task_struct *a, struct task_struct *b, bool in_fi)
177
+
static inline bool prio_less(const struct task_struct *a,
178
+
const struct task_struct *b, bool in_fi)
178
179
{
179
180
180
181
int pa = __task_prio(a), pb = __task_prio(b);
···
195
194
return false;
196
195
}
197
196
198
-
static inline bool __sched_core_less(struct task_struct *a, struct task_struct *b)
197
+
static inline bool __sched_core_less(const struct task_struct *a,
198
+
const struct task_struct *b)
199
199
{
200
200
if (a->core_cookie < b->core_cookie)
201
201
return true;
···
3677
3675
}
3678
3676
3679
3677
/*
3680
-
* Mark the task runnable and perform wakeup-preemption.
3678
+
* Mark the task runnable.
3681
3679
*/
3682
-
static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
3683
-
struct rq_flags *rf)
3680
+
static inline void ttwu_do_wakeup(struct task_struct *p)
3684
3681
{
3685
-
check_preempt_curr(rq, p, wake_flags);
3686
3682
WRITE_ONCE(p->__state, TASK_RUNNING);
3687
3683
trace_sched_wakeup(p);
3684
+
}
3685
+
3686
+
static void
3687
+
ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
3688
+
struct rq_flags *rf)
3689
+
{
3690
+
int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
3691
+
3692
+
lockdep_assert_rq_held(rq);
3693
+
3694
+
if (p->sched_contributes_to_load)
3695
+
rq->nr_uninterruptible--;
3696
+
3697
+
#ifdef CONFIG_SMP
3698
+
if (wake_flags & WF_MIGRATED)
3699
+
en_flags |= ENQUEUE_MIGRATED;
3700
+
else
3701
+
#endif
3702
+
if (p->in_iowait) {
3703
+
delayacct_blkio_end(p);
3704
+
atomic_dec(&task_rq(p)->nr_iowait);
3705
+
}
3706
+
3707
+
activate_task(rq, p, en_flags);
3708
+
check_preempt_curr(rq, p, wake_flags);
3709
+
3710
+
ttwu_do_wakeup(p);
3688
3711
3689
3712
#ifdef CONFIG_SMP
3690
3713
if (p->sched_class->task_woken) {
···
3737
3710
rq->idle_stamp = 0;
3738
3711
}
3739
3712
#endif
3740
-
}
3741
-
3742
-
static void
3743
-
ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
3744
-
struct rq_flags *rf)
3745
-
{
3746
-
int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
3747
-
3748
-
lockdep_assert_rq_held(rq);
3749
-
3750
-
if (p->sched_contributes_to_load)
3751
-
rq->nr_uninterruptible--;
3752
-
3753
-
#ifdef CONFIG_SMP
3754
-
if (wake_flags & WF_MIGRATED)
3755
-
en_flags |= ENQUEUE_MIGRATED;
3756
-
else
3757
-
#endif
3758
-
if (p->in_iowait) {
3759
-
delayacct_blkio_end(p);
3760
-
atomic_dec(&task_rq(p)->nr_iowait);
3761
-
}
3762
-
3763
-
activate_task(rq, p, en_flags);
3764
-
ttwu_do_wakeup(rq, p, wake_flags, rf);
3765
3713
}
3766
3714
3767
3715
/*
···
3772
3770
3773
3771
rq = __task_rq_lock(p, &rf);
3774
3772
if (task_on_rq_queued(p)) {
3775
-
/* check_preempt_curr() may use rq clock */
3776
-
update_rq_clock(rq);
3777
-
ttwu_do_wakeup(rq, p, wake_flags, &rf);
3773
+
if (!task_on_cpu(rq, p)) {
3774
+
/*
3775
+
* When on_rq && !on_cpu the task is preempted, see if
3776
+
* it should preempt the task that is current now.
3777
+
*/
3778
+
update_rq_clock(rq);
3779
+
check_preempt_curr(rq, p, wake_flags);
3780
+
}
3781
+
ttwu_do_wakeup(p);
3778
3782
ret = 1;
3779
3783
}
3780
3784
__task_rq_unlock(rq, &rf);
···
4146
4138
goto out;
4147
4139
4148
4140
trace_sched_waking(p);
4149
-
WRITE_ONCE(p->__state, TASK_RUNNING);
4150
-
trace_sched_wakeup(p);
4141
+
ttwu_do_wakeup(p);
4151
4142
goto out;
4152
4143
}
4153
4144
···
5111
5104
sched_info_switch(rq, prev, next);
5112
5105
perf_event_task_sched_out(prev, next);
5113
5106
rseq_preempt(prev);
5107
+
switch_mm_cid(prev, next);
5114
5108
fire_sched_out_preempt_notifiers(prev, next);
5115
5109
kmap_local_sched_out();
5116
5110
prepare_task(next);
···
6268
6260
{
6269
6261
int i;
6270
6262
6271
-
for_each_cpu_wrap(i, sched_domain_span(sd), cpu) {
6263
+
for_each_cpu_wrap(i, sched_domain_span(sd), cpu + 1) {
6272
6264
if (i == cpu)
6273
6265
continue;
6274
6266
···
11373
11365
{
11374
11366
trace_sched_update_nr_running_tp(rq, count);
11375
11367
}
11368
+
11369
+
#ifdef CONFIG_SCHED_MM_CID
11370
+
void sched_mm_cid_exit_signals(struct task_struct *t)
11371
+
{
11372
+
struct mm_struct *mm = t->mm;
11373
+
unsigned long flags;
11374
+
11375
+
if (!mm)
11376
+
return;
11377
+
local_irq_save(flags);
11378
+
mm_cid_put(mm, t->mm_cid);
11379
+
t->mm_cid = -1;
11380
+
t->mm_cid_active = 0;
11381
+
local_irq_restore(flags);
11382
+
}
11383
+
11384
+
void sched_mm_cid_before_execve(struct task_struct *t)
11385
+
{
11386
+
struct mm_struct *mm = t->mm;
11387
+
unsigned long flags;
11388
+
11389
+
if (!mm)
11390
+
return;
11391
+
local_irq_save(flags);
11392
+
mm_cid_put(mm, t->mm_cid);
11393
+
t->mm_cid = -1;
11394
+
t->mm_cid_active = 0;
11395
+
local_irq_restore(flags);
11396
+
}
11397
+
11398
+
void sched_mm_cid_after_execve(struct task_struct *t)
11399
+
{
11400
+
struct mm_struct *mm = t->mm;
11401
+
unsigned long flags;
11402
+
11403
+
if (!mm)
11404
+
return;
11405
+
local_irq_save(flags);
11406
+
t->mm_cid = mm_cid_get(mm);
11407
+
t->mm_cid_active = 1;
11408
+
local_irq_restore(flags);
11409
+
rseq_set_notify_resume(t);
11410
+
}
11411
+
11412
+
void sched_mm_cid_fork(struct task_struct *t)
11413
+
{
11414
+
WARN_ON_ONCE(!t->mm || t->mm_cid != -1);
11415
+
t->mm_cid_active = 1;
11416
+
}
11417
+
#endif
+23
-20
kernel/sched/cpufreq_schedutil.c
+23
-20
kernel/sched/cpufreq_schedutil.c
···
48
48
49
49
unsigned long util;
50
50
unsigned long bw_dl;
51
-
unsigned long max;
52
51
53
52
/* The field below is for single-CPU policies only: */
54
53
#ifdef CONFIG_NO_HZ_COMMON
···
157
158
{
158
159
struct rq *rq = cpu_rq(sg_cpu->cpu);
159
160
160
-
sg_cpu->max = arch_scale_cpu_capacity(sg_cpu->cpu);
161
161
sg_cpu->bw_dl = cpu_bw_dl(rq);
162
162
sg_cpu->util = effective_cpu_util(sg_cpu->cpu, cpu_util_cfs(sg_cpu->cpu),
163
163
FREQUENCY_UTIL, NULL);
···
236
238
* sugov_iowait_apply() - Apply the IO boost to a CPU.
237
239
* @sg_cpu: the sugov data for the cpu to boost
238
240
* @time: the update time from the caller
241
+
* @max_cap: the max CPU capacity
239
242
*
240
243
* A CPU running a task which woken up after an IO operation can have its
241
244
* utilization boosted to speed up the completion of those IO operations.
···
250
251
* This mechanism is designed to boost high frequently IO waiting tasks, while
251
252
* being more conservative on tasks which does sporadic IO operations.
252
253
*/
253
-
static void sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time)
254
+
static void sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time,
255
+
unsigned long max_cap)
254
256
{
255
257
unsigned long boost;
256
258
···
280
280
* sg_cpu->util is already in capacity scale; convert iowait_boost
281
281
* into the same scale so we can compare.
282
282
*/
283
-
boost = (sg_cpu->iowait_boost * sg_cpu->max) >> SCHED_CAPACITY_SHIFT;
283
+
boost = (sg_cpu->iowait_boost * max_cap) >> SCHED_CAPACITY_SHIFT;
284
284
boost = uclamp_rq_util_with(cpu_rq(sg_cpu->cpu), boost, NULL);
285
285
if (sg_cpu->util < boost)
286
286
sg_cpu->util = boost;
···
310
310
}
311
311
312
312
static inline bool sugov_update_single_common(struct sugov_cpu *sg_cpu,
313
-
u64 time, unsigned int flags)
313
+
u64 time, unsigned long max_cap,
314
+
unsigned int flags)
314
315
{
315
316
sugov_iowait_boost(sg_cpu, time, flags);
316
317
sg_cpu->last_update = time;
···
322
321
return false;
323
322
324
323
sugov_get_util(sg_cpu);
325
-
sugov_iowait_apply(sg_cpu, time);
324
+
sugov_iowait_apply(sg_cpu, time, max_cap);
326
325
327
326
return true;
328
327
}
···
333
332
struct sugov_cpu *sg_cpu = container_of(hook, struct sugov_cpu, update_util);
334
333
struct sugov_policy *sg_policy = sg_cpu->sg_policy;
335
334
unsigned int cached_freq = sg_policy->cached_raw_freq;
335
+
unsigned long max_cap;
336
336
unsigned int next_f;
337
337
338
-
if (!sugov_update_single_common(sg_cpu, time, flags))
338
+
max_cap = arch_scale_cpu_capacity(sg_cpu->cpu);
339
+
340
+
if (!sugov_update_single_common(sg_cpu, time, max_cap, flags))
339
341
return;
340
342
341
-
next_f = get_next_freq(sg_policy, sg_cpu->util, sg_cpu->max);
343
+
next_f = get_next_freq(sg_policy, sg_cpu->util, max_cap);
342
344
/*
343
345
* Do not reduce the frequency if the CPU has not been idle
344
346
* recently, as the reduction is likely to be premature then.
···
378
374
{
379
375
struct sugov_cpu *sg_cpu = container_of(hook, struct sugov_cpu, update_util);
380
376
unsigned long prev_util = sg_cpu->util;
377
+
unsigned long max_cap;
381
378
382
379
/*
383
380
* Fall back to the "frequency" path if frequency invariance is not
···
390
385
return;
391
386
}
392
387
393
-
if (!sugov_update_single_common(sg_cpu, time, flags))
388
+
max_cap = arch_scale_cpu_capacity(sg_cpu->cpu);
389
+
390
+
if (!sugov_update_single_common(sg_cpu, time, max_cap, flags))
394
391
return;
395
392
396
393
/*
···
406
399
sg_cpu->util = prev_util;
407
400
408
401
cpufreq_driver_adjust_perf(sg_cpu->cpu, map_util_perf(sg_cpu->bw_dl),
409
-
map_util_perf(sg_cpu->util), sg_cpu->max);
402
+
map_util_perf(sg_cpu->util), max_cap);
410
403
411
404
sg_cpu->sg_policy->last_freq_update_time = time;
412
405
}
···
415
408
{
416
409
struct sugov_policy *sg_policy = sg_cpu->sg_policy;
417
410
struct cpufreq_policy *policy = sg_policy->policy;
418
-
unsigned long util = 0, max = 1;
411
+
unsigned long util = 0, max_cap;
419
412
unsigned int j;
413
+
414
+
max_cap = arch_scale_cpu_capacity(sg_cpu->cpu);
420
415
421
416
for_each_cpu(j, policy->cpus) {
422
417
struct sugov_cpu *j_sg_cpu = &per_cpu(sugov_cpu, j);
423
-
unsigned long j_util, j_max;
424
418
425
419
sugov_get_util(j_sg_cpu);
426
-
sugov_iowait_apply(j_sg_cpu, time);
427
-
j_util = j_sg_cpu->util;
428
-
j_max = j_sg_cpu->max;
420
+
sugov_iowait_apply(j_sg_cpu, time, max_cap);
429
421
430
-
if (j_util * max > j_max * util) {
431
-
util = j_util;
432
-
max = j_max;
433
-
}
422
+
util = max(j_sg_cpu->util, util);
434
423
}
435
424
436
-
return get_next_freq(sg_policy, util, max);
425
+
return get_next_freq(sg_policy, util, max_cap);
437
426
}
438
427
439
428
static void
+4
kernel/sched/cputime.c
+4
kernel/sched/cputime.c
+28
-16
kernel/sched/deadline.c
+28
-16
kernel/sched/deadline.c
···
2663
2663
static void prio_changed_dl(struct rq *rq, struct task_struct *p,
2664
2664
int oldprio)
2665
2665
{
2666
-
if (task_on_rq_queued(p) || task_current(rq, p)) {
2667
-
#ifdef CONFIG_SMP
2668
-
/*
2669
-
* This might be too much, but unfortunately
2670
-
* we don't have the old deadline value, and
2671
-
* we can't argue if the task is increasing
2672
-
* or lowering its prio, so...
2673
-
*/
2674
-
if (!rq->dl.overloaded)
2675
-
deadline_queue_pull_task(rq);
2666
+
if (!task_on_rq_queued(p))
2667
+
return;
2676
2668
2669
+
#ifdef CONFIG_SMP
2670
+
/*
2671
+
* This might be too much, but unfortunately
2672
+
* we don't have the old deadline value, and
2673
+
* we can't argue if the task is increasing
2674
+
* or lowering its prio, so...
2675
+
*/
2676
+
if (!rq->dl.overloaded)
2677
+
deadline_queue_pull_task(rq);
2678
+
2679
+
if (task_current(rq, p)) {
2677
2680
/*
2678
2681
* If we now have a earlier deadline task than p,
2679
2682
* then reschedule, provided p is still on this
···
2684
2681
*/
2685
2682
if (dl_time_before(rq->dl.earliest_dl.curr, p->dl.deadline))
2686
2683
resched_curr(rq);
2687
-
#else
2684
+
} else {
2688
2685
/*
2689
-
* Again, we don't know if p has a earlier
2690
-
* or later deadline, so let's blindly set a
2691
-
* (maybe not needed) rescheduling point.
2686
+
* Current may not be deadline in case p was throttled but we
2687
+
* have just replenished it (e.g. rt_mutex_setprio()).
2688
+
*
2689
+
* Otherwise, if p was given an earlier deadline, reschedule.
2692
2690
*/
2693
-
resched_curr(rq);
2694
-
#endif /* CONFIG_SMP */
2691
+
if (!dl_task(rq->curr) ||
2692
+
dl_time_before(p->dl.deadline, rq->curr->dl.deadline))
2693
+
resched_curr(rq);
2695
2694
}
2695
+
#else
2696
+
/*
2697
+
* We don't know if p has a earlier or later deadline, so let's blindly
2698
+
* set a (maybe not needed) rescheduling point.
2699
+
*/
2700
+
resched_curr(rq);
2701
+
#endif
2696
2702
}
2697
2703
2698
2704
DEFINE_SCHED_CLASS(dl) = {
+263
-130
kernel/sched/fair.c
+263
-130
kernel/sched/fair.c
···
468
468
return NULL;
469
469
}
470
470
471
-
static inline struct sched_entity *parent_entity(struct sched_entity *se)
471
+
static inline struct sched_entity *parent_entity(const struct sched_entity *se)
472
472
{
473
473
return se->parent;
474
474
}
···
595
595
return min_vruntime;
596
596
}
597
597
598
-
static inline bool entity_before(struct sched_entity *a,
599
-
struct sched_entity *b)
598
+
static inline bool entity_before(const struct sched_entity *a,
599
+
const struct sched_entity *b)
600
600
{
601
601
return (s64)(a->vruntime - b->vruntime) < 0;
602
602
}
···
1804
1804
ns->nr_running += rq->cfs.h_nr_running;
1805
1805
ns->compute_capacity += capacity_of(cpu);
1806
1806
1807
-
if (find_idle && !rq->nr_running && idle_cpu(cpu)) {
1807
+
if (find_idle && idle_core < 0 && !rq->nr_running && idle_cpu(cpu)) {
1808
1808
if (READ_ONCE(rq->numa_migrate_on) ||
1809
1809
!cpumask_test_cpu(cpu, env->p->cpus_ptr))
1810
1810
continue;
···
1836
1836
int start = env->dst_cpu;
1837
1837
1838
1838
/* Find alternative idle CPU. */
1839
-
for_each_cpu_wrap(cpu, cpumask_of_node(env->dst_nid), start) {
1839
+
for_each_cpu_wrap(cpu, cpumask_of_node(env->dst_nid), start + 1) {
1840
1840
if (cpu == env->best_cpu || !idle_cpu(cpu) ||
1841
1841
!cpumask_test_cpu(cpu, env->p->cpus_ptr)) {
1842
1842
continue;
···
4476
4476
*
4477
4477
* For uclamp_max, we can tolerate a drop in performance level as the
4478
4478
* goal is to cap the task. So it's okay if it's getting less.
4479
-
*
4480
-
* In case of capacity inversion we should honour the inverted capacity
4481
-
* for both uclamp_min and uclamp_max all the time.
4482
4479
*/
4483
-
capacity_orig = cpu_in_capacity_inversion(cpu);
4484
-
if (capacity_orig) {
4485
-
capacity_orig_thermal = capacity_orig;
4486
-
} else {
4487
-
capacity_orig = capacity_orig_of(cpu);
4488
-
capacity_orig_thermal = capacity_orig - arch_scale_thermal_pressure(cpu);
4489
-
}
4480
+
capacity_orig = capacity_orig_of(cpu);
4481
+
capacity_orig_thermal = capacity_orig - arch_scale_thermal_pressure(cpu);
4490
4482
4491
4483
/*
4492
4484
* We want to force a task to fit a cpu as implied by uclamp_max.
···
4553
4561
* handle the case uclamp_min > uclamp_max.
4554
4562
*/
4555
4563
uclamp_min = min(uclamp_min, uclamp_max);
4556
-
if (util < uclamp_min && capacity_orig != SCHED_CAPACITY_SCALE)
4557
-
fits = fits && (uclamp_min <= capacity_orig_thermal);
4564
+
if (fits && (util < uclamp_min) && (uclamp_min > capacity_orig_thermal))
4565
+
return -1;
4558
4566
4559
4567
return fits;
4560
4568
}
···
4564
4572
unsigned long uclamp_min = uclamp_eff_value(p, UCLAMP_MIN);
4565
4573
unsigned long uclamp_max = uclamp_eff_value(p, UCLAMP_MAX);
4566
4574
unsigned long util = task_util_est(p);
4567
-
return util_fits_cpu(util, uclamp_min, uclamp_max, cpu);
4575
+
/*
4576
+
* Return true only if the cpu fully fits the task requirements, which
4577
+
* include the utilization but also the performance hints.
4578
+
*/
4579
+
return (util_fits_cpu(util, uclamp_min, uclamp_max, cpu) > 0);
4568
4580
}
4569
4581
4570
4582
static inline void update_misfit_status(struct task_struct *p, struct rq *rq)
···
4652
4656
place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
4653
4657
{
4654
4658
u64 vruntime = cfs_rq->min_vruntime;
4659
+
u64 sleep_time;
4655
4660
4656
4661
/*
4657
4662
* The 'current' period is already promised to the current tasks,
···
4682
4685
vruntime -= thresh;
4683
4686
}
4684
4687
4685
-
/* ensure we never gain time by being placed backwards. */
4686
-
se->vruntime = max_vruntime(se->vruntime, vruntime);
4688
+
/*
4689
+
* Pull vruntime of the entity being placed to the base level of
4690
+
* cfs_rq, to prevent boosting it if placed backwards. If the entity
4691
+
* slept for a long time, don't even try to compare its vruntime with
4692
+
* the base as it may be too far off and the comparison may get
4693
+
* inversed due to s64 overflow.
4694
+
*/
4695
+
sleep_time = rq_clock_task(rq_of(cfs_rq)) - se->exec_start;
4696
+
if ((s64)sleep_time > 60LL * NSEC_PER_SEC)
4697
+
se->vruntime = vruntime;
4698
+
else
4699
+
se->vruntime = max_vruntime(se->vruntime, vruntime);
4687
4700
}
4688
4701
4689
4702
static void check_enqueue_throttle(struct cfs_rq *cfs_rq);
···
4903
4896
struct sched_entity *se;
4904
4897
s64 delta;
4905
4898
4906
-
ideal_runtime = sched_slice(cfs_rq, curr);
4899
+
/*
4900
+
* When many tasks blow up the sched_period; it is possible that
4901
+
* sched_slice() reports unusually large results (when many tasks are
4902
+
* very light for example). Therefore impose a maximum.
4903
+
*/
4904
+
ideal_runtime = min_t(u64, sched_slice(cfs_rq, curr), sysctl_sched_latency);
4905
+
4907
4906
delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime;
4908
4907
if (delta_exec > ideal_runtime) {
4909
4908
resched_curr(rq_of(cfs_rq));
···
5474
5461
resched_curr(rq);
5475
5462
}
5476
5463
5477
-
static void distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
5464
+
#ifdef CONFIG_SMP
5465
+
static void __cfsb_csd_unthrottle(void *arg)
5478
5466
{
5479
-
struct cfs_rq *cfs_rq;
5467
+
struct cfs_rq *cursor, *tmp;
5468
+
struct rq *rq = arg;
5469
+
struct rq_flags rf;
5470
+
5471
+
rq_lock(rq, &rf);
5472
+
5473
+
/*
5474
+
* Since we hold rq lock we're safe from concurrent manipulation of
5475
+
* the CSD list. However, this RCU critical section annotates the
5476
+
* fact that we pair with sched_free_group_rcu(), so that we cannot
5477
+
* race with group being freed in the window between removing it
5478
+
* from the list and advancing to the next entry in the list.
5479
+
*/
5480
+
rcu_read_lock();
5481
+
5482
+
list_for_each_entry_safe(cursor, tmp, &rq->cfsb_csd_list,
5483
+
throttled_csd_list) {
5484
+
list_del_init(&cursor->throttled_csd_list);
5485
+
5486
+
if (cfs_rq_throttled(cursor))
5487
+
unthrottle_cfs_rq(cursor);
5488
+
}
5489
+
5490
+
rcu_read_unlock();
5491
+
5492
+
rq_unlock(rq, &rf);
5493
+
}
5494
+
5495
+
static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
5496
+
{
5497
+
struct rq *rq = rq_of(cfs_rq);
5498
+
bool first;
5499
+
5500
+
if (rq == this_rq()) {
5501
+
unthrottle_cfs_rq(cfs_rq);
5502
+
return;
5503
+
}
5504
+
5505
+
/* Already enqueued */
5506
+
if (SCHED_WARN_ON(!list_empty(&cfs_rq->throttled_csd_list)))
5507
+
return;
5508
+
5509
+
first = list_empty(&rq->cfsb_csd_list);
5510
+
list_add_tail(&cfs_rq->throttled_csd_list, &rq->cfsb_csd_list);
5511
+
if (first)
5512
+
smp_call_function_single_async(cpu_of(rq), &rq->cfsb_csd);
5513
+
}
5514
+
#else
5515
+
static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
5516
+
{
5517
+
unthrottle_cfs_rq(cfs_rq);
5518
+
}
5519
+
#endif
5520
+
5521
+
static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
5522
+
{
5523
+
lockdep_assert_rq_held(rq_of(cfs_rq));
5524
+
5525
+
if (SCHED_WARN_ON(!cfs_rq_throttled(cfs_rq) ||
5526
+
cfs_rq->runtime_remaining <= 0))
5527
+
return;
5528
+
5529
+
__unthrottle_cfs_rq_async(cfs_rq);
5530
+
}
5531
+
5532
+
static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
5533
+
{
5534
+
struct cfs_rq *local_unthrottle = NULL;
5535
+
int this_cpu = smp_processor_id();
5480
5536
u64 runtime, remaining = 1;
5537
+
bool throttled = false;
5538
+
struct cfs_rq *cfs_rq;
5539
+
struct rq_flags rf;
5540
+
struct rq *rq;
5481
5541
5482
5542
rcu_read_lock();
5483
5543
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
5484
5544
throttled_list) {
5485
-
struct rq *rq = rq_of(cfs_rq);
5486
-
struct rq_flags rf;
5545
+
rq = rq_of(cfs_rq);
5546
+
5547
+
if (!remaining) {
5548
+
throttled = true;
5549
+
break;
5550
+
}
5487
5551
5488
5552
rq_lock_irqsave(rq, &rf);
5489
5553
if (!cfs_rq_throttled(cfs_rq))
5490
5554
goto next;
5491
5555
5492
-
/* By the above check, this should never be true */
5556
+
#ifdef CONFIG_SMP
5557
+
/* Already queued for async unthrottle */
5558
+
if (!list_empty(&cfs_rq->throttled_csd_list))
5559
+
goto next;
5560
+
#endif
5561
+
5562
+
/* By the above checks, this should never be true */
5493
5563
SCHED_WARN_ON(cfs_rq->runtime_remaining > 0);
5494
5564
5495
5565
raw_spin_lock(&cfs_b->lock);
···
5586
5490
cfs_rq->runtime_remaining += runtime;
5587
5491
5588
5492
/* we check whether we're throttled above */
5589
-
if (cfs_rq->runtime_remaining > 0)
5590
-
unthrottle_cfs_rq(cfs_rq);
5493
+
if (cfs_rq->runtime_remaining > 0) {
5494
+
if (cpu_of(rq) != this_cpu ||
5495
+
SCHED_WARN_ON(local_unthrottle))
5496
+
unthrottle_cfs_rq_async(cfs_rq);
5497
+
else
5498
+
local_unthrottle = cfs_rq;
5499
+
} else {
5500
+
throttled = true;
5501
+
}
5591
5502
5592
5503
next:
5593
5504
rq_unlock_irqrestore(rq, &rf);
5594
-
5595
-
if (!remaining)
5596
-
break;
5597
5505
}
5598
5506
rcu_read_unlock();
5507
+
5508
+
if (local_unthrottle) {
5509
+
rq = cpu_rq(this_cpu);
5510
+
rq_lock_irqsave(rq, &rf);
5511
+
if (cfs_rq_throttled(local_unthrottle))
5512
+
unthrottle_cfs_rq(local_unthrottle);
5513
+
rq_unlock_irqrestore(rq, &rf);
5514
+
}
5515
+
5516
+
return throttled;
5599
5517
}
5600
5518
5601
5519
/*
···
5654
5544
while (throttled && cfs_b->runtime > 0) {
5655
5545
raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
5656
5546
/* we can't nest cfs_b->lock while distributing bandwidth */
5657
-
distribute_cfs_runtime(cfs_b);
5547
+
throttled = distribute_cfs_runtime(cfs_b);
5658
5548
raw_spin_lock_irqsave(&cfs_b->lock, flags);
5659
-
5660
-
throttled = !list_empty(&cfs_b->throttled_cfs_rq);
5661
5549
}
5662
5550
5663
5551
/*
···
5932
5824
{
5933
5825
cfs_rq->runtime_enabled = 0;
5934
5826
INIT_LIST_HEAD(&cfs_rq->throttled_list);
5827
+
#ifdef CONFIG_SMP
5828
+
INIT_LIST_HEAD(&cfs_rq->throttled_csd_list);
5829
+
#endif
5935
5830
}
5936
5831
5937
5832
void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
···
5951
5840
5952
5841
static void destroy_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
5953
5842
{
5843
+
int __maybe_unused i;
5844
+
5954
5845
/* init_cfs_bandwidth() was not called */
5955
5846
if (!cfs_b->throttled_cfs_rq.next)
5956
5847
return;
5957
5848
5958
5849
hrtimer_cancel(&cfs_b->period_timer);
5959
5850
hrtimer_cancel(&cfs_b->slack_timer);
5851
+
5852
+
/*
5853
+
* It is possible that we still have some cfs_rq's pending on a CSD
5854
+
* list, though this race is very rare. In order for this to occur, we
5855
+
* must have raced with the last task leaving the group while there
5856
+
* exist throttled cfs_rq(s), and the period_timer must have queued the
5857
+
* CSD item but the remote cpu has not yet processed it. To handle this,
5858
+
* we can simply flush all pending CSD work inline here. We're
5859
+
* guaranteed at this point that no additional cfs_rq of this group can
5860
+
* join a CSD list.
5861
+
*/
5862
+
#ifdef CONFIG_SMP
5863
+
for_each_possible_cpu(i) {
5864
+
struct rq *rq = cpu_rq(i);
5865
+
unsigned long flags;
5866
+
5867
+
if (list_empty(&rq->cfsb_csd_list))
5868
+
continue;
5869
+
5870
+
local_irq_save(flags);
5871
+
__cfsb_csd_unthrottle(rq);
5872
+
local_irq_restore(flags);
5873
+
}
5874
+
#endif
5960
5875
}
5961
5876
5962
5877
/*
···
6145
6008
unsigned long rq_util_min = uclamp_rq_get(cpu_rq(cpu), UCLAMP_MIN);
6146
6009
unsigned long rq_util_max = uclamp_rq_get(cpu_rq(cpu), UCLAMP_MAX);
6147
6010
6011
+
/* Return true only if the utilization doesn't fit CPU's capacity */
6148
6012
return !util_fits_cpu(cpu_util_cfs(cpu), rq_util_min, rq_util_max, cpu);
6149
6013
}
6150
6014
···
6939
6801
select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
6940
6802
{
6941
6803
unsigned long task_util, util_min, util_max, best_cap = 0;
6804
+
int fits, best_fits = 0;
6942
6805
int cpu, best_cpu = -1;
6943
6806
struct cpumask *cpus;
6944
6807
···
6950
6811
util_min = uclamp_eff_value(p, UCLAMP_MIN);
6951
6812
util_max = uclamp_eff_value(p, UCLAMP_MAX);
6952
6813
6953
-
for_each_cpu_wrap(cpu, cpus, target) {
6814
+
for_each_cpu_wrap(cpu, cpus, target + 1) {
6954
6815
unsigned long cpu_cap = capacity_of(cpu);
6955
6816
6956
6817
if (!available_idle_cpu(cpu) && !sched_idle_cpu(cpu))
6957
6818
continue;
6958
-
if (util_fits_cpu(task_util, util_min, util_max, cpu))
6959
-
return cpu;
6960
6819
6961
-
if (cpu_cap > best_cap) {
6820
+
fits = util_fits_cpu(task_util, util_min, util_max, cpu);
6821
+
6822
+
/* This CPU fits with all requirements */
6823
+
if (fits > 0)
6824
+
return cpu;
6825
+
/*
6826
+
* Only the min performance hint (i.e. uclamp_min) doesn't fit.
6827
+
* Look for the CPU with best capacity.
6828
+
*/
6829
+
else if (fits < 0)
6830
+
cpu_cap = capacity_orig_of(cpu) - thermal_load_avg(cpu_rq(cpu));
6831
+
6832
+
/*
6833
+
* First, select CPU which fits better (-1 being better than 0).
6834
+
* Then, select the one with best capacity at same level.
6835
+
*/
6836
+
if ((fits < best_fits) ||
6837
+
((fits == best_fits) && (cpu_cap > best_cap))) {
6962
6838
best_cap = cpu_cap;
6963
6839
best_cpu = cpu;
6840
+
best_fits = fits;
6964
6841
}
6965
6842
}
6966
6843
···
6989
6834
int cpu)
6990
6835
{
6991
6836
if (sched_asym_cpucap_active())
6992
-
return util_fits_cpu(util, util_min, util_max, cpu);
6837
+
/*
6838
+
* Return true only if the cpu fully fits the task requirements
6839
+
* which include the utilization and the performance hints.
6840
+
*/
6841
+
return (util_fits_cpu(util, util_min, util_max, cpu) > 0);
6993
6842
6994
6843
return true;
6995
6844
}
···
7360
7201
unsigned long p_util_max = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MAX) : 1024;
7361
7202
struct root_domain *rd = this_rq()->rd;
7362
7203
int cpu, best_energy_cpu, target = -1;
7204
+
int prev_fits = -1, best_fits = -1;
7205
+
unsigned long best_thermal_cap = 0;
7206
+
unsigned long prev_thermal_cap = 0;
7363
7207
struct sched_domain *sd;
7364
7208
struct perf_domain *pd;
7365
7209
struct energy_env eenv;
···
7398
7236
unsigned long prev_spare_cap = 0;
7399
7237
int max_spare_cap_cpu = -1;
7400
7238
unsigned long base_energy;
7239
+
int fits, max_fits = -1;
7401
7240
7402
7241
cpumask_and(cpus, perf_domain_span(pd), cpu_online_mask);
7403
7242
···
7448
7285
util_min = max(rq_util_min, p_util_min);
7449
7286
util_max = max(rq_util_max, p_util_max);
7450
7287
}
7451
-
if (!util_fits_cpu(util, util_min, util_max, cpu))
7288
+
7289
+
fits = util_fits_cpu(util, util_min, util_max, cpu);
7290
+
if (!fits)
7452
7291
continue;
7453
7292
7454
7293
lsub_positive(&cpu_cap, util);
···
7458
7293
if (cpu == prev_cpu) {
7459
7294
/* Always use prev_cpu as a candidate. */
7460
7295
prev_spare_cap = cpu_cap;
7461
-
} else if (cpu_cap > max_spare_cap) {
7296
+
prev_fits = fits;
7297
+
} else if ((fits > max_fits) ||
7298
+
((fits == max_fits) && (cpu_cap > max_spare_cap))) {
7462
7299
/*
7463
7300
* Find the CPU with the maximum spare capacity
7464
7301
* among the remaining CPUs in the performance
···
7468
7301
*/
7469
7302
max_spare_cap = cpu_cap;
7470
7303
max_spare_cap_cpu = cpu;
7304
+
max_fits = fits;
7471
7305
}
7472
7306
}
7473
7307
···
7487
7319
if (prev_delta < base_energy)
7488
7320
goto unlock;
7489
7321
prev_delta -= base_energy;
7322
+
prev_thermal_cap = cpu_thermal_cap;
7490
7323
best_delta = min(best_delta, prev_delta);
7491
7324
}
7492
7325
7493
7326
/* Evaluate the energy impact of using max_spare_cap_cpu. */
7494
7327
if (max_spare_cap_cpu >= 0 && max_spare_cap > prev_spare_cap) {
7328
+
/* Current best energy cpu fits better */
7329
+
if (max_fits < best_fits)
7330
+
continue;
7331
+
7332
+
/*
7333
+
* Both don't fit performance hint (i.e. uclamp_min)
7334
+
* but best energy cpu has better capacity.
7335
+
*/
7336
+
if ((max_fits < 0) &&
7337
+
(cpu_thermal_cap <= best_thermal_cap))
7338
+
continue;
7339
+
7495
7340
cur_delta = compute_energy(&eenv, pd, cpus, p,
7496
7341
max_spare_cap_cpu);
7497
7342
/* CPU utilization has changed */
7498
7343
if (cur_delta < base_energy)
7499
7344
goto unlock;
7500
7345
cur_delta -= base_energy;
7501
-
if (cur_delta < best_delta) {
7502
-
best_delta = cur_delta;
7503
-
best_energy_cpu = max_spare_cap_cpu;
7504
-
}
7346
+
7347
+
/*
7348
+
* Both fit for the task but best energy cpu has lower
7349
+
* energy impact.
7350
+
*/
7351
+
if ((max_fits > 0) && (best_fits > 0) &&
7352
+
(cur_delta >= best_delta))
7353
+
continue;
7354
+
7355
+
best_delta = cur_delta;
7356
+
best_energy_cpu = max_spare_cap_cpu;
7357
+
best_fits = max_fits;
7358
+
best_thermal_cap = cpu_thermal_cap;
7505
7359
}
7506
7360
}
7507
7361
rcu_read_unlock();
7508
7362
7509
-
if (best_delta < prev_delta)
7363
+
if ((best_fits > prev_fits) ||
7364
+
((best_fits > 0) && (best_delta < prev_delta)) ||
7365
+
((best_fits < 0) && (best_thermal_cap > prev_thermal_cap)))
7510
7366
target = best_energy_cpu;
7511
7367
7512
7368
return target;
···
9030
8838
9031
8839
static void update_cpu_capacity(struct sched_domain *sd, int cpu)
9032
8840
{
9033
-
unsigned long capacity_orig = arch_scale_cpu_capacity(cpu);
9034
8841
unsigned long capacity = scale_rt_capacity(cpu);
9035
8842
struct sched_group *sdg = sd->groups;
9036
-
struct rq *rq = cpu_rq(cpu);
9037
8843
9038
-
rq->cpu_capacity_orig = capacity_orig;
8844
+
cpu_rq(cpu)->cpu_capacity_orig = arch_scale_cpu_capacity(cpu);
9039
8845
9040
8846
if (!capacity)
9041
8847
capacity = 1;
9042
8848
9043
-
rq->cpu_capacity = capacity;
9044
-
9045
-
/*
9046
-
* Detect if the performance domain is in capacity inversion state.
9047
-
*
9048
-
* Capacity inversion happens when another perf domain with equal or
9049
-
* lower capacity_orig_of() ends up having higher capacity than this
9050
-
* domain after subtracting thermal pressure.
9051
-
*
9052
-
* We only take into account thermal pressure in this detection as it's
9053
-
* the only metric that actually results in *real* reduction of
9054
-
* capacity due to performance points (OPPs) being dropped/become
9055
-
* unreachable due to thermal throttling.
9056
-
*
9057
-
* We assume:
9058
-
* * That all cpus in a perf domain have the same capacity_orig
9059
-
* (same uArch).
9060
-
* * Thermal pressure will impact all cpus in this perf domain
9061
-
* equally.
9062
-
*/
9063
-
if (sched_energy_enabled()) {
9064
-
unsigned long inv_cap = capacity_orig - thermal_load_avg(rq);
9065
-
struct perf_domain *pd;
9066
-
9067
-
rcu_read_lock();
9068
-
9069
-
pd = rcu_dereference(rq->rd->pd);
9070
-
rq->cpu_capacity_inverted = 0;
9071
-
9072
-
for (; pd; pd = pd->next) {
9073
-
struct cpumask *pd_span = perf_domain_span(pd);
9074
-
unsigned long pd_cap_orig, pd_cap;
9075
-
9076
-
/* We can't be inverted against our own pd */
9077
-
if (cpumask_test_cpu(cpu_of(rq), pd_span))
9078
-
continue;
9079
-
9080
-
cpu = cpumask_any(pd_span);
9081
-
pd_cap_orig = arch_scale_cpu_capacity(cpu);
9082
-
9083
-
if (capacity_orig < pd_cap_orig)
9084
-
continue;
9085
-
9086
-
/*
9087
-
* handle the case of multiple perf domains have the
9088
-
* same capacity_orig but one of them is under higher
9089
-
* thermal pressure. We record it as capacity
9090
-
* inversion.
9091
-
*/
9092
-
if (capacity_orig == pd_cap_orig) {
9093
-
pd_cap = pd_cap_orig - thermal_load_avg(cpu_rq(cpu));
9094
-
9095
-
if (pd_cap > inv_cap) {
9096
-
rq->cpu_capacity_inverted = inv_cap;
9097
-
break;
9098
-
}
9099
-
} else if (pd_cap_orig > inv_cap) {
9100
-
rq->cpu_capacity_inverted = inv_cap;
9101
-
break;
9102
-
}
9103
-
}
9104
-
9105
-
rcu_read_unlock();
9106
-
}
9107
-
9108
-
trace_sched_cpu_capacity_tp(rq);
8849
+
cpu_rq(cpu)->cpu_capacity = capacity;
8850
+
trace_sched_cpu_capacity_tp(cpu_rq(cpu));
9109
8851
9110
8852
sdg->sgc->capacity = capacity;
9111
8853
sdg->sgc->min_capacity = capacity;
···
10267
10141
*/
10268
10142
update_sd_lb_stats(env, &sds);
10269
10143
10144
+
/* There is no busy sibling group to pull tasks from */
10145
+
if (!sds.busiest)
10146
+
goto out_balanced;
10147
+
10148
+
busiest = &sds.busiest_stat;
10149
+
10150
+
/* Misfit tasks should be dealt with regardless of the avg load */
10151
+
if (busiest->group_type == group_misfit_task)
10152
+
goto force_balance;
10153
+
10270
10154
if (sched_energy_enabled()) {
10271
10155
struct root_domain *rd = env->dst_rq->rd;
10272
10156
10273
10157
if (rcu_dereference(rd->pd) && !READ_ONCE(rd->overutilized))
10274
10158
goto out_balanced;
10275
10159
}
10276
-
10277
-
local = &sds.local_stat;
10278
-
busiest = &sds.busiest_stat;
10279
-
10280
-
/* There is no busy sibling group to pull tasks from */
10281
-
if (!sds.busiest)
10282
-
goto out_balanced;
10283
-
10284
-
/* Misfit tasks should be dealt with regardless of the avg load */
10285
-
if (busiest->group_type == group_misfit_task)
10286
-
goto force_balance;
10287
10160
10288
10161
/* ASYM feature bypasses nice load balance check */
10289
10162
if (busiest->group_type == group_asym_packing)
···
10296
10171
if (busiest->group_type == group_imbalanced)
10297
10172
goto force_balance;
10298
10173
10174
+
local = &sds.local_stat;
10299
10175
/*
10300
10176
* If the local group is busier than the selected busiest group
10301
10177
* don't try and pull any tasks.
···
11860
11734
/*
11861
11735
* se_fi_update - Update the cfs_rq->min_vruntime_fi in a CFS hierarchy if needed.
11862
11736
*/
11863
-
static void se_fi_update(struct sched_entity *se, unsigned int fi_seq, bool forceidle)
11737
+
static void se_fi_update(const struct sched_entity *se, unsigned int fi_seq,
11738
+
bool forceidle)
11864
11739
{
11865
11740
for_each_sched_entity(se) {
11866
11741
struct cfs_rq *cfs_rq = cfs_rq_of(se);
···
11886
11759
se_fi_update(se, rq->core->core_forceidle_seq, in_fi);
11887
11760
}
11888
11761
11889
-
bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool in_fi)
11762
+
bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b,
11763
+
bool in_fi)
11890
11764
{
11891
11765
struct rq *rq = task_rq(a);
11892
-
struct sched_entity *sea = &a->se;
11893
-
struct sched_entity *seb = &b->se;
11766
+
const struct sched_entity *sea = &a->se;
11767
+
const struct sched_entity *seb = &b->se;
11894
11768
struct cfs_rq *cfs_rqa;
11895
11769
struct cfs_rq *cfs_rqb;
11896
11770
s64 delta;
···
12608
12480
for_each_possible_cpu(i) {
12609
12481
zalloc_cpumask_var_node(&per_cpu(load_balance_mask, i), GFP_KERNEL, cpu_to_node(i));
12610
12482
zalloc_cpumask_var_node(&per_cpu(select_rq_mask, i), GFP_KERNEL, cpu_to_node(i));
12483
+
12484
+
#ifdef CONFIG_CFS_BANDWIDTH
12485
+
INIT_CSD(&cpu_rq(i)->cfsb_csd, __cfsb_csd_unthrottle, cpu_rq(i));
12486
+
INIT_LIST_HEAD(&cpu_rq(i)->cfsb_csd_list);
12487
+
#endif
12611
12488
}
12612
12489
12613
12490
open_softirq(SCHED_SOFTIRQ, run_rebalance_domains);
+13
-34
kernel/sched/idle.c
+13
-34
kernel/sched/idle.c
···
51
51
52
52
static noinline int __cpuidle cpu_idle_poll(void)
53
53
{
54
+
instrumentation_begin();
54
55
trace_cpu_idle(0, smp_processor_id());
55
56
stop_critical_timings();
56
-
ct_idle_enter();
57
-
local_irq_enable();
57
+
ct_cpuidle_enter();
58
58
59
+
raw_local_irq_enable();
59
60
while (!tif_need_resched() &&
60
61
(cpu_idle_force_poll || tick_check_broadcast_expired()))
61
62
cpu_relax();
63
+
raw_local_irq_disable();
62
64
63
-
ct_idle_exit();
65
+
ct_cpuidle_exit();
64
66
start_critical_timings();
65
67
trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id());
68
+
local_irq_enable();
69
+
instrumentation_end();
66
70
67
71
return 1;
68
72
}
···
79
75
void __weak arch_cpu_idle(void)
80
76
{
81
77
cpu_idle_force_poll = 1;
82
-
raw_local_irq_enable();
83
78
}
84
79
85
80
/**
···
88
85
*/
89
86
void __cpuidle default_idle_call(void)
90
87
{
91
-
if (current_clr_polling_and_test()) {
92
-
local_irq_enable();
93
-
} else {
94
-
88
+
instrumentation_begin();
89
+
if (!current_clr_polling_and_test()) {
95
90
trace_cpu_idle(1, smp_processor_id());
96
91
stop_critical_timings();
97
92
98
-
/*
99
-
* arch_cpu_idle() is supposed to enable IRQs, however
100
-
* we can't do that because of RCU and tracing.
101
-
*
102
-
* Trace IRQs enable here, then switch off RCU, and have
103
-
* arch_cpu_idle() use raw_local_irq_enable(). Note that
104
-
* ct_idle_enter() relies on lockdep IRQ state, so switch that
105
-
* last -- this is very similar to the entry code.
106
-
*/
107
-
trace_hardirqs_on_prepare();
108
-
lockdep_hardirqs_on_prepare();
109
-
ct_idle_enter();
110
-
lockdep_hardirqs_on(_THIS_IP_);
111
-
93
+
ct_cpuidle_enter();
112
94
arch_cpu_idle();
113
-
114
-
/*
115
-
* OK, so IRQs are enabled here, but RCU needs them disabled to
116
-
* turn itself back on.. funny thing is that disabling IRQs
117
-
* will cause tracing, which needs RCU. Jump through hoops to
118
-
* make it 'work'.
119
-
*/
120
-
raw_local_irq_disable();
121
-
lockdep_hardirqs_off(_THIS_IP_);
122
-
ct_idle_exit();
123
-
lockdep_hardirqs_on(_THIS_IP_);
124
-
raw_local_irq_enable();
95
+
ct_cpuidle_exit();
125
96
126
97
start_critical_timings();
127
98
trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id());
128
99
}
100
+
local_irq_enable();
101
+
instrumentation_end();
129
102
}
130
103
131
104
static int call_cpuidle_s2idle(struct cpuidle_driver *drv,
+38
-1
kernel/sched/membarrier.c
+38
-1
kernel/sched/membarrier.c
···
159
159
| MEMBARRIER_CMD_PRIVATE_EXPEDITED \
160
160
| MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED \
161
161
| MEMBARRIER_PRIVATE_EXPEDITED_SYNC_CORE_BITMASK \
162
-
| MEMBARRIER_PRIVATE_EXPEDITED_RSEQ_BITMASK)
162
+
| MEMBARRIER_PRIVATE_EXPEDITED_RSEQ_BITMASK \
163
+
| MEMBARRIER_CMD_GET_REGISTRATIONS)
163
164
164
165
static void ipi_mb(void *info)
165
166
{
···
541
540
return 0;
542
541
}
543
542
543
+
static int membarrier_get_registrations(void)
544
+
{
545
+
struct task_struct *p = current;
546
+
struct mm_struct *mm = p->mm;
547
+
int registrations_mask = 0, membarrier_state, i;
548
+
static const int states[] = {
549
+
MEMBARRIER_STATE_GLOBAL_EXPEDITED |
550
+
MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY,
551
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED |
552
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY,
553
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE |
554
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY,
555
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ |
556
+
MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY
557
+
};
558
+
static const int registration_cmds[] = {
559
+
MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED,
560
+
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
561
+
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
562
+
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ
563
+
};
564
+
BUILD_BUG_ON(ARRAY_SIZE(states) != ARRAY_SIZE(registration_cmds));
565
+
566
+
membarrier_state = atomic_read(&mm->membarrier_state);
567
+
for (i = 0; i < ARRAY_SIZE(states); ++i) {
568
+
if (membarrier_state & states[i]) {
569
+
registrations_mask |= registration_cmds[i];
570
+
membarrier_state &= ~states[i];
571
+
}
572
+
}
573
+
WARN_ON_ONCE(membarrier_state != 0);
574
+
return registrations_mask;
575
+
}
576
+
544
577
/**
545
578
* sys_membarrier - issue memory barriers on a set of threads
546
579
* @cmd: Takes command values defined in enum membarrier_cmd.
···
658
623
return membarrier_private_expedited(MEMBARRIER_FLAG_RSEQ, cpu_id);
659
624
case MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ:
660
625
return membarrier_register_private_expedited(MEMBARRIER_FLAG_RSEQ);
626
+
case MEMBARRIER_CMD_GET_REGISTRATIONS:
627
+
return membarrier_get_registrations();
661
628
default:
662
629
return -EINVAL;
663
630
}
+4
-1
kernel/sched/rt.c
+4
-1
kernel/sched/rt.c
···
1777
1777
BUG_ON(idx >= MAX_RT_PRIO);
1778
1778
1779
1779
queue = array->queue + idx;
1780
+
if (SCHED_WARN_ON(list_empty(queue)))
1781
+
return NULL;
1780
1782
next = list_entry(queue->next, struct sched_rt_entity, run_list);
1781
1783
1782
1784
return next;
···
1791
1789
1792
1790
do {
1793
1791
rt_se = pick_next_rt_entity(rt_rq);
1794
-
BUG_ON(!rt_se);
1792
+
if (unlikely(!rt_se))
1793
+
return NULL;
1795
1794
rt_rq = group_rt_rq(rt_se);
1796
1795
} while (rt_rq);
1797
1796
+76
-31
kernel/sched/sched.h
+76
-31
kernel/sched/sched.h
···
248
248
249
249
#define SCHED_DL_FLAGS (SCHED_FLAG_RECLAIM | SCHED_FLAG_DL_OVERRUN | SCHED_FLAG_SUGOV)
250
250
251
-
static inline bool dl_entity_is_special(struct sched_dl_entity *dl_se)
251
+
static inline bool dl_entity_is_special(const struct sched_dl_entity *dl_se)
252
252
{
253
253
#ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL
254
254
return unlikely(dl_se->flags & SCHED_FLAG_SUGOV);
···
260
260
/*
261
261
* Tells if entity @a should preempt entity @b.
262
262
*/
263
-
static inline bool
264
-
dl_entity_preempt(struct sched_dl_entity *a, struct sched_dl_entity *b)
263
+
static inline bool dl_entity_preempt(const struct sched_dl_entity *a,
264
+
const struct sched_dl_entity *b)
265
265
{
266
266
return dl_entity_is_special(a) ||
267
267
dl_time_before(a->deadline, b->deadline);
···
645
645
int throttled;
646
646
int throttle_count;
647
647
struct list_head throttled_list;
648
+
#ifdef CONFIG_SMP
649
+
struct list_head throttled_csd_list;
650
+
#endif
648
651
#endif /* CONFIG_CFS_BANDWIDTH */
649
652
#endif /* CONFIG_FAIR_GROUP_SCHED */
650
653
};
···
1044
1041
1045
1042
unsigned long cpu_capacity;
1046
1043
unsigned long cpu_capacity_orig;
1047
-
unsigned long cpu_capacity_inverted;
1048
1044
1049
1045
struct balance_callback *balance_callback;
1050
1046
···
1156
1154
1157
1155
/* Scratch cpumask to be temporarily used under rq_lock */
1158
1156
cpumask_var_t scratch_mask;
1157
+
1158
+
#if defined(CONFIG_CFS_BANDWIDTH) && defined(CONFIG_SMP)
1159
+
call_single_data_t cfsb_csd;
1160
+
struct list_head cfsb_csd_list;
1161
+
#endif
1159
1162
};
1160
1163
1161
1164
#ifdef CONFIG_FAIR_GROUP_SCHED
···
1243
1236
return &rq->__lock;
1244
1237
}
1245
1238
1246
-
bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool fi);
1239
+
bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b,
1240
+
bool fi);
1247
1241
1248
1242
/*
1249
1243
* Helpers to check if the CPU's core cookie matches with the task's cookie
···
1423
1415
}
1424
1416
1425
1417
/* runqueue on which this entity is (to be) queued */
1426
-
static inline struct cfs_rq *cfs_rq_of(struct sched_entity *se)
1418
+
static inline struct cfs_rq *cfs_rq_of(const struct sched_entity *se)
1427
1419
{
1428
1420
return se->cfs_rq;
1429
1421
}
···
1436
1428
1437
1429
#else
1438
1430
1439
-
static inline struct task_struct *task_of(struct sched_entity *se)
1440
-
{
1441
-
return container_of(se, struct task_struct, se);
1442
-
}
1431
+
#define task_of(_se) container_of(_se, struct task_struct, se)
1443
1432
1444
-
static inline struct cfs_rq *task_cfs_rq(struct task_struct *p)
1433
+
static inline struct cfs_rq *task_cfs_rq(const struct task_struct *p)
1445
1434
{
1446
1435
return &task_rq(p)->cfs;
1447
1436
}
1448
1437
1449
-
static inline struct cfs_rq *cfs_rq_of(struct sched_entity *se)
1438
+
static inline struct cfs_rq *cfs_rq_of(const struct sched_entity *se)
1450
1439
{
1451
-
struct task_struct *p = task_of(se);
1440
+
const struct task_struct *p = task_of(se);
1452
1441
struct rq *rq = task_rq(p);
1453
1442
1454
1443
return &rq->cfs;
···
2898
2893
return cpu_rq(cpu)->cpu_capacity_orig;
2899
2894
}
2900
2895
2901
-
/*
2902
-
* Returns inverted capacity if the CPU is in capacity inversion state.
2903
-
* 0 otherwise.
2904
-
*
2905
-
* Capacity inversion detection only considers thermal impact where actual
2906
-
* performance points (OPPs) gets dropped.
2907
-
*
2908
-
* Capacity inversion state happens when another performance domain that has
2909
-
* equal or lower capacity_orig_of() becomes effectively larger than the perf
2910
-
* domain this CPU belongs to due to thermal pressure throttling it hard.
2911
-
*
2912
-
* See comment in update_cpu_capacity().
2913
-
*/
2914
-
static inline unsigned long cpu_in_capacity_inversion(int cpu)
2915
-
{
2916
-
return cpu_rq(cpu)->cpu_capacity_inverted;
2917
-
}
2918
-
2919
2896
/**
2920
2897
* enum cpu_util_type - CPU utilization type
2921
2898
* @FREQUENCY_UTIL: Utilization used to select frequency
···
3247
3260
curr->se.exec_start = now;
3248
3261
cgroup_account_cputime(curr, delta_exec);
3249
3262
}
3263
+
3264
+
#ifdef CONFIG_SCHED_MM_CID
3265
+
static inline int __mm_cid_get(struct mm_struct *mm)
3266
+
{
3267
+
struct cpumask *cpumask;
3268
+
int cid;
3269
+
3270
+
cpumask = mm_cidmask(mm);
3271
+
cid = cpumask_first_zero(cpumask);
3272
+
if (cid >= nr_cpu_ids)
3273
+
return -1;
3274
+
__cpumask_set_cpu(cid, cpumask);
3275
+
return cid;
3276
+
}
3277
+
3278
+
static inline void mm_cid_put(struct mm_struct *mm, int cid)
3279
+
{
3280
+
lockdep_assert_irqs_disabled();
3281
+
if (cid < 0)
3282
+
return;
3283
+
raw_spin_lock(&mm->cid_lock);
3284
+
__cpumask_clear_cpu(cid, mm_cidmask(mm));
3285
+
raw_spin_unlock(&mm->cid_lock);
3286
+
}
3287
+
3288
+
static inline int mm_cid_get(struct mm_struct *mm)
3289
+
{
3290
+
int ret;
3291
+
3292
+
lockdep_assert_irqs_disabled();
3293
+
raw_spin_lock(&mm->cid_lock);
3294
+
ret = __mm_cid_get(mm);
3295
+
raw_spin_unlock(&mm->cid_lock);
3296
+
return ret;
3297
+
}
3298
+
3299
+
static inline void switch_mm_cid(struct task_struct *prev, struct task_struct *next)
3300
+
{
3301
+
if (prev->mm_cid_active) {
3302
+
if (next->mm_cid_active && next->mm == prev->mm) {
3303
+
/*
3304
+
* Context switch between threads in same mm, hand over
3305
+
* the mm_cid from prev to next.
3306
+
*/
3307
+
next->mm_cid = prev->mm_cid;
3308
+
prev->mm_cid = -1;
3309
+
return;
3310
+
}
3311
+
mm_cid_put(prev->mm, prev->mm_cid);
3312
+
prev->mm_cid = -1;
3313
+
}
3314
+
if (next->mm_cid_active)
3315
+
next->mm_cid = mm_cid_get(next->mm);
3316
+
}
3317
+
3318
+
#else
3319
+
static inline void switch_mm_cid(struct task_struct *prev, struct task_struct *next) { }
3320
+
#endif
3250
3321
3251
3322
#endif /* _KERNEL_SCHED_SCHED_H */
+2
-2
kernel/sched/topology.c
+2
-2
kernel/sched/topology.c
···
578
578
*/
579
579
struct root_domain def_root_domain;
580
580
581
-
void init_defrootdomain(void)
581
+
void __init init_defrootdomain(void)
582
582
{
583
583
init_rootdomain(&def_root_domain);
584
584
···
2451
2451
* Set up scheduler domains and groups. For now this just excludes isolated
2452
2452
* CPUs, but could be used to exclude other special cases in the future.
2453
2453
*/
2454
-
int sched_init_domains(const struct cpumask *cpu_map)
2454
+
int __init sched_init_domains(const struct cpumask *cpu_map)
2455
2455
{
2456
2456
int err;
2457
2457
+2
kernel/signal.c
+2
kernel/signal.c
···
2951
2951
cgroup_threadgroup_change_begin(tsk);
2952
2952
2953
2953
if (thread_group_empty(tsk) || (tsk->signal->flags & SIGNAL_GROUP_EXIT)) {
2954
+
sched_mm_cid_exit_signals(tsk);
2954
2955
tsk->flags |= PF_EXITING;
2955
2956
cgroup_threadgroup_change_end(tsk);
2956
2957
return;
···
2962
2961
* From now this task is not visible for group-wide signals,
2963
2962
* see wants_signal(), do_signal_stop().
2964
2963
*/
2964
+
sched_mm_cid_exit_signals(tsk);
2965
2965
tsk->flags |= PF_EXITING;
2966
2966
2967
2967
cgroup_threadgroup_change_end(tsk);
+13
-18
kernel/time/tick-broadcast-hrtimer.c
+13
-18
kernel/time/tick-broadcast-hrtimer.c
···
56
56
* hrtimer callback function is currently running, then
57
57
* hrtimer_start() cannot move it and the timer stays on the CPU on
58
58
* which it is assigned at the moment.
59
-
*
60
-
* As this can be called from idle code, the hrtimer_start()
61
-
* invocation has to be wrapped with RCU_NONIDLE() as
62
-
* hrtimer_start() can call into tracing.
63
59
*/
64
-
RCU_NONIDLE( {
65
-
hrtimer_start(&bctimer, expires, HRTIMER_MODE_ABS_PINNED_HARD);
66
-
/*
67
-
* The core tick broadcast mode expects bc->bound_on to be set
68
-
* correctly to prevent a CPU which has the broadcast hrtimer
69
-
* armed from going deep idle.
70
-
*
71
-
* As tick_broadcast_lock is held, nothing can change the cpu
72
-
* base which was just established in hrtimer_start() above. So
73
-
* the below access is safe even without holding the hrtimer
74
-
* base lock.
75
-
*/
76
-
bc->bound_on = bctimer.base->cpu_base->cpu;
77
-
} );
60
+
hrtimer_start(&bctimer, expires, HRTIMER_MODE_ABS_PINNED_HARD);
61
+
/*
62
+
* The core tick broadcast mode expects bc->bound_on to be set
63
+
* correctly to prevent a CPU which has the broadcast hrtimer
64
+
* armed from going deep idle.
65
+
*
66
+
* As tick_broadcast_lock is held, nothing can change the cpu
67
+
* base which was just established in hrtimer_start() above. So
68
+
* the below access is safe even without holding the hrtimer
69
+
* base lock.
70
+
*/
71
+
bc->bound_on = bctimer.base->cpu_base->cpu;
72
+
78
73
return 0;
79
74
}
80
75
+5
-1
kernel/time/tick-broadcast.c
+5
-1
kernel/time/tick-broadcast.c
···
622
622
* to avoid a deep idle transition as we are about to get the
623
623
* broadcast IPI right away.
624
624
*/
625
-
int tick_check_broadcast_expired(void)
625
+
noinstr int tick_check_broadcast_expired(void)
626
626
{
627
+
#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H
628
+
return arch_test_bit(smp_processor_id(), cpumask_bits(tick_broadcast_force_mask));
629
+
#else
627
630
return cpumask_test_cpu(smp_processor_id(), tick_broadcast_force_mask);
631
+
#endif
628
632
}
629
633
630
634
/*
+3
kernel/trace/trace.c
+3
kernel/trace/trace.c
+20
-41
kernel/trace/trace_preemptirq.c
+20
-41
kernel/trace/trace_preemptirq.c
···
15
15
#define CREATE_TRACE_POINTS
16
16
#include <trace/events/preemptirq.h>
17
17
18
+
/*
19
+
* Use regular trace points on architectures that implement noinstr
20
+
* tooling: these calls will only happen with RCU enabled, which can
21
+
* use a regular tracepoint.
22
+
*
23
+
* On older architectures, use the rcuidle tracing methods (which
24
+
* aren't NMI-safe - so exclude NMI contexts):
25
+
*/
26
+
#ifdef CONFIG_ARCH_WANTS_NO_INSTR
27
+
#define trace(point) trace_##point
28
+
#else
29
+
#define trace(point) if (!in_nmi()) trace_##point##_rcuidle
30
+
#endif
31
+
18
32
#ifdef CONFIG_TRACE_IRQFLAGS
19
33
/* Per-cpu variable to prevent redundant calls when IRQs already off */
20
34
static DEFINE_PER_CPU(int, tracing_irq_cpu);
···
42
28
void trace_hardirqs_on_prepare(void)
43
29
{
44
30
if (this_cpu_read(tracing_irq_cpu)) {
45
-
if (!in_nmi())
46
-
trace_irq_enable(CALLER_ADDR0, CALLER_ADDR1);
31
+
trace(irq_enable)(CALLER_ADDR0, CALLER_ADDR1);
47
32
tracer_hardirqs_on(CALLER_ADDR0, CALLER_ADDR1);
48
33
this_cpu_write(tracing_irq_cpu, 0);
49
34
}
···
53
40
void trace_hardirqs_on(void)
54
41
{
55
42
if (this_cpu_read(tracing_irq_cpu)) {
56
-
if (!in_nmi())
57
-
trace_irq_enable_rcuidle(CALLER_ADDR0, CALLER_ADDR1);
43
+
trace(irq_enable)(CALLER_ADDR0, CALLER_ADDR1);
58
44
tracer_hardirqs_on(CALLER_ADDR0, CALLER_ADDR1);
59
45
this_cpu_write(tracing_irq_cpu, 0);
60
46
}
···
75
63
if (!this_cpu_read(tracing_irq_cpu)) {
76
64
this_cpu_write(tracing_irq_cpu, 1);
77
65
tracer_hardirqs_off(CALLER_ADDR0, CALLER_ADDR1);
78
-
if (!in_nmi())
79
-
trace_irq_disable(CALLER_ADDR0, CALLER_ADDR1);
66
+
trace(irq_disable)(CALLER_ADDR0, CALLER_ADDR1);
80
67
}
81
68
82
69
}
···
89
78
if (!this_cpu_read(tracing_irq_cpu)) {
90
79
this_cpu_write(tracing_irq_cpu, 1);
91
80
tracer_hardirqs_off(CALLER_ADDR0, CALLER_ADDR1);
92
-
if (!in_nmi())
93
-
trace_irq_disable_rcuidle(CALLER_ADDR0, CALLER_ADDR1);
81
+
trace(irq_disable)(CALLER_ADDR0, CALLER_ADDR1);
94
82
}
95
83
}
96
84
EXPORT_SYMBOL(trace_hardirqs_off);
97
85
NOKPROBE_SYMBOL(trace_hardirqs_off);
98
-
99
-
__visible void trace_hardirqs_on_caller(unsigned long caller_addr)
100
-
{
101
-
if (this_cpu_read(tracing_irq_cpu)) {
102
-
if (!in_nmi())
103
-
trace_irq_enable_rcuidle(CALLER_ADDR0, caller_addr);
104
-
tracer_hardirqs_on(CALLER_ADDR0, caller_addr);
105
-
this_cpu_write(tracing_irq_cpu, 0);
106
-
}
107
-
108
-
lockdep_hardirqs_on_prepare();
109
-
lockdep_hardirqs_on(caller_addr);
110
-
}
111
-
EXPORT_SYMBOL(trace_hardirqs_on_caller);
112
-
NOKPROBE_SYMBOL(trace_hardirqs_on_caller);
113
-
114
-
__visible void trace_hardirqs_off_caller(unsigned long caller_addr)
115
-
{
116
-
lockdep_hardirqs_off(caller_addr);
117
-
118
-
if (!this_cpu_read(tracing_irq_cpu)) {
119
-
this_cpu_write(tracing_irq_cpu, 1);
120
-
tracer_hardirqs_off(CALLER_ADDR0, caller_addr);
121
-
if (!in_nmi())
122
-
trace_irq_disable_rcuidle(CALLER_ADDR0, caller_addr);
123
-
}
124
-
}
125
-
EXPORT_SYMBOL(trace_hardirqs_off_caller);
126
-
NOKPROBE_SYMBOL(trace_hardirqs_off_caller);
127
86
#endif /* CONFIG_TRACE_IRQFLAGS */
128
87
129
88
#ifdef CONFIG_TRACE_PREEMPT_TOGGLE
130
89
131
90
void trace_preempt_on(unsigned long a0, unsigned long a1)
132
91
{
133
-
if (!in_nmi())
134
-
trace_preempt_enable_rcuidle(a0, a1);
92
+
trace(preempt_enable)(a0, a1);
135
93
tracer_preempt_on(a0, a1);
136
94
}
137
95
138
96
void trace_preempt_off(unsigned long a0, unsigned long a1)
139
97
{
140
-
if (!in_nmi())
141
-
trace_preempt_disable_rcuidle(a0, a1);
98
+
trace(preempt_disable)(a0, a1);
142
99
tracer_preempt_off(a0, a1);
143
100
}
144
101
#endif
+14
-1
lib/bug.c
+14
-1
lib/bug.c
···
47
47
#include <linux/sched.h>
48
48
#include <linux/rculist.h>
49
49
#include <linux/ftrace.h>
50
+
#include <linux/context_tracking.h>
50
51
51
52
extern struct bug_entry __start___bug_table[], __stop___bug_table[];
52
53
···
154
153
return module_find_bug(bugaddr);
155
154
}
156
155
157
-
enum bug_trap_type report_bug(unsigned long bugaddr, struct pt_regs *regs)
156
+
static enum bug_trap_type __report_bug(unsigned long bugaddr, struct pt_regs *regs)
158
157
{
159
158
struct bug_entry *bug;
160
159
const char *file;
···
208
207
(void *)bugaddr);
209
208
210
209
return BUG_TRAP_TYPE_BUG;
210
+
}
211
+
212
+
enum bug_trap_type report_bug(unsigned long bugaddr, struct pt_regs *regs)
213
+
{
214
+
enum bug_trap_type ret;
215
+
bool rcu = false;
216
+
217
+
rcu = warn_rcu_enter();
218
+
ret = __report_bug(bugaddr, regs);
219
+
warn_rcu_exit(rcu);
220
+
221
+
return ret;
211
222
}
212
223
213
224
static void clear_once_table(struct bug_entry *start, struct bug_entry *end)
+4
-1
lib/ubsan.c
+4
-1
lib/ubsan.c
···
339
339
{
340
340
struct invalid_value_data *data = _data;
341
341
char val_str[VALUE_LENGTH];
342
+
unsigned long ua_flags = user_access_save();
342
343
343
344
if (suppress_report(&data->location))
344
-
return;
345
+
goto out;
345
346
346
347
ubsan_prologue(&data->location, "invalid-load");
347
348
···
352
351
val_str, data->type->type_name);
353
352
354
353
ubsan_epilogue();
354
+
out:
355
+
user_access_restore(ua_flags);
355
356
}
356
357
EXPORT_SYMBOL(__ubsan_handle_load_invalid_value);
357
358
+4
mm/kasan/kasan.h
+4
mm/kasan/kasan.h
···
618
618
void __asan_set_shadow_f5(const void *addr, size_t size);
619
619
void __asan_set_shadow_f8(const void *addr, size_t size);
620
620
621
+
void *__asan_memset(void *addr, int c, size_t len);
622
+
void *__asan_memmove(void *dest, const void *src, size_t len);
623
+
void *__asan_memcpy(void *dest, const void *src, size_t len);
624
+
621
625
void __hwasan_load1_noabort(unsigned long addr);
622
626
void __hwasan_store1_noabort(unsigned long addr);
623
627
void __hwasan_load2_noabort(unsigned long addr);
+38
mm/kasan/shadow.c
+38
mm/kasan/shadow.c
···
38
38
}
39
39
EXPORT_SYMBOL(__kasan_check_write);
40
40
41
+
#ifndef CONFIG_GENERIC_ENTRY
42
+
/*
43
+
* CONFIG_GENERIC_ENTRY relies on compiler emitted mem*() calls to not be
44
+
* instrumented. KASAN enabled toolchains should emit __asan_mem*() functions
45
+
* for the sites they want to instrument.
46
+
*/
41
47
#undef memset
42
48
void *memset(void *addr, int c, size_t len)
43
49
{
···
74
68
75
69
return __memcpy(dest, src, len);
76
70
}
71
+
#endif
72
+
73
+
void *__asan_memset(void *addr, int c, size_t len)
74
+
{
75
+
if (!kasan_check_range((unsigned long)addr, len, true, _RET_IP_))
76
+
return NULL;
77
+
78
+
return __memset(addr, c, len);
79
+
}
80
+
EXPORT_SYMBOL(__asan_memset);
81
+
82
+
#ifdef __HAVE_ARCH_MEMMOVE
83
+
void *__asan_memmove(void *dest, const void *src, size_t len)
84
+
{
85
+
if (!kasan_check_range((unsigned long)src, len, false, _RET_IP_) ||
86
+
!kasan_check_range((unsigned long)dest, len, true, _RET_IP_))
87
+
return NULL;
88
+
89
+
return __memmove(dest, src, len);
90
+
}
91
+
EXPORT_SYMBOL(__asan_memmove);
92
+
#endif
93
+
94
+
void *__asan_memcpy(void *dest, const void *src, size_t len)
95
+
{
96
+
if (!kasan_check_range((unsigned long)src, len, false, _RET_IP_) ||
97
+
!kasan_check_range((unsigned long)dest, len, true, _RET_IP_))
98
+
return NULL;
99
+
100
+
return __memcpy(dest, src, len);
101
+
}
102
+
EXPORT_SYMBOL(__asan_memcpy);
77
103
78
104
void kasan_poison(const void *addr, size_t size, u8 value, bool init)
79
105
{
+14
tools/objtool/check.c
+14
tools/objtool/check.c
···
376
376
377
377
if (!strcmp(sec->name, ".noinstr.text") ||
378
378
!strcmp(sec->name, ".entry.text") ||
379
+
!strcmp(sec->name, ".cpuidle.text") ||
379
380
!strncmp(sec->name, ".text.__x86.", 12))
380
381
sec->noinstr = true;
381
382
···
1225
1224
"__ubsan_handle_type_mismatch",
1226
1225
"__ubsan_handle_type_mismatch_v1",
1227
1226
"__ubsan_handle_shift_out_of_bounds",
1227
+
"__ubsan_handle_load_invalid_value",
1228
1228
/* misc */
1229
1229
"csum_partial_copy_generic",
1230
1230
"copy_mc_fragile",
···
3377
3375
return true;
3378
3376
3379
3377
/*
3378
+
* If the symbol is a static_call trampoline, we can't tell.
3379
+
*/
3380
+
if (func->static_call_tramp)
3381
+
return true;
3382
+
3383
+
/*
3380
3384
* The __ubsan_handle_*() calls are like WARN(), they only happen when
3381
3385
* something 'BAD' happened. At the risk of taking the machine down,
3382
3386
* let them proceed to get the message out.
···
4174
4166
}
4175
4167
4176
4168
sec = find_section_by_name(file->elf, ".entry.text");
4169
+
if (sec) {
4170
+
warnings += validate_section(file, sec);
4171
+
warnings += validate_unwind_hints(file, sec);
4172
+
}
4173
+
4174
+
sec = find_section_by_name(file->elf, ".cpuidle.text");
4177
4175
if (sec) {
4178
4176
warnings += validate_section(file, sec);
4179
4177
warnings += validate_unwind_hints(file, sec);
+3
-13
tools/testing/selftests/kvm/rseq_test.c
+3
-13
tools/testing/selftests/kvm/rseq_test.c
···
41
41
GUEST_SYNC(0);
42
42
}
43
43
44
-
/*
45
-
* We have to perform direct system call for getcpu() because it's
46
-
* not available until glic 2.29.
47
-
*/
48
-
static void sys_getcpu(unsigned *cpu)
49
-
{
50
-
int r;
51
-
52
-
r = syscall(__NR_getcpu, cpu, NULL, NULL);
53
-
TEST_ASSERT(!r, "getcpu failed, errno = %d (%s)", errno, strerror(errno));
54
-
}
55
-
56
44
static int next_cpu(int cpu)
57
45
{
58
46
/*
···
237
249
* across the seq_cnt reads.
238
250
*/
239
251
smp_rmb();
240
-
sys_getcpu(&cpu);
252
+
r = sys_getcpu(&cpu, NULL);
253
+
TEST_ASSERT(!r, "getcpu failed, errno = %d (%s)",
254
+
errno, strerror(errno));
241
255
rseq_cpu = rseq_current_cpu_raw();
242
256
smp_rmb();
243
257
} while (snapshot != atomic_read(&seq_cnt));
+33
tools/testing/selftests/membarrier/membarrier_test_impl.h
+33
tools/testing/selftests/membarrier/membarrier_test_impl.h
···
9
9
10
10
#include "../kselftest.h"
11
11
12
+
static int registrations;
13
+
12
14
static int sys_membarrier(int cmd, int flags)
13
15
{
14
16
return syscall(__NR_membarrier, cmd, flags);
17
+
}
18
+
19
+
static int test_membarrier_get_registrations(int cmd)
20
+
{
21
+
int ret, flags = 0;
22
+
const char *test_name =
23
+
"sys membarrier MEMBARRIER_CMD_GET_REGISTRATIONS";
24
+
25
+
registrations |= cmd;
26
+
27
+
ret = sys_membarrier(MEMBARRIER_CMD_GET_REGISTRATIONS, 0);
28
+
if (ret < 0) {
29
+
ksft_exit_fail_msg(
30
+
"%s test: flags = %d, errno = %d\n",
31
+
test_name, flags, errno);
32
+
} else if (ret != registrations) {
33
+
ksft_exit_fail_msg(
34
+
"%s test: flags = %d, ret = %d, registrations = %d\n",
35
+
test_name, flags, ret, registrations);
36
+
}
37
+
ksft_test_result_pass(
38
+
"%s test: flags = %d, ret = %d, registrations = %d\n",
39
+
test_name, flags, ret, registrations);
40
+
41
+
return 0;
15
42
}
16
43
17
44
static int test_membarrier_cmd_fail(void)
···
140
113
ksft_test_result_pass(
141
114
"%s test: flags = %d\n",
142
115
test_name, flags);
116
+
117
+
test_membarrier_get_registrations(cmd);
143
118
return 0;
144
119
}
145
120
···
199
170
ksft_test_result_pass(
200
171
"%s test: flags = %d\n",
201
172
test_name, flags);
173
+
174
+
test_membarrier_get_registrations(cmd);
202
175
return 0;
203
176
}
204
177
···
235
204
ksft_test_result_pass(
236
205
"%s test: flags = %d\n",
237
206
test_name, flags);
207
+
208
+
test_membarrier_get_registrations(cmd);
238
209
return 0;
239
210
}
240
211
+1
-1
tools/testing/selftests/membarrier/membarrier_test_multi_thread.c
+1
-1
tools/testing/selftests/membarrier/membarrier_test_multi_thread.c
+5
-1
tools/testing/selftests/membarrier/membarrier_test_single_thread.c
+5
-1
tools/testing/selftests/membarrier/membarrier_test_single_thread.c
···
12
12
int main(int argc, char **argv)
13
13
{
14
14
ksft_print_header();
15
-
ksft_set_plan(13);
15
+
ksft_set_plan(18);
16
+
17
+
test_membarrier_get_registrations(/*cmd=*/0);
16
18
17
19
test_membarrier_query();
18
20
19
21
test_membarrier_fail();
20
22
21
23
test_membarrier_success();
24
+
25
+
test_membarrier_get_registrations(/*cmd=*/0);
22
26
23
27
return ksft_exit_pass();
24
28
}
+4
tools/testing/selftests/rseq/.gitignore
+4
tools/testing/selftests/rseq/.gitignore
···
1
1
# SPDX-License-Identifier: GPL-2.0-only
2
2
basic_percpu_ops_test
3
+
basic_percpu_ops_mm_cid_test
3
4
basic_test
4
5
basic_rseq_op_test
5
6
param_test
6
7
param_test_benchmark
7
8
param_test_compare_twice
9
+
param_test_mm_cid
10
+
param_test_mm_cid_benchmark
11
+
param_test_mm_cid_compare_twice
+18
-2
tools/testing/selftests/rseq/Makefile
+18
-2
tools/testing/selftests/rseq/Makefile
···
12
12
# still track changes to header files and depend on shared object.
13
13
OVERRIDE_TARGETS = 1
14
14
15
-
TEST_GEN_PROGS = basic_test basic_percpu_ops_test param_test \
16
-
param_test_benchmark param_test_compare_twice
15
+
TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test param_test \
16
+
param_test_benchmark param_test_compare_twice param_test_mm_cid \
17
+
param_test_mm_cid_benchmark param_test_mm_cid_compare_twice
17
18
18
19
TEST_GEN_PROGS_EXTENDED = librseq.so
19
20
···
30
29
$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h
31
30
$(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -o $@
32
31
32
+
$(OUTPUT)/basic_percpu_ops_mm_cid_test: basic_percpu_ops_test.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h
33
+
$(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_MM_CID_ID $< $(LDLIBS) -lrseq -o $@
34
+
33
35
$(OUTPUT)/param_test_benchmark: param_test.c $(TEST_GEN_PROGS_EXTENDED) \
34
36
rseq.h rseq-*.h
35
37
$(CC) $(CFLAGS) -DBENCHMARK $< $(LDLIBS) -lrseq -o $@
···
40
36
$(OUTPUT)/param_test_compare_twice: param_test.c $(TEST_GEN_PROGS_EXTENDED) \
41
37
rseq.h rseq-*.h
42
38
$(CC) $(CFLAGS) -DRSEQ_COMPARE_TWICE $< $(LDLIBS) -lrseq -o $@
39
+
40
+
$(OUTPUT)/param_test_mm_cid: param_test.c $(TEST_GEN_PROGS_EXTENDED) \
41
+
rseq.h rseq-*.h
42
+
$(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_MM_CID $< $(LDLIBS) -lrseq -o $@
43
+
44
+
$(OUTPUT)/param_test_mm_cid_benchmark: param_test.c $(TEST_GEN_PROGS_EXTENDED) \
45
+
rseq.h rseq-*.h
46
+
$(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_MM_CID -DBENCHMARK $< $(LDLIBS) -lrseq -o $@
47
+
48
+
$(OUTPUT)/param_test_mm_cid_compare_twice: param_test.c $(TEST_GEN_PROGS_EXTENDED) \
49
+
rseq.h rseq-*.h
50
+
$(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_MM_CID -DRSEQ_COMPARE_TWICE $< $(LDLIBS) -lrseq -o $@
+39
-7
tools/testing/selftests/rseq/basic_percpu_ops_test.c
+39
-7
tools/testing/selftests/rseq/basic_percpu_ops_test.c
···
12
12
#include "../kselftest.h"
13
13
#include "rseq.h"
14
14
15
+
#ifdef BUILDOPT_RSEQ_PERCPU_MM_CID
16
+
# define RSEQ_PERCPU RSEQ_PERCPU_MM_CID
17
+
static
18
+
int get_current_cpu_id(void)
19
+
{
20
+
return rseq_current_mm_cid();
21
+
}
22
+
static
23
+
bool rseq_validate_cpu_id(void)
24
+
{
25
+
return rseq_mm_cid_available();
26
+
}
27
+
#else
28
+
# define RSEQ_PERCPU RSEQ_PERCPU_CPU_ID
29
+
static
30
+
int get_current_cpu_id(void)
31
+
{
32
+
return rseq_cpu_start();
33
+
}
34
+
static
35
+
bool rseq_validate_cpu_id(void)
36
+
{
37
+
return rseq_current_cpu_raw() >= 0;
38
+
}
39
+
#endif
40
+
15
41
struct percpu_lock_entry {
16
42
intptr_t v;
17
43
} __attribute__((aligned(128)));
···
77
51
for (;;) {
78
52
int ret;
79
53
80
-
cpu = rseq_cpu_start();
81
-
ret = rseq_cmpeqv_storev(&lock->c[cpu].v,
82
-
0, 1, cpu);
54
+
cpu = get_current_cpu_id();
55
+
ret = rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
56
+
&lock->c[cpu].v, 0, 1, cpu);
83
57
if (rseq_likely(!ret))
84
58
break;
85
59
/* Retry if comparison fails or rseq aborts. */
···
167
141
intptr_t *targetptr, newval, expect;
168
142
int ret;
169
143
170
-
cpu = rseq_cpu_start();
144
+
cpu = get_current_cpu_id();
171
145
/* Load list->c[cpu].head with single-copy atomicity. */
172
146
expect = (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head);
173
147
newval = (intptr_t)node;
174
148
targetptr = (intptr_t *)&list->c[cpu].head;
175
149
node->next = (struct percpu_list_node *)expect;
176
-
ret = rseq_cmpeqv_storev(targetptr, expect, newval, cpu);
150
+
ret = rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
151
+
targetptr, expect, newval, cpu);
177
152
if (rseq_likely(!ret))
178
153
break;
179
154
/* Retry if comparison fails or rseq aborts. */
···
197
170
long offset;
198
171
int ret, cpu;
199
172
200
-
cpu = rseq_cpu_start();
173
+
cpu = get_current_cpu_id();
201
174
targetptr = (intptr_t *)&list->c[cpu].head;
202
175
expectnot = (intptr_t)NULL;
203
176
offset = offsetof(struct percpu_list_node, next);
204
177
load = (intptr_t *)&head;
205
-
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,
178
+
ret = rseq_cmpnev_storeoffp_load(RSEQ_MO_RELAXED, RSEQ_PERCPU,
179
+
targetptr, expectnot,
206
180
offset, load, cpu);
207
181
if (rseq_likely(!ret)) {
208
182
if (_cpu)
···
321
293
if (rseq_register_current_thread()) {
322
294
fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n",
323
295
errno, strerror(errno));
296
+
goto error;
297
+
}
298
+
if (!rseq_validate_cpu_id()) {
299
+
fprintf(stderr, "Error: cpu id getter unavailable\n");
324
300
goto error;
325
301
}
326
302
printf("spinlock\n");
+4
tools/testing/selftests/rseq/basic_test.c
+4
tools/testing/selftests/rseq/basic_test.c
···
22
22
CPU_ZERO(&test_affinity);
23
23
for (i = 0; i < CPU_SETSIZE; i++) {
24
24
if (CPU_ISSET(i, &affinity)) {
25
+
int node;
26
+
25
27
CPU_SET(i, &test_affinity);
26
28
sched_setaffinity(0, sizeof(test_affinity),
27
29
&test_affinity);
···
31
29
assert(rseq_current_cpu() == i);
32
30
assert(rseq_current_cpu_raw() == i);
33
31
assert(rseq_cpu_start() == i);
32
+
node = rseq_fallback_current_node();
33
+
assert(rseq_current_node_id() == node);
34
34
CPU_CLR(i, &test_affinity);
35
35
}
36
36
}
+6
tools/testing/selftests/rseq/compiler.h
+6
tools/testing/selftests/rseq/compiler.h
···
27
27
*/
28
28
#define rseq_after_asm_goto() asm volatile ("" : : : "memory")
29
29
30
+
/* Combine two tokens. */
31
+
#define RSEQ__COMBINE_TOKENS(_tokena, _tokenb) \
32
+
_tokena##_tokenb
33
+
#define RSEQ_COMBINE_TOKENS(_tokena, _tokenb) \
34
+
RSEQ__COMBINE_TOKENS(_tokena, _tokenb)
35
+
30
36
#endif /* RSEQ_COMPILER_H_ */
+105
-52
tools/testing/selftests/rseq/param_test.c
+105
-52
tools/testing/selftests/rseq/param_test.c
···
16
16
#include <signal.h>
17
17
#include <errno.h>
18
18
#include <stddef.h>
19
+
#include <stdbool.h>
19
20
20
21
static inline pid_t rseq_gettid(void)
21
22
{
···
37
36
38
37
static int opt_yield, opt_signal, opt_sleep,
39
38
opt_disable_rseq, opt_threads = 200,
40
-
opt_disable_mod = 0, opt_test = 's', opt_mb = 0;
39
+
opt_disable_mod = 0, opt_test = 's';
41
40
42
-
#ifndef RSEQ_SKIP_FASTPATH
43
41
static long long opt_reps = 5000;
44
-
#else
45
-
static long long opt_reps = 100;
46
-
#endif
47
42
48
43
static __thread __attribute__((tls_model("initial-exec")))
49
44
unsigned int signals_delivered;
···
265
268
266
269
#include "rseq.h"
267
270
271
+
static enum rseq_mo opt_mo = RSEQ_MO_RELAXED;
272
+
273
+
#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
274
+
#define TEST_MEMBARRIER
275
+
276
+
static int sys_membarrier(int cmd, int flags, int cpu_id)
277
+
{
278
+
return syscall(__NR_membarrier, cmd, flags, cpu_id);
279
+
}
280
+
#endif
281
+
282
+
#ifdef BUILDOPT_RSEQ_PERCPU_MM_CID
283
+
# define RSEQ_PERCPU RSEQ_PERCPU_MM_CID
284
+
static
285
+
int get_current_cpu_id(void)
286
+
{
287
+
return rseq_current_mm_cid();
288
+
}
289
+
static
290
+
bool rseq_validate_cpu_id(void)
291
+
{
292
+
return rseq_mm_cid_available();
293
+
}
294
+
# ifdef TEST_MEMBARRIER
295
+
/*
296
+
* Membarrier does not currently support targeting a mm_cid, so
297
+
* issue the barrier on all cpus.
298
+
*/
299
+
static
300
+
int rseq_membarrier_expedited(int cpu)
301
+
{
302
+
return sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
303
+
0, 0);
304
+
}
305
+
# endif /* TEST_MEMBARRIER */
306
+
#else
307
+
# define RSEQ_PERCPU RSEQ_PERCPU_CPU_ID
308
+
static
309
+
int get_current_cpu_id(void)
310
+
{
311
+
return rseq_cpu_start();
312
+
}
313
+
static
314
+
bool rseq_validate_cpu_id(void)
315
+
{
316
+
return rseq_current_cpu_raw() >= 0;
317
+
}
318
+
# ifdef TEST_MEMBARRIER
319
+
static
320
+
int rseq_membarrier_expedited(int cpu)
321
+
{
322
+
return sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
323
+
MEMBARRIER_CMD_FLAG_CPU, cpu);
324
+
}
325
+
# endif /* TEST_MEMBARRIER */
326
+
#endif
327
+
268
328
struct percpu_lock_entry {
269
329
intptr_t v;
270
330
} __attribute__((aligned(128)));
···
409
355
for (;;) {
410
356
int ret;
411
357
412
-
cpu = rseq_cpu_start();
413
-
ret = rseq_cmpeqv_storev(&lock->c[cpu].v,
358
+
cpu = get_current_cpu_id();
359
+
if (cpu < 0) {
360
+
fprintf(stderr, "pid: %d: tid: %d, cpu: %d: cid: %d\n",
361
+
getpid(), (int) rseq_gettid(), rseq_current_cpu_raw(), cpu);
362
+
abort();
363
+
}
364
+
ret = rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
365
+
&lock->c[cpu].v,
414
366
0, 1, cpu);
415
367
if (rseq_likely(!ret))
416
368
break;
···
533
473
do {
534
474
int cpu;
535
475
536
-
cpu = rseq_cpu_start();
537
-
ret = rseq_addv(&data->c[cpu].count, 1, cpu);
476
+
cpu = get_current_cpu_id();
477
+
ret = rseq_addv(RSEQ_MO_RELAXED, RSEQ_PERCPU,
478
+
&data->c[cpu].count, 1, cpu);
538
479
} while (rseq_unlikely(ret));
539
480
#ifndef BENCHMARK
540
481
if (i != 0 && !(i % (reps / 10)))
···
604
543
intptr_t *targetptr, newval, expect;
605
544
int ret;
606
545
607
-
cpu = rseq_cpu_start();
546
+
cpu = get_current_cpu_id();
608
547
/* Load list->c[cpu].head with single-copy atomicity. */
609
548
expect = (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head);
610
549
newval = (intptr_t)node;
611
550
targetptr = (intptr_t *)&list->c[cpu].head;
612
551
node->next = (struct percpu_list_node *)expect;
613
-
ret = rseq_cmpeqv_storev(targetptr, expect, newval, cpu);
552
+
ret = rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
553
+
targetptr, expect, newval, cpu);
614
554
if (rseq_likely(!ret))
615
555
break;
616
556
/* Retry if comparison fails or rseq aborts. */
···
637
575
long offset;
638
576
int ret;
639
577
640
-
cpu = rseq_cpu_start();
578
+
cpu = get_current_cpu_id();
641
579
targetptr = (intptr_t *)&list->c[cpu].head;
642
580
expectnot = (intptr_t)NULL;
643
581
offset = offsetof(struct percpu_list_node, next);
644
582
load = (intptr_t *)&head;
645
-
ret = rseq_cmpnev_storeoffp_load(targetptr, expectnot,
646
-
offset, load, cpu);
583
+
ret = rseq_cmpnev_storeoffp_load(RSEQ_MO_RELAXED, RSEQ_PERCPU,
584
+
targetptr, expectnot,
585
+
offset, load, cpu);
647
586
if (rseq_likely(!ret)) {
648
587
node = head;
649
588
break;
···
782
719
intptr_t offset;
783
720
int ret;
784
721
785
-
cpu = rseq_cpu_start();
722
+
cpu = get_current_cpu_id();
786
723
offset = RSEQ_READ_ONCE(buffer->c[cpu].offset);
787
724
if (offset == buffer->c[cpu].buflen)
788
725
break;
···
790
727
targetptr_spec = (intptr_t *)&buffer->c[cpu].array[offset];
791
728
newval_final = offset + 1;
792
729
targetptr_final = &buffer->c[cpu].offset;
793
-
if (opt_mb)
794
-
ret = rseq_cmpeqv_trystorev_storev_release(
795
-
targetptr_final, offset, targetptr_spec,
796
-
newval_spec, newval_final, cpu);
797
-
else
798
-
ret = rseq_cmpeqv_trystorev_storev(targetptr_final,
799
-
offset, targetptr_spec, newval_spec,
800
-
newval_final, cpu);
730
+
ret = rseq_cmpeqv_trystorev_storev(opt_mo, RSEQ_PERCPU,
731
+
targetptr_final, offset, targetptr_spec,
732
+
newval_spec, newval_final, cpu);
801
733
if (rseq_likely(!ret)) {
802
734
result = true;
803
735
break;
···
815
757
intptr_t offset;
816
758
int ret;
817
759
818
-
cpu = rseq_cpu_start();
760
+
cpu = get_current_cpu_id();
819
761
/* Load offset with single-copy atomicity. */
820
762
offset = RSEQ_READ_ONCE(buffer->c[cpu].offset);
821
763
if (offset == 0) {
···
825
767
head = RSEQ_READ_ONCE(buffer->c[cpu].array[offset - 1]);
826
768
newval = offset - 1;
827
769
targetptr = (intptr_t *)&buffer->c[cpu].offset;
828
-
ret = rseq_cmpeqv_cmpeqv_storev(targetptr, offset,
770
+
ret = rseq_cmpeqv_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
771
+
targetptr, offset,
829
772
(intptr_t *)&buffer->c[cpu].array[offset - 1],
830
773
(intptr_t)head, newval, cpu);
831
774
if (rseq_likely(!ret))
···
983
924
size_t copylen;
984
925
int ret;
985
926
986
-
cpu = rseq_cpu_start();
927
+
cpu = get_current_cpu_id();
987
928
/* Load offset with single-copy atomicity. */
988
929
offset = RSEQ_READ_ONCE(buffer->c[cpu].offset);
989
930
if (offset == buffer->c[cpu].buflen)
···
994
935
copylen = sizeof(item);
995
936
newval_final = offset + 1;
996
937
targetptr_final = &buffer->c[cpu].offset;
997
-
if (opt_mb)
998
-
ret = rseq_cmpeqv_trymemcpy_storev_release(
999
-
targetptr_final, offset,
1000
-
destptr, srcptr, copylen,
1001
-
newval_final, cpu);
1002
-
else
1003
-
ret = rseq_cmpeqv_trymemcpy_storev(targetptr_final,
1004
-
offset, destptr, srcptr, copylen,
1005
-
newval_final, cpu);
938
+
ret = rseq_cmpeqv_trymemcpy_storev(
939
+
opt_mo, RSEQ_PERCPU,
940
+
targetptr_final, offset,
941
+
destptr, srcptr, copylen,
942
+
newval_final, cpu);
1006
943
if (rseq_likely(!ret)) {
1007
944
result = true;
1008
945
break;
···
1023
968
size_t copylen;
1024
969
int ret;
1025
970
1026
-
cpu = rseq_cpu_start();
971
+
cpu = get_current_cpu_id();
1027
972
/* Load offset with single-copy atomicity. */
1028
973
offset = RSEQ_READ_ONCE(buffer->c[cpu].offset);
1029
974
if (offset == 0)
···
1034
979
copylen = sizeof(*item);
1035
980
newval_final = offset - 1;
1036
981
targetptr_final = &buffer->c[cpu].offset;
1037
-
ret = rseq_cmpeqv_trymemcpy_storev(targetptr_final,
1038
-
offset, destptr, srcptr, copylen,
982
+
ret = rseq_cmpeqv_trymemcpy_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU,
983
+
targetptr_final, offset, destptr, srcptr, copylen,
1039
984
newval_final, cpu);
1040
985
if (rseq_likely(!ret)) {
1041
986
result = true;
···
1210
1155
}
1211
1156
1212
1157
/* Test MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU membarrier command. */
1213
-
#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
1158
+
#ifdef TEST_MEMBARRIER
1214
1159
struct test_membarrier_thread_args {
1215
1160
int stop;
1216
1161
intptr_t percpu_list_ptr;
···
1237
1182
int ret;
1238
1183
1239
1184
do {
1240
-
int cpu = rseq_cpu_start();
1185
+
int cpu = get_current_cpu_id();
1241
1186
1242
-
ret = rseq_offset_deref_addv(&args->percpu_list_ptr,
1187
+
ret = rseq_offset_deref_addv(RSEQ_MO_RELAXED, RSEQ_PERCPU,
1188
+
&args->percpu_list_ptr,
1243
1189
sizeof(struct percpu_list_entry) * cpu, 1, cpu);
1244
1190
} while (rseq_unlikely(ret));
1245
1191
}
···
1275
1219
1276
1220
for (i = 0; i < CPU_SETSIZE; i++)
1277
1221
free(list->c[i].head);
1278
-
}
1279
-
1280
-
static int sys_membarrier(int cmd, int flags, int cpu_id)
1281
-
{
1282
-
return syscall(__NR_membarrier, cmd, flags, cpu_id);
1283
1222
}
1284
1223
1285
1224
/*
···
1315
1264
1316
1265
/* Make list_b "active". */
1317
1266
atomic_store(&args->percpu_list_ptr, (intptr_t)&list_b);
1318
-
if (sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
1319
-
MEMBARRIER_CMD_FLAG_CPU, cpu_a) &&
1267
+
if (rseq_membarrier_expedited(cpu_a) &&
1320
1268
errno != ENXIO /* missing CPU */) {
1321
1269
perror("sys_membarrier");
1322
1270
abort();
···
1338
1288
1339
1289
/* Make list_a "active". */
1340
1290
atomic_store(&args->percpu_list_ptr, (intptr_t)&list_a);
1341
-
if (sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
1342
-
MEMBARRIER_CMD_FLAG_CPU, cpu_b) &&
1291
+
if (rseq_membarrier_expedited(cpu_b) &&
1343
1292
errno != ENXIO /* missing CPU*/) {
1344
1293
perror("sys_membarrier");
1345
1294
abort();
···
1409
1360
abort();
1410
1361
}
1411
1362
}
1412
-
#else /* RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV */
1363
+
#else /* TEST_MEMBARRIER */
1413
1364
void test_membarrier(void)
1414
1365
{
1415
1366
fprintf(stderr, "rseq_offset_deref_addv is not implemented on this architecture. "
···
1566
1517
verbose = 1;
1567
1518
break;
1568
1519
case 'M':
1569
-
opt_mb = 1;
1520
+
opt_mo = RSEQ_MO_RELEASE;
1570
1521
break;
1571
1522
default:
1572
1523
show_usage(argc, argv);
···
1586
1537
1587
1538
if (!opt_disable_rseq && rseq_register_current_thread())
1588
1539
goto error;
1540
+
if (!opt_disable_rseq && !rseq_validate_cpu_id()) {
1541
+
fprintf(stderr, "Error: cpu id getter unavailable\n");
1542
+
goto error;
1543
+
}
1589
1544
switch (opt_test) {
1590
1545
case 's':
1591
1546
printf_verbose("spinlock\n");
+22
tools/testing/selftests/rseq/rseq-abi.h
+22
tools/testing/selftests/rseq/rseq-abi.h
···
146
146
* this thread.
147
147
*/
148
148
__u32 flags;
149
+
150
+
/*
151
+
* Restartable sequences node_id field. Updated by the kernel. Read by
152
+
* user-space with single-copy atomicity semantics. This field should
153
+
* only be read by the thread which registered this data structure.
154
+
* Aligned on 32-bit. Contains the current NUMA node ID.
155
+
*/
156
+
__u32 node_id;
157
+
158
+
/*
159
+
* Restartable sequences mm_cid field. Updated by the kernel. Read by
160
+
* user-space with single-copy atomicity semantics. This field should
161
+
* only be read by the thread which registered this data structure.
162
+
* Aligned on 32-bit. Contains the current thread's concurrency ID
163
+
* (allocated uniquely within a memory map).
164
+
*/
165
+
__u32 mm_cid;
166
+
167
+
/*
168
+
* Flexible array member at end of structure, after last feature field.
169
+
*/
170
+
char end[];
149
171
} __attribute__((aligned(4 * sizeof(__u64))));
150
172
151
173
#endif /* _RSEQ_ABI_H */
+505
tools/testing/selftests/rseq/rseq-arm-bits.h
+505
tools/testing/selftests/rseq/rseq-arm-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-arm-bits.h
4
+
*
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
*/
7
+
8
+
#include "rseq-bits-template.h"
9
+
10
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
11
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
12
+
13
+
static inline __attribute__((always_inline))
14
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
15
+
{
16
+
RSEQ_INJECT_C(9)
17
+
18
+
__asm__ __volatile__ goto (
19
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
20
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
21
+
#ifdef RSEQ_COMPARE_TWICE
22
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
23
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
24
+
#endif
25
+
/* Start rseq by storing table entry pointer into rseq_cs. */
26
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
27
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
28
+
RSEQ_INJECT_ASM(3)
29
+
"ldr r0, %[v]\n\t"
30
+
"cmp %[expect], r0\n\t"
31
+
"bne %l[cmpfail]\n\t"
32
+
RSEQ_INJECT_ASM(4)
33
+
#ifdef RSEQ_COMPARE_TWICE
34
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
35
+
"ldr r0, %[v]\n\t"
36
+
"cmp %[expect], r0\n\t"
37
+
"bne %l[error2]\n\t"
38
+
#endif
39
+
/* final store */
40
+
"str %[newv], %[v]\n\t"
41
+
"2:\n\t"
42
+
RSEQ_INJECT_ASM(5)
43
+
"b 5f\n\t"
44
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
45
+
"5:\n\t"
46
+
: /* gcc asm goto does not allow outputs */
47
+
: [cpu_id] "r" (cpu),
48
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
49
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
50
+
[v] "m" (*v),
51
+
[expect] "r" (expect),
52
+
[newv] "r" (newv)
53
+
RSEQ_INJECT_INPUT
54
+
: "r0", "memory", "cc"
55
+
RSEQ_INJECT_CLOBBER
56
+
: abort, cmpfail
57
+
#ifdef RSEQ_COMPARE_TWICE
58
+
, error1, error2
59
+
#endif
60
+
);
61
+
rseq_after_asm_goto();
62
+
return 0;
63
+
abort:
64
+
rseq_after_asm_goto();
65
+
RSEQ_INJECT_FAILED
66
+
return -1;
67
+
cmpfail:
68
+
rseq_after_asm_goto();
69
+
return 1;
70
+
#ifdef RSEQ_COMPARE_TWICE
71
+
error1:
72
+
rseq_after_asm_goto();
73
+
rseq_bug("cpu_id comparison failed");
74
+
error2:
75
+
rseq_after_asm_goto();
76
+
rseq_bug("expected value comparison failed");
77
+
#endif
78
+
}
79
+
80
+
static inline __attribute__((always_inline))
81
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
82
+
long voffp, intptr_t *load, int cpu)
83
+
{
84
+
RSEQ_INJECT_C(9)
85
+
86
+
__asm__ __volatile__ goto (
87
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
88
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
89
+
#ifdef RSEQ_COMPARE_TWICE
90
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
91
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
92
+
#endif
93
+
/* Start rseq by storing table entry pointer into rseq_cs. */
94
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
95
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
96
+
RSEQ_INJECT_ASM(3)
97
+
"ldr r0, %[v]\n\t"
98
+
"cmp %[expectnot], r0\n\t"
99
+
"beq %l[cmpfail]\n\t"
100
+
RSEQ_INJECT_ASM(4)
101
+
#ifdef RSEQ_COMPARE_TWICE
102
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
103
+
"ldr r0, %[v]\n\t"
104
+
"cmp %[expectnot], r0\n\t"
105
+
"beq %l[error2]\n\t"
106
+
#endif
107
+
"str r0, %[load]\n\t"
108
+
"add r0, %[voffp]\n\t"
109
+
"ldr r0, [r0]\n\t"
110
+
/* final store */
111
+
"str r0, %[v]\n\t"
112
+
"2:\n\t"
113
+
RSEQ_INJECT_ASM(5)
114
+
"b 5f\n\t"
115
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
116
+
"5:\n\t"
117
+
: /* gcc asm goto does not allow outputs */
118
+
: [cpu_id] "r" (cpu),
119
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
120
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
121
+
/* final store input */
122
+
[v] "m" (*v),
123
+
[expectnot] "r" (expectnot),
124
+
[voffp] "Ir" (voffp),
125
+
[load] "m" (*load)
126
+
RSEQ_INJECT_INPUT
127
+
: "r0", "memory", "cc"
128
+
RSEQ_INJECT_CLOBBER
129
+
: abort, cmpfail
130
+
#ifdef RSEQ_COMPARE_TWICE
131
+
, error1, error2
132
+
#endif
133
+
);
134
+
rseq_after_asm_goto();
135
+
return 0;
136
+
abort:
137
+
rseq_after_asm_goto();
138
+
RSEQ_INJECT_FAILED
139
+
return -1;
140
+
cmpfail:
141
+
rseq_after_asm_goto();
142
+
return 1;
143
+
#ifdef RSEQ_COMPARE_TWICE
144
+
error1:
145
+
rseq_after_asm_goto();
146
+
rseq_bug("cpu_id comparison failed");
147
+
error2:
148
+
rseq_after_asm_goto();
149
+
rseq_bug("expected value comparison failed");
150
+
#endif
151
+
}
152
+
153
+
static inline __attribute__((always_inline))
154
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
155
+
{
156
+
RSEQ_INJECT_C(9)
157
+
158
+
__asm__ __volatile__ goto (
159
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
160
+
#ifdef RSEQ_COMPARE_TWICE
161
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
162
+
#endif
163
+
/* Start rseq by storing table entry pointer into rseq_cs. */
164
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
165
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
166
+
RSEQ_INJECT_ASM(3)
167
+
#ifdef RSEQ_COMPARE_TWICE
168
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
169
+
#endif
170
+
"ldr r0, %[v]\n\t"
171
+
"add r0, %[count]\n\t"
172
+
/* final store */
173
+
"str r0, %[v]\n\t"
174
+
"2:\n\t"
175
+
RSEQ_INJECT_ASM(4)
176
+
"b 5f\n\t"
177
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
178
+
"5:\n\t"
179
+
: /* gcc asm goto does not allow outputs */
180
+
: [cpu_id] "r" (cpu),
181
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
182
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
183
+
[v] "m" (*v),
184
+
[count] "Ir" (count)
185
+
RSEQ_INJECT_INPUT
186
+
: "r0", "memory", "cc"
187
+
RSEQ_INJECT_CLOBBER
188
+
: abort
189
+
#ifdef RSEQ_COMPARE_TWICE
190
+
, error1
191
+
#endif
192
+
);
193
+
rseq_after_asm_goto();
194
+
return 0;
195
+
abort:
196
+
rseq_after_asm_goto();
197
+
RSEQ_INJECT_FAILED
198
+
return -1;
199
+
#ifdef RSEQ_COMPARE_TWICE
200
+
error1:
201
+
rseq_after_asm_goto();
202
+
rseq_bug("cpu_id comparison failed");
203
+
#endif
204
+
}
205
+
206
+
static inline __attribute__((always_inline))
207
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
208
+
intptr_t *v2, intptr_t expect2,
209
+
intptr_t newv, int cpu)
210
+
{
211
+
RSEQ_INJECT_C(9)
212
+
213
+
__asm__ __volatile__ goto (
214
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
215
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
216
+
#ifdef RSEQ_COMPARE_TWICE
217
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
218
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
219
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
220
+
#endif
221
+
/* Start rseq by storing table entry pointer into rseq_cs. */
222
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
223
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
224
+
RSEQ_INJECT_ASM(3)
225
+
"ldr r0, %[v]\n\t"
226
+
"cmp %[expect], r0\n\t"
227
+
"bne %l[cmpfail]\n\t"
228
+
RSEQ_INJECT_ASM(4)
229
+
"ldr r0, %[v2]\n\t"
230
+
"cmp %[expect2], r0\n\t"
231
+
"bne %l[cmpfail]\n\t"
232
+
RSEQ_INJECT_ASM(5)
233
+
#ifdef RSEQ_COMPARE_TWICE
234
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
235
+
"ldr r0, %[v]\n\t"
236
+
"cmp %[expect], r0\n\t"
237
+
"bne %l[error2]\n\t"
238
+
"ldr r0, %[v2]\n\t"
239
+
"cmp %[expect2], r0\n\t"
240
+
"bne %l[error3]\n\t"
241
+
#endif
242
+
/* final store */
243
+
"str %[newv], %[v]\n\t"
244
+
"2:\n\t"
245
+
RSEQ_INJECT_ASM(6)
246
+
"b 5f\n\t"
247
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
248
+
"5:\n\t"
249
+
: /* gcc asm goto does not allow outputs */
250
+
: [cpu_id] "r" (cpu),
251
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
252
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
253
+
/* cmp2 input */
254
+
[v2] "m" (*v2),
255
+
[expect2] "r" (expect2),
256
+
/* final store input */
257
+
[v] "m" (*v),
258
+
[expect] "r" (expect),
259
+
[newv] "r" (newv)
260
+
RSEQ_INJECT_INPUT
261
+
: "r0", "memory", "cc"
262
+
RSEQ_INJECT_CLOBBER
263
+
: abort, cmpfail
264
+
#ifdef RSEQ_COMPARE_TWICE
265
+
, error1, error2, error3
266
+
#endif
267
+
);
268
+
rseq_after_asm_goto();
269
+
return 0;
270
+
abort:
271
+
rseq_after_asm_goto();
272
+
RSEQ_INJECT_FAILED
273
+
return -1;
274
+
cmpfail:
275
+
rseq_after_asm_goto();
276
+
return 1;
277
+
#ifdef RSEQ_COMPARE_TWICE
278
+
error1:
279
+
rseq_after_asm_goto();
280
+
rseq_bug("cpu_id comparison failed");
281
+
error2:
282
+
rseq_after_asm_goto();
283
+
rseq_bug("1st expected value comparison failed");
284
+
error3:
285
+
rseq_after_asm_goto();
286
+
rseq_bug("2nd expected value comparison failed");
287
+
#endif
288
+
}
289
+
290
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
291
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
292
+
293
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
294
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
295
+
296
+
static inline __attribute__((always_inline))
297
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
298
+
intptr_t *v2, intptr_t newv2,
299
+
intptr_t newv, int cpu)
300
+
{
301
+
RSEQ_INJECT_C(9)
302
+
303
+
__asm__ __volatile__ goto (
304
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
305
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
306
+
#ifdef RSEQ_COMPARE_TWICE
307
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
308
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
309
+
#endif
310
+
/* Start rseq by storing table entry pointer into rseq_cs. */
311
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
312
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
313
+
RSEQ_INJECT_ASM(3)
314
+
"ldr r0, %[v]\n\t"
315
+
"cmp %[expect], r0\n\t"
316
+
"bne %l[cmpfail]\n\t"
317
+
RSEQ_INJECT_ASM(4)
318
+
#ifdef RSEQ_COMPARE_TWICE
319
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
320
+
"ldr r0, %[v]\n\t"
321
+
"cmp %[expect], r0\n\t"
322
+
"bne %l[error2]\n\t"
323
+
#endif
324
+
/* try store */
325
+
"str %[newv2], %[v2]\n\t"
326
+
RSEQ_INJECT_ASM(5)
327
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
328
+
"dmb\n\t" /* full mb provides store-release */
329
+
#endif
330
+
/* final store */
331
+
"str %[newv], %[v]\n\t"
332
+
"2:\n\t"
333
+
RSEQ_INJECT_ASM(6)
334
+
"b 5f\n\t"
335
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
336
+
"5:\n\t"
337
+
: /* gcc asm goto does not allow outputs */
338
+
: [cpu_id] "r" (cpu),
339
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
340
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
341
+
/* try store input */
342
+
[v2] "m" (*v2),
343
+
[newv2] "r" (newv2),
344
+
/* final store input */
345
+
[v] "m" (*v),
346
+
[expect] "r" (expect),
347
+
[newv] "r" (newv)
348
+
RSEQ_INJECT_INPUT
349
+
: "r0", "memory", "cc"
350
+
RSEQ_INJECT_CLOBBER
351
+
: abort, cmpfail
352
+
#ifdef RSEQ_COMPARE_TWICE
353
+
, error1, error2
354
+
#endif
355
+
);
356
+
rseq_after_asm_goto();
357
+
return 0;
358
+
abort:
359
+
rseq_after_asm_goto();
360
+
RSEQ_INJECT_FAILED
361
+
return -1;
362
+
cmpfail:
363
+
rseq_after_asm_goto();
364
+
return 1;
365
+
#ifdef RSEQ_COMPARE_TWICE
366
+
error1:
367
+
rseq_after_asm_goto();
368
+
rseq_bug("cpu_id comparison failed");
369
+
error2:
370
+
rseq_after_asm_goto();
371
+
rseq_bug("expected value comparison failed");
372
+
#endif
373
+
}
374
+
375
+
376
+
static inline __attribute__((always_inline))
377
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
378
+
void *dst, void *src, size_t len,
379
+
intptr_t newv, int cpu)
380
+
{
381
+
uint32_t rseq_scratch[3];
382
+
383
+
RSEQ_INJECT_C(9)
384
+
385
+
__asm__ __volatile__ goto (
386
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
387
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
388
+
#ifdef RSEQ_COMPARE_TWICE
389
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
390
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
391
+
#endif
392
+
"str %[src], %[rseq_scratch0]\n\t"
393
+
"str %[dst], %[rseq_scratch1]\n\t"
394
+
"str %[len], %[rseq_scratch2]\n\t"
395
+
/* Start rseq by storing table entry pointer into rseq_cs. */
396
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
397
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
398
+
RSEQ_INJECT_ASM(3)
399
+
"ldr r0, %[v]\n\t"
400
+
"cmp %[expect], r0\n\t"
401
+
"bne 5f\n\t"
402
+
RSEQ_INJECT_ASM(4)
403
+
#ifdef RSEQ_COMPARE_TWICE
404
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
405
+
"ldr r0, %[v]\n\t"
406
+
"cmp %[expect], r0\n\t"
407
+
"bne 7f\n\t"
408
+
#endif
409
+
/* try memcpy */
410
+
"cmp %[len], #0\n\t" \
411
+
"beq 333f\n\t" \
412
+
"222:\n\t" \
413
+
"ldrb %%r0, [%[src]]\n\t" \
414
+
"strb %%r0, [%[dst]]\n\t" \
415
+
"adds %[src], #1\n\t" \
416
+
"adds %[dst], #1\n\t" \
417
+
"subs %[len], #1\n\t" \
418
+
"bne 222b\n\t" \
419
+
"333:\n\t" \
420
+
RSEQ_INJECT_ASM(5)
421
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
422
+
"dmb\n\t" /* full mb provides store-release */
423
+
#endif
424
+
/* final store */
425
+
"str %[newv], %[v]\n\t"
426
+
"2:\n\t"
427
+
RSEQ_INJECT_ASM(6)
428
+
/* teardown */
429
+
"ldr %[len], %[rseq_scratch2]\n\t"
430
+
"ldr %[dst], %[rseq_scratch1]\n\t"
431
+
"ldr %[src], %[rseq_scratch0]\n\t"
432
+
"b 8f\n\t"
433
+
RSEQ_ASM_DEFINE_ABORT(3, 4,
434
+
/* teardown */
435
+
"ldr %[len], %[rseq_scratch2]\n\t"
436
+
"ldr %[dst], %[rseq_scratch1]\n\t"
437
+
"ldr %[src], %[rseq_scratch0]\n\t",
438
+
abort, 1b, 2b, 4f)
439
+
RSEQ_ASM_DEFINE_CMPFAIL(5,
440
+
/* teardown */
441
+
"ldr %[len], %[rseq_scratch2]\n\t"
442
+
"ldr %[dst], %[rseq_scratch1]\n\t"
443
+
"ldr %[src], %[rseq_scratch0]\n\t",
444
+
cmpfail)
445
+
#ifdef RSEQ_COMPARE_TWICE
446
+
RSEQ_ASM_DEFINE_CMPFAIL(6,
447
+
/* teardown */
448
+
"ldr %[len], %[rseq_scratch2]\n\t"
449
+
"ldr %[dst], %[rseq_scratch1]\n\t"
450
+
"ldr %[src], %[rseq_scratch0]\n\t",
451
+
error1)
452
+
RSEQ_ASM_DEFINE_CMPFAIL(7,
453
+
/* teardown */
454
+
"ldr %[len], %[rseq_scratch2]\n\t"
455
+
"ldr %[dst], %[rseq_scratch1]\n\t"
456
+
"ldr %[src], %[rseq_scratch0]\n\t",
457
+
error2)
458
+
#endif
459
+
"8:\n\t"
460
+
: /* gcc asm goto does not allow outputs */
461
+
: [cpu_id] "r" (cpu),
462
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
463
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
464
+
/* final store input */
465
+
[v] "m" (*v),
466
+
[expect] "r" (expect),
467
+
[newv] "r" (newv),
468
+
/* try memcpy input */
469
+
[dst] "r" (dst),
470
+
[src] "r" (src),
471
+
[len] "r" (len),
472
+
[rseq_scratch0] "m" (rseq_scratch[0]),
473
+
[rseq_scratch1] "m" (rseq_scratch[1]),
474
+
[rseq_scratch2] "m" (rseq_scratch[2])
475
+
RSEQ_INJECT_INPUT
476
+
: "r0", "memory", "cc"
477
+
RSEQ_INJECT_CLOBBER
478
+
: abort, cmpfail
479
+
#ifdef RSEQ_COMPARE_TWICE
480
+
, error1, error2
481
+
#endif
482
+
);
483
+
rseq_after_asm_goto();
484
+
return 0;
485
+
abort:
486
+
rseq_after_asm_goto();
487
+
RSEQ_INJECT_FAILED
488
+
return -1;
489
+
cmpfail:
490
+
rseq_after_asm_goto();
491
+
return 1;
492
+
#ifdef RSEQ_COMPARE_TWICE
493
+
error1:
494
+
rseq_after_asm_goto();
495
+
rseq_bug("cpu_id comparison failed");
496
+
error2:
497
+
rseq_after_asm_goto();
498
+
rseq_bug("expected value comparison failed");
499
+
#endif
500
+
}
501
+
502
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
503
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
504
+
505
+
#include "rseq-bits-reset.h"
+25
-676
tools/testing/selftests/rseq/rseq-arm.h
+25
-676
tools/testing/selftests/rseq/rseq-arm.h
···
2
2
/*
3
3
* rseq-arm.h
4
4
*
5
-
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
6
*/
7
7
8
8
/*
···
79
79
RSEQ_WRITE_ONCE(*p, v); \
80
80
} while (0)
81
81
82
-
#ifdef RSEQ_SKIP_FASTPATH
83
-
#include "rseq-skip.h"
84
-
#else /* !RSEQ_SKIP_FASTPATH */
85
-
86
82
#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \
87
83
post_commit_offset, abort_ip) \
88
84
".pushsection __rseq_cs, \"aw\"\n\t" \
···
143
147
teardown \
144
148
"b %l[" __rseq_str(cmpfail_label) "]\n\t"
145
149
146
-
static inline __attribute__((always_inline))
147
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
148
-
{
149
-
RSEQ_INJECT_C(9)
150
+
/* Per-cpu-id indexing. */
150
151
151
-
__asm__ __volatile__ goto (
152
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
153
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
154
-
#ifdef RSEQ_COMPARE_TWICE
155
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
156
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
157
-
#endif
158
-
/* Start rseq by storing table entry pointer into rseq_cs. */
159
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
160
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
161
-
RSEQ_INJECT_ASM(3)
162
-
"ldr r0, %[v]\n\t"
163
-
"cmp %[expect], r0\n\t"
164
-
"bne %l[cmpfail]\n\t"
165
-
RSEQ_INJECT_ASM(4)
166
-
#ifdef RSEQ_COMPARE_TWICE
167
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
168
-
"ldr r0, %[v]\n\t"
169
-
"cmp %[expect], r0\n\t"
170
-
"bne %l[error2]\n\t"
171
-
#endif
172
-
/* final store */
173
-
"str %[newv], %[v]\n\t"
174
-
"2:\n\t"
175
-
RSEQ_INJECT_ASM(5)
176
-
"b 5f\n\t"
177
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
178
-
"5:\n\t"
179
-
: /* gcc asm goto does not allow outputs */
180
-
: [cpu_id] "r" (cpu),
181
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
182
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
183
-
[v] "m" (*v),
184
-
[expect] "r" (expect),
185
-
[newv] "r" (newv)
186
-
RSEQ_INJECT_INPUT
187
-
: "r0", "memory", "cc"
188
-
RSEQ_INJECT_CLOBBER
189
-
: abort, cmpfail
190
-
#ifdef RSEQ_COMPARE_TWICE
191
-
, error1, error2
192
-
#endif
193
-
);
194
-
rseq_after_asm_goto();
195
-
return 0;
196
-
abort:
197
-
rseq_after_asm_goto();
198
-
RSEQ_INJECT_FAILED
199
-
return -1;
200
-
cmpfail:
201
-
rseq_after_asm_goto();
202
-
return 1;
203
-
#ifdef RSEQ_COMPARE_TWICE
204
-
error1:
205
-
rseq_after_asm_goto();
206
-
rseq_bug("cpu_id comparison failed");
207
-
error2:
208
-
rseq_after_asm_goto();
209
-
rseq_bug("expected value comparison failed");
210
-
#endif
211
-
}
152
+
#define RSEQ_TEMPLATE_CPU_ID
153
+
#define RSEQ_TEMPLATE_MO_RELAXED
154
+
#include "rseq-arm-bits.h"
155
+
#undef RSEQ_TEMPLATE_MO_RELAXED
212
156
213
-
static inline __attribute__((always_inline))
214
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
215
-
long voffp, intptr_t *load, int cpu)
216
-
{
217
-
RSEQ_INJECT_C(9)
157
+
#define RSEQ_TEMPLATE_MO_RELEASE
158
+
#include "rseq-arm-bits.h"
159
+
#undef RSEQ_TEMPLATE_MO_RELEASE
160
+
#undef RSEQ_TEMPLATE_CPU_ID
218
161
219
-
__asm__ __volatile__ goto (
220
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
221
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
222
-
#ifdef RSEQ_COMPARE_TWICE
223
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
224
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
225
-
#endif
226
-
/* Start rseq by storing table entry pointer into rseq_cs. */
227
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
228
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
229
-
RSEQ_INJECT_ASM(3)
230
-
"ldr r0, %[v]\n\t"
231
-
"cmp %[expectnot], r0\n\t"
232
-
"beq %l[cmpfail]\n\t"
233
-
RSEQ_INJECT_ASM(4)
234
-
#ifdef RSEQ_COMPARE_TWICE
235
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
236
-
"ldr r0, %[v]\n\t"
237
-
"cmp %[expectnot], r0\n\t"
238
-
"beq %l[error2]\n\t"
239
-
#endif
240
-
"str r0, %[load]\n\t"
241
-
"add r0, %[voffp]\n\t"
242
-
"ldr r0, [r0]\n\t"
243
-
/* final store */
244
-
"str r0, %[v]\n\t"
245
-
"2:\n\t"
246
-
RSEQ_INJECT_ASM(5)
247
-
"b 5f\n\t"
248
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
249
-
"5:\n\t"
250
-
: /* gcc asm goto does not allow outputs */
251
-
: [cpu_id] "r" (cpu),
252
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
253
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
254
-
/* final store input */
255
-
[v] "m" (*v),
256
-
[expectnot] "r" (expectnot),
257
-
[voffp] "Ir" (voffp),
258
-
[load] "m" (*load)
259
-
RSEQ_INJECT_INPUT
260
-
: "r0", "memory", "cc"
261
-
RSEQ_INJECT_CLOBBER
262
-
: abort, cmpfail
263
-
#ifdef RSEQ_COMPARE_TWICE
264
-
, error1, error2
265
-
#endif
266
-
);
267
-
rseq_after_asm_goto();
268
-
return 0;
269
-
abort:
270
-
rseq_after_asm_goto();
271
-
RSEQ_INJECT_FAILED
272
-
return -1;
273
-
cmpfail:
274
-
rseq_after_asm_goto();
275
-
return 1;
276
-
#ifdef RSEQ_COMPARE_TWICE
277
-
error1:
278
-
rseq_after_asm_goto();
279
-
rseq_bug("cpu_id comparison failed");
280
-
error2:
281
-
rseq_after_asm_goto();
282
-
rseq_bug("expected value comparison failed");
283
-
#endif
284
-
}
162
+
/* Per-mm-cid indexing. */
285
163
286
-
static inline __attribute__((always_inline))
287
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
288
-
{
289
-
RSEQ_INJECT_C(9)
164
+
#define RSEQ_TEMPLATE_MM_CID
165
+
#define RSEQ_TEMPLATE_MO_RELAXED
166
+
#include "rseq-arm-bits.h"
167
+
#undef RSEQ_TEMPLATE_MO_RELAXED
290
168
291
-
__asm__ __volatile__ goto (
292
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
293
-
#ifdef RSEQ_COMPARE_TWICE
294
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
295
-
#endif
296
-
/* Start rseq by storing table entry pointer into rseq_cs. */
297
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
298
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
299
-
RSEQ_INJECT_ASM(3)
300
-
#ifdef RSEQ_COMPARE_TWICE
301
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
302
-
#endif
303
-
"ldr r0, %[v]\n\t"
304
-
"add r0, %[count]\n\t"
305
-
/* final store */
306
-
"str r0, %[v]\n\t"
307
-
"2:\n\t"
308
-
RSEQ_INJECT_ASM(4)
309
-
"b 5f\n\t"
310
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
311
-
"5:\n\t"
312
-
: /* gcc asm goto does not allow outputs */
313
-
: [cpu_id] "r" (cpu),
314
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
315
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
316
-
[v] "m" (*v),
317
-
[count] "Ir" (count)
318
-
RSEQ_INJECT_INPUT
319
-
: "r0", "memory", "cc"
320
-
RSEQ_INJECT_CLOBBER
321
-
: abort
322
-
#ifdef RSEQ_COMPARE_TWICE
323
-
, error1
324
-
#endif
325
-
);
326
-
rseq_after_asm_goto();
327
-
return 0;
328
-
abort:
329
-
rseq_after_asm_goto();
330
-
RSEQ_INJECT_FAILED
331
-
return -1;
332
-
#ifdef RSEQ_COMPARE_TWICE
333
-
error1:
334
-
rseq_after_asm_goto();
335
-
rseq_bug("cpu_id comparison failed");
336
-
#endif
337
-
}
169
+
#define RSEQ_TEMPLATE_MO_RELEASE
170
+
#include "rseq-arm-bits.h"
171
+
#undef RSEQ_TEMPLATE_MO_RELEASE
172
+
#undef RSEQ_TEMPLATE_MM_CID
338
173
339
-
static inline __attribute__((always_inline))
340
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
341
-
intptr_t *v2, intptr_t newv2,
342
-
intptr_t newv, int cpu)
343
-
{
344
-
RSEQ_INJECT_C(9)
174
+
/* APIs which are not based on cpu ids. */
345
175
346
-
__asm__ __volatile__ goto (
347
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
348
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
349
-
#ifdef RSEQ_COMPARE_TWICE
350
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
351
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
352
-
#endif
353
-
/* Start rseq by storing table entry pointer into rseq_cs. */
354
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
355
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
356
-
RSEQ_INJECT_ASM(3)
357
-
"ldr r0, %[v]\n\t"
358
-
"cmp %[expect], r0\n\t"
359
-
"bne %l[cmpfail]\n\t"
360
-
RSEQ_INJECT_ASM(4)
361
-
#ifdef RSEQ_COMPARE_TWICE
362
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
363
-
"ldr r0, %[v]\n\t"
364
-
"cmp %[expect], r0\n\t"
365
-
"bne %l[error2]\n\t"
366
-
#endif
367
-
/* try store */
368
-
"str %[newv2], %[v2]\n\t"
369
-
RSEQ_INJECT_ASM(5)
370
-
/* final store */
371
-
"str %[newv], %[v]\n\t"
372
-
"2:\n\t"
373
-
RSEQ_INJECT_ASM(6)
374
-
"b 5f\n\t"
375
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
376
-
"5:\n\t"
377
-
: /* gcc asm goto does not allow outputs */
378
-
: [cpu_id] "r" (cpu),
379
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
380
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
381
-
/* try store input */
382
-
[v2] "m" (*v2),
383
-
[newv2] "r" (newv2),
384
-
/* final store input */
385
-
[v] "m" (*v),
386
-
[expect] "r" (expect),
387
-
[newv] "r" (newv)
388
-
RSEQ_INJECT_INPUT
389
-
: "r0", "memory", "cc"
390
-
RSEQ_INJECT_CLOBBER
391
-
: abort, cmpfail
392
-
#ifdef RSEQ_COMPARE_TWICE
393
-
, error1, error2
394
-
#endif
395
-
);
396
-
rseq_after_asm_goto();
397
-
return 0;
398
-
abort:
399
-
rseq_after_asm_goto();
400
-
RSEQ_INJECT_FAILED
401
-
return -1;
402
-
cmpfail:
403
-
rseq_after_asm_goto();
404
-
return 1;
405
-
#ifdef RSEQ_COMPARE_TWICE
406
-
error1:
407
-
rseq_after_asm_goto();
408
-
rseq_bug("cpu_id comparison failed");
409
-
error2:
410
-
rseq_after_asm_goto();
411
-
rseq_bug("expected value comparison failed");
412
-
#endif
413
-
}
414
-
415
-
static inline __attribute__((always_inline))
416
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
417
-
intptr_t *v2, intptr_t newv2,
418
-
intptr_t newv, int cpu)
419
-
{
420
-
RSEQ_INJECT_C(9)
421
-
422
-
__asm__ __volatile__ goto (
423
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
424
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
425
-
#ifdef RSEQ_COMPARE_TWICE
426
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
427
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
428
-
#endif
429
-
/* Start rseq by storing table entry pointer into rseq_cs. */
430
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
431
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
432
-
RSEQ_INJECT_ASM(3)
433
-
"ldr r0, %[v]\n\t"
434
-
"cmp %[expect], r0\n\t"
435
-
"bne %l[cmpfail]\n\t"
436
-
RSEQ_INJECT_ASM(4)
437
-
#ifdef RSEQ_COMPARE_TWICE
438
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
439
-
"ldr r0, %[v]\n\t"
440
-
"cmp %[expect], r0\n\t"
441
-
"bne %l[error2]\n\t"
442
-
#endif
443
-
/* try store */
444
-
"str %[newv2], %[v2]\n\t"
445
-
RSEQ_INJECT_ASM(5)
446
-
"dmb\n\t" /* full mb provides store-release */
447
-
/* final store */
448
-
"str %[newv], %[v]\n\t"
449
-
"2:\n\t"
450
-
RSEQ_INJECT_ASM(6)
451
-
"b 5f\n\t"
452
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
453
-
"5:\n\t"
454
-
: /* gcc asm goto does not allow outputs */
455
-
: [cpu_id] "r" (cpu),
456
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
457
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
458
-
/* try store input */
459
-
[v2] "m" (*v2),
460
-
[newv2] "r" (newv2),
461
-
/* final store input */
462
-
[v] "m" (*v),
463
-
[expect] "r" (expect),
464
-
[newv] "r" (newv)
465
-
RSEQ_INJECT_INPUT
466
-
: "r0", "memory", "cc"
467
-
RSEQ_INJECT_CLOBBER
468
-
: abort, cmpfail
469
-
#ifdef RSEQ_COMPARE_TWICE
470
-
, error1, error2
471
-
#endif
472
-
);
473
-
rseq_after_asm_goto();
474
-
return 0;
475
-
abort:
476
-
rseq_after_asm_goto();
477
-
RSEQ_INJECT_FAILED
478
-
return -1;
479
-
cmpfail:
480
-
rseq_after_asm_goto();
481
-
return 1;
482
-
#ifdef RSEQ_COMPARE_TWICE
483
-
error1:
484
-
rseq_after_asm_goto();
485
-
rseq_bug("cpu_id comparison failed");
486
-
error2:
487
-
rseq_after_asm_goto();
488
-
rseq_bug("expected value comparison failed");
489
-
#endif
490
-
}
491
-
492
-
static inline __attribute__((always_inline))
493
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
494
-
intptr_t *v2, intptr_t expect2,
495
-
intptr_t newv, int cpu)
496
-
{
497
-
RSEQ_INJECT_C(9)
498
-
499
-
__asm__ __volatile__ goto (
500
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
501
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
502
-
#ifdef RSEQ_COMPARE_TWICE
503
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
504
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
505
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
506
-
#endif
507
-
/* Start rseq by storing table entry pointer into rseq_cs. */
508
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
509
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
510
-
RSEQ_INJECT_ASM(3)
511
-
"ldr r0, %[v]\n\t"
512
-
"cmp %[expect], r0\n\t"
513
-
"bne %l[cmpfail]\n\t"
514
-
RSEQ_INJECT_ASM(4)
515
-
"ldr r0, %[v2]\n\t"
516
-
"cmp %[expect2], r0\n\t"
517
-
"bne %l[cmpfail]\n\t"
518
-
RSEQ_INJECT_ASM(5)
519
-
#ifdef RSEQ_COMPARE_TWICE
520
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
521
-
"ldr r0, %[v]\n\t"
522
-
"cmp %[expect], r0\n\t"
523
-
"bne %l[error2]\n\t"
524
-
"ldr r0, %[v2]\n\t"
525
-
"cmp %[expect2], r0\n\t"
526
-
"bne %l[error3]\n\t"
527
-
#endif
528
-
/* final store */
529
-
"str %[newv], %[v]\n\t"
530
-
"2:\n\t"
531
-
RSEQ_INJECT_ASM(6)
532
-
"b 5f\n\t"
533
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
534
-
"5:\n\t"
535
-
: /* gcc asm goto does not allow outputs */
536
-
: [cpu_id] "r" (cpu),
537
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
538
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
539
-
/* cmp2 input */
540
-
[v2] "m" (*v2),
541
-
[expect2] "r" (expect2),
542
-
/* final store input */
543
-
[v] "m" (*v),
544
-
[expect] "r" (expect),
545
-
[newv] "r" (newv)
546
-
RSEQ_INJECT_INPUT
547
-
: "r0", "memory", "cc"
548
-
RSEQ_INJECT_CLOBBER
549
-
: abort, cmpfail
550
-
#ifdef RSEQ_COMPARE_TWICE
551
-
, error1, error2, error3
552
-
#endif
553
-
);
554
-
rseq_after_asm_goto();
555
-
return 0;
556
-
abort:
557
-
rseq_after_asm_goto();
558
-
RSEQ_INJECT_FAILED
559
-
return -1;
560
-
cmpfail:
561
-
rseq_after_asm_goto();
562
-
return 1;
563
-
#ifdef RSEQ_COMPARE_TWICE
564
-
error1:
565
-
rseq_after_asm_goto();
566
-
rseq_bug("cpu_id comparison failed");
567
-
error2:
568
-
rseq_after_asm_goto();
569
-
rseq_bug("1st expected value comparison failed");
570
-
error3:
571
-
rseq_after_asm_goto();
572
-
rseq_bug("2nd expected value comparison failed");
573
-
#endif
574
-
}
575
-
576
-
static inline __attribute__((always_inline))
577
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
578
-
void *dst, void *src, size_t len,
579
-
intptr_t newv, int cpu)
580
-
{
581
-
uint32_t rseq_scratch[3];
582
-
583
-
RSEQ_INJECT_C(9)
584
-
585
-
__asm__ __volatile__ goto (
586
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
587
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
588
-
#ifdef RSEQ_COMPARE_TWICE
589
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
590
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
591
-
#endif
592
-
"str %[src], %[rseq_scratch0]\n\t"
593
-
"str %[dst], %[rseq_scratch1]\n\t"
594
-
"str %[len], %[rseq_scratch2]\n\t"
595
-
/* Start rseq by storing table entry pointer into rseq_cs. */
596
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
597
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
598
-
RSEQ_INJECT_ASM(3)
599
-
"ldr r0, %[v]\n\t"
600
-
"cmp %[expect], r0\n\t"
601
-
"bne 5f\n\t"
602
-
RSEQ_INJECT_ASM(4)
603
-
#ifdef RSEQ_COMPARE_TWICE
604
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
605
-
"ldr r0, %[v]\n\t"
606
-
"cmp %[expect], r0\n\t"
607
-
"bne 7f\n\t"
608
-
#endif
609
-
/* try memcpy */
610
-
"cmp %[len], #0\n\t" \
611
-
"beq 333f\n\t" \
612
-
"222:\n\t" \
613
-
"ldrb %%r0, [%[src]]\n\t" \
614
-
"strb %%r0, [%[dst]]\n\t" \
615
-
"adds %[src], #1\n\t" \
616
-
"adds %[dst], #1\n\t" \
617
-
"subs %[len], #1\n\t" \
618
-
"bne 222b\n\t" \
619
-
"333:\n\t" \
620
-
RSEQ_INJECT_ASM(5)
621
-
/* final store */
622
-
"str %[newv], %[v]\n\t"
623
-
"2:\n\t"
624
-
RSEQ_INJECT_ASM(6)
625
-
/* teardown */
626
-
"ldr %[len], %[rseq_scratch2]\n\t"
627
-
"ldr %[dst], %[rseq_scratch1]\n\t"
628
-
"ldr %[src], %[rseq_scratch0]\n\t"
629
-
"b 8f\n\t"
630
-
RSEQ_ASM_DEFINE_ABORT(3, 4,
631
-
/* teardown */
632
-
"ldr %[len], %[rseq_scratch2]\n\t"
633
-
"ldr %[dst], %[rseq_scratch1]\n\t"
634
-
"ldr %[src], %[rseq_scratch0]\n\t",
635
-
abort, 1b, 2b, 4f)
636
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
637
-
/* teardown */
638
-
"ldr %[len], %[rseq_scratch2]\n\t"
639
-
"ldr %[dst], %[rseq_scratch1]\n\t"
640
-
"ldr %[src], %[rseq_scratch0]\n\t",
641
-
cmpfail)
642
-
#ifdef RSEQ_COMPARE_TWICE
643
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
644
-
/* teardown */
645
-
"ldr %[len], %[rseq_scratch2]\n\t"
646
-
"ldr %[dst], %[rseq_scratch1]\n\t"
647
-
"ldr %[src], %[rseq_scratch0]\n\t",
648
-
error1)
649
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
650
-
/* teardown */
651
-
"ldr %[len], %[rseq_scratch2]\n\t"
652
-
"ldr %[dst], %[rseq_scratch1]\n\t"
653
-
"ldr %[src], %[rseq_scratch0]\n\t",
654
-
error2)
655
-
#endif
656
-
"8:\n\t"
657
-
: /* gcc asm goto does not allow outputs */
658
-
: [cpu_id] "r" (cpu),
659
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
660
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
661
-
/* final store input */
662
-
[v] "m" (*v),
663
-
[expect] "r" (expect),
664
-
[newv] "r" (newv),
665
-
/* try memcpy input */
666
-
[dst] "r" (dst),
667
-
[src] "r" (src),
668
-
[len] "r" (len),
669
-
[rseq_scratch0] "m" (rseq_scratch[0]),
670
-
[rseq_scratch1] "m" (rseq_scratch[1]),
671
-
[rseq_scratch2] "m" (rseq_scratch[2])
672
-
RSEQ_INJECT_INPUT
673
-
: "r0", "memory", "cc"
674
-
RSEQ_INJECT_CLOBBER
675
-
: abort, cmpfail
676
-
#ifdef RSEQ_COMPARE_TWICE
677
-
, error1, error2
678
-
#endif
679
-
);
680
-
rseq_after_asm_goto();
681
-
return 0;
682
-
abort:
683
-
rseq_after_asm_goto();
684
-
RSEQ_INJECT_FAILED
685
-
return -1;
686
-
cmpfail:
687
-
rseq_after_asm_goto();
688
-
return 1;
689
-
#ifdef RSEQ_COMPARE_TWICE
690
-
error1:
691
-
rseq_after_asm_goto();
692
-
rseq_bug("cpu_id comparison failed");
693
-
error2:
694
-
rseq_after_asm_goto();
695
-
rseq_bug("expected value comparison failed");
696
-
#endif
697
-
}
698
-
699
-
static inline __attribute__((always_inline))
700
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
701
-
void *dst, void *src, size_t len,
702
-
intptr_t newv, int cpu)
703
-
{
704
-
uint32_t rseq_scratch[3];
705
-
706
-
RSEQ_INJECT_C(9)
707
-
708
-
__asm__ __volatile__ goto (
709
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
710
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
711
-
#ifdef RSEQ_COMPARE_TWICE
712
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
713
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
714
-
#endif
715
-
"str %[src], %[rseq_scratch0]\n\t"
716
-
"str %[dst], %[rseq_scratch1]\n\t"
717
-
"str %[len], %[rseq_scratch2]\n\t"
718
-
/* Start rseq by storing table entry pointer into rseq_cs. */
719
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
720
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
721
-
RSEQ_INJECT_ASM(3)
722
-
"ldr r0, %[v]\n\t"
723
-
"cmp %[expect], r0\n\t"
724
-
"bne 5f\n\t"
725
-
RSEQ_INJECT_ASM(4)
726
-
#ifdef RSEQ_COMPARE_TWICE
727
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
728
-
"ldr r0, %[v]\n\t"
729
-
"cmp %[expect], r0\n\t"
730
-
"bne 7f\n\t"
731
-
#endif
732
-
/* try memcpy */
733
-
"cmp %[len], #0\n\t" \
734
-
"beq 333f\n\t" \
735
-
"222:\n\t" \
736
-
"ldrb %%r0, [%[src]]\n\t" \
737
-
"strb %%r0, [%[dst]]\n\t" \
738
-
"adds %[src], #1\n\t" \
739
-
"adds %[dst], #1\n\t" \
740
-
"subs %[len], #1\n\t" \
741
-
"bne 222b\n\t" \
742
-
"333:\n\t" \
743
-
RSEQ_INJECT_ASM(5)
744
-
"dmb\n\t" /* full mb provides store-release */
745
-
/* final store */
746
-
"str %[newv], %[v]\n\t"
747
-
"2:\n\t"
748
-
RSEQ_INJECT_ASM(6)
749
-
/* teardown */
750
-
"ldr %[len], %[rseq_scratch2]\n\t"
751
-
"ldr %[dst], %[rseq_scratch1]\n\t"
752
-
"ldr %[src], %[rseq_scratch0]\n\t"
753
-
"b 8f\n\t"
754
-
RSEQ_ASM_DEFINE_ABORT(3, 4,
755
-
/* teardown */
756
-
"ldr %[len], %[rseq_scratch2]\n\t"
757
-
"ldr %[dst], %[rseq_scratch1]\n\t"
758
-
"ldr %[src], %[rseq_scratch0]\n\t",
759
-
abort, 1b, 2b, 4f)
760
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
761
-
/* teardown */
762
-
"ldr %[len], %[rseq_scratch2]\n\t"
763
-
"ldr %[dst], %[rseq_scratch1]\n\t"
764
-
"ldr %[src], %[rseq_scratch0]\n\t",
765
-
cmpfail)
766
-
#ifdef RSEQ_COMPARE_TWICE
767
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
768
-
/* teardown */
769
-
"ldr %[len], %[rseq_scratch2]\n\t"
770
-
"ldr %[dst], %[rseq_scratch1]\n\t"
771
-
"ldr %[src], %[rseq_scratch0]\n\t",
772
-
error1)
773
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
774
-
/* teardown */
775
-
"ldr %[len], %[rseq_scratch2]\n\t"
776
-
"ldr %[dst], %[rseq_scratch1]\n\t"
777
-
"ldr %[src], %[rseq_scratch0]\n\t",
778
-
error2)
779
-
#endif
780
-
"8:\n\t"
781
-
: /* gcc asm goto does not allow outputs */
782
-
: [cpu_id] "r" (cpu),
783
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
784
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
785
-
/* final store input */
786
-
[v] "m" (*v),
787
-
[expect] "r" (expect),
788
-
[newv] "r" (newv),
789
-
/* try memcpy input */
790
-
[dst] "r" (dst),
791
-
[src] "r" (src),
792
-
[len] "r" (len),
793
-
[rseq_scratch0] "m" (rseq_scratch[0]),
794
-
[rseq_scratch1] "m" (rseq_scratch[1]),
795
-
[rseq_scratch2] "m" (rseq_scratch[2])
796
-
RSEQ_INJECT_INPUT
797
-
: "r0", "memory", "cc"
798
-
RSEQ_INJECT_CLOBBER
799
-
: abort, cmpfail
800
-
#ifdef RSEQ_COMPARE_TWICE
801
-
, error1, error2
802
-
#endif
803
-
);
804
-
rseq_after_asm_goto();
805
-
return 0;
806
-
abort:
807
-
rseq_after_asm_goto();
808
-
RSEQ_INJECT_FAILED
809
-
return -1;
810
-
cmpfail:
811
-
rseq_after_asm_goto();
812
-
return 1;
813
-
#ifdef RSEQ_COMPARE_TWICE
814
-
error1:
815
-
rseq_after_asm_goto();
816
-
rseq_bug("cpu_id comparison failed");
817
-
error2:
818
-
rseq_after_asm_goto();
819
-
rseq_bug("expected value comparison failed");
820
-
#endif
821
-
}
822
-
823
-
#endif /* !RSEQ_SKIP_FASTPATH */
176
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
177
+
#define RSEQ_TEMPLATE_MO_RELAXED
178
+
#include "rseq-arm-bits.h"
179
+
#undef RSEQ_TEMPLATE_MO_RELAXED
180
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+392
tools/testing/selftests/rseq/rseq-arm64-bits.h
+392
tools/testing/selftests/rseq/rseq-arm64-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-arm64-bits.h
4
+
*
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
* (C) Copyright 2018 - Will Deacon <will.deacon@arm.com>
7
+
*/
8
+
9
+
#include "rseq-bits-template.h"
10
+
11
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
12
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
13
+
14
+
static inline __attribute__((always_inline))
15
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
16
+
{
17
+
RSEQ_INJECT_C(9)
18
+
19
+
__asm__ __volatile__ goto (
20
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
21
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
22
+
#ifdef RSEQ_COMPARE_TWICE
23
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
24
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
25
+
#endif
26
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
27
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
28
+
RSEQ_INJECT_ASM(3)
29
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
30
+
RSEQ_INJECT_ASM(4)
31
+
#ifdef RSEQ_COMPARE_TWICE
32
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
33
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
34
+
#endif
35
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
36
+
RSEQ_INJECT_ASM(5)
37
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
38
+
: /* gcc asm goto does not allow outputs */
39
+
: [cpu_id] "r" (cpu),
40
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
41
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
42
+
[v] "Qo" (*v),
43
+
[expect] "r" (expect),
44
+
[newv] "r" (newv)
45
+
RSEQ_INJECT_INPUT
46
+
: "memory", RSEQ_ASM_TMP_REG
47
+
: abort, cmpfail
48
+
#ifdef RSEQ_COMPARE_TWICE
49
+
, error1, error2
50
+
#endif
51
+
);
52
+
rseq_after_asm_goto();
53
+
return 0;
54
+
abort:
55
+
rseq_after_asm_goto();
56
+
RSEQ_INJECT_FAILED
57
+
return -1;
58
+
cmpfail:
59
+
rseq_after_asm_goto();
60
+
return 1;
61
+
#ifdef RSEQ_COMPARE_TWICE
62
+
error1:
63
+
rseq_after_asm_goto();
64
+
rseq_bug("cpu_id comparison failed");
65
+
error2:
66
+
rseq_after_asm_goto();
67
+
rseq_bug("expected value comparison failed");
68
+
#endif
69
+
}
70
+
71
+
static inline __attribute__((always_inline))
72
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
73
+
long voffp, intptr_t *load, int cpu)
74
+
{
75
+
RSEQ_INJECT_C(9)
76
+
77
+
__asm__ __volatile__ goto (
78
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
79
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
80
+
#ifdef RSEQ_COMPARE_TWICE
81
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
82
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
83
+
#endif
84
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
85
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
86
+
RSEQ_INJECT_ASM(3)
87
+
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail])
88
+
RSEQ_INJECT_ASM(4)
89
+
#ifdef RSEQ_COMPARE_TWICE
90
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
91
+
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2])
92
+
#endif
93
+
RSEQ_ASM_OP_R_LOAD(v)
94
+
RSEQ_ASM_OP_R_STORE(load)
95
+
RSEQ_ASM_OP_R_LOAD_OFF(voffp)
96
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
97
+
RSEQ_INJECT_ASM(5)
98
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
99
+
: /* gcc asm goto does not allow outputs */
100
+
: [cpu_id] "r" (cpu),
101
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
102
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
103
+
[v] "Qo" (*v),
104
+
[expectnot] "r" (expectnot),
105
+
[load] "Qo" (*load),
106
+
[voffp] "r" (voffp)
107
+
RSEQ_INJECT_INPUT
108
+
: "memory", RSEQ_ASM_TMP_REG
109
+
: abort, cmpfail
110
+
#ifdef RSEQ_COMPARE_TWICE
111
+
, error1, error2
112
+
#endif
113
+
);
114
+
rseq_after_asm_goto();
115
+
return 0;
116
+
abort:
117
+
rseq_after_asm_goto();
118
+
RSEQ_INJECT_FAILED
119
+
return -1;
120
+
cmpfail:
121
+
rseq_after_asm_goto();
122
+
return 1;
123
+
#ifdef RSEQ_COMPARE_TWICE
124
+
error1:
125
+
rseq_after_asm_goto();
126
+
rseq_bug("cpu_id comparison failed");
127
+
error2:
128
+
rseq_after_asm_goto();
129
+
rseq_bug("expected value comparison failed");
130
+
#endif
131
+
}
132
+
133
+
static inline __attribute__((always_inline))
134
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
135
+
{
136
+
RSEQ_INJECT_C(9)
137
+
138
+
__asm__ __volatile__ goto (
139
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
140
+
#ifdef RSEQ_COMPARE_TWICE
141
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
142
+
#endif
143
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
144
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
145
+
RSEQ_INJECT_ASM(3)
146
+
#ifdef RSEQ_COMPARE_TWICE
147
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
148
+
#endif
149
+
RSEQ_ASM_OP_R_LOAD(v)
150
+
RSEQ_ASM_OP_R_ADD(count)
151
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
152
+
RSEQ_INJECT_ASM(4)
153
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
154
+
: /* gcc asm goto does not allow outputs */
155
+
: [cpu_id] "r" (cpu),
156
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
157
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
158
+
[v] "Qo" (*v),
159
+
[count] "r" (count)
160
+
RSEQ_INJECT_INPUT
161
+
: "memory", RSEQ_ASM_TMP_REG
162
+
: abort
163
+
#ifdef RSEQ_COMPARE_TWICE
164
+
, error1
165
+
#endif
166
+
);
167
+
rseq_after_asm_goto();
168
+
return 0;
169
+
abort:
170
+
rseq_after_asm_goto();
171
+
RSEQ_INJECT_FAILED
172
+
return -1;
173
+
#ifdef RSEQ_COMPARE_TWICE
174
+
error1:
175
+
rseq_after_asm_goto();
176
+
rseq_bug("cpu_id comparison failed");
177
+
#endif
178
+
}
179
+
180
+
static inline __attribute__((always_inline))
181
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
182
+
intptr_t *v2, intptr_t expect2,
183
+
intptr_t newv, int cpu)
184
+
{
185
+
RSEQ_INJECT_C(9)
186
+
187
+
__asm__ __volatile__ goto (
188
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
189
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
190
+
#ifdef RSEQ_COMPARE_TWICE
191
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
192
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
193
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3])
194
+
#endif
195
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
196
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
197
+
RSEQ_INJECT_ASM(3)
198
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
199
+
RSEQ_INJECT_ASM(4)
200
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail])
201
+
RSEQ_INJECT_ASM(5)
202
+
#ifdef RSEQ_COMPARE_TWICE
203
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
204
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
205
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3])
206
+
#endif
207
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
208
+
RSEQ_INJECT_ASM(6)
209
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
210
+
: /* gcc asm goto does not allow outputs */
211
+
: [cpu_id] "r" (cpu),
212
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
213
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
214
+
[v] "Qo" (*v),
215
+
[expect] "r" (expect),
216
+
[v2] "Qo" (*v2),
217
+
[expect2] "r" (expect2),
218
+
[newv] "r" (newv)
219
+
RSEQ_INJECT_INPUT
220
+
: "memory", RSEQ_ASM_TMP_REG
221
+
: abort, cmpfail
222
+
#ifdef RSEQ_COMPARE_TWICE
223
+
, error1, error2, error3
224
+
#endif
225
+
);
226
+
rseq_after_asm_goto();
227
+
return 0;
228
+
abort:
229
+
rseq_after_asm_goto();
230
+
RSEQ_INJECT_FAILED
231
+
return -1;
232
+
cmpfail:
233
+
rseq_after_asm_goto();
234
+
return 1;
235
+
#ifdef RSEQ_COMPARE_TWICE
236
+
error1:
237
+
rseq_after_asm_goto();
238
+
rseq_bug("cpu_id comparison failed");
239
+
error2:
240
+
rseq_after_asm_goto();
241
+
rseq_bug("expected value comparison failed");
242
+
error3:
243
+
rseq_after_asm_goto();
244
+
rseq_bug("2nd expected value comparison failed");
245
+
#endif
246
+
}
247
+
248
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
249
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
250
+
251
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
252
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
253
+
254
+
static inline __attribute__((always_inline))
255
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
256
+
intptr_t *v2, intptr_t newv2,
257
+
intptr_t newv, int cpu)
258
+
{
259
+
RSEQ_INJECT_C(9)
260
+
261
+
__asm__ __volatile__ goto (
262
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
263
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
264
+
#ifdef RSEQ_COMPARE_TWICE
265
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
266
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
267
+
#endif
268
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
269
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
270
+
RSEQ_INJECT_ASM(3)
271
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
272
+
RSEQ_INJECT_ASM(4)
273
+
#ifdef RSEQ_COMPARE_TWICE
274
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
275
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
276
+
#endif
277
+
RSEQ_ASM_OP_STORE(newv2, v2)
278
+
RSEQ_INJECT_ASM(5)
279
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
280
+
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
281
+
#else
282
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
283
+
#endif
284
+
RSEQ_INJECT_ASM(6)
285
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
286
+
: /* gcc asm goto does not allow outputs */
287
+
: [cpu_id] "r" (cpu),
288
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
289
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
290
+
[expect] "r" (expect),
291
+
[v] "Qo" (*v),
292
+
[newv] "r" (newv),
293
+
[v2] "Qo" (*v2),
294
+
[newv2] "r" (newv2)
295
+
RSEQ_INJECT_INPUT
296
+
: "memory", RSEQ_ASM_TMP_REG
297
+
: abort, cmpfail
298
+
#ifdef RSEQ_COMPARE_TWICE
299
+
, error1, error2
300
+
#endif
301
+
);
302
+
rseq_after_asm_goto();
303
+
return 0;
304
+
abort:
305
+
rseq_after_asm_goto();
306
+
RSEQ_INJECT_FAILED
307
+
return -1;
308
+
cmpfail:
309
+
rseq_after_asm_goto();
310
+
return 1;
311
+
#ifdef RSEQ_COMPARE_TWICE
312
+
error1:
313
+
rseq_after_asm_goto();
314
+
rseq_bug("cpu_id comparison failed");
315
+
error2:
316
+
rseq_after_asm_goto();
317
+
rseq_bug("expected value comparison failed");
318
+
#endif
319
+
}
320
+
321
+
static inline __attribute__((always_inline))
322
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
323
+
void *dst, void *src, size_t len,
324
+
intptr_t newv, int cpu)
325
+
{
326
+
RSEQ_INJECT_C(9)
327
+
328
+
__asm__ __volatile__ goto (
329
+
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
330
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
331
+
#ifdef RSEQ_COMPARE_TWICE
332
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
333
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
334
+
#endif
335
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
336
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
337
+
RSEQ_INJECT_ASM(3)
338
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
339
+
RSEQ_INJECT_ASM(4)
340
+
#ifdef RSEQ_COMPARE_TWICE
341
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
342
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
343
+
#endif
344
+
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
345
+
RSEQ_INJECT_ASM(5)
346
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
347
+
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
348
+
#else
349
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
350
+
#endif
351
+
RSEQ_INJECT_ASM(6)
352
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
353
+
: /* gcc asm goto does not allow outputs */
354
+
: [cpu_id] "r" (cpu),
355
+
[current_cpu_id] "Qo" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
356
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
357
+
[expect] "r" (expect),
358
+
[v] "Qo" (*v),
359
+
[newv] "r" (newv),
360
+
[dst] "r" (dst),
361
+
[src] "r" (src),
362
+
[len] "r" (len)
363
+
RSEQ_INJECT_INPUT
364
+
: "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2
365
+
: abort, cmpfail
366
+
#ifdef RSEQ_COMPARE_TWICE
367
+
, error1, error2
368
+
#endif
369
+
);
370
+
rseq_after_asm_goto();
371
+
return 0;
372
+
abort:
373
+
rseq_after_asm_goto();
374
+
RSEQ_INJECT_FAILED
375
+
return -1;
376
+
cmpfail:
377
+
rseq_after_asm_goto();
378
+
return 1;
379
+
#ifdef RSEQ_COMPARE_TWICE
380
+
error1:
381
+
rseq_after_asm_goto();
382
+
rseq_bug("cpu_id comparison failed");
383
+
error2:
384
+
rseq_after_asm_goto();
385
+
rseq_bug("expected value comparison failed");
386
+
#endif
387
+
}
388
+
389
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
390
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
391
+
392
+
#include "rseq-bits-reset.h"
+25
-487
tools/testing/selftests/rseq/rseq-arm64.h
+25
-487
tools/testing/selftests/rseq/rseq-arm64.h
···
2
2
/*
3
3
* rseq-arm64.h
4
4
*
5
-
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
6
* (C) Copyright 2018 - Will Deacon <will.deacon@arm.com>
7
7
*/
8
8
···
84
84
break; \
85
85
} \
86
86
} while (0)
87
-
88
-
#ifdef RSEQ_SKIP_FASTPATH
89
-
#include "rseq-skip.h"
90
-
#else /* !RSEQ_SKIP_FASTPATH */
91
87
92
88
#define RSEQ_ASM_TMP_REG32 "w15"
93
89
#define RSEQ_ASM_TMP_REG "x15"
···
200
204
" cbnz " RSEQ_ASM_TMP_REG_2 ", 222b\n" \
201
205
"333:\n"
202
206
203
-
static inline __attribute__((always_inline))
204
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
205
-
{
206
-
RSEQ_INJECT_C(9)
207
+
/* Per-cpu-id indexing. */
207
208
208
-
__asm__ __volatile__ goto (
209
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
210
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
211
-
#ifdef RSEQ_COMPARE_TWICE
212
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
213
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
214
-
#endif
215
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
216
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
217
-
RSEQ_INJECT_ASM(3)
218
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
219
-
RSEQ_INJECT_ASM(4)
220
-
#ifdef RSEQ_COMPARE_TWICE
221
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
222
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
223
-
#endif
224
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
225
-
RSEQ_INJECT_ASM(5)
226
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
227
-
: /* gcc asm goto does not allow outputs */
228
-
: [cpu_id] "r" (cpu),
229
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
230
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
231
-
[v] "Qo" (*v),
232
-
[expect] "r" (expect),
233
-
[newv] "r" (newv)
234
-
RSEQ_INJECT_INPUT
235
-
: "memory", RSEQ_ASM_TMP_REG
236
-
: abort, cmpfail
237
-
#ifdef RSEQ_COMPARE_TWICE
238
-
, error1, error2
239
-
#endif
240
-
);
241
-
rseq_after_asm_goto();
242
-
return 0;
243
-
abort:
244
-
rseq_after_asm_goto();
245
-
RSEQ_INJECT_FAILED
246
-
return -1;
247
-
cmpfail:
248
-
rseq_after_asm_goto();
249
-
return 1;
250
-
#ifdef RSEQ_COMPARE_TWICE
251
-
error1:
252
-
rseq_after_asm_goto();
253
-
rseq_bug("cpu_id comparison failed");
254
-
error2:
255
-
rseq_after_asm_goto();
256
-
rseq_bug("expected value comparison failed");
257
-
#endif
258
-
}
209
+
#define RSEQ_TEMPLATE_CPU_ID
210
+
#define RSEQ_TEMPLATE_MO_RELAXED
211
+
#include "rseq-arm64-bits.h"
212
+
#undef RSEQ_TEMPLATE_MO_RELAXED
259
213
260
-
static inline __attribute__((always_inline))
261
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
262
-
long voffp, intptr_t *load, int cpu)
263
-
{
264
-
RSEQ_INJECT_C(9)
214
+
#define RSEQ_TEMPLATE_MO_RELEASE
215
+
#include "rseq-arm64-bits.h"
216
+
#undef RSEQ_TEMPLATE_MO_RELEASE
217
+
#undef RSEQ_TEMPLATE_CPU_ID
265
218
266
-
__asm__ __volatile__ goto (
267
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
268
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
269
-
#ifdef RSEQ_COMPARE_TWICE
270
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
271
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
272
-
#endif
273
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
274
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
275
-
RSEQ_INJECT_ASM(3)
276
-
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail])
277
-
RSEQ_INJECT_ASM(4)
278
-
#ifdef RSEQ_COMPARE_TWICE
279
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
280
-
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2])
281
-
#endif
282
-
RSEQ_ASM_OP_R_LOAD(v)
283
-
RSEQ_ASM_OP_R_STORE(load)
284
-
RSEQ_ASM_OP_R_LOAD_OFF(voffp)
285
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
286
-
RSEQ_INJECT_ASM(5)
287
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
288
-
: /* gcc asm goto does not allow outputs */
289
-
: [cpu_id] "r" (cpu),
290
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
291
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
292
-
[v] "Qo" (*v),
293
-
[expectnot] "r" (expectnot),
294
-
[load] "Qo" (*load),
295
-
[voffp] "r" (voffp)
296
-
RSEQ_INJECT_INPUT
297
-
: "memory", RSEQ_ASM_TMP_REG
298
-
: abort, cmpfail
299
-
#ifdef RSEQ_COMPARE_TWICE
300
-
, error1, error2
301
-
#endif
302
-
);
303
-
rseq_after_asm_goto();
304
-
return 0;
305
-
abort:
306
-
rseq_after_asm_goto();
307
-
RSEQ_INJECT_FAILED
308
-
return -1;
309
-
cmpfail:
310
-
rseq_after_asm_goto();
311
-
return 1;
312
-
#ifdef RSEQ_COMPARE_TWICE
313
-
error1:
314
-
rseq_after_asm_goto();
315
-
rseq_bug("cpu_id comparison failed");
316
-
error2:
317
-
rseq_after_asm_goto();
318
-
rseq_bug("expected value comparison failed");
319
-
#endif
320
-
}
219
+
/* Per-mm-cid indexing. */
321
220
322
-
static inline __attribute__((always_inline))
323
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
324
-
{
325
-
RSEQ_INJECT_C(9)
221
+
#define RSEQ_TEMPLATE_MM_CID
222
+
#define RSEQ_TEMPLATE_MO_RELAXED
223
+
#include "rseq-arm64-bits.h"
224
+
#undef RSEQ_TEMPLATE_MO_RELAXED
326
225
327
-
__asm__ __volatile__ goto (
328
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
329
-
#ifdef RSEQ_COMPARE_TWICE
330
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
331
-
#endif
332
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
333
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
334
-
RSEQ_INJECT_ASM(3)
335
-
#ifdef RSEQ_COMPARE_TWICE
336
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
337
-
#endif
338
-
RSEQ_ASM_OP_R_LOAD(v)
339
-
RSEQ_ASM_OP_R_ADD(count)
340
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
341
-
RSEQ_INJECT_ASM(4)
342
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
343
-
: /* gcc asm goto does not allow outputs */
344
-
: [cpu_id] "r" (cpu),
345
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
346
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
347
-
[v] "Qo" (*v),
348
-
[count] "r" (count)
349
-
RSEQ_INJECT_INPUT
350
-
: "memory", RSEQ_ASM_TMP_REG
351
-
: abort
352
-
#ifdef RSEQ_COMPARE_TWICE
353
-
, error1
354
-
#endif
355
-
);
356
-
rseq_after_asm_goto();
357
-
return 0;
358
-
abort:
359
-
rseq_after_asm_goto();
360
-
RSEQ_INJECT_FAILED
361
-
return -1;
362
-
#ifdef RSEQ_COMPARE_TWICE
363
-
error1:
364
-
rseq_after_asm_goto();
365
-
rseq_bug("cpu_id comparison failed");
366
-
#endif
367
-
}
226
+
#define RSEQ_TEMPLATE_MO_RELEASE
227
+
#include "rseq-arm64-bits.h"
228
+
#undef RSEQ_TEMPLATE_MO_RELEASE
229
+
#undef RSEQ_TEMPLATE_MM_CID
368
230
369
-
static inline __attribute__((always_inline))
370
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
371
-
intptr_t *v2, intptr_t newv2,
372
-
intptr_t newv, int cpu)
373
-
{
374
-
RSEQ_INJECT_C(9)
231
+
/* APIs which are not based on cpu ids. */
375
232
376
-
__asm__ __volatile__ goto (
377
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
378
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
379
-
#ifdef RSEQ_COMPARE_TWICE
380
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
381
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
382
-
#endif
383
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
384
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
385
-
RSEQ_INJECT_ASM(3)
386
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
387
-
RSEQ_INJECT_ASM(4)
388
-
#ifdef RSEQ_COMPARE_TWICE
389
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
390
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
391
-
#endif
392
-
RSEQ_ASM_OP_STORE(newv2, v2)
393
-
RSEQ_INJECT_ASM(5)
394
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
395
-
RSEQ_INJECT_ASM(6)
396
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
397
-
: /* gcc asm goto does not allow outputs */
398
-
: [cpu_id] "r" (cpu),
399
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
400
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
401
-
[expect] "r" (expect),
402
-
[v] "Qo" (*v),
403
-
[newv] "r" (newv),
404
-
[v2] "Qo" (*v2),
405
-
[newv2] "r" (newv2)
406
-
RSEQ_INJECT_INPUT
407
-
: "memory", RSEQ_ASM_TMP_REG
408
-
: abort, cmpfail
409
-
#ifdef RSEQ_COMPARE_TWICE
410
-
, error1, error2
411
-
#endif
412
-
);
413
-
rseq_after_asm_goto();
414
-
return 0;
415
-
abort:
416
-
rseq_after_asm_goto();
417
-
RSEQ_INJECT_FAILED
418
-
return -1;
419
-
cmpfail:
420
-
rseq_after_asm_goto();
421
-
return 1;
422
-
#ifdef RSEQ_COMPARE_TWICE
423
-
error1:
424
-
rseq_after_asm_goto();
425
-
rseq_bug("cpu_id comparison failed");
426
-
error2:
427
-
rseq_after_asm_goto();
428
-
rseq_bug("expected value comparison failed");
429
-
#endif
430
-
}
431
-
432
-
static inline __attribute__((always_inline))
433
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
434
-
intptr_t *v2, intptr_t newv2,
435
-
intptr_t newv, int cpu)
436
-
{
437
-
RSEQ_INJECT_C(9)
438
-
439
-
__asm__ __volatile__ goto (
440
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
441
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
442
-
#ifdef RSEQ_COMPARE_TWICE
443
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
444
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
445
-
#endif
446
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
447
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
448
-
RSEQ_INJECT_ASM(3)
449
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
450
-
RSEQ_INJECT_ASM(4)
451
-
#ifdef RSEQ_COMPARE_TWICE
452
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
453
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
454
-
#endif
455
-
RSEQ_ASM_OP_STORE(newv2, v2)
456
-
RSEQ_INJECT_ASM(5)
457
-
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
458
-
RSEQ_INJECT_ASM(6)
459
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
460
-
: /* gcc asm goto does not allow outputs */
461
-
: [cpu_id] "r" (cpu),
462
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
463
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
464
-
[expect] "r" (expect),
465
-
[v] "Qo" (*v),
466
-
[newv] "r" (newv),
467
-
[v2] "Qo" (*v2),
468
-
[newv2] "r" (newv2)
469
-
RSEQ_INJECT_INPUT
470
-
: "memory", RSEQ_ASM_TMP_REG
471
-
: abort, cmpfail
472
-
#ifdef RSEQ_COMPARE_TWICE
473
-
, error1, error2
474
-
#endif
475
-
);
476
-
rseq_after_asm_goto();
477
-
return 0;
478
-
abort:
479
-
rseq_after_asm_goto();
480
-
RSEQ_INJECT_FAILED
481
-
return -1;
482
-
cmpfail:
483
-
rseq_after_asm_goto();
484
-
return 1;
485
-
#ifdef RSEQ_COMPARE_TWICE
486
-
error1:
487
-
rseq_after_asm_goto();
488
-
rseq_bug("cpu_id comparison failed");
489
-
error2:
490
-
rseq_after_asm_goto();
491
-
rseq_bug("expected value comparison failed");
492
-
#endif
493
-
}
494
-
495
-
static inline __attribute__((always_inline))
496
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
497
-
intptr_t *v2, intptr_t expect2,
498
-
intptr_t newv, int cpu)
499
-
{
500
-
RSEQ_INJECT_C(9)
501
-
502
-
__asm__ __volatile__ goto (
503
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
504
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
505
-
#ifdef RSEQ_COMPARE_TWICE
506
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
507
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
508
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3])
509
-
#endif
510
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
511
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
512
-
RSEQ_INJECT_ASM(3)
513
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
514
-
RSEQ_INJECT_ASM(4)
515
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail])
516
-
RSEQ_INJECT_ASM(5)
517
-
#ifdef RSEQ_COMPARE_TWICE
518
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
519
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
520
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3])
521
-
#endif
522
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
523
-
RSEQ_INJECT_ASM(6)
524
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
525
-
: /* gcc asm goto does not allow outputs */
526
-
: [cpu_id] "r" (cpu),
527
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
528
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
529
-
[v] "Qo" (*v),
530
-
[expect] "r" (expect),
531
-
[v2] "Qo" (*v2),
532
-
[expect2] "r" (expect2),
533
-
[newv] "r" (newv)
534
-
RSEQ_INJECT_INPUT
535
-
: "memory", RSEQ_ASM_TMP_REG
536
-
: abort, cmpfail
537
-
#ifdef RSEQ_COMPARE_TWICE
538
-
, error1, error2, error3
539
-
#endif
540
-
);
541
-
rseq_after_asm_goto();
542
-
return 0;
543
-
abort:
544
-
rseq_after_asm_goto();
545
-
RSEQ_INJECT_FAILED
546
-
return -1;
547
-
cmpfail:
548
-
rseq_after_asm_goto();
549
-
return 1;
550
-
#ifdef RSEQ_COMPARE_TWICE
551
-
error1:
552
-
rseq_after_asm_goto();
553
-
rseq_bug("cpu_id comparison failed");
554
-
error2:
555
-
rseq_after_asm_goto();
556
-
rseq_bug("expected value comparison failed");
557
-
error3:
558
-
rseq_after_asm_goto();
559
-
rseq_bug("2nd expected value comparison failed");
560
-
#endif
561
-
}
562
-
563
-
static inline __attribute__((always_inline))
564
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
565
-
void *dst, void *src, size_t len,
566
-
intptr_t newv, int cpu)
567
-
{
568
-
RSEQ_INJECT_C(9)
569
-
570
-
__asm__ __volatile__ goto (
571
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
572
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
573
-
#ifdef RSEQ_COMPARE_TWICE
574
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
575
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
576
-
#endif
577
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
578
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
579
-
RSEQ_INJECT_ASM(3)
580
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
581
-
RSEQ_INJECT_ASM(4)
582
-
#ifdef RSEQ_COMPARE_TWICE
583
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
584
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
585
-
#endif
586
-
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
587
-
RSEQ_INJECT_ASM(5)
588
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
589
-
RSEQ_INJECT_ASM(6)
590
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
591
-
: /* gcc asm goto does not allow outputs */
592
-
: [cpu_id] "r" (cpu),
593
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
594
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
595
-
[expect] "r" (expect),
596
-
[v] "Qo" (*v),
597
-
[newv] "r" (newv),
598
-
[dst] "r" (dst),
599
-
[src] "r" (src),
600
-
[len] "r" (len)
601
-
RSEQ_INJECT_INPUT
602
-
: "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2
603
-
: abort, cmpfail
604
-
#ifdef RSEQ_COMPARE_TWICE
605
-
, error1, error2
606
-
#endif
607
-
);
608
-
rseq_after_asm_goto();
609
-
return 0;
610
-
abort:
611
-
rseq_after_asm_goto();
612
-
RSEQ_INJECT_FAILED
613
-
return -1;
614
-
cmpfail:
615
-
rseq_after_asm_goto();
616
-
return 1;
617
-
#ifdef RSEQ_COMPARE_TWICE
618
-
error1:
619
-
rseq_after_asm_goto();
620
-
rseq_bug("cpu_id comparison failed");
621
-
error2:
622
-
rseq_after_asm_goto();
623
-
rseq_bug("expected value comparison failed");
624
-
#endif
625
-
}
626
-
627
-
static inline __attribute__((always_inline))
628
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
629
-
void *dst, void *src, size_t len,
630
-
intptr_t newv, int cpu)
631
-
{
632
-
RSEQ_INJECT_C(9)
633
-
634
-
__asm__ __volatile__ goto (
635
-
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
636
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])
637
-
#ifdef RSEQ_COMPARE_TWICE
638
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])
639
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])
640
-
#endif
641
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
642
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
643
-
RSEQ_INJECT_ASM(3)
644
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
645
-
RSEQ_INJECT_ASM(4)
646
-
#ifdef RSEQ_COMPARE_TWICE
647
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
648
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
649
-
#endif
650
-
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
651
-
RSEQ_INJECT_ASM(5)
652
-
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
653
-
RSEQ_INJECT_ASM(6)
654
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
655
-
: /* gcc asm goto does not allow outputs */
656
-
: [cpu_id] "r" (cpu),
657
-
[current_cpu_id] "Qo" (rseq_get_abi()->cpu_id),
658
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
659
-
[expect] "r" (expect),
660
-
[v] "Qo" (*v),
661
-
[newv] "r" (newv),
662
-
[dst] "r" (dst),
663
-
[src] "r" (src),
664
-
[len] "r" (len)
665
-
RSEQ_INJECT_INPUT
666
-
: "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2
667
-
: abort, cmpfail
668
-
#ifdef RSEQ_COMPARE_TWICE
669
-
, error1, error2
670
-
#endif
671
-
);
672
-
rseq_after_asm_goto();
673
-
return 0;
674
-
abort:
675
-
rseq_after_asm_goto();
676
-
RSEQ_INJECT_FAILED
677
-
return -1;
678
-
cmpfail:
679
-
rseq_after_asm_goto();
680
-
return 1;
681
-
#ifdef RSEQ_COMPARE_TWICE
682
-
error1:
683
-
rseq_after_asm_goto();
684
-
rseq_bug("cpu_id comparison failed");
685
-
error2:
686
-
rseq_after_asm_goto();
687
-
rseq_bug("expected value comparison failed");
688
-
#endif
689
-
}
690
-
691
-
#endif /* !RSEQ_SKIP_FASTPATH */
233
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
234
+
#define RSEQ_TEMPLATE_MO_RELAXED
235
+
#include "rseq-arm64-bits.h"
236
+
#undef RSEQ_TEMPLATE_MO_RELAXED
237
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+11
tools/testing/selftests/rseq/rseq-bits-reset.h
+11
tools/testing/selftests/rseq/rseq-bits-reset.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-bits-reset.h
4
+
*
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
*/
7
+
8
+
#undef RSEQ_TEMPLATE_IDENTIFIER
9
+
#undef RSEQ_TEMPLATE_CPU_ID_FIELD
10
+
#undef RSEQ_TEMPLATE_CPU_ID_OFFSET
11
+
#undef RSEQ_TEMPLATE_SUFFIX
+41
tools/testing/selftests/rseq/rseq-bits-template.h
+41
tools/testing/selftests/rseq/rseq-bits-template.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-bits-template.h
4
+
*
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
*/
7
+
8
+
#ifdef RSEQ_TEMPLATE_CPU_ID
9
+
# define RSEQ_TEMPLATE_CPU_ID_OFFSET RSEQ_CPU_ID_OFFSET
10
+
# define RSEQ_TEMPLATE_CPU_ID_FIELD cpu_id
11
+
# ifdef RSEQ_TEMPLATE_MO_RELEASE
12
+
# define RSEQ_TEMPLATE_SUFFIX _release_cpu_id
13
+
# elif defined (RSEQ_TEMPLATE_MO_RELAXED)
14
+
# define RSEQ_TEMPLATE_SUFFIX _relaxed_cpu_id
15
+
# else
16
+
# error "Never use <rseq-bits-template.h> directly; include <rseq.h> instead."
17
+
# endif
18
+
#elif defined(RSEQ_TEMPLATE_MM_CID)
19
+
# define RSEQ_TEMPLATE_CPU_ID_OFFSET RSEQ_MM_CID_OFFSET
20
+
# define RSEQ_TEMPLATE_CPU_ID_FIELD mm_cid
21
+
# ifdef RSEQ_TEMPLATE_MO_RELEASE
22
+
# define RSEQ_TEMPLATE_SUFFIX _release_mm_cid
23
+
# elif defined (RSEQ_TEMPLATE_MO_RELAXED)
24
+
# define RSEQ_TEMPLATE_SUFFIX _relaxed_mm_cid
25
+
# else
26
+
# error "Never use <rseq-bits-template.h> directly; include <rseq.h> instead."
27
+
# endif
28
+
#elif defined (RSEQ_TEMPLATE_CPU_ID_NONE)
29
+
# ifdef RSEQ_TEMPLATE_MO_RELEASE
30
+
# define RSEQ_TEMPLATE_SUFFIX _release
31
+
# elif defined (RSEQ_TEMPLATE_MO_RELAXED)
32
+
# define RSEQ_TEMPLATE_SUFFIX _relaxed
33
+
# else
34
+
# error "Never use <rseq-bits-template.h> directly; include <rseq.h> instead."
35
+
# endif
36
+
#else
37
+
# error "Never use <rseq-bits-template.h> directly; include <rseq.h> instead."
38
+
#endif
39
+
40
+
#define RSEQ_TEMPLATE_IDENTIFIER(x) RSEQ_COMBINE_TOKENS(x, RSEQ_TEMPLATE_SUFFIX)
41
+
+462
tools/testing/selftests/rseq/rseq-mips-bits.h
+462
tools/testing/selftests/rseq/rseq-mips-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* Author: Paul Burton <paul.burton@mips.com>
4
+
* (C) Copyright 2018 MIPS Tech LLC
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
*/
7
+
8
+
#include "rseq-bits-template.h"
9
+
10
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
11
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
12
+
13
+
static inline __attribute__((always_inline))
14
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
15
+
{
16
+
RSEQ_INJECT_C(9)
17
+
18
+
__asm__ __volatile__ goto (
19
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
20
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
21
+
#ifdef RSEQ_COMPARE_TWICE
22
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
23
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
24
+
#endif
25
+
/* Start rseq by storing table entry pointer into rseq_cs. */
26
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
27
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
28
+
RSEQ_INJECT_ASM(3)
29
+
LONG_L " $4, %[v]\n\t"
30
+
"bne $4, %[expect], %l[cmpfail]\n\t"
31
+
RSEQ_INJECT_ASM(4)
32
+
#ifdef RSEQ_COMPARE_TWICE
33
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
34
+
LONG_L " $4, %[v]\n\t"
35
+
"bne $4, %[expect], %l[error2]\n\t"
36
+
#endif
37
+
/* final store */
38
+
LONG_S " %[newv], %[v]\n\t"
39
+
"2:\n\t"
40
+
RSEQ_INJECT_ASM(5)
41
+
"b 5f\n\t"
42
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
43
+
"5:\n\t"
44
+
: /* gcc asm goto does not allow outputs */
45
+
: [cpu_id] "r" (cpu),
46
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
47
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
48
+
[v] "m" (*v),
49
+
[expect] "r" (expect),
50
+
[newv] "r" (newv)
51
+
RSEQ_INJECT_INPUT
52
+
: "$4", "memory"
53
+
RSEQ_INJECT_CLOBBER
54
+
: abort, cmpfail
55
+
#ifdef RSEQ_COMPARE_TWICE
56
+
, error1, error2
57
+
#endif
58
+
);
59
+
return 0;
60
+
abort:
61
+
RSEQ_INJECT_FAILED
62
+
return -1;
63
+
cmpfail:
64
+
return 1;
65
+
#ifdef RSEQ_COMPARE_TWICE
66
+
error1:
67
+
rseq_bug("cpu_id comparison failed");
68
+
error2:
69
+
rseq_bug("expected value comparison failed");
70
+
#endif
71
+
}
72
+
73
+
static inline __attribute__((always_inline))
74
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
75
+
long voffp, intptr_t *load, int cpu)
76
+
{
77
+
RSEQ_INJECT_C(9)
78
+
79
+
__asm__ __volatile__ goto (
80
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
81
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
82
+
#ifdef RSEQ_COMPARE_TWICE
83
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
84
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
85
+
#endif
86
+
/* Start rseq by storing table entry pointer into rseq_cs. */
87
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
88
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
89
+
RSEQ_INJECT_ASM(3)
90
+
LONG_L " $4, %[v]\n\t"
91
+
"beq $4, %[expectnot], %l[cmpfail]\n\t"
92
+
RSEQ_INJECT_ASM(4)
93
+
#ifdef RSEQ_COMPARE_TWICE
94
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
95
+
LONG_L " $4, %[v]\n\t"
96
+
"beq $4, %[expectnot], %l[error2]\n\t"
97
+
#endif
98
+
LONG_S " $4, %[load]\n\t"
99
+
LONG_ADDI " $4, %[voffp]\n\t"
100
+
LONG_L " $4, 0($4)\n\t"
101
+
/* final store */
102
+
LONG_S " $4, %[v]\n\t"
103
+
"2:\n\t"
104
+
RSEQ_INJECT_ASM(5)
105
+
"b 5f\n\t"
106
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
107
+
"5:\n\t"
108
+
: /* gcc asm goto does not allow outputs */
109
+
: [cpu_id] "r" (cpu),
110
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
111
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
112
+
/* final store input */
113
+
[v] "m" (*v),
114
+
[expectnot] "r" (expectnot),
115
+
[voffp] "Ir" (voffp),
116
+
[load] "m" (*load)
117
+
RSEQ_INJECT_INPUT
118
+
: "$4", "memory"
119
+
RSEQ_INJECT_CLOBBER
120
+
: abort, cmpfail
121
+
#ifdef RSEQ_COMPARE_TWICE
122
+
, error1, error2
123
+
#endif
124
+
);
125
+
return 0;
126
+
abort:
127
+
RSEQ_INJECT_FAILED
128
+
return -1;
129
+
cmpfail:
130
+
return 1;
131
+
#ifdef RSEQ_COMPARE_TWICE
132
+
error1:
133
+
rseq_bug("cpu_id comparison failed");
134
+
error2:
135
+
rseq_bug("expected value comparison failed");
136
+
#endif
137
+
}
138
+
139
+
static inline __attribute__((always_inline))
140
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
141
+
{
142
+
RSEQ_INJECT_C(9)
143
+
144
+
__asm__ __volatile__ goto (
145
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
146
+
#ifdef RSEQ_COMPARE_TWICE
147
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
148
+
#endif
149
+
/* Start rseq by storing table entry pointer into rseq_cs. */
150
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
151
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
152
+
RSEQ_INJECT_ASM(3)
153
+
#ifdef RSEQ_COMPARE_TWICE
154
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
155
+
#endif
156
+
LONG_L " $4, %[v]\n\t"
157
+
LONG_ADDI " $4, %[count]\n\t"
158
+
/* final store */
159
+
LONG_S " $4, %[v]\n\t"
160
+
"2:\n\t"
161
+
RSEQ_INJECT_ASM(4)
162
+
"b 5f\n\t"
163
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
164
+
"5:\n\t"
165
+
: /* gcc asm goto does not allow outputs */
166
+
: [cpu_id] "r" (cpu),
167
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
168
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
169
+
[v] "m" (*v),
170
+
[count] "Ir" (count)
171
+
RSEQ_INJECT_INPUT
172
+
: "$4", "memory"
173
+
RSEQ_INJECT_CLOBBER
174
+
: abort
175
+
#ifdef RSEQ_COMPARE_TWICE
176
+
, error1
177
+
#endif
178
+
);
179
+
return 0;
180
+
abort:
181
+
RSEQ_INJECT_FAILED
182
+
return -1;
183
+
#ifdef RSEQ_COMPARE_TWICE
184
+
error1:
185
+
rseq_bug("cpu_id comparison failed");
186
+
#endif
187
+
}
188
+
189
+
static inline __attribute__((always_inline))
190
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
191
+
intptr_t *v2, intptr_t expect2,
192
+
intptr_t newv, int cpu)
193
+
{
194
+
RSEQ_INJECT_C(9)
195
+
196
+
__asm__ __volatile__ goto (
197
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
198
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
199
+
#ifdef RSEQ_COMPARE_TWICE
200
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
201
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
202
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
203
+
#endif
204
+
/* Start rseq by storing table entry pointer into rseq_cs. */
205
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
206
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
207
+
RSEQ_INJECT_ASM(3)
208
+
LONG_L " $4, %[v]\n\t"
209
+
"bne $4, %[expect], %l[cmpfail]\n\t"
210
+
RSEQ_INJECT_ASM(4)
211
+
LONG_L " $4, %[v2]\n\t"
212
+
"bne $4, %[expect2], %l[cmpfail]\n\t"
213
+
RSEQ_INJECT_ASM(5)
214
+
#ifdef RSEQ_COMPARE_TWICE
215
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
216
+
LONG_L " $4, %[v]\n\t"
217
+
"bne $4, %[expect], %l[error2]\n\t"
218
+
LONG_L " $4, %[v2]\n\t"
219
+
"bne $4, %[expect2], %l[error3]\n\t"
220
+
#endif
221
+
/* final store */
222
+
LONG_S " %[newv], %[v]\n\t"
223
+
"2:\n\t"
224
+
RSEQ_INJECT_ASM(6)
225
+
"b 5f\n\t"
226
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
227
+
"5:\n\t"
228
+
: /* gcc asm goto does not allow outputs */
229
+
: [cpu_id] "r" (cpu),
230
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
231
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
232
+
/* cmp2 input */
233
+
[v2] "m" (*v2),
234
+
[expect2] "r" (expect2),
235
+
/* final store input */
236
+
[v] "m" (*v),
237
+
[expect] "r" (expect),
238
+
[newv] "r" (newv)
239
+
RSEQ_INJECT_INPUT
240
+
: "$4", "memory"
241
+
RSEQ_INJECT_CLOBBER
242
+
: abort, cmpfail
243
+
#ifdef RSEQ_COMPARE_TWICE
244
+
, error1, error2, error3
245
+
#endif
246
+
);
247
+
return 0;
248
+
abort:
249
+
RSEQ_INJECT_FAILED
250
+
return -1;
251
+
cmpfail:
252
+
return 1;
253
+
#ifdef RSEQ_COMPARE_TWICE
254
+
error1:
255
+
rseq_bug("cpu_id comparison failed");
256
+
error2:
257
+
rseq_bug("1st expected value comparison failed");
258
+
error3:
259
+
rseq_bug("2nd expected value comparison failed");
260
+
#endif
261
+
}
262
+
263
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
264
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
265
+
266
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
267
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
268
+
269
+
static inline __attribute__((always_inline))
270
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
271
+
intptr_t *v2, intptr_t newv2,
272
+
intptr_t newv, int cpu)
273
+
{
274
+
RSEQ_INJECT_C(9)
275
+
276
+
__asm__ __volatile__ goto (
277
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
278
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
279
+
#ifdef RSEQ_COMPARE_TWICE
280
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
281
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
282
+
#endif
283
+
/* Start rseq by storing table entry pointer into rseq_cs. */
284
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
285
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
286
+
RSEQ_INJECT_ASM(3)
287
+
LONG_L " $4, %[v]\n\t"
288
+
"bne $4, %[expect], %l[cmpfail]\n\t"
289
+
RSEQ_INJECT_ASM(4)
290
+
#ifdef RSEQ_COMPARE_TWICE
291
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
292
+
LONG_L " $4, %[v]\n\t"
293
+
"bne $4, %[expect], %l[error2]\n\t"
294
+
#endif
295
+
/* try store */
296
+
LONG_S " %[newv2], %[v2]\n\t"
297
+
RSEQ_INJECT_ASM(5)
298
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
299
+
"sync\n\t" /* full sync provides store-release */
300
+
#endif
301
+
/* final store */
302
+
LONG_S " %[newv], %[v]\n\t"
303
+
"2:\n\t"
304
+
RSEQ_INJECT_ASM(6)
305
+
"b 5f\n\t"
306
+
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
307
+
"5:\n\t"
308
+
: /* gcc asm goto does not allow outputs */
309
+
: [cpu_id] "r" (cpu),
310
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
311
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
312
+
/* try store input */
313
+
[v2] "m" (*v2),
314
+
[newv2] "r" (newv2),
315
+
/* final store input */
316
+
[v] "m" (*v),
317
+
[expect] "r" (expect),
318
+
[newv] "r" (newv)
319
+
RSEQ_INJECT_INPUT
320
+
: "$4", "memory"
321
+
RSEQ_INJECT_CLOBBER
322
+
: abort, cmpfail
323
+
#ifdef RSEQ_COMPARE_TWICE
324
+
, error1, error2
325
+
#endif
326
+
);
327
+
return 0;
328
+
abort:
329
+
RSEQ_INJECT_FAILED
330
+
return -1;
331
+
cmpfail:
332
+
return 1;
333
+
#ifdef RSEQ_COMPARE_TWICE
334
+
error1:
335
+
rseq_bug("cpu_id comparison failed");
336
+
error2:
337
+
rseq_bug("expected value comparison failed");
338
+
#endif
339
+
}
340
+
341
+
static inline __attribute__((always_inline))
342
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
343
+
void *dst, void *src, size_t len,
344
+
intptr_t newv, int cpu)
345
+
{
346
+
uintptr_t rseq_scratch[3];
347
+
348
+
RSEQ_INJECT_C(9)
349
+
350
+
__asm__ __volatile__ goto (
351
+
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
352
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
353
+
#ifdef RSEQ_COMPARE_TWICE
354
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
355
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
356
+
#endif
357
+
LONG_S " %[src], %[rseq_scratch0]\n\t"
358
+
LONG_S " %[dst], %[rseq_scratch1]\n\t"
359
+
LONG_S " %[len], %[rseq_scratch2]\n\t"
360
+
/* Start rseq by storing table entry pointer into rseq_cs. */
361
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
362
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
363
+
RSEQ_INJECT_ASM(3)
364
+
LONG_L " $4, %[v]\n\t"
365
+
"bne $4, %[expect], 5f\n\t"
366
+
RSEQ_INJECT_ASM(4)
367
+
#ifdef RSEQ_COMPARE_TWICE
368
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
369
+
LONG_L " $4, %[v]\n\t"
370
+
"bne $4, %[expect], 7f\n\t"
371
+
#endif
372
+
/* try memcpy */
373
+
"beqz %[len], 333f\n\t" \
374
+
"222:\n\t" \
375
+
"lb $4, 0(%[src])\n\t" \
376
+
"sb $4, 0(%[dst])\n\t" \
377
+
LONG_ADDI " %[src], 1\n\t" \
378
+
LONG_ADDI " %[dst], 1\n\t" \
379
+
LONG_ADDI " %[len], -1\n\t" \
380
+
"bnez %[len], 222b\n\t" \
381
+
"333:\n\t" \
382
+
RSEQ_INJECT_ASM(5)
383
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
384
+
"sync\n\t" /* full sync provides store-release */
385
+
#endif
386
+
/* final store */
387
+
LONG_S " %[newv], %[v]\n\t"
388
+
"2:\n\t"
389
+
RSEQ_INJECT_ASM(6)
390
+
/* teardown */
391
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
392
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
393
+
LONG_L " %[src], %[rseq_scratch0]\n\t"
394
+
"b 8f\n\t"
395
+
RSEQ_ASM_DEFINE_ABORT(3, 4,
396
+
/* teardown */
397
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
398
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
399
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
400
+
abort, 1b, 2b, 4f)
401
+
RSEQ_ASM_DEFINE_CMPFAIL(5,
402
+
/* teardown */
403
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
404
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
405
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
406
+
cmpfail)
407
+
#ifdef RSEQ_COMPARE_TWICE
408
+
RSEQ_ASM_DEFINE_CMPFAIL(6,
409
+
/* teardown */
410
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
411
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
412
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
413
+
error1)
414
+
RSEQ_ASM_DEFINE_CMPFAIL(7,
415
+
/* teardown */
416
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
417
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
418
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
419
+
error2)
420
+
#endif
421
+
"8:\n\t"
422
+
: /* gcc asm goto does not allow outputs */
423
+
: [cpu_id] "r" (cpu),
424
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
425
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
426
+
/* final store input */
427
+
[v] "m" (*v),
428
+
[expect] "r" (expect),
429
+
[newv] "r" (newv),
430
+
/* try memcpy input */
431
+
[dst] "r" (dst),
432
+
[src] "r" (src),
433
+
[len] "r" (len),
434
+
[rseq_scratch0] "m" (rseq_scratch[0]),
435
+
[rseq_scratch1] "m" (rseq_scratch[1]),
436
+
[rseq_scratch2] "m" (rseq_scratch[2])
437
+
RSEQ_INJECT_INPUT
438
+
: "$4", "memory"
439
+
RSEQ_INJECT_CLOBBER
440
+
: abort, cmpfail
441
+
#ifdef RSEQ_COMPARE_TWICE
442
+
, error1, error2
443
+
#endif
444
+
);
445
+
return 0;
446
+
abort:
447
+
RSEQ_INJECT_FAILED
448
+
return -1;
449
+
cmpfail:
450
+
return 1;
451
+
#ifdef RSEQ_COMPARE_TWICE
452
+
error1:
453
+
rseq_bug("cpu_id comparison failed");
454
+
error2:
455
+
rseq_bug("expected value comparison failed");
456
+
#endif
457
+
}
458
+
459
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
460
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
461
+
462
+
#include "rseq-bits-reset.h"
+25
-621
tools/testing/selftests/rseq/rseq-mips.h
+25
-621
tools/testing/selftests/rseq/rseq-mips.h
···
2
2
/*
3
3
* Author: Paul Burton <paul.burton@mips.com>
4
4
* (C) Copyright 2018 MIPS Tech LLC
5
-
*
6
-
* Based on rseq-arm.h:
7
-
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
8
6
*/
9
7
10
8
/*
···
57
59
rseq_smp_mb(); \
58
60
RSEQ_WRITE_ONCE(*p, v); \
59
61
} while (0)
60
-
61
-
#ifdef RSEQ_SKIP_FASTPATH
62
-
#include "rseq-skip.h"
63
-
#else /* !RSEQ_SKIP_FASTPATH */
64
62
65
63
#if _MIPS_SZLONG == 64
66
64
# define LONG ".dword"
···
148
154
teardown \
149
155
"b %l[" __rseq_str(cmpfail_label) "]\n\t"
150
156
151
-
static inline __attribute__((always_inline))
152
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
153
-
{
154
-
RSEQ_INJECT_C(9)
157
+
/* Per-cpu-id indexing. */
155
158
156
-
__asm__ __volatile__ goto (
157
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
158
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
159
-
#ifdef RSEQ_COMPARE_TWICE
160
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
161
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
162
-
#endif
163
-
/* Start rseq by storing table entry pointer into rseq_cs. */
164
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
165
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
166
-
RSEQ_INJECT_ASM(3)
167
-
LONG_L " $4, %[v]\n\t"
168
-
"bne $4, %[expect], %l[cmpfail]\n\t"
169
-
RSEQ_INJECT_ASM(4)
170
-
#ifdef RSEQ_COMPARE_TWICE
171
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
172
-
LONG_L " $4, %[v]\n\t"
173
-
"bne $4, %[expect], %l[error2]\n\t"
174
-
#endif
175
-
/* final store */
176
-
LONG_S " %[newv], %[v]\n\t"
177
-
"2:\n\t"
178
-
RSEQ_INJECT_ASM(5)
179
-
"b 5f\n\t"
180
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
181
-
"5:\n\t"
182
-
: /* gcc asm goto does not allow outputs */
183
-
: [cpu_id] "r" (cpu),
184
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
185
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
186
-
[v] "m" (*v),
187
-
[expect] "r" (expect),
188
-
[newv] "r" (newv)
189
-
RSEQ_INJECT_INPUT
190
-
: "$4", "memory"
191
-
RSEQ_INJECT_CLOBBER
192
-
: abort, cmpfail
193
-
#ifdef RSEQ_COMPARE_TWICE
194
-
, error1, error2
195
-
#endif
196
-
);
197
-
return 0;
198
-
abort:
199
-
RSEQ_INJECT_FAILED
200
-
return -1;
201
-
cmpfail:
202
-
return 1;
203
-
#ifdef RSEQ_COMPARE_TWICE
204
-
error1:
205
-
rseq_bug("cpu_id comparison failed");
206
-
error2:
207
-
rseq_bug("expected value comparison failed");
208
-
#endif
209
-
}
159
+
#define RSEQ_TEMPLATE_CPU_ID
160
+
#define RSEQ_TEMPLATE_MO_RELAXED
161
+
#include "rseq-mips-bits.h"
162
+
#undef RSEQ_TEMPLATE_MO_RELAXED
210
163
211
-
static inline __attribute__((always_inline))
212
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
213
-
long voffp, intptr_t *load, int cpu)
214
-
{
215
-
RSEQ_INJECT_C(9)
164
+
#define RSEQ_TEMPLATE_MO_RELEASE
165
+
#include "rseq-mips-bits.h"
166
+
#undef RSEQ_TEMPLATE_MO_RELEASE
167
+
#undef RSEQ_TEMPLATE_CPU_ID
216
168
217
-
__asm__ __volatile__ goto (
218
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
219
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
220
-
#ifdef RSEQ_COMPARE_TWICE
221
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
222
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
223
-
#endif
224
-
/* Start rseq by storing table entry pointer into rseq_cs. */
225
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
226
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
227
-
RSEQ_INJECT_ASM(3)
228
-
LONG_L " $4, %[v]\n\t"
229
-
"beq $4, %[expectnot], %l[cmpfail]\n\t"
230
-
RSEQ_INJECT_ASM(4)
231
-
#ifdef RSEQ_COMPARE_TWICE
232
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
233
-
LONG_L " $4, %[v]\n\t"
234
-
"beq $4, %[expectnot], %l[error2]\n\t"
235
-
#endif
236
-
LONG_S " $4, %[load]\n\t"
237
-
LONG_ADDI " $4, %[voffp]\n\t"
238
-
LONG_L " $4, 0($4)\n\t"
239
-
/* final store */
240
-
LONG_S " $4, %[v]\n\t"
241
-
"2:\n\t"
242
-
RSEQ_INJECT_ASM(5)
243
-
"b 5f\n\t"
244
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
245
-
"5:\n\t"
246
-
: /* gcc asm goto does not allow outputs */
247
-
: [cpu_id] "r" (cpu),
248
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
249
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
250
-
/* final store input */
251
-
[v] "m" (*v),
252
-
[expectnot] "r" (expectnot),
253
-
[voffp] "Ir" (voffp),
254
-
[load] "m" (*load)
255
-
RSEQ_INJECT_INPUT
256
-
: "$4", "memory"
257
-
RSEQ_INJECT_CLOBBER
258
-
: abort, cmpfail
259
-
#ifdef RSEQ_COMPARE_TWICE
260
-
, error1, error2
261
-
#endif
262
-
);
263
-
return 0;
264
-
abort:
265
-
RSEQ_INJECT_FAILED
266
-
return -1;
267
-
cmpfail:
268
-
return 1;
269
-
#ifdef RSEQ_COMPARE_TWICE
270
-
error1:
271
-
rseq_bug("cpu_id comparison failed");
272
-
error2:
273
-
rseq_bug("expected value comparison failed");
274
-
#endif
275
-
}
169
+
/* Per-mm-cid indexing. */
276
170
277
-
static inline __attribute__((always_inline))
278
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
279
-
{
280
-
RSEQ_INJECT_C(9)
171
+
#define RSEQ_TEMPLATE_MM_CID
172
+
#define RSEQ_TEMPLATE_MO_RELAXED
173
+
#include "rseq-mips-bits.h"
174
+
#undef RSEQ_TEMPLATE_MO_RELAXED
281
175
282
-
__asm__ __volatile__ goto (
283
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
284
-
#ifdef RSEQ_COMPARE_TWICE
285
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
286
-
#endif
287
-
/* Start rseq by storing table entry pointer into rseq_cs. */
288
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
289
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
290
-
RSEQ_INJECT_ASM(3)
291
-
#ifdef RSEQ_COMPARE_TWICE
292
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
293
-
#endif
294
-
LONG_L " $4, %[v]\n\t"
295
-
LONG_ADDI " $4, %[count]\n\t"
296
-
/* final store */
297
-
LONG_S " $4, %[v]\n\t"
298
-
"2:\n\t"
299
-
RSEQ_INJECT_ASM(4)
300
-
"b 5f\n\t"
301
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
302
-
"5:\n\t"
303
-
: /* gcc asm goto does not allow outputs */
304
-
: [cpu_id] "r" (cpu),
305
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
306
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
307
-
[v] "m" (*v),
308
-
[count] "Ir" (count)
309
-
RSEQ_INJECT_INPUT
310
-
: "$4", "memory"
311
-
RSEQ_INJECT_CLOBBER
312
-
: abort
313
-
#ifdef RSEQ_COMPARE_TWICE
314
-
, error1
315
-
#endif
316
-
);
317
-
return 0;
318
-
abort:
319
-
RSEQ_INJECT_FAILED
320
-
return -1;
321
-
#ifdef RSEQ_COMPARE_TWICE
322
-
error1:
323
-
rseq_bug("cpu_id comparison failed");
324
-
#endif
325
-
}
176
+
#define RSEQ_TEMPLATE_MO_RELEASE
177
+
#include "rseq-mips-bits.h"
178
+
#undef RSEQ_TEMPLATE_MO_RELEASE
179
+
#undef RSEQ_TEMPLATE_MM_CID
326
180
327
-
static inline __attribute__((always_inline))
328
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
329
-
intptr_t *v2, intptr_t newv2,
330
-
intptr_t newv, int cpu)
331
-
{
332
-
RSEQ_INJECT_C(9)
181
+
/* APIs which are not based on cpu ids. */
333
182
334
-
__asm__ __volatile__ goto (
335
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
336
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
337
-
#ifdef RSEQ_COMPARE_TWICE
338
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
339
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
340
-
#endif
341
-
/* Start rseq by storing table entry pointer into rseq_cs. */
342
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
343
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
344
-
RSEQ_INJECT_ASM(3)
345
-
LONG_L " $4, %[v]\n\t"
346
-
"bne $4, %[expect], %l[cmpfail]\n\t"
347
-
RSEQ_INJECT_ASM(4)
348
-
#ifdef RSEQ_COMPARE_TWICE
349
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
350
-
LONG_L " $4, %[v]\n\t"
351
-
"bne $4, %[expect], %l[error2]\n\t"
352
-
#endif
353
-
/* try store */
354
-
LONG_S " %[newv2], %[v2]\n\t"
355
-
RSEQ_INJECT_ASM(5)
356
-
/* final store */
357
-
LONG_S " %[newv], %[v]\n\t"
358
-
"2:\n\t"
359
-
RSEQ_INJECT_ASM(6)
360
-
"b 5f\n\t"
361
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
362
-
"5:\n\t"
363
-
: /* gcc asm goto does not allow outputs */
364
-
: [cpu_id] "r" (cpu),
365
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
366
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
367
-
/* try store input */
368
-
[v2] "m" (*v2),
369
-
[newv2] "r" (newv2),
370
-
/* final store input */
371
-
[v] "m" (*v),
372
-
[expect] "r" (expect),
373
-
[newv] "r" (newv)
374
-
RSEQ_INJECT_INPUT
375
-
: "$4", "memory"
376
-
RSEQ_INJECT_CLOBBER
377
-
: abort, cmpfail
378
-
#ifdef RSEQ_COMPARE_TWICE
379
-
, error1, error2
380
-
#endif
381
-
);
382
-
return 0;
383
-
abort:
384
-
RSEQ_INJECT_FAILED
385
-
return -1;
386
-
cmpfail:
387
-
return 1;
388
-
#ifdef RSEQ_COMPARE_TWICE
389
-
error1:
390
-
rseq_bug("cpu_id comparison failed");
391
-
error2:
392
-
rseq_bug("expected value comparison failed");
393
-
#endif
394
-
}
395
-
396
-
static inline __attribute__((always_inline))
397
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
398
-
intptr_t *v2, intptr_t newv2,
399
-
intptr_t newv, int cpu)
400
-
{
401
-
RSEQ_INJECT_C(9)
402
-
403
-
__asm__ __volatile__ goto (
404
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
405
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
406
-
#ifdef RSEQ_COMPARE_TWICE
407
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
408
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
409
-
#endif
410
-
/* Start rseq by storing table entry pointer into rseq_cs. */
411
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
412
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
413
-
RSEQ_INJECT_ASM(3)
414
-
LONG_L " $4, %[v]\n\t"
415
-
"bne $4, %[expect], %l[cmpfail]\n\t"
416
-
RSEQ_INJECT_ASM(4)
417
-
#ifdef RSEQ_COMPARE_TWICE
418
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
419
-
LONG_L " $4, %[v]\n\t"
420
-
"bne $4, %[expect], %l[error2]\n\t"
421
-
#endif
422
-
/* try store */
423
-
LONG_S " %[newv2], %[v2]\n\t"
424
-
RSEQ_INJECT_ASM(5)
425
-
"sync\n\t" /* full sync provides store-release */
426
-
/* final store */
427
-
LONG_S " %[newv], %[v]\n\t"
428
-
"2:\n\t"
429
-
RSEQ_INJECT_ASM(6)
430
-
"b 5f\n\t"
431
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
432
-
"5:\n\t"
433
-
: /* gcc asm goto does not allow outputs */
434
-
: [cpu_id] "r" (cpu),
435
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
436
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
437
-
/* try store input */
438
-
[v2] "m" (*v2),
439
-
[newv2] "r" (newv2),
440
-
/* final store input */
441
-
[v] "m" (*v),
442
-
[expect] "r" (expect),
443
-
[newv] "r" (newv)
444
-
RSEQ_INJECT_INPUT
445
-
: "$4", "memory"
446
-
RSEQ_INJECT_CLOBBER
447
-
: abort, cmpfail
448
-
#ifdef RSEQ_COMPARE_TWICE
449
-
, error1, error2
450
-
#endif
451
-
);
452
-
return 0;
453
-
abort:
454
-
RSEQ_INJECT_FAILED
455
-
return -1;
456
-
cmpfail:
457
-
return 1;
458
-
#ifdef RSEQ_COMPARE_TWICE
459
-
error1:
460
-
rseq_bug("cpu_id comparison failed");
461
-
error2:
462
-
rseq_bug("expected value comparison failed");
463
-
#endif
464
-
}
465
-
466
-
static inline __attribute__((always_inline))
467
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
468
-
intptr_t *v2, intptr_t expect2,
469
-
intptr_t newv, int cpu)
470
-
{
471
-
RSEQ_INJECT_C(9)
472
-
473
-
__asm__ __volatile__ goto (
474
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
475
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
476
-
#ifdef RSEQ_COMPARE_TWICE
477
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
478
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
479
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
480
-
#endif
481
-
/* Start rseq by storing table entry pointer into rseq_cs. */
482
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
483
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
484
-
RSEQ_INJECT_ASM(3)
485
-
LONG_L " $4, %[v]\n\t"
486
-
"bne $4, %[expect], %l[cmpfail]\n\t"
487
-
RSEQ_INJECT_ASM(4)
488
-
LONG_L " $4, %[v2]\n\t"
489
-
"bne $4, %[expect2], %l[cmpfail]\n\t"
490
-
RSEQ_INJECT_ASM(5)
491
-
#ifdef RSEQ_COMPARE_TWICE
492
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
493
-
LONG_L " $4, %[v]\n\t"
494
-
"bne $4, %[expect], %l[error2]\n\t"
495
-
LONG_L " $4, %[v2]\n\t"
496
-
"bne $4, %[expect2], %l[error3]\n\t"
497
-
#endif
498
-
/* final store */
499
-
LONG_S " %[newv], %[v]\n\t"
500
-
"2:\n\t"
501
-
RSEQ_INJECT_ASM(6)
502
-
"b 5f\n\t"
503
-
RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
504
-
"5:\n\t"
505
-
: /* gcc asm goto does not allow outputs */
506
-
: [cpu_id] "r" (cpu),
507
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
508
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
509
-
/* cmp2 input */
510
-
[v2] "m" (*v2),
511
-
[expect2] "r" (expect2),
512
-
/* final store input */
513
-
[v] "m" (*v),
514
-
[expect] "r" (expect),
515
-
[newv] "r" (newv)
516
-
RSEQ_INJECT_INPUT
517
-
: "$4", "memory"
518
-
RSEQ_INJECT_CLOBBER
519
-
: abort, cmpfail
520
-
#ifdef RSEQ_COMPARE_TWICE
521
-
, error1, error2, error3
522
-
#endif
523
-
);
524
-
return 0;
525
-
abort:
526
-
RSEQ_INJECT_FAILED
527
-
return -1;
528
-
cmpfail:
529
-
return 1;
530
-
#ifdef RSEQ_COMPARE_TWICE
531
-
error1:
532
-
rseq_bug("cpu_id comparison failed");
533
-
error2:
534
-
rseq_bug("1st expected value comparison failed");
535
-
error3:
536
-
rseq_bug("2nd expected value comparison failed");
537
-
#endif
538
-
}
539
-
540
-
static inline __attribute__((always_inline))
541
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
542
-
void *dst, void *src, size_t len,
543
-
intptr_t newv, int cpu)
544
-
{
545
-
uintptr_t rseq_scratch[3];
546
-
547
-
RSEQ_INJECT_C(9)
548
-
549
-
__asm__ __volatile__ goto (
550
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
551
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
552
-
#ifdef RSEQ_COMPARE_TWICE
553
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
554
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
555
-
#endif
556
-
LONG_S " %[src], %[rseq_scratch0]\n\t"
557
-
LONG_S " %[dst], %[rseq_scratch1]\n\t"
558
-
LONG_S " %[len], %[rseq_scratch2]\n\t"
559
-
/* Start rseq by storing table entry pointer into rseq_cs. */
560
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
561
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
562
-
RSEQ_INJECT_ASM(3)
563
-
LONG_L " $4, %[v]\n\t"
564
-
"bne $4, %[expect], 5f\n\t"
565
-
RSEQ_INJECT_ASM(4)
566
-
#ifdef RSEQ_COMPARE_TWICE
567
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
568
-
LONG_L " $4, %[v]\n\t"
569
-
"bne $4, %[expect], 7f\n\t"
570
-
#endif
571
-
/* try memcpy */
572
-
"beqz %[len], 333f\n\t" \
573
-
"222:\n\t" \
574
-
"lb $4, 0(%[src])\n\t" \
575
-
"sb $4, 0(%[dst])\n\t" \
576
-
LONG_ADDI " %[src], 1\n\t" \
577
-
LONG_ADDI " %[dst], 1\n\t" \
578
-
LONG_ADDI " %[len], -1\n\t" \
579
-
"bnez %[len], 222b\n\t" \
580
-
"333:\n\t" \
581
-
RSEQ_INJECT_ASM(5)
582
-
/* final store */
583
-
LONG_S " %[newv], %[v]\n\t"
584
-
"2:\n\t"
585
-
RSEQ_INJECT_ASM(6)
586
-
/* teardown */
587
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
588
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
589
-
LONG_L " %[src], %[rseq_scratch0]\n\t"
590
-
"b 8f\n\t"
591
-
RSEQ_ASM_DEFINE_ABORT(3, 4,
592
-
/* teardown */
593
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
594
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
595
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
596
-
abort, 1b, 2b, 4f)
597
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
598
-
/* teardown */
599
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
600
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
601
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
602
-
cmpfail)
603
-
#ifdef RSEQ_COMPARE_TWICE
604
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
605
-
/* teardown */
606
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
607
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
608
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
609
-
error1)
610
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
611
-
/* teardown */
612
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
613
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
614
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
615
-
error2)
616
-
#endif
617
-
"8:\n\t"
618
-
: /* gcc asm goto does not allow outputs */
619
-
: [cpu_id] "r" (cpu),
620
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
621
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
622
-
/* final store input */
623
-
[v] "m" (*v),
624
-
[expect] "r" (expect),
625
-
[newv] "r" (newv),
626
-
/* try memcpy input */
627
-
[dst] "r" (dst),
628
-
[src] "r" (src),
629
-
[len] "r" (len),
630
-
[rseq_scratch0] "m" (rseq_scratch[0]),
631
-
[rseq_scratch1] "m" (rseq_scratch[1]),
632
-
[rseq_scratch2] "m" (rseq_scratch[2])
633
-
RSEQ_INJECT_INPUT
634
-
: "$4", "memory"
635
-
RSEQ_INJECT_CLOBBER
636
-
: abort, cmpfail
637
-
#ifdef RSEQ_COMPARE_TWICE
638
-
, error1, error2
639
-
#endif
640
-
);
641
-
return 0;
642
-
abort:
643
-
RSEQ_INJECT_FAILED
644
-
return -1;
645
-
cmpfail:
646
-
return 1;
647
-
#ifdef RSEQ_COMPARE_TWICE
648
-
error1:
649
-
rseq_bug("cpu_id comparison failed");
650
-
error2:
651
-
rseq_bug("expected value comparison failed");
652
-
#endif
653
-
}
654
-
655
-
static inline __attribute__((always_inline))
656
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
657
-
void *dst, void *src, size_t len,
658
-
intptr_t newv, int cpu)
659
-
{
660
-
uintptr_t rseq_scratch[3];
661
-
662
-
RSEQ_INJECT_C(9)
663
-
664
-
__asm__ __volatile__ goto (
665
-
RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */
666
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
667
-
#ifdef RSEQ_COMPARE_TWICE
668
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
669
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
670
-
#endif
671
-
LONG_S " %[src], %[rseq_scratch0]\n\t"
672
-
LONG_S " %[dst], %[rseq_scratch1]\n\t"
673
-
LONG_S " %[len], %[rseq_scratch2]\n\t"
674
-
/* Start rseq by storing table entry pointer into rseq_cs. */
675
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
676
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
677
-
RSEQ_INJECT_ASM(3)
678
-
LONG_L " $4, %[v]\n\t"
679
-
"bne $4, %[expect], 5f\n\t"
680
-
RSEQ_INJECT_ASM(4)
681
-
#ifdef RSEQ_COMPARE_TWICE
682
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
683
-
LONG_L " $4, %[v]\n\t"
684
-
"bne $4, %[expect], 7f\n\t"
685
-
#endif
686
-
/* try memcpy */
687
-
"beqz %[len], 333f\n\t" \
688
-
"222:\n\t" \
689
-
"lb $4, 0(%[src])\n\t" \
690
-
"sb $4, 0(%[dst])\n\t" \
691
-
LONG_ADDI " %[src], 1\n\t" \
692
-
LONG_ADDI " %[dst], 1\n\t" \
693
-
LONG_ADDI " %[len], -1\n\t" \
694
-
"bnez %[len], 222b\n\t" \
695
-
"333:\n\t" \
696
-
RSEQ_INJECT_ASM(5)
697
-
"sync\n\t" /* full sync provides store-release */
698
-
/* final store */
699
-
LONG_S " %[newv], %[v]\n\t"
700
-
"2:\n\t"
701
-
RSEQ_INJECT_ASM(6)
702
-
/* teardown */
703
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
704
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
705
-
LONG_L " %[src], %[rseq_scratch0]\n\t"
706
-
"b 8f\n\t"
707
-
RSEQ_ASM_DEFINE_ABORT(3, 4,
708
-
/* teardown */
709
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
710
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
711
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
712
-
abort, 1b, 2b, 4f)
713
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
714
-
/* teardown */
715
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
716
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
717
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
718
-
cmpfail)
719
-
#ifdef RSEQ_COMPARE_TWICE
720
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
721
-
/* teardown */
722
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
723
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
724
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
725
-
error1)
726
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
727
-
/* teardown */
728
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
729
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
730
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
731
-
error2)
732
-
#endif
733
-
"8:\n\t"
734
-
: /* gcc asm goto does not allow outputs */
735
-
: [cpu_id] "r" (cpu),
736
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
737
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
738
-
/* final store input */
739
-
[v] "m" (*v),
740
-
[expect] "r" (expect),
741
-
[newv] "r" (newv),
742
-
/* try memcpy input */
743
-
[dst] "r" (dst),
744
-
[src] "r" (src),
745
-
[len] "r" (len),
746
-
[rseq_scratch0] "m" (rseq_scratch[0]),
747
-
[rseq_scratch1] "m" (rseq_scratch[1]),
748
-
[rseq_scratch2] "m" (rseq_scratch[2])
749
-
RSEQ_INJECT_INPUT
750
-
: "$4", "memory"
751
-
RSEQ_INJECT_CLOBBER
752
-
: abort, cmpfail
753
-
#ifdef RSEQ_COMPARE_TWICE
754
-
, error1, error2
755
-
#endif
756
-
);
757
-
return 0;
758
-
abort:
759
-
RSEQ_INJECT_FAILED
760
-
return -1;
761
-
cmpfail:
762
-
return 1;
763
-
#ifdef RSEQ_COMPARE_TWICE
764
-
error1:
765
-
rseq_bug("cpu_id comparison failed");
766
-
error2:
767
-
rseq_bug("expected value comparison failed");
768
-
#endif
769
-
}
770
-
771
-
#endif /* !RSEQ_SKIP_FASTPATH */
183
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
184
+
#define RSEQ_TEMPLATE_MO_RELAXED
185
+
#include "rseq-mips-bits.h"
186
+
#undef RSEQ_TEMPLATE_MO_RELAXED
187
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+454
tools/testing/selftests/rseq/rseq-ppc-bits.h
+454
tools/testing/selftests/rseq/rseq-ppc-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-ppc-bits.h
4
+
*
5
+
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
* (C) Copyright 2016-2018 - Boqun Feng <boqun.feng@gmail.com>
7
+
*/
8
+
9
+
#include "rseq-bits-template.h"
10
+
11
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
12
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
13
+
14
+
static inline __attribute__((always_inline))
15
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
16
+
{
17
+
RSEQ_INJECT_C(9)
18
+
19
+
__asm__ __volatile__ goto (
20
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
21
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
22
+
#ifdef RSEQ_COMPARE_TWICE
23
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
24
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
25
+
#endif
26
+
/* Start rseq by storing table entry pointer into rseq_cs. */
27
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
28
+
/* cmp cpuid */
29
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
30
+
RSEQ_INJECT_ASM(3)
31
+
/* cmp @v equal to @expect */
32
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
33
+
RSEQ_INJECT_ASM(4)
34
+
#ifdef RSEQ_COMPARE_TWICE
35
+
/* cmp cpuid */
36
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
37
+
/* cmp @v equal to @expect */
38
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
39
+
#endif
40
+
/* final store */
41
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
42
+
RSEQ_INJECT_ASM(5)
43
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
44
+
: /* gcc asm goto does not allow outputs */
45
+
: [cpu_id] "r" (cpu),
46
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
47
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
48
+
[v] "m" (*v),
49
+
[expect] "r" (expect),
50
+
[newv] "r" (newv)
51
+
RSEQ_INJECT_INPUT
52
+
: "memory", "cc", "r17"
53
+
RSEQ_INJECT_CLOBBER
54
+
: abort, cmpfail
55
+
#ifdef RSEQ_COMPARE_TWICE
56
+
, error1, error2
57
+
#endif
58
+
);
59
+
rseq_after_asm_goto();
60
+
return 0;
61
+
abort:
62
+
rseq_after_asm_goto();
63
+
RSEQ_INJECT_FAILED
64
+
return -1;
65
+
cmpfail:
66
+
rseq_after_asm_goto();
67
+
return 1;
68
+
#ifdef RSEQ_COMPARE_TWICE
69
+
error1:
70
+
rseq_after_asm_goto();
71
+
rseq_bug("cpu_id comparison failed");
72
+
error2:
73
+
rseq_after_asm_goto();
74
+
rseq_bug("expected value comparison failed");
75
+
#endif
76
+
}
77
+
78
+
static inline __attribute__((always_inline))
79
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
80
+
long voffp, intptr_t *load, int cpu)
81
+
{
82
+
RSEQ_INJECT_C(9)
83
+
84
+
__asm__ __volatile__ goto (
85
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
86
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
87
+
#ifdef RSEQ_COMPARE_TWICE
88
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
89
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
90
+
#endif
91
+
/* Start rseq by storing table entry pointer into rseq_cs. */
92
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
93
+
/* cmp cpuid */
94
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
95
+
RSEQ_INJECT_ASM(3)
96
+
/* cmp @v not equal to @expectnot */
97
+
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail])
98
+
RSEQ_INJECT_ASM(4)
99
+
#ifdef RSEQ_COMPARE_TWICE
100
+
/* cmp cpuid */
101
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
102
+
/* cmp @v not equal to @expectnot */
103
+
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2])
104
+
#endif
105
+
/* load the value of @v */
106
+
RSEQ_ASM_OP_R_LOAD(v)
107
+
/* store it in @load */
108
+
RSEQ_ASM_OP_R_STORE(load)
109
+
/* dereference voffp(v) */
110
+
RSEQ_ASM_OP_R_LOADX(voffp)
111
+
/* final store the value at voffp(v) */
112
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 2)
113
+
RSEQ_INJECT_ASM(5)
114
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
115
+
: /* gcc asm goto does not allow outputs */
116
+
: [cpu_id] "r" (cpu),
117
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
118
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
119
+
/* final store input */
120
+
[v] "m" (*v),
121
+
[expectnot] "r" (expectnot),
122
+
[voffp] "b" (voffp),
123
+
[load] "m" (*load)
124
+
RSEQ_INJECT_INPUT
125
+
: "memory", "cc", "r17"
126
+
RSEQ_INJECT_CLOBBER
127
+
: abort, cmpfail
128
+
#ifdef RSEQ_COMPARE_TWICE
129
+
, error1, error2
130
+
#endif
131
+
);
132
+
rseq_after_asm_goto();
133
+
return 0;
134
+
abort:
135
+
rseq_after_asm_goto();
136
+
RSEQ_INJECT_FAILED
137
+
return -1;
138
+
cmpfail:
139
+
rseq_after_asm_goto();
140
+
return 1;
141
+
#ifdef RSEQ_COMPARE_TWICE
142
+
error1:
143
+
rseq_after_asm_goto();
144
+
rseq_bug("cpu_id comparison failed");
145
+
error2:
146
+
rseq_after_asm_goto();
147
+
rseq_bug("expected value comparison failed");
148
+
#endif
149
+
}
150
+
151
+
static inline __attribute__((always_inline))
152
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
153
+
{
154
+
RSEQ_INJECT_C(9)
155
+
156
+
__asm__ __volatile__ goto (
157
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
158
+
#ifdef RSEQ_COMPARE_TWICE
159
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
160
+
#endif
161
+
/* Start rseq by storing table entry pointer into rseq_cs. */
162
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
163
+
/* cmp cpuid */
164
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
165
+
RSEQ_INJECT_ASM(3)
166
+
#ifdef RSEQ_COMPARE_TWICE
167
+
/* cmp cpuid */
168
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
169
+
#endif
170
+
/* load the value of @v */
171
+
RSEQ_ASM_OP_R_LOAD(v)
172
+
/* add @count to it */
173
+
RSEQ_ASM_OP_R_ADD(count)
174
+
/* final store */
175
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 2)
176
+
RSEQ_INJECT_ASM(4)
177
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
178
+
: /* gcc asm goto does not allow outputs */
179
+
: [cpu_id] "r" (cpu),
180
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
181
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
182
+
/* final store input */
183
+
[v] "m" (*v),
184
+
[count] "r" (count)
185
+
RSEQ_INJECT_INPUT
186
+
: "memory", "cc", "r17"
187
+
RSEQ_INJECT_CLOBBER
188
+
: abort
189
+
#ifdef RSEQ_COMPARE_TWICE
190
+
, error1
191
+
#endif
192
+
);
193
+
rseq_after_asm_goto();
194
+
return 0;
195
+
abort:
196
+
rseq_after_asm_goto();
197
+
RSEQ_INJECT_FAILED
198
+
return -1;
199
+
#ifdef RSEQ_COMPARE_TWICE
200
+
error1:
201
+
rseq_after_asm_goto();
202
+
rseq_bug("cpu_id comparison failed");
203
+
#endif
204
+
}
205
+
206
+
static inline __attribute__((always_inline))
207
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
208
+
intptr_t *v2, intptr_t expect2,
209
+
intptr_t newv, int cpu)
210
+
{
211
+
RSEQ_INJECT_C(9)
212
+
213
+
__asm__ __volatile__ goto (
214
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
215
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
216
+
#ifdef RSEQ_COMPARE_TWICE
217
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
218
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
219
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
220
+
#endif
221
+
/* Start rseq by storing table entry pointer into rseq_cs. */
222
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
223
+
/* cmp cpuid */
224
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
225
+
RSEQ_INJECT_ASM(3)
226
+
/* cmp @v equal to @expect */
227
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
228
+
RSEQ_INJECT_ASM(4)
229
+
/* cmp @v2 equal to @expct2 */
230
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail])
231
+
RSEQ_INJECT_ASM(5)
232
+
#ifdef RSEQ_COMPARE_TWICE
233
+
/* cmp cpuid */
234
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
235
+
/* cmp @v equal to @expect */
236
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
237
+
/* cmp @v2 equal to @expct2 */
238
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3])
239
+
#endif
240
+
/* final store */
241
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
242
+
RSEQ_INJECT_ASM(6)
243
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
244
+
: /* gcc asm goto does not allow outputs */
245
+
: [cpu_id] "r" (cpu),
246
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
247
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
248
+
/* cmp2 input */
249
+
[v2] "m" (*v2),
250
+
[expect2] "r" (expect2),
251
+
/* final store input */
252
+
[v] "m" (*v),
253
+
[expect] "r" (expect),
254
+
[newv] "r" (newv)
255
+
RSEQ_INJECT_INPUT
256
+
: "memory", "cc", "r17"
257
+
RSEQ_INJECT_CLOBBER
258
+
: abort, cmpfail
259
+
#ifdef RSEQ_COMPARE_TWICE
260
+
, error1, error2, error3
261
+
#endif
262
+
);
263
+
rseq_after_asm_goto();
264
+
return 0;
265
+
abort:
266
+
rseq_after_asm_goto();
267
+
RSEQ_INJECT_FAILED
268
+
return -1;
269
+
cmpfail:
270
+
rseq_after_asm_goto();
271
+
return 1;
272
+
#ifdef RSEQ_COMPARE_TWICE
273
+
error1:
274
+
rseq_after_asm_goto();
275
+
rseq_bug("cpu_id comparison failed");
276
+
error2:
277
+
rseq_after_asm_goto();
278
+
rseq_bug("1st expected value comparison failed");
279
+
error3:
280
+
rseq_after_asm_goto();
281
+
rseq_bug("2nd expected value comparison failed");
282
+
#endif
283
+
}
284
+
285
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
286
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
287
+
288
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
289
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
290
+
291
+
static inline __attribute__((always_inline))
292
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
293
+
intptr_t *v2, intptr_t newv2,
294
+
intptr_t newv, int cpu)
295
+
{
296
+
RSEQ_INJECT_C(9)
297
+
298
+
__asm__ __volatile__ goto (
299
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
300
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
301
+
#ifdef RSEQ_COMPARE_TWICE
302
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
303
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
304
+
#endif
305
+
/* Start rseq by storing table entry pointer into rseq_cs. */
306
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
307
+
/* cmp cpuid */
308
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
309
+
RSEQ_INJECT_ASM(3)
310
+
/* cmp @v equal to @expect */
311
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
312
+
RSEQ_INJECT_ASM(4)
313
+
#ifdef RSEQ_COMPARE_TWICE
314
+
/* cmp cpuid */
315
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
316
+
/* cmp @v equal to @expect */
317
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
318
+
#endif
319
+
/* try store */
320
+
RSEQ_ASM_OP_STORE(newv2, v2)
321
+
RSEQ_INJECT_ASM(5)
322
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
323
+
/* for 'release' */
324
+
"lwsync\n\t"
325
+
#endif
326
+
/* final store */
327
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
328
+
RSEQ_INJECT_ASM(6)
329
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
330
+
: /* gcc asm goto does not allow outputs */
331
+
: [cpu_id] "r" (cpu),
332
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
333
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
334
+
/* try store input */
335
+
[v2] "m" (*v2),
336
+
[newv2] "r" (newv2),
337
+
/* final store input */
338
+
[v] "m" (*v),
339
+
[expect] "r" (expect),
340
+
[newv] "r" (newv)
341
+
RSEQ_INJECT_INPUT
342
+
: "memory", "cc", "r17"
343
+
RSEQ_INJECT_CLOBBER
344
+
: abort, cmpfail
345
+
#ifdef RSEQ_COMPARE_TWICE
346
+
, error1, error2
347
+
#endif
348
+
);
349
+
rseq_after_asm_goto();
350
+
return 0;
351
+
abort:
352
+
rseq_after_asm_goto();
353
+
RSEQ_INJECT_FAILED
354
+
return -1;
355
+
cmpfail:
356
+
rseq_after_asm_goto();
357
+
return 1;
358
+
#ifdef RSEQ_COMPARE_TWICE
359
+
error1:
360
+
rseq_after_asm_goto();
361
+
rseq_bug("cpu_id comparison failed");
362
+
error2:
363
+
rseq_after_asm_goto();
364
+
rseq_bug("expected value comparison failed");
365
+
#endif
366
+
}
367
+
368
+
static inline __attribute__((always_inline))
369
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
370
+
void *dst, void *src, size_t len,
371
+
intptr_t newv, int cpu)
372
+
{
373
+
RSEQ_INJECT_C(9)
374
+
375
+
__asm__ __volatile__ goto (
376
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
377
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
378
+
#ifdef RSEQ_COMPARE_TWICE
379
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
380
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
381
+
#endif
382
+
/* setup for mempcy */
383
+
"mr %%r19, %[len]\n\t"
384
+
"mr %%r20, %[src]\n\t"
385
+
"mr %%r21, %[dst]\n\t"
386
+
/* Start rseq by storing table entry pointer into rseq_cs. */
387
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
388
+
/* cmp cpuid */
389
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
390
+
RSEQ_INJECT_ASM(3)
391
+
/* cmp @v equal to @expect */
392
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
393
+
RSEQ_INJECT_ASM(4)
394
+
#ifdef RSEQ_COMPARE_TWICE
395
+
/* cmp cpuid */
396
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
397
+
/* cmp @v equal to @expect */
398
+
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
399
+
#endif
400
+
/* try memcpy */
401
+
RSEQ_ASM_OP_R_MEMCPY()
402
+
RSEQ_INJECT_ASM(5)
403
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
404
+
/* for 'release' */
405
+
"lwsync\n\t"
406
+
#endif
407
+
/* final store */
408
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
409
+
RSEQ_INJECT_ASM(6)
410
+
/* teardown */
411
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
412
+
: /* gcc asm goto does not allow outputs */
413
+
: [cpu_id] "r" (cpu),
414
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
415
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
416
+
/* final store input */
417
+
[v] "m" (*v),
418
+
[expect] "r" (expect),
419
+
[newv] "r" (newv),
420
+
/* try memcpy input */
421
+
[dst] "r" (dst),
422
+
[src] "r" (src),
423
+
[len] "r" (len)
424
+
RSEQ_INJECT_INPUT
425
+
: "memory", "cc", "r17", "r18", "r19", "r20", "r21"
426
+
RSEQ_INJECT_CLOBBER
427
+
: abort, cmpfail
428
+
#ifdef RSEQ_COMPARE_TWICE
429
+
, error1, error2
430
+
#endif
431
+
);
432
+
rseq_after_asm_goto();
433
+
return 0;
434
+
abort:
435
+
rseq_after_asm_goto();
436
+
RSEQ_INJECT_FAILED
437
+
return -1;
438
+
cmpfail:
439
+
rseq_after_asm_goto();
440
+
return 1;
441
+
#ifdef RSEQ_COMPARE_TWICE
442
+
error1:
443
+
rseq_after_asm_goto();
444
+
rseq_bug("cpu_id comparison failed");
445
+
error2:
446
+
rseq_after_asm_goto();
447
+
rseq_bug("expected value comparison failed");
448
+
#endif
449
+
}
450
+
451
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
452
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
453
+
454
+
#include "rseq-bits-reset.h"
+25
-578
tools/testing/selftests/rseq/rseq-ppc.h
+25
-578
tools/testing/selftests/rseq/rseq-ppc.h
···
2
2
/*
3
3
* rseq-ppc.h
4
4
*
5
-
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
6
* (C) Copyright 2016-2018 - Boqun Feng <boqun.feng@gmail.com>
7
7
*/
8
8
···
35
35
rseq_smp_lwsync(); \
36
36
RSEQ_WRITE_ONCE(*p, v); \
37
37
} while (0)
38
-
39
-
#ifdef RSEQ_SKIP_FASTPATH
40
-
#include "rseq-skip.h"
41
-
#else /* !RSEQ_SKIP_FASTPATH */
42
38
43
39
/*
44
40
* The __rseq_cs_ptr_array and __rseq_cs sections can be used by debuggers to
···
205
209
RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \
206
210
__rseq_str(post_commit_label) ":\n\t"
207
211
208
-
static inline __attribute__((always_inline))
209
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
210
-
{
211
-
RSEQ_INJECT_C(9)
212
+
/* Per-cpu-id indexing. */
212
213
213
-
__asm__ __volatile__ goto (
214
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
215
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
216
-
#ifdef RSEQ_COMPARE_TWICE
217
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
218
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
219
-
#endif
220
-
/* Start rseq by storing table entry pointer into rseq_cs. */
221
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
222
-
/* cmp cpuid */
223
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
224
-
RSEQ_INJECT_ASM(3)
225
-
/* cmp @v equal to @expect */
226
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
227
-
RSEQ_INJECT_ASM(4)
228
-
#ifdef RSEQ_COMPARE_TWICE
229
-
/* cmp cpuid */
230
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
231
-
/* cmp @v equal to @expect */
232
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
233
-
#endif
234
-
/* final store */
235
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
236
-
RSEQ_INJECT_ASM(5)
237
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
238
-
: /* gcc asm goto does not allow outputs */
239
-
: [cpu_id] "r" (cpu),
240
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
241
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
242
-
[v] "m" (*v),
243
-
[expect] "r" (expect),
244
-
[newv] "r" (newv)
245
-
RSEQ_INJECT_INPUT
246
-
: "memory", "cc", "r17"
247
-
RSEQ_INJECT_CLOBBER
248
-
: abort, cmpfail
249
-
#ifdef RSEQ_COMPARE_TWICE
250
-
, error1, error2
251
-
#endif
252
-
);
253
-
rseq_after_asm_goto();
254
-
return 0;
255
-
abort:
256
-
rseq_after_asm_goto();
257
-
RSEQ_INJECT_FAILED
258
-
return -1;
259
-
cmpfail:
260
-
rseq_after_asm_goto();
261
-
return 1;
262
-
#ifdef RSEQ_COMPARE_TWICE
263
-
error1:
264
-
rseq_after_asm_goto();
265
-
rseq_bug("cpu_id comparison failed");
266
-
error2:
267
-
rseq_after_asm_goto();
268
-
rseq_bug("expected value comparison failed");
269
-
#endif
270
-
}
214
+
#define RSEQ_TEMPLATE_CPU_ID
215
+
#define RSEQ_TEMPLATE_MO_RELAXED
216
+
#include "rseq-ppc-bits.h"
217
+
#undef RSEQ_TEMPLATE_MO_RELAXED
271
218
272
-
static inline __attribute__((always_inline))
273
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
274
-
long voffp, intptr_t *load, int cpu)
275
-
{
276
-
RSEQ_INJECT_C(9)
219
+
#define RSEQ_TEMPLATE_MO_RELEASE
220
+
#include "rseq-ppc-bits.h"
221
+
#undef RSEQ_TEMPLATE_MO_RELEASE
222
+
#undef RSEQ_TEMPLATE_CPU_ID
277
223
278
-
__asm__ __volatile__ goto (
279
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
280
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
281
-
#ifdef RSEQ_COMPARE_TWICE
282
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
283
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
284
-
#endif
285
-
/* Start rseq by storing table entry pointer into rseq_cs. */
286
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
287
-
/* cmp cpuid */
288
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
289
-
RSEQ_INJECT_ASM(3)
290
-
/* cmp @v not equal to @expectnot */
291
-
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail])
292
-
RSEQ_INJECT_ASM(4)
293
-
#ifdef RSEQ_COMPARE_TWICE
294
-
/* cmp cpuid */
295
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
296
-
/* cmp @v not equal to @expectnot */
297
-
RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2])
298
-
#endif
299
-
/* load the value of @v */
300
-
RSEQ_ASM_OP_R_LOAD(v)
301
-
/* store it in @load */
302
-
RSEQ_ASM_OP_R_STORE(load)
303
-
/* dereference voffp(v) */
304
-
RSEQ_ASM_OP_R_LOADX(voffp)
305
-
/* final store the value at voffp(v) */
306
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 2)
307
-
RSEQ_INJECT_ASM(5)
308
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
309
-
: /* gcc asm goto does not allow outputs */
310
-
: [cpu_id] "r" (cpu),
311
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
312
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
313
-
/* final store input */
314
-
[v] "m" (*v),
315
-
[expectnot] "r" (expectnot),
316
-
[voffp] "b" (voffp),
317
-
[load] "m" (*load)
318
-
RSEQ_INJECT_INPUT
319
-
: "memory", "cc", "r17"
320
-
RSEQ_INJECT_CLOBBER
321
-
: abort, cmpfail
322
-
#ifdef RSEQ_COMPARE_TWICE
323
-
, error1, error2
324
-
#endif
325
-
);
326
-
rseq_after_asm_goto();
327
-
return 0;
328
-
abort:
329
-
rseq_after_asm_goto();
330
-
RSEQ_INJECT_FAILED
331
-
return -1;
332
-
cmpfail:
333
-
rseq_after_asm_goto();
334
-
return 1;
335
-
#ifdef RSEQ_COMPARE_TWICE
336
-
error1:
337
-
rseq_after_asm_goto();
338
-
rseq_bug("cpu_id comparison failed");
339
-
error2:
340
-
rseq_after_asm_goto();
341
-
rseq_bug("expected value comparison failed");
342
-
#endif
343
-
}
224
+
/* Per-mm-cid indexing. */
344
225
345
-
static inline __attribute__((always_inline))
346
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
347
-
{
348
-
RSEQ_INJECT_C(9)
226
+
#define RSEQ_TEMPLATE_MM_CID
227
+
#define RSEQ_TEMPLATE_MO_RELAXED
228
+
#include "rseq-ppc-bits.h"
229
+
#undef RSEQ_TEMPLATE_MO_RELAXED
349
230
350
-
__asm__ __volatile__ goto (
351
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
352
-
#ifdef RSEQ_COMPARE_TWICE
353
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
354
-
#endif
355
-
/* Start rseq by storing table entry pointer into rseq_cs. */
356
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
357
-
/* cmp cpuid */
358
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
359
-
RSEQ_INJECT_ASM(3)
360
-
#ifdef RSEQ_COMPARE_TWICE
361
-
/* cmp cpuid */
362
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
363
-
#endif
364
-
/* load the value of @v */
365
-
RSEQ_ASM_OP_R_LOAD(v)
366
-
/* add @count to it */
367
-
RSEQ_ASM_OP_R_ADD(count)
368
-
/* final store */
369
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 2)
370
-
RSEQ_INJECT_ASM(4)
371
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
372
-
: /* gcc asm goto does not allow outputs */
373
-
: [cpu_id] "r" (cpu),
374
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
375
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
376
-
/* final store input */
377
-
[v] "m" (*v),
378
-
[count] "r" (count)
379
-
RSEQ_INJECT_INPUT
380
-
: "memory", "cc", "r17"
381
-
RSEQ_INJECT_CLOBBER
382
-
: abort
383
-
#ifdef RSEQ_COMPARE_TWICE
384
-
, error1
385
-
#endif
386
-
);
387
-
rseq_after_asm_goto();
388
-
return 0;
389
-
abort:
390
-
rseq_after_asm_goto();
391
-
RSEQ_INJECT_FAILED
392
-
return -1;
393
-
#ifdef RSEQ_COMPARE_TWICE
394
-
error1:
395
-
rseq_after_asm_goto();
396
-
rseq_bug("cpu_id comparison failed");
397
-
#endif
398
-
}
231
+
#define RSEQ_TEMPLATE_MO_RELEASE
232
+
#include "rseq-ppc-bits.h"
233
+
#undef RSEQ_TEMPLATE_MO_RELEASE
234
+
#undef RSEQ_TEMPLATE_MM_CID
399
235
400
-
static inline __attribute__((always_inline))
401
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
402
-
intptr_t *v2, intptr_t newv2,
403
-
intptr_t newv, int cpu)
404
-
{
405
-
RSEQ_INJECT_C(9)
236
+
/* APIs which are not based on cpu ids. */
406
237
407
-
__asm__ __volatile__ goto (
408
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
409
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
410
-
#ifdef RSEQ_COMPARE_TWICE
411
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
412
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
413
-
#endif
414
-
/* Start rseq by storing table entry pointer into rseq_cs. */
415
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
416
-
/* cmp cpuid */
417
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
418
-
RSEQ_INJECT_ASM(3)
419
-
/* cmp @v equal to @expect */
420
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
421
-
RSEQ_INJECT_ASM(4)
422
-
#ifdef RSEQ_COMPARE_TWICE
423
-
/* cmp cpuid */
424
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
425
-
/* cmp @v equal to @expect */
426
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
427
-
#endif
428
-
/* try store */
429
-
RSEQ_ASM_OP_STORE(newv2, v2)
430
-
RSEQ_INJECT_ASM(5)
431
-
/* final store */
432
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
433
-
RSEQ_INJECT_ASM(6)
434
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
435
-
: /* gcc asm goto does not allow outputs */
436
-
: [cpu_id] "r" (cpu),
437
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
438
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
439
-
/* try store input */
440
-
[v2] "m" (*v2),
441
-
[newv2] "r" (newv2),
442
-
/* final store input */
443
-
[v] "m" (*v),
444
-
[expect] "r" (expect),
445
-
[newv] "r" (newv)
446
-
RSEQ_INJECT_INPUT
447
-
: "memory", "cc", "r17"
448
-
RSEQ_INJECT_CLOBBER
449
-
: abort, cmpfail
450
-
#ifdef RSEQ_COMPARE_TWICE
451
-
, error1, error2
452
-
#endif
453
-
);
454
-
rseq_after_asm_goto();
455
-
return 0;
456
-
abort:
457
-
rseq_after_asm_goto();
458
-
RSEQ_INJECT_FAILED
459
-
return -1;
460
-
cmpfail:
461
-
rseq_after_asm_goto();
462
-
return 1;
463
-
#ifdef RSEQ_COMPARE_TWICE
464
-
error1:
465
-
rseq_after_asm_goto();
466
-
rseq_bug("cpu_id comparison failed");
467
-
error2:
468
-
rseq_after_asm_goto();
469
-
rseq_bug("expected value comparison failed");
470
-
#endif
471
-
}
472
-
473
-
static inline __attribute__((always_inline))
474
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
475
-
intptr_t *v2, intptr_t newv2,
476
-
intptr_t newv, int cpu)
477
-
{
478
-
RSEQ_INJECT_C(9)
479
-
480
-
__asm__ __volatile__ goto (
481
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
482
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
483
-
#ifdef RSEQ_COMPARE_TWICE
484
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
485
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
486
-
#endif
487
-
/* Start rseq by storing table entry pointer into rseq_cs. */
488
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
489
-
/* cmp cpuid */
490
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
491
-
RSEQ_INJECT_ASM(3)
492
-
/* cmp @v equal to @expect */
493
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
494
-
RSEQ_INJECT_ASM(4)
495
-
#ifdef RSEQ_COMPARE_TWICE
496
-
/* cmp cpuid */
497
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
498
-
/* cmp @v equal to @expect */
499
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
500
-
#endif
501
-
/* try store */
502
-
RSEQ_ASM_OP_STORE(newv2, v2)
503
-
RSEQ_INJECT_ASM(5)
504
-
/* for 'release' */
505
-
"lwsync\n\t"
506
-
/* final store */
507
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
508
-
RSEQ_INJECT_ASM(6)
509
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
510
-
: /* gcc asm goto does not allow outputs */
511
-
: [cpu_id] "r" (cpu),
512
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
513
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
514
-
/* try store input */
515
-
[v2] "m" (*v2),
516
-
[newv2] "r" (newv2),
517
-
/* final store input */
518
-
[v] "m" (*v),
519
-
[expect] "r" (expect),
520
-
[newv] "r" (newv)
521
-
RSEQ_INJECT_INPUT
522
-
: "memory", "cc", "r17"
523
-
RSEQ_INJECT_CLOBBER
524
-
: abort, cmpfail
525
-
#ifdef RSEQ_COMPARE_TWICE
526
-
, error1, error2
527
-
#endif
528
-
);
529
-
rseq_after_asm_goto();
530
-
return 0;
531
-
abort:
532
-
rseq_after_asm_goto();
533
-
RSEQ_INJECT_FAILED
534
-
return -1;
535
-
cmpfail:
536
-
rseq_after_asm_goto();
537
-
return 1;
538
-
#ifdef RSEQ_COMPARE_TWICE
539
-
error1:
540
-
rseq_after_asm_goto();
541
-
rseq_bug("cpu_id comparison failed");
542
-
error2:
543
-
rseq_after_asm_goto();
544
-
rseq_bug("expected value comparison failed");
545
-
#endif
546
-
}
547
-
548
-
static inline __attribute__((always_inline))
549
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
550
-
intptr_t *v2, intptr_t expect2,
551
-
intptr_t newv, int cpu)
552
-
{
553
-
RSEQ_INJECT_C(9)
554
-
555
-
__asm__ __volatile__ goto (
556
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
557
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
558
-
#ifdef RSEQ_COMPARE_TWICE
559
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
560
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
561
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
562
-
#endif
563
-
/* Start rseq by storing table entry pointer into rseq_cs. */
564
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
565
-
/* cmp cpuid */
566
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
567
-
RSEQ_INJECT_ASM(3)
568
-
/* cmp @v equal to @expect */
569
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
570
-
RSEQ_INJECT_ASM(4)
571
-
/* cmp @v2 equal to @expct2 */
572
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail])
573
-
RSEQ_INJECT_ASM(5)
574
-
#ifdef RSEQ_COMPARE_TWICE
575
-
/* cmp cpuid */
576
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
577
-
/* cmp @v equal to @expect */
578
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
579
-
/* cmp @v2 equal to @expct2 */
580
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3])
581
-
#endif
582
-
/* final store */
583
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
584
-
RSEQ_INJECT_ASM(6)
585
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
586
-
: /* gcc asm goto does not allow outputs */
587
-
: [cpu_id] "r" (cpu),
588
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
589
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
590
-
/* cmp2 input */
591
-
[v2] "m" (*v2),
592
-
[expect2] "r" (expect2),
593
-
/* final store input */
594
-
[v] "m" (*v),
595
-
[expect] "r" (expect),
596
-
[newv] "r" (newv)
597
-
RSEQ_INJECT_INPUT
598
-
: "memory", "cc", "r17"
599
-
RSEQ_INJECT_CLOBBER
600
-
: abort, cmpfail
601
-
#ifdef RSEQ_COMPARE_TWICE
602
-
, error1, error2, error3
603
-
#endif
604
-
);
605
-
rseq_after_asm_goto();
606
-
return 0;
607
-
abort:
608
-
rseq_after_asm_goto();
609
-
RSEQ_INJECT_FAILED
610
-
return -1;
611
-
cmpfail:
612
-
rseq_after_asm_goto();
613
-
return 1;
614
-
#ifdef RSEQ_COMPARE_TWICE
615
-
error1:
616
-
rseq_after_asm_goto();
617
-
rseq_bug("cpu_id comparison failed");
618
-
error2:
619
-
rseq_after_asm_goto();
620
-
rseq_bug("1st expected value comparison failed");
621
-
error3:
622
-
rseq_after_asm_goto();
623
-
rseq_bug("2nd expected value comparison failed");
624
-
#endif
625
-
}
626
-
627
-
static inline __attribute__((always_inline))
628
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
629
-
void *dst, void *src, size_t len,
630
-
intptr_t newv, int cpu)
631
-
{
632
-
RSEQ_INJECT_C(9)
633
-
634
-
__asm__ __volatile__ goto (
635
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
636
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
637
-
#ifdef RSEQ_COMPARE_TWICE
638
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
639
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
640
-
#endif
641
-
/* setup for mempcy */
642
-
"mr %%r19, %[len]\n\t"
643
-
"mr %%r20, %[src]\n\t"
644
-
"mr %%r21, %[dst]\n\t"
645
-
/* Start rseq by storing table entry pointer into rseq_cs. */
646
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
647
-
/* cmp cpuid */
648
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
649
-
RSEQ_INJECT_ASM(3)
650
-
/* cmp @v equal to @expect */
651
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
652
-
RSEQ_INJECT_ASM(4)
653
-
#ifdef RSEQ_COMPARE_TWICE
654
-
/* cmp cpuid */
655
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
656
-
/* cmp @v equal to @expect */
657
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
658
-
#endif
659
-
/* try memcpy */
660
-
RSEQ_ASM_OP_R_MEMCPY()
661
-
RSEQ_INJECT_ASM(5)
662
-
/* final store */
663
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
664
-
RSEQ_INJECT_ASM(6)
665
-
/* teardown */
666
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
667
-
: /* gcc asm goto does not allow outputs */
668
-
: [cpu_id] "r" (cpu),
669
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
670
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
671
-
/* final store input */
672
-
[v] "m" (*v),
673
-
[expect] "r" (expect),
674
-
[newv] "r" (newv),
675
-
/* try memcpy input */
676
-
[dst] "r" (dst),
677
-
[src] "r" (src),
678
-
[len] "r" (len)
679
-
RSEQ_INJECT_INPUT
680
-
: "memory", "cc", "r17", "r18", "r19", "r20", "r21"
681
-
RSEQ_INJECT_CLOBBER
682
-
: abort, cmpfail
683
-
#ifdef RSEQ_COMPARE_TWICE
684
-
, error1, error2
685
-
#endif
686
-
);
687
-
rseq_after_asm_goto();
688
-
return 0;
689
-
abort:
690
-
rseq_after_asm_goto();
691
-
RSEQ_INJECT_FAILED
692
-
return -1;
693
-
cmpfail:
694
-
rseq_after_asm_goto();
695
-
return 1;
696
-
#ifdef RSEQ_COMPARE_TWICE
697
-
error1:
698
-
rseq_after_asm_goto();
699
-
rseq_bug("cpu_id comparison failed");
700
-
error2:
701
-
rseq_after_asm_goto();
702
-
rseq_bug("expected value comparison failed");
703
-
#endif
704
-
}
705
-
706
-
static inline __attribute__((always_inline))
707
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
708
-
void *dst, void *src, size_t len,
709
-
intptr_t newv, int cpu)
710
-
{
711
-
RSEQ_INJECT_C(9)
712
-
713
-
__asm__ __volatile__ goto (
714
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
715
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
716
-
#ifdef RSEQ_COMPARE_TWICE
717
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
718
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
719
-
#endif
720
-
/* setup for mempcy */
721
-
"mr %%r19, %[len]\n\t"
722
-
"mr %%r20, %[src]\n\t"
723
-
"mr %%r21, %[dst]\n\t"
724
-
/* Start rseq by storing table entry pointer into rseq_cs. */
725
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
726
-
/* cmp cpuid */
727
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
728
-
RSEQ_INJECT_ASM(3)
729
-
/* cmp @v equal to @expect */
730
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])
731
-
RSEQ_INJECT_ASM(4)
732
-
#ifdef RSEQ_COMPARE_TWICE
733
-
/* cmp cpuid */
734
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
735
-
/* cmp @v equal to @expect */
736
-
RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])
737
-
#endif
738
-
/* try memcpy */
739
-
RSEQ_ASM_OP_R_MEMCPY()
740
-
RSEQ_INJECT_ASM(5)
741
-
/* for 'release' */
742
-
"lwsync\n\t"
743
-
/* final store */
744
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 2)
745
-
RSEQ_INJECT_ASM(6)
746
-
/* teardown */
747
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
748
-
: /* gcc asm goto does not allow outputs */
749
-
: [cpu_id] "r" (cpu),
750
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
751
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
752
-
/* final store input */
753
-
[v] "m" (*v),
754
-
[expect] "r" (expect),
755
-
[newv] "r" (newv),
756
-
/* try memcpy input */
757
-
[dst] "r" (dst),
758
-
[src] "r" (src),
759
-
[len] "r" (len)
760
-
RSEQ_INJECT_INPUT
761
-
: "memory", "cc", "r17", "r18", "r19", "r20", "r21"
762
-
RSEQ_INJECT_CLOBBER
763
-
: abort, cmpfail
764
-
#ifdef RSEQ_COMPARE_TWICE
765
-
, error1, error2
766
-
#endif
767
-
);
768
-
rseq_after_asm_goto();
769
-
return 0;
770
-
abort:
771
-
rseq_after_asm_goto();
772
-
RSEQ_INJECT_FAILED
773
-
return -1;
774
-
cmpfail:
775
-
rseq_after_asm_goto();
776
-
return 1;
777
-
#ifdef RSEQ_COMPARE_TWICE
778
-
error1:
779
-
rseq_after_asm_goto();
780
-
rseq_bug("cpu_id comparison failed");
781
-
error2:
782
-
rseq_after_asm_goto();
783
-
rseq_bug("expected value comparison failed");
784
-
#endif
785
-
}
786
-
787
-
#endif /* !RSEQ_SKIP_FASTPATH */
238
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
239
+
#define RSEQ_TEMPLATE_MO_RELAXED
240
+
#include "rseq-ppc-bits.h"
241
+
#undef RSEQ_TEMPLATE_MO_RELAXED
242
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+410
tools/testing/selftests/rseq/rseq-riscv-bits.h
+410
tools/testing/selftests/rseq/rseq-riscv-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
3
+
#include "rseq-bits-template.h"
4
+
5
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
6
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
7
+
8
+
static inline __always_inline
9
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
10
+
{
11
+
RSEQ_INJECT_C(9)
12
+
13
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
14
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
15
+
#ifdef RSEQ_COMPARE_TWICE
16
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
17
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
18
+
#endif
19
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
20
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
21
+
RSEQ_INJECT_ASM(3)
22
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
23
+
RSEQ_INJECT_ASM(4)
24
+
#ifdef RSEQ_COMPARE_TWICE
25
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
26
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
27
+
#endif
28
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
29
+
RSEQ_INJECT_ASM(5)
30
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
31
+
: /* gcc asm goto does not allow outputs */
32
+
: [cpu_id] "r" (cpu),
33
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
34
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
35
+
[v] "m" (*v),
36
+
[expect] "r" (expect),
37
+
[newv] "r" (newv)
38
+
RSEQ_INJECT_INPUT
39
+
: "memory", RSEQ_ASM_TMP_REG_1
40
+
RSEQ_INJECT_CLOBBER
41
+
: abort, cmpfail
42
+
#ifdef RSEQ_COMPARE_TWICE
43
+
, error1, error2
44
+
#endif
45
+
);
46
+
47
+
return 0;
48
+
abort:
49
+
RSEQ_INJECT_FAILED
50
+
return -1;
51
+
cmpfail:
52
+
return 1;
53
+
#ifdef RSEQ_COMPARE_TWICE
54
+
error1:
55
+
rseq_bug("cpu_id comparison failed");
56
+
error2:
57
+
rseq_bug("expected value comparison failed");
58
+
#endif
59
+
}
60
+
61
+
static inline __always_inline
62
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
63
+
off_t voffp, intptr_t *load, int cpu)
64
+
{
65
+
RSEQ_INJECT_C(9)
66
+
67
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
68
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
69
+
#ifdef RSEQ_COMPARE_TWICE
70
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
71
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
72
+
#endif
73
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
74
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
75
+
RSEQ_INJECT_ASM(3)
76
+
RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[cmpfail]")
77
+
RSEQ_INJECT_ASM(4)
78
+
#ifdef RSEQ_COMPARE_TWICE
79
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
80
+
RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[error2]")
81
+
#endif
82
+
RSEQ_ASM_OP_R_LOAD(v)
83
+
RSEQ_ASM_OP_R_STORE(load)
84
+
RSEQ_ASM_OP_R_LOAD_OFF(voffp)
85
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
86
+
RSEQ_INJECT_ASM(5)
87
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
88
+
: /* gcc asm goto does not allow outputs */
89
+
: [cpu_id] "r" (cpu),
90
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
91
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
92
+
[v] "m" (*v),
93
+
[expectnot] "r" (expectnot),
94
+
[load] "m" (*load),
95
+
[voffp] "r" (voffp)
96
+
RSEQ_INJECT_INPUT
97
+
: "memory", RSEQ_ASM_TMP_REG_1
98
+
RSEQ_INJECT_CLOBBER
99
+
: abort, cmpfail
100
+
#ifdef RSEQ_COMPARE_TWICE
101
+
, error1, error2
102
+
#endif
103
+
);
104
+
return 0;
105
+
abort:
106
+
RSEQ_INJECT_FAILED
107
+
return -1;
108
+
cmpfail:
109
+
return 1;
110
+
#ifdef RSEQ_COMPARE_TWICE
111
+
error1:
112
+
rseq_bug("cpu_id comparison failed");
113
+
error2:
114
+
rseq_bug("expected value comparison failed");
115
+
#endif
116
+
}
117
+
118
+
static inline __always_inline
119
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
120
+
{
121
+
RSEQ_INJECT_C(9)
122
+
123
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
124
+
#ifdef RSEQ_COMPARE_TWICE
125
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
126
+
#endif
127
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
128
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
129
+
RSEQ_INJECT_ASM(3)
130
+
#ifdef RSEQ_COMPARE_TWICE
131
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
132
+
#endif
133
+
RSEQ_ASM_OP_R_LOAD(v)
134
+
RSEQ_ASM_OP_R_ADD(count)
135
+
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
136
+
RSEQ_INJECT_ASM(4)
137
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
138
+
: /* gcc asm goto does not allow outputs */
139
+
: [cpu_id] "r" (cpu),
140
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
141
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
142
+
[v] "m" (*v),
143
+
[count] "r" (count)
144
+
RSEQ_INJECT_INPUT
145
+
: "memory", RSEQ_ASM_TMP_REG_1
146
+
RSEQ_INJECT_CLOBBER
147
+
: abort
148
+
#ifdef RSEQ_COMPARE_TWICE
149
+
, error1
150
+
#endif
151
+
);
152
+
return 0;
153
+
abort:
154
+
RSEQ_INJECT_FAILED
155
+
return -1;
156
+
#ifdef RSEQ_COMPARE_TWICE
157
+
error1:
158
+
rseq_bug("cpu_id comparison failed");
159
+
#endif
160
+
}
161
+
162
+
static inline __always_inline
163
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
164
+
intptr_t *v2, intptr_t expect2,
165
+
intptr_t newv, int cpu)
166
+
{
167
+
RSEQ_INJECT_C(9)
168
+
169
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
170
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
171
+
#ifdef RSEQ_COMPARE_TWICE
172
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
173
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
174
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error3]")
175
+
#endif
176
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
177
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
178
+
RSEQ_INJECT_ASM(3)
179
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
180
+
RSEQ_INJECT_ASM(4)
181
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[cmpfail]")
182
+
RSEQ_INJECT_ASM(5)
183
+
#ifdef RSEQ_COMPARE_TWICE
184
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
185
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
186
+
RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[error3]")
187
+
#endif
188
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
189
+
RSEQ_INJECT_ASM(6)
190
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
191
+
: /* gcc asm goto does not allow outputs */
192
+
: [cpu_id] "r" (cpu),
193
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
194
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
195
+
[v] "m" (*v),
196
+
[expect] "r" (expect),
197
+
[v2] "m" (*v2),
198
+
[expect2] "r" (expect2),
199
+
[newv] "r" (newv)
200
+
RSEQ_INJECT_INPUT
201
+
: "memory", RSEQ_ASM_TMP_REG_1
202
+
RSEQ_INJECT_CLOBBER
203
+
: abort, cmpfail
204
+
#ifdef RSEQ_COMPARE_TWICE
205
+
, error1, error2, error3
206
+
#endif
207
+
);
208
+
209
+
return 0;
210
+
abort:
211
+
RSEQ_INJECT_FAILED
212
+
return -1;
213
+
cmpfail:
214
+
return 1;
215
+
#ifdef RSEQ_COMPARE_TWICE
216
+
error1:
217
+
rseq_bug("cpu_id comparison failed");
218
+
error2:
219
+
rseq_bug("expected value comparison failed");
220
+
error3:
221
+
rseq_bug("2nd expected value comparison failed");
222
+
#endif
223
+
}
224
+
225
+
#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
226
+
227
+
/*
228
+
* pval = *(ptr+off)
229
+
* *pval += inc;
230
+
*/
231
+
static inline __always_inline
232
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_offset_deref_addv)(intptr_t *ptr, off_t off, intptr_t inc, int cpu)
233
+
{
234
+
RSEQ_INJECT_C(9)
235
+
236
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
237
+
#ifdef RSEQ_COMPARE_TWICE
238
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
239
+
#endif
240
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
241
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
242
+
RSEQ_INJECT_ASM(3)
243
+
#ifdef RSEQ_COMPARE_TWICE
244
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
245
+
#endif
246
+
RSEQ_ASM_OP_R_DEREF_ADDV(ptr, off, 3)
247
+
RSEQ_INJECT_ASM(4)
248
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
249
+
: /* gcc asm goto does not allow outputs */
250
+
: [cpu_id] "r" (cpu),
251
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
252
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
253
+
[ptr] "r" (ptr),
254
+
[off] "er" (off),
255
+
[inc] "er" (inc)
256
+
RSEQ_INJECT_INPUT
257
+
: "memory", RSEQ_ASM_TMP_REG_1
258
+
RSEQ_INJECT_CLOBBER
259
+
: abort
260
+
#ifdef RSEQ_COMPARE_TWICE
261
+
, error1
262
+
#endif
263
+
);
264
+
return 0;
265
+
abort:
266
+
RSEQ_INJECT_FAILED
267
+
return -1;
268
+
#ifdef RSEQ_COMPARE_TWICE
269
+
error1:
270
+
rseq_bug("cpu_id comparison failed");
271
+
#endif
272
+
}
273
+
274
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
275
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
276
+
277
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
278
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
279
+
280
+
static inline __always_inline
281
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
282
+
intptr_t *v2, intptr_t newv2,
283
+
intptr_t newv, int cpu)
284
+
{
285
+
RSEQ_INJECT_C(9)
286
+
287
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
288
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
289
+
#ifdef RSEQ_COMPARE_TWICE
290
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
291
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
292
+
#endif
293
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
294
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
295
+
RSEQ_INJECT_ASM(3)
296
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
297
+
RSEQ_INJECT_ASM(4)
298
+
#ifdef RSEQ_COMPARE_TWICE
299
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
300
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
301
+
#endif
302
+
RSEQ_ASM_OP_STORE(newv2, v2)
303
+
RSEQ_INJECT_ASM(5)
304
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
305
+
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
306
+
#else
307
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
308
+
#endif
309
+
RSEQ_INJECT_ASM(6)
310
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
311
+
: /* gcc asm goto does not allow outputs */
312
+
: [cpu_id] "r" (cpu),
313
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
314
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
315
+
[expect] "r" (expect),
316
+
[v] "m" (*v),
317
+
[newv] "r" (newv),
318
+
[v2] "m" (*v2),
319
+
[newv2] "r" (newv2)
320
+
RSEQ_INJECT_INPUT
321
+
: "memory", RSEQ_ASM_TMP_REG_1
322
+
RSEQ_INJECT_CLOBBER
323
+
: abort, cmpfail
324
+
#ifdef RSEQ_COMPARE_TWICE
325
+
, error1, error2
326
+
#endif
327
+
);
328
+
329
+
return 0;
330
+
abort:
331
+
RSEQ_INJECT_FAILED
332
+
return -1;
333
+
cmpfail:
334
+
return 1;
335
+
#ifdef RSEQ_COMPARE_TWICE
336
+
error1:
337
+
rseq_bug("cpu_id comparison failed");
338
+
error2:
339
+
rseq_bug("expected value comparison failed");
340
+
#endif
341
+
}
342
+
343
+
static inline __always_inline
344
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
345
+
void *dst, void *src, size_t len,
346
+
intptr_t newv, int cpu)
347
+
{
348
+
RSEQ_INJECT_C(9)
349
+
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
350
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
351
+
#ifdef RSEQ_COMPARE_TWICE
352
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
353
+
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
354
+
#endif
355
+
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
356
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
357
+
RSEQ_INJECT_ASM(3)
358
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
359
+
RSEQ_INJECT_ASM(4)
360
+
#ifdef RSEQ_COMPARE_TWICE
361
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
362
+
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
363
+
#endif
364
+
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
365
+
RSEQ_INJECT_ASM(5)
366
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
367
+
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
368
+
#else
369
+
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
370
+
#endif
371
+
RSEQ_INJECT_ASM(6)
372
+
RSEQ_ASM_DEFINE_ABORT(4, abort)
373
+
: /* gcc asm goto does not allow outputs */
374
+
: [cpu_id] "r" (cpu),
375
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
376
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
377
+
[expect] "r" (expect),
378
+
[v] "m" (*v),
379
+
[newv] "r" (newv),
380
+
[dst] "r" (dst),
381
+
[src] "r" (src),
382
+
[len] "r" (len)
383
+
RSEQ_INJECT_INPUT
384
+
: "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2,
385
+
RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4
386
+
RSEQ_INJECT_CLOBBER
387
+
: abort, cmpfail
388
+
#ifdef RSEQ_COMPARE_TWICE
389
+
, error1, error2
390
+
#endif
391
+
);
392
+
393
+
return 0;
394
+
abort:
395
+
RSEQ_INJECT_FAILED
396
+
return -1;
397
+
cmpfail:
398
+
return 1;
399
+
#ifdef RSEQ_COMPARE_TWICE
400
+
error1:
401
+
rseq_bug("cpu_id comparison failed");
402
+
error2:
403
+
rseq_bug("expected value comparison failed");
404
+
#endif
405
+
}
406
+
407
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
408
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
409
+
410
+
#include "rseq-bits-reset.h"
+24
-503
tools/testing/selftests/rseq/rseq-riscv.h
+24
-503
tools/testing/selftests/rseq/rseq-riscv.h
···
49
49
RSEQ_WRITE_ONCE(*(p), v); \
50
50
} while (0)
51
51
52
-
#ifdef RSEQ_SKIP_FASTPATH
53
-
#include "rseq-skip.h"
54
-
#else /* !RSEQ_SKIP_FASTPATH */
55
-
56
52
#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \
57
53
post_commit_offset, abort_ip) \
58
54
".pushsection __rseq_cs, \"aw\"\n" \
···
165
169
RSEQ_ASM_OP_R_ADD(inc) \
166
170
__rseq_str(post_commit_label) ":\n"
167
171
168
-
static inline __always_inline
169
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
170
-
{
171
-
RSEQ_INJECT_C(9)
172
+
/* Per-cpu-id indexing. */
172
173
173
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
174
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
175
-
#ifdef RSEQ_COMPARE_TWICE
176
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
177
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
178
-
#endif
179
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
180
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
181
-
RSEQ_INJECT_ASM(3)
182
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
183
-
RSEQ_INJECT_ASM(4)
184
-
#ifdef RSEQ_COMPARE_TWICE
185
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
186
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
187
-
#endif
188
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
189
-
RSEQ_INJECT_ASM(5)
190
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
191
-
: /* gcc asm goto does not allow outputs */
192
-
: [cpu_id] "r" (cpu),
193
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
194
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
195
-
[v] "m" (*v),
196
-
[expect] "r" (expect),
197
-
[newv] "r" (newv)
198
-
RSEQ_INJECT_INPUT
199
-
: "memory", RSEQ_ASM_TMP_REG_1
200
-
RSEQ_INJECT_CLOBBER
201
-
: abort, cmpfail
202
-
#ifdef RSEQ_COMPARE_TWICE
203
-
, error1, error2
204
-
#endif
205
-
);
174
+
#define RSEQ_TEMPLATE_CPU_ID
175
+
#define RSEQ_TEMPLATE_MO_RELAXED
176
+
#include "rseq-riscv-bits.h"
177
+
#undef RSEQ_TEMPLATE_MO_RELAXED
206
178
207
-
return 0;
208
-
abort:
209
-
RSEQ_INJECT_FAILED
210
-
return -1;
211
-
cmpfail:
212
-
return 1;
213
-
#ifdef RSEQ_COMPARE_TWICE
214
-
error1:
215
-
rseq_bug("cpu_id comparison failed");
216
-
error2:
217
-
rseq_bug("expected value comparison failed");
218
-
#endif
219
-
}
179
+
#define RSEQ_TEMPLATE_MO_RELEASE
180
+
#include "rseq-riscv-bits.h"
181
+
#undef RSEQ_TEMPLATE_MO_RELEASE
182
+
#undef RSEQ_TEMPLATE_CPU_ID
220
183
221
-
static inline __always_inline
222
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
223
-
off_t voffp, intptr_t *load, int cpu)
224
-
{
225
-
RSEQ_INJECT_C(9)
184
+
/* Per-mm-cid indexing. */
226
185
227
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
228
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
229
-
#ifdef RSEQ_COMPARE_TWICE
230
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
231
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
232
-
#endif
233
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
234
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
235
-
RSEQ_INJECT_ASM(3)
236
-
RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[cmpfail]")
237
-
RSEQ_INJECT_ASM(4)
238
-
#ifdef RSEQ_COMPARE_TWICE
239
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
240
-
RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[error2]")
241
-
#endif
242
-
RSEQ_ASM_OP_R_LOAD(v)
243
-
RSEQ_ASM_OP_R_STORE(load)
244
-
RSEQ_ASM_OP_R_LOAD_OFF(voffp)
245
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
246
-
RSEQ_INJECT_ASM(5)
247
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
248
-
: /* gcc asm goto does not allow outputs */
249
-
: [cpu_id] "r" (cpu),
250
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
251
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
252
-
[v] "m" (*v),
253
-
[expectnot] "r" (expectnot),
254
-
[load] "m" (*load),
255
-
[voffp] "r" (voffp)
256
-
RSEQ_INJECT_INPUT
257
-
: "memory", RSEQ_ASM_TMP_REG_1
258
-
RSEQ_INJECT_CLOBBER
259
-
: abort, cmpfail
260
-
#ifdef RSEQ_COMPARE_TWICE
261
-
, error1, error2
262
-
#endif
263
-
);
264
-
return 0;
265
-
abort:
266
-
RSEQ_INJECT_FAILED
267
-
return -1;
268
-
cmpfail:
269
-
return 1;
270
-
#ifdef RSEQ_COMPARE_TWICE
271
-
error1:
272
-
rseq_bug("cpu_id comparison failed");
273
-
error2:
274
-
rseq_bug("expected value comparison failed");
275
-
#endif
276
-
}
186
+
#define RSEQ_TEMPLATE_MM_CID
187
+
#define RSEQ_TEMPLATE_MO_RELAXED
188
+
#include "rseq-riscv-bits.h"
189
+
#undef RSEQ_TEMPLATE_MO_RELAXED
277
190
278
-
static inline __always_inline
279
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
280
-
{
281
-
RSEQ_INJECT_C(9)
191
+
#define RSEQ_TEMPLATE_MO_RELEASE
192
+
#include "rseq-riscv-bits.h"
193
+
#undef RSEQ_TEMPLATE_MO_RELEASE
194
+
#undef RSEQ_TEMPLATE_MM_CID
282
195
283
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
284
-
#ifdef RSEQ_COMPARE_TWICE
285
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
286
-
#endif
287
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
288
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
289
-
RSEQ_INJECT_ASM(3)
290
-
#ifdef RSEQ_COMPARE_TWICE
291
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
292
-
#endif
293
-
RSEQ_ASM_OP_R_LOAD(v)
294
-
RSEQ_ASM_OP_R_ADD(count)
295
-
RSEQ_ASM_OP_R_FINAL_STORE(v, 3)
296
-
RSEQ_INJECT_ASM(4)
297
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
298
-
: /* gcc asm goto does not allow outputs */
299
-
: [cpu_id] "r" (cpu),
300
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
301
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
302
-
[v] "m" (*v),
303
-
[count] "r" (count)
304
-
RSEQ_INJECT_INPUT
305
-
: "memory", RSEQ_ASM_TMP_REG_1
306
-
RSEQ_INJECT_CLOBBER
307
-
: abort
308
-
#ifdef RSEQ_COMPARE_TWICE
309
-
, error1
310
-
#endif
311
-
);
312
-
return 0;
313
-
abort:
314
-
RSEQ_INJECT_FAILED
315
-
return -1;
316
-
#ifdef RSEQ_COMPARE_TWICE
317
-
error1:
318
-
rseq_bug("cpu_id comparison failed");
319
-
#endif
320
-
}
196
+
/* APIs which are not based on cpu ids. */
321
197
322
-
static inline __always_inline
323
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
324
-
intptr_t *v2, intptr_t newv2,
325
-
intptr_t newv, int cpu)
326
-
{
327
-
RSEQ_INJECT_C(9)
328
-
329
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
330
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
331
-
#ifdef RSEQ_COMPARE_TWICE
332
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
333
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
334
-
#endif
335
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
336
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
337
-
RSEQ_INJECT_ASM(3)
338
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
339
-
RSEQ_INJECT_ASM(4)
340
-
#ifdef RSEQ_COMPARE_TWICE
341
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
342
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
343
-
#endif
344
-
RSEQ_ASM_OP_STORE(newv2, v2)
345
-
RSEQ_INJECT_ASM(5)
346
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
347
-
RSEQ_INJECT_ASM(6)
348
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
349
-
: /* gcc asm goto does not allow outputs */
350
-
: [cpu_id] "r" (cpu),
351
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
352
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
353
-
[expect] "r" (expect),
354
-
[v] "m" (*v),
355
-
[newv] "r" (newv),
356
-
[v2] "m" (*v2),
357
-
[newv2] "r" (newv2)
358
-
RSEQ_INJECT_INPUT
359
-
: "memory", RSEQ_ASM_TMP_REG_1
360
-
RSEQ_INJECT_CLOBBER
361
-
: abort, cmpfail
362
-
#ifdef RSEQ_COMPARE_TWICE
363
-
, error1, error2
364
-
#endif
365
-
);
366
-
367
-
return 0;
368
-
abort:
369
-
RSEQ_INJECT_FAILED
370
-
return -1;
371
-
cmpfail:
372
-
return 1;
373
-
#ifdef RSEQ_COMPARE_TWICE
374
-
error1:
375
-
rseq_bug("cpu_id comparison failed");
376
-
error2:
377
-
rseq_bug("expected value comparison failed");
378
-
#endif
379
-
}
380
-
381
-
static inline __always_inline
382
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
383
-
intptr_t *v2, intptr_t newv2,
384
-
intptr_t newv, int cpu)
385
-
{
386
-
RSEQ_INJECT_C(9)
387
-
388
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
389
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
390
-
#ifdef RSEQ_COMPARE_TWICE
391
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
392
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
393
-
#endif
394
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
395
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
396
-
RSEQ_INJECT_ASM(3)
397
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
398
-
RSEQ_INJECT_ASM(4)
399
-
#ifdef RSEQ_COMPARE_TWICE
400
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
401
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
402
-
#endif
403
-
RSEQ_ASM_OP_STORE(newv2, v2)
404
-
RSEQ_INJECT_ASM(5)
405
-
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
406
-
RSEQ_INJECT_ASM(6)
407
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
408
-
: /* gcc asm goto does not allow outputs */
409
-
: [cpu_id] "r" (cpu),
410
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
411
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
412
-
[expect] "r" (expect),
413
-
[v] "m" (*v),
414
-
[newv] "r" (newv),
415
-
[v2] "m" (*v2),
416
-
[newv2] "r" (newv2)
417
-
RSEQ_INJECT_INPUT
418
-
: "memory", RSEQ_ASM_TMP_REG_1
419
-
RSEQ_INJECT_CLOBBER
420
-
: abort, cmpfail
421
-
#ifdef RSEQ_COMPARE_TWICE
422
-
, error1, error2
423
-
#endif
424
-
);
425
-
426
-
return 0;
427
-
abort:
428
-
RSEQ_INJECT_FAILED
429
-
return -1;
430
-
cmpfail:
431
-
return 1;
432
-
#ifdef RSEQ_COMPARE_TWICE
433
-
error1:
434
-
rseq_bug("cpu_id comparison failed");
435
-
error2:
436
-
rseq_bug("expected value comparison failed");
437
-
#endif
438
-
}
439
-
440
-
static inline __always_inline
441
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
442
-
intptr_t *v2, intptr_t expect2,
443
-
intptr_t newv, int cpu)
444
-
{
445
-
RSEQ_INJECT_C(9)
446
-
447
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
448
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
449
-
#ifdef RSEQ_COMPARE_TWICE
450
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
451
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
452
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error3]")
453
-
#endif
454
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
455
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
456
-
RSEQ_INJECT_ASM(3)
457
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
458
-
RSEQ_INJECT_ASM(4)
459
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[cmpfail]")
460
-
RSEQ_INJECT_ASM(5)
461
-
#ifdef RSEQ_COMPARE_TWICE
462
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
463
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
464
-
RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[error3]")
465
-
#endif
466
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
467
-
RSEQ_INJECT_ASM(6)
468
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
469
-
: /* gcc asm goto does not allow outputs */
470
-
: [cpu_id] "r" (cpu),
471
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
472
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
473
-
[v] "m" (*v),
474
-
[expect] "r" (expect),
475
-
[v2] "m" (*v2),
476
-
[expect2] "r" (expect2),
477
-
[newv] "r" (newv)
478
-
RSEQ_INJECT_INPUT
479
-
: "memory", RSEQ_ASM_TMP_REG_1
480
-
RSEQ_INJECT_CLOBBER
481
-
: abort, cmpfail
482
-
#ifdef RSEQ_COMPARE_TWICE
483
-
, error1, error2, error3
484
-
#endif
485
-
);
486
-
487
-
return 0;
488
-
abort:
489
-
RSEQ_INJECT_FAILED
490
-
return -1;
491
-
cmpfail:
492
-
return 1;
493
-
#ifdef RSEQ_COMPARE_TWICE
494
-
error1:
495
-
rseq_bug("cpu_id comparison failed");
496
-
error2:
497
-
rseq_bug("expected value comparison failed");
498
-
error3:
499
-
rseq_bug("2nd expected value comparison failed");
500
-
#endif
501
-
}
502
-
503
-
static inline __always_inline
504
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
505
-
void *dst, void *src, size_t len,
506
-
intptr_t newv, int cpu)
507
-
{
508
-
RSEQ_INJECT_C(9)
509
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
510
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
511
-
#ifdef RSEQ_COMPARE_TWICE
512
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
513
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
514
-
#endif
515
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
516
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
517
-
RSEQ_INJECT_ASM(3)
518
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
519
-
RSEQ_INJECT_ASM(4)
520
-
#ifdef RSEQ_COMPARE_TWICE
521
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
522
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
523
-
#endif
524
-
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
525
-
RSEQ_INJECT_ASM(5)
526
-
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)
527
-
RSEQ_INJECT_ASM(6)
528
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
529
-
: /* gcc asm goto does not allow outputs */
530
-
: [cpu_id] "r" (cpu),
531
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
532
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
533
-
[expect] "r" (expect),
534
-
[v] "m" (*v),
535
-
[newv] "r" (newv),
536
-
[dst] "r" (dst),
537
-
[src] "r" (src),
538
-
[len] "r" (len)
539
-
RSEQ_INJECT_INPUT
540
-
: "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2,
541
-
RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4
542
-
RSEQ_INJECT_CLOBBER
543
-
: abort, cmpfail
544
-
#ifdef RSEQ_COMPARE_TWICE
545
-
, error1, error2
546
-
#endif
547
-
);
548
-
549
-
return 0;
550
-
abort:
551
-
RSEQ_INJECT_FAILED
552
-
return -1;
553
-
cmpfail:
554
-
return 1;
555
-
#ifdef RSEQ_COMPARE_TWICE
556
-
error1:
557
-
rseq_bug("cpu_id comparison failed");
558
-
error2:
559
-
rseq_bug("expected value comparison failed");
560
-
#endif
561
-
}
562
-
563
-
static inline __always_inline
564
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
565
-
void *dst, void *src, size_t len,
566
-
intptr_t newv, int cpu)
567
-
{
568
-
RSEQ_INJECT_C(9)
569
-
570
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
571
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]")
572
-
#ifdef RSEQ_COMPARE_TWICE
573
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
574
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]")
575
-
#endif
576
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
577
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
578
-
RSEQ_INJECT_ASM(3)
579
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]")
580
-
RSEQ_INJECT_ASM(4)
581
-
#ifdef RSEQ_COMPARE_TWICE
582
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
583
-
RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]")
584
-
#endif
585
-
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)
586
-
RSEQ_INJECT_ASM(5)
587
-
RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)
588
-
RSEQ_INJECT_ASM(6)
589
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
590
-
: /* gcc asm goto does not allow outputs */
591
-
: [cpu_id] "r" (cpu),
592
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
593
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
594
-
[expect] "r" (expect),
595
-
[v] "m" (*v),
596
-
[newv] "r" (newv),
597
-
[dst] "r" (dst),
598
-
[src] "r" (src),
599
-
[len] "r" (len)
600
-
RSEQ_INJECT_INPUT
601
-
: "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2,
602
-
RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4
603
-
RSEQ_INJECT_CLOBBER
604
-
: abort, cmpfail
605
-
#ifdef RSEQ_COMPARE_TWICE
606
-
, error1, error2
607
-
#endif
608
-
);
609
-
610
-
return 0;
611
-
abort:
612
-
RSEQ_INJECT_FAILED
613
-
return -1;
614
-
cmpfail:
615
-
return 1;
616
-
#ifdef RSEQ_COMPARE_TWICE
617
-
error1:
618
-
rseq_bug("cpu_id comparison failed");
619
-
error2:
620
-
rseq_bug("expected value comparison failed");
621
-
#endif
622
-
}
623
-
624
-
#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
625
-
626
-
/*
627
-
* pval = *(ptr+off)
628
-
* *pval += inc;
629
-
*/
630
-
static inline __always_inline
631
-
int rseq_offset_deref_addv(intptr_t *ptr, off_t off, intptr_t inc, int cpu)
632
-
{
633
-
RSEQ_INJECT_C(9)
634
-
635
-
__asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)
636
-
#ifdef RSEQ_COMPARE_TWICE
637
-
RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]")
638
-
#endif
639
-
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)
640
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
641
-
RSEQ_INJECT_ASM(3)
642
-
#ifdef RSEQ_COMPARE_TWICE
643
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]")
644
-
#endif
645
-
RSEQ_ASM_OP_R_DEREF_ADDV(ptr, off, 3)
646
-
RSEQ_INJECT_ASM(4)
647
-
RSEQ_ASM_DEFINE_ABORT(4, abort)
648
-
: /* gcc asm goto does not allow outputs */
649
-
: [cpu_id] "r" (cpu),
650
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
651
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
652
-
[ptr] "r" (ptr),
653
-
[off] "er" (off),
654
-
[inc] "er" (inc)
655
-
RSEQ_INJECT_INPUT
656
-
: "memory", RSEQ_ASM_TMP_REG_1
657
-
RSEQ_INJECT_CLOBBER
658
-
: abort
659
-
#ifdef RSEQ_COMPARE_TWICE
660
-
, error1
661
-
#endif
662
-
);
663
-
return 0;
664
-
abort:
665
-
RSEQ_INJECT_FAILED
666
-
return -1;
667
-
#ifdef RSEQ_COMPARE_TWICE
668
-
error1:
669
-
rseq_bug("cpu_id comparison failed");
670
-
#endif
671
-
}
672
-
673
-
#endif /* !RSEQ_SKIP_FASTPATH */
198
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
199
+
#define RSEQ_TEMPLATE_MO_RELAXED
200
+
#include "rseq-riscv-bits.h"
201
+
#undef RSEQ_TEMPLATE_MO_RELAXED
202
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+474
tools/testing/selftests/rseq/rseq-s390-bits.h
+474
tools/testing/selftests/rseq/rseq-s390-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
3
+
#include "rseq-bits-template.h"
4
+
5
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
6
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
7
+
8
+
static inline __attribute__((always_inline))
9
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
10
+
{
11
+
RSEQ_INJECT_C(9)
12
+
13
+
__asm__ __volatile__ goto (
14
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
15
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
16
+
#ifdef RSEQ_COMPARE_TWICE
17
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
18
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
19
+
#endif
20
+
/* Start rseq by storing table entry pointer into rseq_cs. */
21
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
22
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
23
+
RSEQ_INJECT_ASM(3)
24
+
LONG_CMP " %[expect], %[v]\n\t"
25
+
"jnz %l[cmpfail]\n\t"
26
+
RSEQ_INJECT_ASM(4)
27
+
#ifdef RSEQ_COMPARE_TWICE
28
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
29
+
LONG_CMP " %[expect], %[v]\n\t"
30
+
"jnz %l[error2]\n\t"
31
+
#endif
32
+
/* final store */
33
+
LONG_S " %[newv], %[v]\n\t"
34
+
"2:\n\t"
35
+
RSEQ_INJECT_ASM(5)
36
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
37
+
: /* gcc asm goto does not allow outputs */
38
+
: [cpu_id] "r" (cpu),
39
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
40
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
41
+
[v] "m" (*v),
42
+
[expect] "r" (expect),
43
+
[newv] "r" (newv)
44
+
RSEQ_INJECT_INPUT
45
+
: "memory", "cc", "r0"
46
+
RSEQ_INJECT_CLOBBER
47
+
: abort, cmpfail
48
+
#ifdef RSEQ_COMPARE_TWICE
49
+
, error1, error2
50
+
#endif
51
+
);
52
+
rseq_after_asm_goto();
53
+
return 0;
54
+
abort:
55
+
rseq_after_asm_goto();
56
+
RSEQ_INJECT_FAILED
57
+
return -1;
58
+
cmpfail:
59
+
rseq_after_asm_goto();
60
+
return 1;
61
+
#ifdef RSEQ_COMPARE_TWICE
62
+
error1:
63
+
rseq_after_asm_goto();
64
+
rseq_bug("cpu_id comparison failed");
65
+
error2:
66
+
rseq_after_asm_goto();
67
+
rseq_bug("expected value comparison failed");
68
+
#endif
69
+
}
70
+
71
+
/*
72
+
* Compare @v against @expectnot. When it does _not_ match, load @v
73
+
* into @load, and store the content of *@v + voffp into @v.
74
+
*/
75
+
static inline __attribute__((always_inline))
76
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
77
+
long voffp, intptr_t *load, int cpu)
78
+
{
79
+
RSEQ_INJECT_C(9)
80
+
81
+
__asm__ __volatile__ goto (
82
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
83
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
84
+
#ifdef RSEQ_COMPARE_TWICE
85
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
86
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
87
+
#endif
88
+
/* Start rseq by storing table entry pointer into rseq_cs. */
89
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
90
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
91
+
RSEQ_INJECT_ASM(3)
92
+
LONG_L " %%r1, %[v]\n\t"
93
+
LONG_CMP_R " %%r1, %[expectnot]\n\t"
94
+
"je %l[cmpfail]\n\t"
95
+
RSEQ_INJECT_ASM(4)
96
+
#ifdef RSEQ_COMPARE_TWICE
97
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
98
+
LONG_L " %%r1, %[v]\n\t"
99
+
LONG_CMP_R " %%r1, %[expectnot]\n\t"
100
+
"je %l[error2]\n\t"
101
+
#endif
102
+
LONG_S " %%r1, %[load]\n\t"
103
+
LONG_ADD_R " %%r1, %[voffp]\n\t"
104
+
LONG_L " %%r1, 0(%%r1)\n\t"
105
+
/* final store */
106
+
LONG_S " %%r1, %[v]\n\t"
107
+
"2:\n\t"
108
+
RSEQ_INJECT_ASM(5)
109
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
110
+
: /* gcc asm goto does not allow outputs */
111
+
: [cpu_id] "r" (cpu),
112
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
113
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
114
+
/* final store input */
115
+
[v] "m" (*v),
116
+
[expectnot] "r" (expectnot),
117
+
[voffp] "r" (voffp),
118
+
[load] "m" (*load)
119
+
RSEQ_INJECT_INPUT
120
+
: "memory", "cc", "r0", "r1"
121
+
RSEQ_INJECT_CLOBBER
122
+
: abort, cmpfail
123
+
#ifdef RSEQ_COMPARE_TWICE
124
+
, error1, error2
125
+
#endif
126
+
);
127
+
rseq_after_asm_goto();
128
+
return 0;
129
+
abort:
130
+
rseq_after_asm_goto();
131
+
RSEQ_INJECT_FAILED
132
+
return -1;
133
+
cmpfail:
134
+
rseq_after_asm_goto();
135
+
return 1;
136
+
#ifdef RSEQ_COMPARE_TWICE
137
+
error1:
138
+
rseq_after_asm_goto();
139
+
rseq_bug("cpu_id comparison failed");
140
+
error2:
141
+
rseq_after_asm_goto();
142
+
rseq_bug("expected value comparison failed");
143
+
#endif
144
+
}
145
+
146
+
static inline __attribute__((always_inline))
147
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
148
+
{
149
+
RSEQ_INJECT_C(9)
150
+
151
+
__asm__ __volatile__ goto (
152
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
153
+
#ifdef RSEQ_COMPARE_TWICE
154
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
155
+
#endif
156
+
/* Start rseq by storing table entry pointer into rseq_cs. */
157
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
158
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
159
+
RSEQ_INJECT_ASM(3)
160
+
#ifdef RSEQ_COMPARE_TWICE
161
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
162
+
#endif
163
+
LONG_L " %%r0, %[v]\n\t"
164
+
LONG_ADD_R " %%r0, %[count]\n\t"
165
+
/* final store */
166
+
LONG_S " %%r0, %[v]\n\t"
167
+
"2:\n\t"
168
+
RSEQ_INJECT_ASM(4)
169
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
170
+
: /* gcc asm goto does not allow outputs */
171
+
: [cpu_id] "r" (cpu),
172
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
173
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
174
+
/* final store input */
175
+
[v] "m" (*v),
176
+
[count] "r" (count)
177
+
RSEQ_INJECT_INPUT
178
+
: "memory", "cc", "r0"
179
+
RSEQ_INJECT_CLOBBER
180
+
: abort
181
+
#ifdef RSEQ_COMPARE_TWICE
182
+
, error1
183
+
#endif
184
+
);
185
+
rseq_after_asm_goto();
186
+
return 0;
187
+
abort:
188
+
rseq_after_asm_goto();
189
+
RSEQ_INJECT_FAILED
190
+
return -1;
191
+
#ifdef RSEQ_COMPARE_TWICE
192
+
error1:
193
+
rseq_after_asm_goto();
194
+
rseq_bug("cpu_id comparison failed");
195
+
#endif
196
+
}
197
+
198
+
static inline __attribute__((always_inline))
199
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
200
+
intptr_t *v2, intptr_t expect2,
201
+
intptr_t newv, int cpu)
202
+
{
203
+
RSEQ_INJECT_C(9)
204
+
205
+
__asm__ __volatile__ goto (
206
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
207
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
208
+
#ifdef RSEQ_COMPARE_TWICE
209
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
210
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
211
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
212
+
#endif
213
+
/* Start rseq by storing table entry pointer into rseq_cs. */
214
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
215
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
216
+
RSEQ_INJECT_ASM(3)
217
+
LONG_CMP " %[expect], %[v]\n\t"
218
+
"jnz %l[cmpfail]\n\t"
219
+
RSEQ_INJECT_ASM(4)
220
+
LONG_CMP " %[expect2], %[v2]\n\t"
221
+
"jnz %l[cmpfail]\n\t"
222
+
RSEQ_INJECT_ASM(5)
223
+
#ifdef RSEQ_COMPARE_TWICE
224
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
225
+
LONG_CMP " %[expect], %[v]\n\t"
226
+
"jnz %l[error2]\n\t"
227
+
LONG_CMP " %[expect2], %[v2]\n\t"
228
+
"jnz %l[error3]\n\t"
229
+
#endif
230
+
/* final store */
231
+
LONG_S " %[newv], %[v]\n\t"
232
+
"2:\n\t"
233
+
RSEQ_INJECT_ASM(6)
234
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
235
+
: /* gcc asm goto does not allow outputs */
236
+
: [cpu_id] "r" (cpu),
237
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
238
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
239
+
/* cmp2 input */
240
+
[v2] "m" (*v2),
241
+
[expect2] "r" (expect2),
242
+
/* final store input */
243
+
[v] "m" (*v),
244
+
[expect] "r" (expect),
245
+
[newv] "r" (newv)
246
+
RSEQ_INJECT_INPUT
247
+
: "memory", "cc", "r0"
248
+
RSEQ_INJECT_CLOBBER
249
+
: abort, cmpfail
250
+
#ifdef RSEQ_COMPARE_TWICE
251
+
, error1, error2, error3
252
+
#endif
253
+
);
254
+
rseq_after_asm_goto();
255
+
return 0;
256
+
abort:
257
+
rseq_after_asm_goto();
258
+
RSEQ_INJECT_FAILED
259
+
return -1;
260
+
cmpfail:
261
+
rseq_after_asm_goto();
262
+
return 1;
263
+
#ifdef RSEQ_COMPARE_TWICE
264
+
error1:
265
+
rseq_after_asm_goto();
266
+
rseq_bug("cpu_id comparison failed");
267
+
error2:
268
+
rseq_after_asm_goto();
269
+
rseq_bug("1st expected value comparison failed");
270
+
error3:
271
+
rseq_after_asm_goto();
272
+
rseq_bug("2nd expected value comparison failed");
273
+
#endif
274
+
}
275
+
276
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
277
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
278
+
279
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
280
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
281
+
282
+
/* s390 is TSO. */
283
+
static inline __attribute__((always_inline))
284
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
285
+
intptr_t *v2, intptr_t newv2,
286
+
intptr_t newv, int cpu)
287
+
{
288
+
RSEQ_INJECT_C(9)
289
+
290
+
__asm__ __volatile__ goto (
291
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
292
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
293
+
#ifdef RSEQ_COMPARE_TWICE
294
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
295
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
296
+
#endif
297
+
/* Start rseq by storing table entry pointer into rseq_cs. */
298
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
299
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
300
+
RSEQ_INJECT_ASM(3)
301
+
LONG_CMP " %[expect], %[v]\n\t"
302
+
"jnz %l[cmpfail]\n\t"
303
+
RSEQ_INJECT_ASM(4)
304
+
#ifdef RSEQ_COMPARE_TWICE
305
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
306
+
LONG_CMP " %[expect], %[v]\n\t"
307
+
"jnz %l[error2]\n\t"
308
+
#endif
309
+
/* try store */
310
+
LONG_S " %[newv2], %[v2]\n\t"
311
+
RSEQ_INJECT_ASM(5)
312
+
/* final store */
313
+
LONG_S " %[newv], %[v]\n\t"
314
+
"2:\n\t"
315
+
RSEQ_INJECT_ASM(6)
316
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
317
+
: /* gcc asm goto does not allow outputs */
318
+
: [cpu_id] "r" (cpu),
319
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
320
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
321
+
/* try store input */
322
+
[v2] "m" (*v2),
323
+
[newv2] "r" (newv2),
324
+
/* final store input */
325
+
[v] "m" (*v),
326
+
[expect] "r" (expect),
327
+
[newv] "r" (newv)
328
+
RSEQ_INJECT_INPUT
329
+
: "memory", "cc", "r0"
330
+
RSEQ_INJECT_CLOBBER
331
+
: abort, cmpfail
332
+
#ifdef RSEQ_COMPARE_TWICE
333
+
, error1, error2
334
+
#endif
335
+
);
336
+
rseq_after_asm_goto();
337
+
return 0;
338
+
abort:
339
+
rseq_after_asm_goto();
340
+
RSEQ_INJECT_FAILED
341
+
return -1;
342
+
cmpfail:
343
+
rseq_after_asm_goto();
344
+
return 1;
345
+
#ifdef RSEQ_COMPARE_TWICE
346
+
error1:
347
+
rseq_after_asm_goto();
348
+
rseq_bug("cpu_id comparison failed");
349
+
error2:
350
+
rseq_after_asm_goto();
351
+
rseq_bug("expected value comparison failed");
352
+
#endif
353
+
}
354
+
355
+
/* s390 is TSO. */
356
+
static inline __attribute__((always_inline))
357
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
358
+
void *dst, void *src, size_t len,
359
+
intptr_t newv, int cpu)
360
+
{
361
+
uint64_t rseq_scratch[3];
362
+
363
+
RSEQ_INJECT_C(9)
364
+
365
+
__asm__ __volatile__ goto (
366
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
367
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
368
+
#ifdef RSEQ_COMPARE_TWICE
369
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
370
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
371
+
#endif
372
+
LONG_S " %[src], %[rseq_scratch0]\n\t"
373
+
LONG_S " %[dst], %[rseq_scratch1]\n\t"
374
+
LONG_S " %[len], %[rseq_scratch2]\n\t"
375
+
/* Start rseq by storing table entry pointer into rseq_cs. */
376
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
377
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
378
+
RSEQ_INJECT_ASM(3)
379
+
LONG_CMP " %[expect], %[v]\n\t"
380
+
"jnz 5f\n\t"
381
+
RSEQ_INJECT_ASM(4)
382
+
#ifdef RSEQ_COMPARE_TWICE
383
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
384
+
LONG_CMP " %[expect], %[v]\n\t"
385
+
"jnz 7f\n\t"
386
+
#endif
387
+
/* try memcpy */
388
+
LONG_LT_R " %[len], %[len]\n\t"
389
+
"jz 333f\n\t"
390
+
"222:\n\t"
391
+
"ic %%r0,0(%[src])\n\t"
392
+
"stc %%r0,0(%[dst])\n\t"
393
+
LONG_ADDI " %[src], 1\n\t"
394
+
LONG_ADDI " %[dst], 1\n\t"
395
+
LONG_ADDI " %[len], -1\n\t"
396
+
"jnz 222b\n\t"
397
+
"333:\n\t"
398
+
RSEQ_INJECT_ASM(5)
399
+
/* final store */
400
+
LONG_S " %[newv], %[v]\n\t"
401
+
"2:\n\t"
402
+
RSEQ_INJECT_ASM(6)
403
+
/* teardown */
404
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
405
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
406
+
LONG_L " %[src], %[rseq_scratch0]\n\t"
407
+
RSEQ_ASM_DEFINE_ABORT(4,
408
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
409
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
410
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
411
+
abort)
412
+
RSEQ_ASM_DEFINE_CMPFAIL(5,
413
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
414
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
415
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
416
+
cmpfail)
417
+
#ifdef RSEQ_COMPARE_TWICE
418
+
RSEQ_ASM_DEFINE_CMPFAIL(6,
419
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
420
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
421
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
422
+
error1)
423
+
RSEQ_ASM_DEFINE_CMPFAIL(7,
424
+
LONG_L " %[len], %[rseq_scratch2]\n\t"
425
+
LONG_L " %[dst], %[rseq_scratch1]\n\t"
426
+
LONG_L " %[src], %[rseq_scratch0]\n\t",
427
+
error2)
428
+
#endif
429
+
: /* gcc asm goto does not allow outputs */
430
+
: [cpu_id] "r" (cpu),
431
+
[current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD),
432
+
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
433
+
/* final store input */
434
+
[v] "m" (*v),
435
+
[expect] "r" (expect),
436
+
[newv] "r" (newv),
437
+
/* try memcpy input */
438
+
[dst] "r" (dst),
439
+
[src] "r" (src),
440
+
[len] "r" (len),
441
+
[rseq_scratch0] "m" (rseq_scratch[0]),
442
+
[rseq_scratch1] "m" (rseq_scratch[1]),
443
+
[rseq_scratch2] "m" (rseq_scratch[2])
444
+
RSEQ_INJECT_INPUT
445
+
: "memory", "cc", "r0"
446
+
RSEQ_INJECT_CLOBBER
447
+
: abort, cmpfail
448
+
#ifdef RSEQ_COMPARE_TWICE
449
+
, error1, error2
450
+
#endif
451
+
);
452
+
rseq_after_asm_goto();
453
+
return 0;
454
+
abort:
455
+
rseq_after_asm_goto();
456
+
RSEQ_INJECT_FAILED
457
+
return -1;
458
+
cmpfail:
459
+
rseq_after_asm_goto();
460
+
return 1;
461
+
#ifdef RSEQ_COMPARE_TWICE
462
+
error1:
463
+
rseq_after_asm_goto();
464
+
rseq_bug("cpu_id comparison failed");
465
+
error2:
466
+
rseq_after_asm_goto();
467
+
rseq_bug("expected value comparison failed");
468
+
#endif
469
+
}
470
+
471
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
472
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
473
+
474
+
#include "rseq-bits-reset.h"
+24
-471
tools/testing/selftests/rseq/rseq-s390.h
+24
-471
tools/testing/selftests/rseq/rseq-s390.h
···
28
28
RSEQ_WRITE_ONCE(*p, v); \
29
29
} while (0)
30
30
31
-
#ifdef RSEQ_SKIP_FASTPATH
32
-
#include "rseq-skip.h"
33
-
#else /* !RSEQ_SKIP_FASTPATH */
34
-
35
31
#ifdef __s390x__
36
32
37
33
#define LONG_L "lg"
···
130
134
"jg %l[" __rseq_str(cmpfail_label) "]\n\t" \
131
135
".popsection\n\t"
132
136
133
-
static inline __attribute__((always_inline))
134
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
135
-
{
136
-
RSEQ_INJECT_C(9)
137
+
/* Per-cpu-id indexing. */
137
138
138
-
__asm__ __volatile__ goto (
139
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
140
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
141
-
#ifdef RSEQ_COMPARE_TWICE
142
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
143
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
144
-
#endif
145
-
/* Start rseq by storing table entry pointer into rseq_cs. */
146
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
147
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
148
-
RSEQ_INJECT_ASM(3)
149
-
LONG_CMP " %[expect], %[v]\n\t"
150
-
"jnz %l[cmpfail]\n\t"
151
-
RSEQ_INJECT_ASM(4)
152
-
#ifdef RSEQ_COMPARE_TWICE
153
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
154
-
LONG_CMP " %[expect], %[v]\n\t"
155
-
"jnz %l[error2]\n\t"
156
-
#endif
157
-
/* final store */
158
-
LONG_S " %[newv], %[v]\n\t"
159
-
"2:\n\t"
160
-
RSEQ_INJECT_ASM(5)
161
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
162
-
: /* gcc asm goto does not allow outputs */
163
-
: [cpu_id] "r" (cpu),
164
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
165
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
166
-
[v] "m" (*v),
167
-
[expect] "r" (expect),
168
-
[newv] "r" (newv)
169
-
RSEQ_INJECT_INPUT
170
-
: "memory", "cc", "r0"
171
-
RSEQ_INJECT_CLOBBER
172
-
: abort, cmpfail
173
-
#ifdef RSEQ_COMPARE_TWICE
174
-
, error1, error2
175
-
#endif
176
-
);
177
-
rseq_after_asm_goto();
178
-
return 0;
179
-
abort:
180
-
rseq_after_asm_goto();
181
-
RSEQ_INJECT_FAILED
182
-
return -1;
183
-
cmpfail:
184
-
rseq_after_asm_goto();
185
-
return 1;
186
-
#ifdef RSEQ_COMPARE_TWICE
187
-
error1:
188
-
rseq_after_asm_goto();
189
-
rseq_bug("cpu_id comparison failed");
190
-
error2:
191
-
rseq_after_asm_goto();
192
-
rseq_bug("expected value comparison failed");
193
-
#endif
194
-
}
139
+
#define RSEQ_TEMPLATE_CPU_ID
140
+
#define RSEQ_TEMPLATE_MO_RELAXED
141
+
#include "rseq-s390-bits.h"
142
+
#undef RSEQ_TEMPLATE_MO_RELAXED
195
143
196
-
/*
197
-
* Compare @v against @expectnot. When it does _not_ match, load @v
198
-
* into @load, and store the content of *@v + voffp into @v.
199
-
*/
200
-
static inline __attribute__((always_inline))
201
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
202
-
long voffp, intptr_t *load, int cpu)
203
-
{
204
-
RSEQ_INJECT_C(9)
144
+
#define RSEQ_TEMPLATE_MO_RELEASE
145
+
#include "rseq-s390-bits.h"
146
+
#undef RSEQ_TEMPLATE_MO_RELEASE
147
+
#undef RSEQ_TEMPLATE_CPU_ID
205
148
206
-
__asm__ __volatile__ goto (
207
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
208
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
209
-
#ifdef RSEQ_COMPARE_TWICE
210
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
211
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
212
-
#endif
213
-
/* Start rseq by storing table entry pointer into rseq_cs. */
214
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
215
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
216
-
RSEQ_INJECT_ASM(3)
217
-
LONG_L " %%r1, %[v]\n\t"
218
-
LONG_CMP_R " %%r1, %[expectnot]\n\t"
219
-
"je %l[cmpfail]\n\t"
220
-
RSEQ_INJECT_ASM(4)
221
-
#ifdef RSEQ_COMPARE_TWICE
222
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
223
-
LONG_L " %%r1, %[v]\n\t"
224
-
LONG_CMP_R " %%r1, %[expectnot]\n\t"
225
-
"je %l[error2]\n\t"
226
-
#endif
227
-
LONG_S " %%r1, %[load]\n\t"
228
-
LONG_ADD_R " %%r1, %[voffp]\n\t"
229
-
LONG_L " %%r1, 0(%%r1)\n\t"
230
-
/* final store */
231
-
LONG_S " %%r1, %[v]\n\t"
232
-
"2:\n\t"
233
-
RSEQ_INJECT_ASM(5)
234
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
235
-
: /* gcc asm goto does not allow outputs */
236
-
: [cpu_id] "r" (cpu),
237
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
238
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
239
-
/* final store input */
240
-
[v] "m" (*v),
241
-
[expectnot] "r" (expectnot),
242
-
[voffp] "r" (voffp),
243
-
[load] "m" (*load)
244
-
RSEQ_INJECT_INPUT
245
-
: "memory", "cc", "r0", "r1"
246
-
RSEQ_INJECT_CLOBBER
247
-
: abort, cmpfail
248
-
#ifdef RSEQ_COMPARE_TWICE
249
-
, error1, error2
250
-
#endif
251
-
);
252
-
rseq_after_asm_goto();
253
-
return 0;
254
-
abort:
255
-
rseq_after_asm_goto();
256
-
RSEQ_INJECT_FAILED
257
-
return -1;
258
-
cmpfail:
259
-
rseq_after_asm_goto();
260
-
return 1;
261
-
#ifdef RSEQ_COMPARE_TWICE
262
-
error1:
263
-
rseq_after_asm_goto();
264
-
rseq_bug("cpu_id comparison failed");
265
-
error2:
266
-
rseq_after_asm_goto();
267
-
rseq_bug("expected value comparison failed");
268
-
#endif
269
-
}
149
+
/* Per-mm-cid indexing. */
270
150
271
-
static inline __attribute__((always_inline))
272
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
273
-
{
274
-
RSEQ_INJECT_C(9)
151
+
#define RSEQ_TEMPLATE_MM_CID
152
+
#define RSEQ_TEMPLATE_MO_RELAXED
153
+
#include "rseq-s390-bits.h"
154
+
#undef RSEQ_TEMPLATE_MO_RELAXED
275
155
276
-
__asm__ __volatile__ goto (
277
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
278
-
#ifdef RSEQ_COMPARE_TWICE
279
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
280
-
#endif
281
-
/* Start rseq by storing table entry pointer into rseq_cs. */
282
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
283
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
284
-
RSEQ_INJECT_ASM(3)
285
-
#ifdef RSEQ_COMPARE_TWICE
286
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
287
-
#endif
288
-
LONG_L " %%r0, %[v]\n\t"
289
-
LONG_ADD_R " %%r0, %[count]\n\t"
290
-
/* final store */
291
-
LONG_S " %%r0, %[v]\n\t"
292
-
"2:\n\t"
293
-
RSEQ_INJECT_ASM(4)
294
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
295
-
: /* gcc asm goto does not allow outputs */
296
-
: [cpu_id] "r" (cpu),
297
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
298
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
299
-
/* final store input */
300
-
[v] "m" (*v),
301
-
[count] "r" (count)
302
-
RSEQ_INJECT_INPUT
303
-
: "memory", "cc", "r0"
304
-
RSEQ_INJECT_CLOBBER
305
-
: abort
306
-
#ifdef RSEQ_COMPARE_TWICE
307
-
, error1
308
-
#endif
309
-
);
310
-
rseq_after_asm_goto();
311
-
return 0;
312
-
abort:
313
-
rseq_after_asm_goto();
314
-
RSEQ_INJECT_FAILED
315
-
return -1;
316
-
#ifdef RSEQ_COMPARE_TWICE
317
-
error1:
318
-
rseq_after_asm_goto();
319
-
rseq_bug("cpu_id comparison failed");
320
-
#endif
321
-
}
156
+
#define RSEQ_TEMPLATE_MO_RELEASE
157
+
#include "rseq-s390-bits.h"
158
+
#undef RSEQ_TEMPLATE_MO_RELEASE
159
+
#undef RSEQ_TEMPLATE_MM_CID
322
160
323
-
static inline __attribute__((always_inline))
324
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
325
-
intptr_t *v2, intptr_t newv2,
326
-
intptr_t newv, int cpu)
327
-
{
328
-
RSEQ_INJECT_C(9)
161
+
/* APIs which are not based on cpu ids. */
329
162
330
-
__asm__ __volatile__ goto (
331
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
332
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
333
-
#ifdef RSEQ_COMPARE_TWICE
334
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
335
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
336
-
#endif
337
-
/* Start rseq by storing table entry pointer into rseq_cs. */
338
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
339
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
340
-
RSEQ_INJECT_ASM(3)
341
-
LONG_CMP " %[expect], %[v]\n\t"
342
-
"jnz %l[cmpfail]\n\t"
343
-
RSEQ_INJECT_ASM(4)
344
-
#ifdef RSEQ_COMPARE_TWICE
345
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
346
-
LONG_CMP " %[expect], %[v]\n\t"
347
-
"jnz %l[error2]\n\t"
348
-
#endif
349
-
/* try store */
350
-
LONG_S " %[newv2], %[v2]\n\t"
351
-
RSEQ_INJECT_ASM(5)
352
-
/* final store */
353
-
LONG_S " %[newv], %[v]\n\t"
354
-
"2:\n\t"
355
-
RSEQ_INJECT_ASM(6)
356
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
357
-
: /* gcc asm goto does not allow outputs */
358
-
: [cpu_id] "r" (cpu),
359
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
360
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
361
-
/* try store input */
362
-
[v2] "m" (*v2),
363
-
[newv2] "r" (newv2),
364
-
/* final store input */
365
-
[v] "m" (*v),
366
-
[expect] "r" (expect),
367
-
[newv] "r" (newv)
368
-
RSEQ_INJECT_INPUT
369
-
: "memory", "cc", "r0"
370
-
RSEQ_INJECT_CLOBBER
371
-
: abort, cmpfail
372
-
#ifdef RSEQ_COMPARE_TWICE
373
-
, error1, error2
374
-
#endif
375
-
);
376
-
rseq_after_asm_goto();
377
-
return 0;
378
-
abort:
379
-
rseq_after_asm_goto();
380
-
RSEQ_INJECT_FAILED
381
-
return -1;
382
-
cmpfail:
383
-
rseq_after_asm_goto();
384
-
return 1;
385
-
#ifdef RSEQ_COMPARE_TWICE
386
-
error1:
387
-
rseq_after_asm_goto();
388
-
rseq_bug("cpu_id comparison failed");
389
-
error2:
390
-
rseq_after_asm_goto();
391
-
rseq_bug("expected value comparison failed");
392
-
#endif
393
-
}
394
-
395
-
/* s390 is TSO. */
396
-
static inline __attribute__((always_inline))
397
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
398
-
intptr_t *v2, intptr_t newv2,
399
-
intptr_t newv, int cpu)
400
-
{
401
-
return rseq_cmpeqv_trystorev_storev(v, expect, v2, newv2, newv, cpu);
402
-
}
403
-
404
-
static inline __attribute__((always_inline))
405
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
406
-
intptr_t *v2, intptr_t expect2,
407
-
intptr_t newv, int cpu)
408
-
{
409
-
RSEQ_INJECT_C(9)
410
-
411
-
__asm__ __volatile__ goto (
412
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
413
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
414
-
#ifdef RSEQ_COMPARE_TWICE
415
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
416
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
417
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
418
-
#endif
419
-
/* Start rseq by storing table entry pointer into rseq_cs. */
420
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
421
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
422
-
RSEQ_INJECT_ASM(3)
423
-
LONG_CMP " %[expect], %[v]\n\t"
424
-
"jnz %l[cmpfail]\n\t"
425
-
RSEQ_INJECT_ASM(4)
426
-
LONG_CMP " %[expect2], %[v2]\n\t"
427
-
"jnz %l[cmpfail]\n\t"
428
-
RSEQ_INJECT_ASM(5)
429
-
#ifdef RSEQ_COMPARE_TWICE
430
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
431
-
LONG_CMP " %[expect], %[v]\n\t"
432
-
"jnz %l[error2]\n\t"
433
-
LONG_CMP " %[expect2], %[v2]\n\t"
434
-
"jnz %l[error3]\n\t"
435
-
#endif
436
-
/* final store */
437
-
LONG_S " %[newv], %[v]\n\t"
438
-
"2:\n\t"
439
-
RSEQ_INJECT_ASM(6)
440
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
441
-
: /* gcc asm goto does not allow outputs */
442
-
: [cpu_id] "r" (cpu),
443
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
444
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
445
-
/* cmp2 input */
446
-
[v2] "m" (*v2),
447
-
[expect2] "r" (expect2),
448
-
/* final store input */
449
-
[v] "m" (*v),
450
-
[expect] "r" (expect),
451
-
[newv] "r" (newv)
452
-
RSEQ_INJECT_INPUT
453
-
: "memory", "cc", "r0"
454
-
RSEQ_INJECT_CLOBBER
455
-
: abort, cmpfail
456
-
#ifdef RSEQ_COMPARE_TWICE
457
-
, error1, error2, error3
458
-
#endif
459
-
);
460
-
rseq_after_asm_goto();
461
-
return 0;
462
-
abort:
463
-
rseq_after_asm_goto();
464
-
RSEQ_INJECT_FAILED
465
-
return -1;
466
-
cmpfail:
467
-
rseq_after_asm_goto();
468
-
return 1;
469
-
#ifdef RSEQ_COMPARE_TWICE
470
-
error1:
471
-
rseq_after_asm_goto();
472
-
rseq_bug("cpu_id comparison failed");
473
-
error2:
474
-
rseq_after_asm_goto();
475
-
rseq_bug("1st expected value comparison failed");
476
-
error3:
477
-
rseq_after_asm_goto();
478
-
rseq_bug("2nd expected value comparison failed");
479
-
#endif
480
-
}
481
-
482
-
static inline __attribute__((always_inline))
483
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
484
-
void *dst, void *src, size_t len,
485
-
intptr_t newv, int cpu)
486
-
{
487
-
uint64_t rseq_scratch[3];
488
-
489
-
RSEQ_INJECT_C(9)
490
-
491
-
__asm__ __volatile__ goto (
492
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
493
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
494
-
#ifdef RSEQ_COMPARE_TWICE
495
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
496
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
497
-
#endif
498
-
LONG_S " %[src], %[rseq_scratch0]\n\t"
499
-
LONG_S " %[dst], %[rseq_scratch1]\n\t"
500
-
LONG_S " %[len], %[rseq_scratch2]\n\t"
501
-
/* Start rseq by storing table entry pointer into rseq_cs. */
502
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs)
503
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
504
-
RSEQ_INJECT_ASM(3)
505
-
LONG_CMP " %[expect], %[v]\n\t"
506
-
"jnz 5f\n\t"
507
-
RSEQ_INJECT_ASM(4)
508
-
#ifdef RSEQ_COMPARE_TWICE
509
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
510
-
LONG_CMP " %[expect], %[v]\n\t"
511
-
"jnz 7f\n\t"
512
-
#endif
513
-
/* try memcpy */
514
-
LONG_LT_R " %[len], %[len]\n\t"
515
-
"jz 333f\n\t"
516
-
"222:\n\t"
517
-
"ic %%r0,0(%[src])\n\t"
518
-
"stc %%r0,0(%[dst])\n\t"
519
-
LONG_ADDI " %[src], 1\n\t"
520
-
LONG_ADDI " %[dst], 1\n\t"
521
-
LONG_ADDI " %[len], -1\n\t"
522
-
"jnz 222b\n\t"
523
-
"333:\n\t"
524
-
RSEQ_INJECT_ASM(5)
525
-
/* final store */
526
-
LONG_S " %[newv], %[v]\n\t"
527
-
"2:\n\t"
528
-
RSEQ_INJECT_ASM(6)
529
-
/* teardown */
530
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
531
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
532
-
LONG_L " %[src], %[rseq_scratch0]\n\t"
533
-
RSEQ_ASM_DEFINE_ABORT(4,
534
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
535
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
536
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
537
-
abort)
538
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
539
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
540
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
541
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
542
-
cmpfail)
543
-
#ifdef RSEQ_COMPARE_TWICE
544
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
545
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
546
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
547
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
548
-
error1)
549
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
550
-
LONG_L " %[len], %[rseq_scratch2]\n\t"
551
-
LONG_L " %[dst], %[rseq_scratch1]\n\t"
552
-
LONG_L " %[src], %[rseq_scratch0]\n\t",
553
-
error2)
554
-
#endif
555
-
: /* gcc asm goto does not allow outputs */
556
-
: [cpu_id] "r" (cpu),
557
-
[current_cpu_id] "m" (rseq_get_abi()->cpu_id),
558
-
[rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr),
559
-
/* final store input */
560
-
[v] "m" (*v),
561
-
[expect] "r" (expect),
562
-
[newv] "r" (newv),
563
-
/* try memcpy input */
564
-
[dst] "r" (dst),
565
-
[src] "r" (src),
566
-
[len] "r" (len),
567
-
[rseq_scratch0] "m" (rseq_scratch[0]),
568
-
[rseq_scratch1] "m" (rseq_scratch[1]),
569
-
[rseq_scratch2] "m" (rseq_scratch[2])
570
-
RSEQ_INJECT_INPUT
571
-
: "memory", "cc", "r0"
572
-
RSEQ_INJECT_CLOBBER
573
-
: abort, cmpfail
574
-
#ifdef RSEQ_COMPARE_TWICE
575
-
, error1, error2
576
-
#endif
577
-
);
578
-
rseq_after_asm_goto();
579
-
return 0;
580
-
abort:
581
-
rseq_after_asm_goto();
582
-
RSEQ_INJECT_FAILED
583
-
return -1;
584
-
cmpfail:
585
-
rseq_after_asm_goto();
586
-
return 1;
587
-
#ifdef RSEQ_COMPARE_TWICE
588
-
error1:
589
-
rseq_after_asm_goto();
590
-
rseq_bug("cpu_id comparison failed");
591
-
error2:
592
-
rseq_after_asm_goto();
593
-
rseq_bug("expected value comparison failed");
594
-
#endif
595
-
}
596
-
597
-
/* s390 is TSO. */
598
-
static inline __attribute__((always_inline))
599
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
600
-
void *dst, void *src, size_t len,
601
-
intptr_t newv, int cpu)
602
-
{
603
-
return rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len,
604
-
newv, cpu);
605
-
}
606
-
#endif /* !RSEQ_SKIP_FASTPATH */
163
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
164
+
#define RSEQ_TEMPLATE_MO_RELAXED
165
+
#include "rseq-s390-bits.h"
166
+
#undef RSEQ_TEMPLATE_MO_RELAXED
167
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
-65
tools/testing/selftests/rseq/rseq-skip.h
-65
tools/testing/selftests/rseq/rseq-skip.h
···
1
-
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
-
/*
3
-
* rseq-skip.h
4
-
*
5
-
* (C) Copyright 2017-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
-
*/
7
-
8
-
static inline __attribute__((always_inline))
9
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
10
-
{
11
-
return -1;
12
-
}
13
-
14
-
static inline __attribute__((always_inline))
15
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
16
-
long voffp, intptr_t *load, int cpu)
17
-
{
18
-
return -1;
19
-
}
20
-
21
-
static inline __attribute__((always_inline))
22
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
23
-
{
24
-
return -1;
25
-
}
26
-
27
-
static inline __attribute__((always_inline))
28
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
29
-
intptr_t *v2, intptr_t newv2,
30
-
intptr_t newv, int cpu)
31
-
{
32
-
return -1;
33
-
}
34
-
35
-
static inline __attribute__((always_inline))
36
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
37
-
intptr_t *v2, intptr_t newv2,
38
-
intptr_t newv, int cpu)
39
-
{
40
-
return -1;
41
-
}
42
-
43
-
static inline __attribute__((always_inline))
44
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
45
-
intptr_t *v2, intptr_t expect2,
46
-
intptr_t newv, int cpu)
47
-
{
48
-
return -1;
49
-
}
50
-
51
-
static inline __attribute__((always_inline))
52
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
53
-
void *dst, void *src, size_t len,
54
-
intptr_t newv, int cpu)
55
-
{
56
-
return -1;
57
-
}
58
-
59
-
static inline __attribute__((always_inline))
60
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
61
-
void *dst, void *src, size_t len,
62
-
intptr_t newv, int cpu)
63
-
{
64
-
return -1;
65
-
}
+993
tools/testing/selftests/rseq/rseq-x86-bits.h
+993
tools/testing/selftests/rseq/rseq-x86-bits.h
···
1
+
/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
2
+
/*
3
+
* rseq-x86-bits.h
4
+
*
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
+
*/
7
+
8
+
#include "rseq-bits-template.h"
9
+
10
+
#ifdef __x86_64__
11
+
12
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
13
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
14
+
15
+
static inline __attribute__((always_inline))
16
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
17
+
{
18
+
RSEQ_INJECT_C(9)
19
+
20
+
__asm__ __volatile__ goto (
21
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
22
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
23
+
#ifdef RSEQ_COMPARE_TWICE
24
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
25
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
26
+
#endif
27
+
/* Start rseq by storing table entry pointer into rseq_cs. */
28
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
29
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
30
+
RSEQ_INJECT_ASM(3)
31
+
"cmpq %[v], %[expect]\n\t"
32
+
"jnz %l[cmpfail]\n\t"
33
+
RSEQ_INJECT_ASM(4)
34
+
#ifdef RSEQ_COMPARE_TWICE
35
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
36
+
"cmpq %[v], %[expect]\n\t"
37
+
"jnz %l[error2]\n\t"
38
+
#endif
39
+
/* final store */
40
+
"movq %[newv], %[v]\n\t"
41
+
"2:\n\t"
42
+
RSEQ_INJECT_ASM(5)
43
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
44
+
: /* gcc asm goto does not allow outputs */
45
+
: [cpu_id] "r" (cpu),
46
+
[rseq_offset] "r" (rseq_offset),
47
+
[v] "m" (*v),
48
+
[expect] "r" (expect),
49
+
[newv] "r" (newv)
50
+
: "memory", "cc", "rax"
51
+
RSEQ_INJECT_CLOBBER
52
+
: abort, cmpfail
53
+
#ifdef RSEQ_COMPARE_TWICE
54
+
, error1, error2
55
+
#endif
56
+
);
57
+
rseq_after_asm_goto();
58
+
return 0;
59
+
abort:
60
+
rseq_after_asm_goto();
61
+
RSEQ_INJECT_FAILED
62
+
return -1;
63
+
cmpfail:
64
+
rseq_after_asm_goto();
65
+
return 1;
66
+
#ifdef RSEQ_COMPARE_TWICE
67
+
error1:
68
+
rseq_after_asm_goto();
69
+
rseq_bug("cpu_id comparison failed");
70
+
error2:
71
+
rseq_after_asm_goto();
72
+
rseq_bug("expected value comparison failed");
73
+
#endif
74
+
}
75
+
76
+
/*
77
+
* Compare @v against @expectnot. When it does _not_ match, load @v
78
+
* into @load, and store the content of *@v + voffp into @v.
79
+
*/
80
+
static inline __attribute__((always_inline))
81
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
82
+
long voffp, intptr_t *load, int cpu)
83
+
{
84
+
RSEQ_INJECT_C(9)
85
+
86
+
__asm__ __volatile__ goto (
87
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
88
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
89
+
#ifdef RSEQ_COMPARE_TWICE
90
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
91
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
92
+
#endif
93
+
/* Start rseq by storing table entry pointer into rseq_cs. */
94
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
95
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
96
+
RSEQ_INJECT_ASM(3)
97
+
"movq %[v], %%rbx\n\t"
98
+
"cmpq %%rbx, %[expectnot]\n\t"
99
+
"je %l[cmpfail]\n\t"
100
+
RSEQ_INJECT_ASM(4)
101
+
#ifdef RSEQ_COMPARE_TWICE
102
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
103
+
"movq %[v], %%rbx\n\t"
104
+
"cmpq %%rbx, %[expectnot]\n\t"
105
+
"je %l[error2]\n\t"
106
+
#endif
107
+
"movq %%rbx, %[load]\n\t"
108
+
"addq %[voffp], %%rbx\n\t"
109
+
"movq (%%rbx), %%rbx\n\t"
110
+
/* final store */
111
+
"movq %%rbx, %[v]\n\t"
112
+
"2:\n\t"
113
+
RSEQ_INJECT_ASM(5)
114
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
115
+
: /* gcc asm goto does not allow outputs */
116
+
: [cpu_id] "r" (cpu),
117
+
[rseq_offset] "r" (rseq_offset),
118
+
/* final store input */
119
+
[v] "m" (*v),
120
+
[expectnot] "r" (expectnot),
121
+
[voffp] "er" (voffp),
122
+
[load] "m" (*load)
123
+
: "memory", "cc", "rax", "rbx"
124
+
RSEQ_INJECT_CLOBBER
125
+
: abort, cmpfail
126
+
#ifdef RSEQ_COMPARE_TWICE
127
+
, error1, error2
128
+
#endif
129
+
);
130
+
rseq_after_asm_goto();
131
+
return 0;
132
+
abort:
133
+
rseq_after_asm_goto();
134
+
RSEQ_INJECT_FAILED
135
+
return -1;
136
+
cmpfail:
137
+
rseq_after_asm_goto();
138
+
return 1;
139
+
#ifdef RSEQ_COMPARE_TWICE
140
+
error1:
141
+
rseq_after_asm_goto();
142
+
rseq_bug("cpu_id comparison failed");
143
+
error2:
144
+
rseq_after_asm_goto();
145
+
rseq_bug("expected value comparison failed");
146
+
#endif
147
+
}
148
+
149
+
static inline __attribute__((always_inline))
150
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
151
+
{
152
+
RSEQ_INJECT_C(9)
153
+
154
+
__asm__ __volatile__ goto (
155
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
156
+
#ifdef RSEQ_COMPARE_TWICE
157
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
158
+
#endif
159
+
/* Start rseq by storing table entry pointer into rseq_cs. */
160
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
161
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
162
+
RSEQ_INJECT_ASM(3)
163
+
#ifdef RSEQ_COMPARE_TWICE
164
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
165
+
#endif
166
+
/* final store */
167
+
"addq %[count], %[v]\n\t"
168
+
"2:\n\t"
169
+
RSEQ_INJECT_ASM(4)
170
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
171
+
: /* gcc asm goto does not allow outputs */
172
+
: [cpu_id] "r" (cpu),
173
+
[rseq_offset] "r" (rseq_offset),
174
+
/* final store input */
175
+
[v] "m" (*v),
176
+
[count] "er" (count)
177
+
: "memory", "cc", "rax"
178
+
RSEQ_INJECT_CLOBBER
179
+
: abort
180
+
#ifdef RSEQ_COMPARE_TWICE
181
+
, error1
182
+
#endif
183
+
);
184
+
rseq_after_asm_goto();
185
+
return 0;
186
+
abort:
187
+
rseq_after_asm_goto();
188
+
RSEQ_INJECT_FAILED
189
+
return -1;
190
+
#ifdef RSEQ_COMPARE_TWICE
191
+
error1:
192
+
rseq_after_asm_goto();
193
+
rseq_bug("cpu_id comparison failed");
194
+
#endif
195
+
}
196
+
197
+
#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
198
+
199
+
/*
200
+
* pval = *(ptr+off)
201
+
* *pval += inc;
202
+
*/
203
+
static inline __attribute__((always_inline))
204
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_offset_deref_addv)(intptr_t *ptr, long off, intptr_t inc, int cpu)
205
+
{
206
+
RSEQ_INJECT_C(9)
207
+
208
+
__asm__ __volatile__ goto (
209
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
210
+
#ifdef RSEQ_COMPARE_TWICE
211
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
212
+
#endif
213
+
/* Start rseq by storing table entry pointer into rseq_cs. */
214
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
215
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
216
+
RSEQ_INJECT_ASM(3)
217
+
#ifdef RSEQ_COMPARE_TWICE
218
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
219
+
#endif
220
+
/* get p+v */
221
+
"movq %[ptr], %%rbx\n\t"
222
+
"addq %[off], %%rbx\n\t"
223
+
/* get pv */
224
+
"movq (%%rbx), %%rcx\n\t"
225
+
/* *pv += inc */
226
+
"addq %[inc], (%%rcx)\n\t"
227
+
"2:\n\t"
228
+
RSEQ_INJECT_ASM(4)
229
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
230
+
: /* gcc asm goto does not allow outputs */
231
+
: [cpu_id] "r" (cpu),
232
+
[rseq_offset] "r" (rseq_offset),
233
+
/* final store input */
234
+
[ptr] "m" (*ptr),
235
+
[off] "er" (off),
236
+
[inc] "er" (inc)
237
+
: "memory", "cc", "rax", "rbx", "rcx"
238
+
RSEQ_INJECT_CLOBBER
239
+
: abort
240
+
#ifdef RSEQ_COMPARE_TWICE
241
+
, error1
242
+
#endif
243
+
);
244
+
return 0;
245
+
abort:
246
+
RSEQ_INJECT_FAILED
247
+
return -1;
248
+
#ifdef RSEQ_COMPARE_TWICE
249
+
error1:
250
+
rseq_bug("cpu_id comparison failed");
251
+
#endif
252
+
}
253
+
254
+
static inline __attribute__((always_inline))
255
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
256
+
intptr_t *v2, intptr_t expect2,
257
+
intptr_t newv, int cpu)
258
+
{
259
+
RSEQ_INJECT_C(9)
260
+
261
+
__asm__ __volatile__ goto (
262
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
263
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
264
+
#ifdef RSEQ_COMPARE_TWICE
265
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
266
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
267
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
268
+
#endif
269
+
/* Start rseq by storing table entry pointer into rseq_cs. */
270
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
271
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
272
+
RSEQ_INJECT_ASM(3)
273
+
"cmpq %[v], %[expect]\n\t"
274
+
"jnz %l[cmpfail]\n\t"
275
+
RSEQ_INJECT_ASM(4)
276
+
"cmpq %[v2], %[expect2]\n\t"
277
+
"jnz %l[cmpfail]\n\t"
278
+
RSEQ_INJECT_ASM(5)
279
+
#ifdef RSEQ_COMPARE_TWICE
280
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
281
+
"cmpq %[v], %[expect]\n\t"
282
+
"jnz %l[error2]\n\t"
283
+
"cmpq %[v2], %[expect2]\n\t"
284
+
"jnz %l[error3]\n\t"
285
+
#endif
286
+
/* final store */
287
+
"movq %[newv], %[v]\n\t"
288
+
"2:\n\t"
289
+
RSEQ_INJECT_ASM(6)
290
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
291
+
: /* gcc asm goto does not allow outputs */
292
+
: [cpu_id] "r" (cpu),
293
+
[rseq_offset] "r" (rseq_offset),
294
+
/* cmp2 input */
295
+
[v2] "m" (*v2),
296
+
[expect2] "r" (expect2),
297
+
/* final store input */
298
+
[v] "m" (*v),
299
+
[expect] "r" (expect),
300
+
[newv] "r" (newv)
301
+
: "memory", "cc", "rax"
302
+
RSEQ_INJECT_CLOBBER
303
+
: abort, cmpfail
304
+
#ifdef RSEQ_COMPARE_TWICE
305
+
, error1, error2, error3
306
+
#endif
307
+
);
308
+
rseq_after_asm_goto();
309
+
return 0;
310
+
abort:
311
+
rseq_after_asm_goto();
312
+
RSEQ_INJECT_FAILED
313
+
return -1;
314
+
cmpfail:
315
+
rseq_after_asm_goto();
316
+
return 1;
317
+
#ifdef RSEQ_COMPARE_TWICE
318
+
error1:
319
+
rseq_after_asm_goto();
320
+
rseq_bug("cpu_id comparison failed");
321
+
error2:
322
+
rseq_after_asm_goto();
323
+
rseq_bug("1st expected value comparison failed");
324
+
error3:
325
+
rseq_after_asm_goto();
326
+
rseq_bug("2nd expected value comparison failed");
327
+
#endif
328
+
}
329
+
330
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
331
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
332
+
333
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
334
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
335
+
336
+
static inline __attribute__((always_inline))
337
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
338
+
intptr_t *v2, intptr_t newv2,
339
+
intptr_t newv, int cpu)
340
+
{
341
+
RSEQ_INJECT_C(9)
342
+
343
+
__asm__ __volatile__ goto (
344
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
345
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
346
+
#ifdef RSEQ_COMPARE_TWICE
347
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
348
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
349
+
#endif
350
+
/* Start rseq by storing table entry pointer into rseq_cs. */
351
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
352
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
353
+
RSEQ_INJECT_ASM(3)
354
+
"cmpq %[v], %[expect]\n\t"
355
+
"jnz %l[cmpfail]\n\t"
356
+
RSEQ_INJECT_ASM(4)
357
+
#ifdef RSEQ_COMPARE_TWICE
358
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
359
+
"cmpq %[v], %[expect]\n\t"
360
+
"jnz %l[error2]\n\t"
361
+
#endif
362
+
/* try store */
363
+
"movq %[newv2], %[v2]\n\t"
364
+
RSEQ_INJECT_ASM(5)
365
+
/* final store */
366
+
"movq %[newv], %[v]\n\t"
367
+
"2:\n\t"
368
+
RSEQ_INJECT_ASM(6)
369
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
370
+
: /* gcc asm goto does not allow outputs */
371
+
: [cpu_id] "r" (cpu),
372
+
[rseq_offset] "r" (rseq_offset),
373
+
/* try store input */
374
+
[v2] "m" (*v2),
375
+
[newv2] "r" (newv2),
376
+
/* final store input */
377
+
[v] "m" (*v),
378
+
[expect] "r" (expect),
379
+
[newv] "r" (newv)
380
+
: "memory", "cc", "rax"
381
+
RSEQ_INJECT_CLOBBER
382
+
: abort, cmpfail
383
+
#ifdef RSEQ_COMPARE_TWICE
384
+
, error1, error2
385
+
#endif
386
+
);
387
+
rseq_after_asm_goto();
388
+
return 0;
389
+
abort:
390
+
rseq_after_asm_goto();
391
+
RSEQ_INJECT_FAILED
392
+
return -1;
393
+
cmpfail:
394
+
rseq_after_asm_goto();
395
+
return 1;
396
+
#ifdef RSEQ_COMPARE_TWICE
397
+
error1:
398
+
rseq_after_asm_goto();
399
+
rseq_bug("cpu_id comparison failed");
400
+
error2:
401
+
rseq_after_asm_goto();
402
+
rseq_bug("expected value comparison failed");
403
+
#endif
404
+
}
405
+
406
+
static inline __attribute__((always_inline))
407
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
408
+
void *dst, void *src, size_t len,
409
+
intptr_t newv, int cpu)
410
+
{
411
+
uint64_t rseq_scratch[3];
412
+
413
+
RSEQ_INJECT_C(9)
414
+
415
+
__asm__ __volatile__ goto (
416
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
417
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
418
+
#ifdef RSEQ_COMPARE_TWICE
419
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
420
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
421
+
#endif
422
+
"movq %[src], %[rseq_scratch0]\n\t"
423
+
"movq %[dst], %[rseq_scratch1]\n\t"
424
+
"movq %[len], %[rseq_scratch2]\n\t"
425
+
/* Start rseq by storing table entry pointer into rseq_cs. */
426
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
427
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
428
+
RSEQ_INJECT_ASM(3)
429
+
"cmpq %[v], %[expect]\n\t"
430
+
"jnz 5f\n\t"
431
+
RSEQ_INJECT_ASM(4)
432
+
#ifdef RSEQ_COMPARE_TWICE
433
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 6f)
434
+
"cmpq %[v], %[expect]\n\t"
435
+
"jnz 7f\n\t"
436
+
#endif
437
+
/* try memcpy */
438
+
"test %[len], %[len]\n\t" \
439
+
"jz 333f\n\t" \
440
+
"222:\n\t" \
441
+
"movb (%[src]), %%al\n\t" \
442
+
"movb %%al, (%[dst])\n\t" \
443
+
"inc %[src]\n\t" \
444
+
"inc %[dst]\n\t" \
445
+
"dec %[len]\n\t" \
446
+
"jnz 222b\n\t" \
447
+
"333:\n\t" \
448
+
RSEQ_INJECT_ASM(5)
449
+
/* final store */
450
+
"movq %[newv], %[v]\n\t"
451
+
"2:\n\t"
452
+
RSEQ_INJECT_ASM(6)
453
+
/* teardown */
454
+
"movq %[rseq_scratch2], %[len]\n\t"
455
+
"movq %[rseq_scratch1], %[dst]\n\t"
456
+
"movq %[rseq_scratch0], %[src]\n\t"
457
+
RSEQ_ASM_DEFINE_ABORT(4,
458
+
"movq %[rseq_scratch2], %[len]\n\t"
459
+
"movq %[rseq_scratch1], %[dst]\n\t"
460
+
"movq %[rseq_scratch0], %[src]\n\t",
461
+
abort)
462
+
RSEQ_ASM_DEFINE_CMPFAIL(5,
463
+
"movq %[rseq_scratch2], %[len]\n\t"
464
+
"movq %[rseq_scratch1], %[dst]\n\t"
465
+
"movq %[rseq_scratch0], %[src]\n\t",
466
+
cmpfail)
467
+
#ifdef RSEQ_COMPARE_TWICE
468
+
RSEQ_ASM_DEFINE_CMPFAIL(6,
469
+
"movq %[rseq_scratch2], %[len]\n\t"
470
+
"movq %[rseq_scratch1], %[dst]\n\t"
471
+
"movq %[rseq_scratch0], %[src]\n\t",
472
+
error1)
473
+
RSEQ_ASM_DEFINE_CMPFAIL(7,
474
+
"movq %[rseq_scratch2], %[len]\n\t"
475
+
"movq %[rseq_scratch1], %[dst]\n\t"
476
+
"movq %[rseq_scratch0], %[src]\n\t",
477
+
error2)
478
+
#endif
479
+
: /* gcc asm goto does not allow outputs */
480
+
: [cpu_id] "r" (cpu),
481
+
[rseq_offset] "r" (rseq_offset),
482
+
/* final store input */
483
+
[v] "m" (*v),
484
+
[expect] "r" (expect),
485
+
[newv] "r" (newv),
486
+
/* try memcpy input */
487
+
[dst] "r" (dst),
488
+
[src] "r" (src),
489
+
[len] "r" (len),
490
+
[rseq_scratch0] "m" (rseq_scratch[0]),
491
+
[rseq_scratch1] "m" (rseq_scratch[1]),
492
+
[rseq_scratch2] "m" (rseq_scratch[2])
493
+
: "memory", "cc", "rax"
494
+
RSEQ_INJECT_CLOBBER
495
+
: abort, cmpfail
496
+
#ifdef RSEQ_COMPARE_TWICE
497
+
, error1, error2
498
+
#endif
499
+
);
500
+
rseq_after_asm_goto();
501
+
return 0;
502
+
abort:
503
+
rseq_after_asm_goto();
504
+
RSEQ_INJECT_FAILED
505
+
return -1;
506
+
cmpfail:
507
+
rseq_after_asm_goto();
508
+
return 1;
509
+
#ifdef RSEQ_COMPARE_TWICE
510
+
error1:
511
+
rseq_after_asm_goto();
512
+
rseq_bug("cpu_id comparison failed");
513
+
error2:
514
+
rseq_after_asm_goto();
515
+
rseq_bug("expected value comparison failed");
516
+
#endif
517
+
}
518
+
519
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
520
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
521
+
522
+
#elif defined(__i386__)
523
+
524
+
#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \
525
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
526
+
527
+
static inline __attribute__((always_inline))
528
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
529
+
{
530
+
RSEQ_INJECT_C(9)
531
+
532
+
__asm__ __volatile__ goto (
533
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
534
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
535
+
#ifdef RSEQ_COMPARE_TWICE
536
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
537
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
538
+
#endif
539
+
/* Start rseq by storing table entry pointer into rseq_cs. */
540
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
541
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
542
+
RSEQ_INJECT_ASM(3)
543
+
"cmpl %[v], %[expect]\n\t"
544
+
"jnz %l[cmpfail]\n\t"
545
+
RSEQ_INJECT_ASM(4)
546
+
#ifdef RSEQ_COMPARE_TWICE
547
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
548
+
"cmpl %[v], %[expect]\n\t"
549
+
"jnz %l[error2]\n\t"
550
+
#endif
551
+
/* final store */
552
+
"movl %[newv], %[v]\n\t"
553
+
"2:\n\t"
554
+
RSEQ_INJECT_ASM(5)
555
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
556
+
: /* gcc asm goto does not allow outputs */
557
+
: [cpu_id] "r" (cpu),
558
+
[rseq_offset] "r" (rseq_offset),
559
+
[v] "m" (*v),
560
+
[expect] "r" (expect),
561
+
[newv] "r" (newv)
562
+
: "memory", "cc", "eax"
563
+
RSEQ_INJECT_CLOBBER
564
+
: abort, cmpfail
565
+
#ifdef RSEQ_COMPARE_TWICE
566
+
, error1, error2
567
+
#endif
568
+
);
569
+
rseq_after_asm_goto();
570
+
return 0;
571
+
abort:
572
+
rseq_after_asm_goto();
573
+
RSEQ_INJECT_FAILED
574
+
return -1;
575
+
cmpfail:
576
+
rseq_after_asm_goto();
577
+
return 1;
578
+
#ifdef RSEQ_COMPARE_TWICE
579
+
error1:
580
+
rseq_after_asm_goto();
581
+
rseq_bug("cpu_id comparison failed");
582
+
error2:
583
+
rseq_after_asm_goto();
584
+
rseq_bug("expected value comparison failed");
585
+
#endif
586
+
}
587
+
588
+
/*
589
+
* Compare @v against @expectnot. When it does _not_ match, load @v
590
+
* into @load, and store the content of *@v + voffp into @v.
591
+
*/
592
+
static inline __attribute__((always_inline))
593
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot,
594
+
long voffp, intptr_t *load, int cpu)
595
+
{
596
+
RSEQ_INJECT_C(9)
597
+
598
+
__asm__ __volatile__ goto (
599
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
600
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
601
+
#ifdef RSEQ_COMPARE_TWICE
602
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
603
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
604
+
#endif
605
+
/* Start rseq by storing table entry pointer into rseq_cs. */
606
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
607
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
608
+
RSEQ_INJECT_ASM(3)
609
+
"movl %[v], %%ebx\n\t"
610
+
"cmpl %%ebx, %[expectnot]\n\t"
611
+
"je %l[cmpfail]\n\t"
612
+
RSEQ_INJECT_ASM(4)
613
+
#ifdef RSEQ_COMPARE_TWICE
614
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
615
+
"movl %[v], %%ebx\n\t"
616
+
"cmpl %%ebx, %[expectnot]\n\t"
617
+
"je %l[error2]\n\t"
618
+
#endif
619
+
"movl %%ebx, %[load]\n\t"
620
+
"addl %[voffp], %%ebx\n\t"
621
+
"movl (%%ebx), %%ebx\n\t"
622
+
/* final store */
623
+
"movl %%ebx, %[v]\n\t"
624
+
"2:\n\t"
625
+
RSEQ_INJECT_ASM(5)
626
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
627
+
: /* gcc asm goto does not allow outputs */
628
+
: [cpu_id] "r" (cpu),
629
+
[rseq_offset] "r" (rseq_offset),
630
+
/* final store input */
631
+
[v] "m" (*v),
632
+
[expectnot] "r" (expectnot),
633
+
[voffp] "ir" (voffp),
634
+
[load] "m" (*load)
635
+
: "memory", "cc", "eax", "ebx"
636
+
RSEQ_INJECT_CLOBBER
637
+
: abort, cmpfail
638
+
#ifdef RSEQ_COMPARE_TWICE
639
+
, error1, error2
640
+
#endif
641
+
);
642
+
rseq_after_asm_goto();
643
+
return 0;
644
+
abort:
645
+
rseq_after_asm_goto();
646
+
RSEQ_INJECT_FAILED
647
+
return -1;
648
+
cmpfail:
649
+
rseq_after_asm_goto();
650
+
return 1;
651
+
#ifdef RSEQ_COMPARE_TWICE
652
+
error1:
653
+
rseq_after_asm_goto();
654
+
rseq_bug("cpu_id comparison failed");
655
+
error2:
656
+
rseq_after_asm_goto();
657
+
rseq_bug("expected value comparison failed");
658
+
#endif
659
+
}
660
+
661
+
static inline __attribute__((always_inline))
662
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu)
663
+
{
664
+
RSEQ_INJECT_C(9)
665
+
666
+
__asm__ __volatile__ goto (
667
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
668
+
#ifdef RSEQ_COMPARE_TWICE
669
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
670
+
#endif
671
+
/* Start rseq by storing table entry pointer into rseq_cs. */
672
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
673
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
674
+
RSEQ_INJECT_ASM(3)
675
+
#ifdef RSEQ_COMPARE_TWICE
676
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
677
+
#endif
678
+
/* final store */
679
+
"addl %[count], %[v]\n\t"
680
+
"2:\n\t"
681
+
RSEQ_INJECT_ASM(4)
682
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
683
+
: /* gcc asm goto does not allow outputs */
684
+
: [cpu_id] "r" (cpu),
685
+
[rseq_offset] "r" (rseq_offset),
686
+
/* final store input */
687
+
[v] "m" (*v),
688
+
[count] "ir" (count)
689
+
: "memory", "cc", "eax"
690
+
RSEQ_INJECT_CLOBBER
691
+
: abort
692
+
#ifdef RSEQ_COMPARE_TWICE
693
+
, error1
694
+
#endif
695
+
);
696
+
rseq_after_asm_goto();
697
+
return 0;
698
+
abort:
699
+
rseq_after_asm_goto();
700
+
RSEQ_INJECT_FAILED
701
+
return -1;
702
+
#ifdef RSEQ_COMPARE_TWICE
703
+
error1:
704
+
rseq_after_asm_goto();
705
+
rseq_bug("cpu_id comparison failed");
706
+
#endif
707
+
}
708
+
709
+
static inline __attribute__((always_inline))
710
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect,
711
+
intptr_t *v2, intptr_t expect2,
712
+
intptr_t newv, int cpu)
713
+
{
714
+
RSEQ_INJECT_C(9)
715
+
716
+
__asm__ __volatile__ goto (
717
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
718
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
719
+
#ifdef RSEQ_COMPARE_TWICE
720
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
721
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
722
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
723
+
#endif
724
+
/* Start rseq by storing table entry pointer into rseq_cs. */
725
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
726
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
727
+
RSEQ_INJECT_ASM(3)
728
+
"cmpl %[v], %[expect]\n\t"
729
+
"jnz %l[cmpfail]\n\t"
730
+
RSEQ_INJECT_ASM(4)
731
+
"cmpl %[expect2], %[v2]\n\t"
732
+
"jnz %l[cmpfail]\n\t"
733
+
RSEQ_INJECT_ASM(5)
734
+
#ifdef RSEQ_COMPARE_TWICE
735
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
736
+
"cmpl %[v], %[expect]\n\t"
737
+
"jnz %l[error2]\n\t"
738
+
"cmpl %[expect2], %[v2]\n\t"
739
+
"jnz %l[error3]\n\t"
740
+
#endif
741
+
"movl %[newv], %%eax\n\t"
742
+
/* final store */
743
+
"movl %%eax, %[v]\n\t"
744
+
"2:\n\t"
745
+
RSEQ_INJECT_ASM(6)
746
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
747
+
: /* gcc asm goto does not allow outputs */
748
+
: [cpu_id] "r" (cpu),
749
+
[rseq_offset] "r" (rseq_offset),
750
+
/* cmp2 input */
751
+
[v2] "m" (*v2),
752
+
[expect2] "r" (expect2),
753
+
/* final store input */
754
+
[v] "m" (*v),
755
+
[expect] "r" (expect),
756
+
[newv] "m" (newv)
757
+
: "memory", "cc", "eax"
758
+
RSEQ_INJECT_CLOBBER
759
+
: abort, cmpfail
760
+
#ifdef RSEQ_COMPARE_TWICE
761
+
, error1, error2, error3
762
+
#endif
763
+
);
764
+
rseq_after_asm_goto();
765
+
return 0;
766
+
abort:
767
+
rseq_after_asm_goto();
768
+
RSEQ_INJECT_FAILED
769
+
return -1;
770
+
cmpfail:
771
+
rseq_after_asm_goto();
772
+
return 1;
773
+
#ifdef RSEQ_COMPARE_TWICE
774
+
error1:
775
+
rseq_after_asm_goto();
776
+
rseq_bug("cpu_id comparison failed");
777
+
error2:
778
+
rseq_after_asm_goto();
779
+
rseq_bug("1st expected value comparison failed");
780
+
error3:
781
+
rseq_after_asm_goto();
782
+
rseq_bug("2nd expected value comparison failed");
783
+
#endif
784
+
}
785
+
786
+
#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) &&
787
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
788
+
789
+
#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \
790
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID))
791
+
792
+
static inline __attribute__((always_inline))
793
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect,
794
+
intptr_t *v2, intptr_t newv2,
795
+
intptr_t newv, int cpu)
796
+
{
797
+
RSEQ_INJECT_C(9)
798
+
799
+
__asm__ __volatile__ goto (
800
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
801
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
802
+
#ifdef RSEQ_COMPARE_TWICE
803
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
804
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
805
+
#endif
806
+
/* Start rseq by storing table entry pointer into rseq_cs. */
807
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
808
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
809
+
RSEQ_INJECT_ASM(3)
810
+
"movl %[expect], %%eax\n\t"
811
+
"cmpl %[v], %%eax\n\t"
812
+
"jnz %l[cmpfail]\n\t"
813
+
RSEQ_INJECT_ASM(4)
814
+
#ifdef RSEQ_COMPARE_TWICE
815
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
816
+
"movl %[expect], %%eax\n\t"
817
+
"cmpl %[v], %%eax\n\t"
818
+
"jnz %l[error2]\n\t"
819
+
#endif
820
+
/* try store */
821
+
"movl %[newv2], %[v2]\n\t"
822
+
RSEQ_INJECT_ASM(5)
823
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
824
+
"lock; addl $0,-128(%%esp)\n\t"
825
+
#endif
826
+
/* final store */
827
+
"movl %[newv], %[v]\n\t"
828
+
"2:\n\t"
829
+
RSEQ_INJECT_ASM(6)
830
+
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
831
+
: /* gcc asm goto does not allow outputs */
832
+
: [cpu_id] "r" (cpu),
833
+
[rseq_offset] "r" (rseq_offset),
834
+
/* try store input */
835
+
[v2] "m" (*v2),
836
+
[newv2] "r" (newv2),
837
+
/* final store input */
838
+
[v] "m" (*v),
839
+
[expect] "m" (expect),
840
+
[newv] "r" (newv)
841
+
: "memory", "cc", "eax"
842
+
RSEQ_INJECT_CLOBBER
843
+
: abort, cmpfail
844
+
#ifdef RSEQ_COMPARE_TWICE
845
+
, error1, error2
846
+
#endif
847
+
);
848
+
rseq_after_asm_goto();
849
+
return 0;
850
+
abort:
851
+
rseq_after_asm_goto();
852
+
RSEQ_INJECT_FAILED
853
+
return -1;
854
+
cmpfail:
855
+
rseq_after_asm_goto();
856
+
return 1;
857
+
#ifdef RSEQ_COMPARE_TWICE
858
+
error1:
859
+
rseq_after_asm_goto();
860
+
rseq_bug("cpu_id comparison failed");
861
+
error2:
862
+
rseq_after_asm_goto();
863
+
rseq_bug("expected value comparison failed");
864
+
#endif
865
+
866
+
}
867
+
868
+
/* TODO: implement a faster memcpy. */
869
+
static inline __attribute__((always_inline))
870
+
int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect,
871
+
void *dst, void *src, size_t len,
872
+
intptr_t newv, int cpu)
873
+
{
874
+
uint32_t rseq_scratch[3];
875
+
876
+
RSEQ_INJECT_C(9)
877
+
878
+
__asm__ __volatile__ goto (
879
+
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
880
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
881
+
#ifdef RSEQ_COMPARE_TWICE
882
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
883
+
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
884
+
#endif
885
+
"movl %[src], %[rseq_scratch0]\n\t"
886
+
"movl %[dst], %[rseq_scratch1]\n\t"
887
+
"movl %[len], %[rseq_scratch2]\n\t"
888
+
/* Start rseq by storing table entry pointer into rseq_cs. */
889
+
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
890
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 4f)
891
+
RSEQ_INJECT_ASM(3)
892
+
"movl %[expect], %%eax\n\t"
893
+
"cmpl %%eax, %[v]\n\t"
894
+
"jnz 5f\n\t"
895
+
RSEQ_INJECT_ASM(4)
896
+
#ifdef RSEQ_COMPARE_TWICE
897
+
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFFSET(%[rseq_offset]), 6f)
898
+
"movl %[expect], %%eax\n\t"
899
+
"cmpl %%eax, %[v]\n\t"
900
+
"jnz 7f\n\t"
901
+
#endif
902
+
/* try memcpy */
903
+
"test %[len], %[len]\n\t" \
904
+
"jz 333f\n\t" \
905
+
"222:\n\t" \
906
+
"movb (%[src]), %%al\n\t" \
907
+
"movb %%al, (%[dst])\n\t" \
908
+
"inc %[src]\n\t" \
909
+
"inc %[dst]\n\t" \
910
+
"dec %[len]\n\t" \
911
+
"jnz 222b\n\t" \
912
+
"333:\n\t" \
913
+
RSEQ_INJECT_ASM(5)
914
+
#ifdef RSEQ_TEMPLATE_MO_RELEASE
915
+
"lock; addl $0,-128(%%esp)\n\t"
916
+
#endif
917
+
"movl %[newv], %%eax\n\t"
918
+
/* final store */
919
+
"movl %%eax, %[v]\n\t"
920
+
"2:\n\t"
921
+
RSEQ_INJECT_ASM(6)
922
+
/* teardown */
923
+
"movl %[rseq_scratch2], %[len]\n\t"
924
+
"movl %[rseq_scratch1], %[dst]\n\t"
925
+
"movl %[rseq_scratch0], %[src]\n\t"
926
+
RSEQ_ASM_DEFINE_ABORT(4,
927
+
"movl %[rseq_scratch2], %[len]\n\t"
928
+
"movl %[rseq_scratch1], %[dst]\n\t"
929
+
"movl %[rseq_scratch0], %[src]\n\t",
930
+
abort)
931
+
RSEQ_ASM_DEFINE_CMPFAIL(5,
932
+
"movl %[rseq_scratch2], %[len]\n\t"
933
+
"movl %[rseq_scratch1], %[dst]\n\t"
934
+
"movl %[rseq_scratch0], %[src]\n\t",
935
+
cmpfail)
936
+
#ifdef RSEQ_COMPARE_TWICE
937
+
RSEQ_ASM_DEFINE_CMPFAIL(6,
938
+
"movl %[rseq_scratch2], %[len]\n\t"
939
+
"movl %[rseq_scratch1], %[dst]\n\t"
940
+
"movl %[rseq_scratch0], %[src]\n\t",
941
+
error1)
942
+
RSEQ_ASM_DEFINE_CMPFAIL(7,
943
+
"movl %[rseq_scratch2], %[len]\n\t"
944
+
"movl %[rseq_scratch1], %[dst]\n\t"
945
+
"movl %[rseq_scratch0], %[src]\n\t",
946
+
error2)
947
+
#endif
948
+
: /* gcc asm goto does not allow outputs */
949
+
: [cpu_id] "r" (cpu),
950
+
[rseq_offset] "r" (rseq_offset),
951
+
/* final store input */
952
+
[v] "m" (*v),
953
+
[expect] "m" (expect),
954
+
[newv] "m" (newv),
955
+
/* try memcpy input */
956
+
[dst] "r" (dst),
957
+
[src] "r" (src),
958
+
[len] "r" (len),
959
+
[rseq_scratch0] "m" (rseq_scratch[0]),
960
+
[rseq_scratch1] "m" (rseq_scratch[1]),
961
+
[rseq_scratch2] "m" (rseq_scratch[2])
962
+
: "memory", "cc", "eax"
963
+
RSEQ_INJECT_CLOBBER
964
+
: abort, cmpfail
965
+
#ifdef RSEQ_COMPARE_TWICE
966
+
, error1, error2
967
+
#endif
968
+
);
969
+
rseq_after_asm_goto();
970
+
return 0;
971
+
abort:
972
+
rseq_after_asm_goto();
973
+
RSEQ_INJECT_FAILED
974
+
return -1;
975
+
cmpfail:
976
+
rseq_after_asm_goto();
977
+
return 1;
978
+
#ifdef RSEQ_COMPARE_TWICE
979
+
error1:
980
+
rseq_after_asm_goto();
981
+
rseq_bug("cpu_id comparison failed");
982
+
error2:
983
+
rseq_after_asm_goto();
984
+
rseq_bug("expected value comparison failed");
985
+
#endif
986
+
}
987
+
988
+
#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) &&
989
+
(defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */
990
+
991
+
#endif
992
+
993
+
#include "rseq-bits-reset.h"
+31
-1162
tools/testing/selftests/rseq/rseq-x86.h
+31
-1162
tools/testing/selftests/rseq/rseq-x86.h
···
2
2
/*
3
3
* rseq-x86.h
4
4
*
5
-
* (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
5
+
* (C) Copyright 2016-2022 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
6
6
*/
7
+
8
+
#ifndef RSEQ_H
9
+
#error "Never use <rseq-x86.h> directly; include <rseq.h> instead."
10
+
#endif
7
11
8
12
#include <stdint.h>
9
13
···
26
22
* address through a "r" input operand.
27
23
*/
28
24
29
-
/* Offset of cpu_id and rseq_cs fields in struct rseq. */
25
+
/* Offset of cpu_id, rseq_cs, and mm_cid fields in struct rseq. */
30
26
#define RSEQ_CPU_ID_OFFSET 4
31
27
#define RSEQ_CS_OFFSET 8
28
+
#define RSEQ_MM_CID_OFFSET 24
32
29
33
30
#ifdef __x86_64__
34
31
···
54
49
rseq_barrier(); \
55
50
RSEQ_WRITE_ONCE(*p, v); \
56
51
} while (0)
57
-
58
-
#ifdef RSEQ_SKIP_FASTPATH
59
-
#include "rseq-skip.h"
60
-
#else /* !RSEQ_SKIP_FASTPATH */
61
52
62
53
#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, \
63
54
start_ip, post_commit_offset, abort_ip) \
···
113
112
"jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \
114
113
".popsection\n\t"
115
114
116
-
static inline __attribute__((always_inline))
117
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
118
-
{
119
-
RSEQ_INJECT_C(9)
120
-
121
-
__asm__ __volatile__ goto (
122
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
123
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
124
-
#ifdef RSEQ_COMPARE_TWICE
125
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
126
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
127
-
#endif
128
-
/* Start rseq by storing table entry pointer into rseq_cs. */
129
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
130
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
131
-
RSEQ_INJECT_ASM(3)
132
-
"cmpq %[v], %[expect]\n\t"
133
-
"jnz %l[cmpfail]\n\t"
134
-
RSEQ_INJECT_ASM(4)
135
-
#ifdef RSEQ_COMPARE_TWICE
136
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
137
-
"cmpq %[v], %[expect]\n\t"
138
-
"jnz %l[error2]\n\t"
139
-
#endif
140
-
/* final store */
141
-
"movq %[newv], %[v]\n\t"
142
-
"2:\n\t"
143
-
RSEQ_INJECT_ASM(5)
144
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
145
-
: /* gcc asm goto does not allow outputs */
146
-
: [cpu_id] "r" (cpu),
147
-
[rseq_offset] "r" (rseq_offset),
148
-
[v] "m" (*v),
149
-
[expect] "r" (expect),
150
-
[newv] "r" (newv)
151
-
: "memory", "cc", "rax"
152
-
RSEQ_INJECT_CLOBBER
153
-
: abort, cmpfail
154
-
#ifdef RSEQ_COMPARE_TWICE
155
-
, error1, error2
156
-
#endif
157
-
);
158
-
rseq_after_asm_goto();
159
-
return 0;
160
-
abort:
161
-
rseq_after_asm_goto();
162
-
RSEQ_INJECT_FAILED
163
-
return -1;
164
-
cmpfail:
165
-
rseq_after_asm_goto();
166
-
return 1;
167
-
#ifdef RSEQ_COMPARE_TWICE
168
-
error1:
169
-
rseq_after_asm_goto();
170
-
rseq_bug("cpu_id comparison failed");
171
-
error2:
172
-
rseq_after_asm_goto();
173
-
rseq_bug("expected value comparison failed");
174
-
#endif
175
-
}
176
-
177
-
/*
178
-
* Compare @v against @expectnot. When it does _not_ match, load @v
179
-
* into @load, and store the content of *@v + voffp into @v.
180
-
*/
181
-
static inline __attribute__((always_inline))
182
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
183
-
long voffp, intptr_t *load, int cpu)
184
-
{
185
-
RSEQ_INJECT_C(9)
186
-
187
-
__asm__ __volatile__ goto (
188
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
189
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
190
-
#ifdef RSEQ_COMPARE_TWICE
191
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
192
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
193
-
#endif
194
-
/* Start rseq by storing table entry pointer into rseq_cs. */
195
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
196
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
197
-
RSEQ_INJECT_ASM(3)
198
-
"movq %[v], %%rbx\n\t"
199
-
"cmpq %%rbx, %[expectnot]\n\t"
200
-
"je %l[cmpfail]\n\t"
201
-
RSEQ_INJECT_ASM(4)
202
-
#ifdef RSEQ_COMPARE_TWICE
203
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
204
-
"movq %[v], %%rbx\n\t"
205
-
"cmpq %%rbx, %[expectnot]\n\t"
206
-
"je %l[error2]\n\t"
207
-
#endif
208
-
"movq %%rbx, %[load]\n\t"
209
-
"addq %[voffp], %%rbx\n\t"
210
-
"movq (%%rbx), %%rbx\n\t"
211
-
/* final store */
212
-
"movq %%rbx, %[v]\n\t"
213
-
"2:\n\t"
214
-
RSEQ_INJECT_ASM(5)
215
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
216
-
: /* gcc asm goto does not allow outputs */
217
-
: [cpu_id] "r" (cpu),
218
-
[rseq_offset] "r" (rseq_offset),
219
-
/* final store input */
220
-
[v] "m" (*v),
221
-
[expectnot] "r" (expectnot),
222
-
[voffp] "er" (voffp),
223
-
[load] "m" (*load)
224
-
: "memory", "cc", "rax", "rbx"
225
-
RSEQ_INJECT_CLOBBER
226
-
: abort, cmpfail
227
-
#ifdef RSEQ_COMPARE_TWICE
228
-
, error1, error2
229
-
#endif
230
-
);
231
-
rseq_after_asm_goto();
232
-
return 0;
233
-
abort:
234
-
rseq_after_asm_goto();
235
-
RSEQ_INJECT_FAILED
236
-
return -1;
237
-
cmpfail:
238
-
rseq_after_asm_goto();
239
-
return 1;
240
-
#ifdef RSEQ_COMPARE_TWICE
241
-
error1:
242
-
rseq_after_asm_goto();
243
-
rseq_bug("cpu_id comparison failed");
244
-
error2:
245
-
rseq_after_asm_goto();
246
-
rseq_bug("expected value comparison failed");
247
-
#endif
248
-
}
249
-
250
-
static inline __attribute__((always_inline))
251
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
252
-
{
253
-
RSEQ_INJECT_C(9)
254
-
255
-
__asm__ __volatile__ goto (
256
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
257
-
#ifdef RSEQ_COMPARE_TWICE
258
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
259
-
#endif
260
-
/* Start rseq by storing table entry pointer into rseq_cs. */
261
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
262
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
263
-
RSEQ_INJECT_ASM(3)
264
-
#ifdef RSEQ_COMPARE_TWICE
265
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
266
-
#endif
267
-
/* final store */
268
-
"addq %[count], %[v]\n\t"
269
-
"2:\n\t"
270
-
RSEQ_INJECT_ASM(4)
271
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
272
-
: /* gcc asm goto does not allow outputs */
273
-
: [cpu_id] "r" (cpu),
274
-
[rseq_offset] "r" (rseq_offset),
275
-
/* final store input */
276
-
[v] "m" (*v),
277
-
[count] "er" (count)
278
-
: "memory", "cc", "rax"
279
-
RSEQ_INJECT_CLOBBER
280
-
: abort
281
-
#ifdef RSEQ_COMPARE_TWICE
282
-
, error1
283
-
#endif
284
-
);
285
-
rseq_after_asm_goto();
286
-
return 0;
287
-
abort:
288
-
rseq_after_asm_goto();
289
-
RSEQ_INJECT_FAILED
290
-
return -1;
291
-
#ifdef RSEQ_COMPARE_TWICE
292
-
error1:
293
-
rseq_after_asm_goto();
294
-
rseq_bug("cpu_id comparison failed");
295
-
#endif
296
-
}
297
-
298
-
#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
299
-
300
-
/*
301
-
* pval = *(ptr+off)
302
-
* *pval += inc;
303
-
*/
304
-
static inline __attribute__((always_inline))
305
-
int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu)
306
-
{
307
-
RSEQ_INJECT_C(9)
308
-
309
-
__asm__ __volatile__ goto (
310
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
311
-
#ifdef RSEQ_COMPARE_TWICE
312
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
313
-
#endif
314
-
/* Start rseq by storing table entry pointer into rseq_cs. */
315
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
316
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
317
-
RSEQ_INJECT_ASM(3)
318
-
#ifdef RSEQ_COMPARE_TWICE
319
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
320
-
#endif
321
-
/* get p+v */
322
-
"movq %[ptr], %%rbx\n\t"
323
-
"addq %[off], %%rbx\n\t"
324
-
/* get pv */
325
-
"movq (%%rbx), %%rcx\n\t"
326
-
/* *pv += inc */
327
-
"addq %[inc], (%%rcx)\n\t"
328
-
"2:\n\t"
329
-
RSEQ_INJECT_ASM(4)
330
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
331
-
: /* gcc asm goto does not allow outputs */
332
-
: [cpu_id] "r" (cpu),
333
-
[rseq_offset] "r" (rseq_offset),
334
-
/* final store input */
335
-
[ptr] "m" (*ptr),
336
-
[off] "er" (off),
337
-
[inc] "er" (inc)
338
-
: "memory", "cc", "rax", "rbx", "rcx"
339
-
RSEQ_INJECT_CLOBBER
340
-
: abort
341
-
#ifdef RSEQ_COMPARE_TWICE
342
-
, error1
343
-
#endif
344
-
);
345
-
return 0;
346
-
abort:
347
-
RSEQ_INJECT_FAILED
348
-
return -1;
349
-
#ifdef RSEQ_COMPARE_TWICE
350
-
error1:
351
-
rseq_bug("cpu_id comparison failed");
352
-
#endif
353
-
}
354
-
355
-
static inline __attribute__((always_inline))
356
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
357
-
intptr_t *v2, intptr_t newv2,
358
-
intptr_t newv, int cpu)
359
-
{
360
-
RSEQ_INJECT_C(9)
361
-
362
-
__asm__ __volatile__ goto (
363
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
364
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
365
-
#ifdef RSEQ_COMPARE_TWICE
366
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
367
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
368
-
#endif
369
-
/* Start rseq by storing table entry pointer into rseq_cs. */
370
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
371
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
372
-
RSEQ_INJECT_ASM(3)
373
-
"cmpq %[v], %[expect]\n\t"
374
-
"jnz %l[cmpfail]\n\t"
375
-
RSEQ_INJECT_ASM(4)
376
-
#ifdef RSEQ_COMPARE_TWICE
377
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
378
-
"cmpq %[v], %[expect]\n\t"
379
-
"jnz %l[error2]\n\t"
380
-
#endif
381
-
/* try store */
382
-
"movq %[newv2], %[v2]\n\t"
383
-
RSEQ_INJECT_ASM(5)
384
-
/* final store */
385
-
"movq %[newv], %[v]\n\t"
386
-
"2:\n\t"
387
-
RSEQ_INJECT_ASM(6)
388
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
389
-
: /* gcc asm goto does not allow outputs */
390
-
: [cpu_id] "r" (cpu),
391
-
[rseq_offset] "r" (rseq_offset),
392
-
/* try store input */
393
-
[v2] "m" (*v2),
394
-
[newv2] "r" (newv2),
395
-
/* final store input */
396
-
[v] "m" (*v),
397
-
[expect] "r" (expect),
398
-
[newv] "r" (newv)
399
-
: "memory", "cc", "rax"
400
-
RSEQ_INJECT_CLOBBER
401
-
: abort, cmpfail
402
-
#ifdef RSEQ_COMPARE_TWICE
403
-
, error1, error2
404
-
#endif
405
-
);
406
-
rseq_after_asm_goto();
407
-
return 0;
408
-
abort:
409
-
rseq_after_asm_goto();
410
-
RSEQ_INJECT_FAILED
411
-
return -1;
412
-
cmpfail:
413
-
rseq_after_asm_goto();
414
-
return 1;
415
-
#ifdef RSEQ_COMPARE_TWICE
416
-
error1:
417
-
rseq_after_asm_goto();
418
-
rseq_bug("cpu_id comparison failed");
419
-
error2:
420
-
rseq_after_asm_goto();
421
-
rseq_bug("expected value comparison failed");
422
-
#endif
423
-
}
424
-
425
-
/* x86-64 is TSO. */
426
-
static inline __attribute__((always_inline))
427
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
428
-
intptr_t *v2, intptr_t newv2,
429
-
intptr_t newv, int cpu)
430
-
{
431
-
return rseq_cmpeqv_trystorev_storev(v, expect, v2, newv2, newv, cpu);
432
-
}
433
-
434
-
static inline __attribute__((always_inline))
435
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
436
-
intptr_t *v2, intptr_t expect2,
437
-
intptr_t newv, int cpu)
438
-
{
439
-
RSEQ_INJECT_C(9)
440
-
441
-
__asm__ __volatile__ goto (
442
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
443
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
444
-
#ifdef RSEQ_COMPARE_TWICE
445
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
446
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
447
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
448
-
#endif
449
-
/* Start rseq by storing table entry pointer into rseq_cs. */
450
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
451
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
452
-
RSEQ_INJECT_ASM(3)
453
-
"cmpq %[v], %[expect]\n\t"
454
-
"jnz %l[cmpfail]\n\t"
455
-
RSEQ_INJECT_ASM(4)
456
-
"cmpq %[v2], %[expect2]\n\t"
457
-
"jnz %l[cmpfail]\n\t"
458
-
RSEQ_INJECT_ASM(5)
459
-
#ifdef RSEQ_COMPARE_TWICE
460
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
461
-
"cmpq %[v], %[expect]\n\t"
462
-
"jnz %l[error2]\n\t"
463
-
"cmpq %[v2], %[expect2]\n\t"
464
-
"jnz %l[error3]\n\t"
465
-
#endif
466
-
/* final store */
467
-
"movq %[newv], %[v]\n\t"
468
-
"2:\n\t"
469
-
RSEQ_INJECT_ASM(6)
470
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
471
-
: /* gcc asm goto does not allow outputs */
472
-
: [cpu_id] "r" (cpu),
473
-
[rseq_offset] "r" (rseq_offset),
474
-
/* cmp2 input */
475
-
[v2] "m" (*v2),
476
-
[expect2] "r" (expect2),
477
-
/* final store input */
478
-
[v] "m" (*v),
479
-
[expect] "r" (expect),
480
-
[newv] "r" (newv)
481
-
: "memory", "cc", "rax"
482
-
RSEQ_INJECT_CLOBBER
483
-
: abort, cmpfail
484
-
#ifdef RSEQ_COMPARE_TWICE
485
-
, error1, error2, error3
486
-
#endif
487
-
);
488
-
rseq_after_asm_goto();
489
-
return 0;
490
-
abort:
491
-
rseq_after_asm_goto();
492
-
RSEQ_INJECT_FAILED
493
-
return -1;
494
-
cmpfail:
495
-
rseq_after_asm_goto();
496
-
return 1;
497
-
#ifdef RSEQ_COMPARE_TWICE
498
-
error1:
499
-
rseq_after_asm_goto();
500
-
rseq_bug("cpu_id comparison failed");
501
-
error2:
502
-
rseq_after_asm_goto();
503
-
rseq_bug("1st expected value comparison failed");
504
-
error3:
505
-
rseq_after_asm_goto();
506
-
rseq_bug("2nd expected value comparison failed");
507
-
#endif
508
-
}
509
-
510
-
static inline __attribute__((always_inline))
511
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
512
-
void *dst, void *src, size_t len,
513
-
intptr_t newv, int cpu)
514
-
{
515
-
uint64_t rseq_scratch[3];
516
-
517
-
RSEQ_INJECT_C(9)
518
-
519
-
__asm__ __volatile__ goto (
520
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
521
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
522
-
#ifdef RSEQ_COMPARE_TWICE
523
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
524
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
525
-
#endif
526
-
"movq %[src], %[rseq_scratch0]\n\t"
527
-
"movq %[dst], %[rseq_scratch1]\n\t"
528
-
"movq %[len], %[rseq_scratch2]\n\t"
529
-
/* Start rseq by storing table entry pointer into rseq_cs. */
530
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
531
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
532
-
RSEQ_INJECT_ASM(3)
533
-
"cmpq %[v], %[expect]\n\t"
534
-
"jnz 5f\n\t"
535
-
RSEQ_INJECT_ASM(4)
536
-
#ifdef RSEQ_COMPARE_TWICE
537
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)
538
-
"cmpq %[v], %[expect]\n\t"
539
-
"jnz 7f\n\t"
540
-
#endif
541
-
/* try memcpy */
542
-
"test %[len], %[len]\n\t" \
543
-
"jz 333f\n\t" \
544
-
"222:\n\t" \
545
-
"movb (%[src]), %%al\n\t" \
546
-
"movb %%al, (%[dst])\n\t" \
547
-
"inc %[src]\n\t" \
548
-
"inc %[dst]\n\t" \
549
-
"dec %[len]\n\t" \
550
-
"jnz 222b\n\t" \
551
-
"333:\n\t" \
552
-
RSEQ_INJECT_ASM(5)
553
-
/* final store */
554
-
"movq %[newv], %[v]\n\t"
555
-
"2:\n\t"
556
-
RSEQ_INJECT_ASM(6)
557
-
/* teardown */
558
-
"movq %[rseq_scratch2], %[len]\n\t"
559
-
"movq %[rseq_scratch1], %[dst]\n\t"
560
-
"movq %[rseq_scratch0], %[src]\n\t"
561
-
RSEQ_ASM_DEFINE_ABORT(4,
562
-
"movq %[rseq_scratch2], %[len]\n\t"
563
-
"movq %[rseq_scratch1], %[dst]\n\t"
564
-
"movq %[rseq_scratch0], %[src]\n\t",
565
-
abort)
566
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
567
-
"movq %[rseq_scratch2], %[len]\n\t"
568
-
"movq %[rseq_scratch1], %[dst]\n\t"
569
-
"movq %[rseq_scratch0], %[src]\n\t",
570
-
cmpfail)
571
-
#ifdef RSEQ_COMPARE_TWICE
572
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
573
-
"movq %[rseq_scratch2], %[len]\n\t"
574
-
"movq %[rseq_scratch1], %[dst]\n\t"
575
-
"movq %[rseq_scratch0], %[src]\n\t",
576
-
error1)
577
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
578
-
"movq %[rseq_scratch2], %[len]\n\t"
579
-
"movq %[rseq_scratch1], %[dst]\n\t"
580
-
"movq %[rseq_scratch0], %[src]\n\t",
581
-
error2)
582
-
#endif
583
-
: /* gcc asm goto does not allow outputs */
584
-
: [cpu_id] "r" (cpu),
585
-
[rseq_offset] "r" (rseq_offset),
586
-
/* final store input */
587
-
[v] "m" (*v),
588
-
[expect] "r" (expect),
589
-
[newv] "r" (newv),
590
-
/* try memcpy input */
591
-
[dst] "r" (dst),
592
-
[src] "r" (src),
593
-
[len] "r" (len),
594
-
[rseq_scratch0] "m" (rseq_scratch[0]),
595
-
[rseq_scratch1] "m" (rseq_scratch[1]),
596
-
[rseq_scratch2] "m" (rseq_scratch[2])
597
-
: "memory", "cc", "rax"
598
-
RSEQ_INJECT_CLOBBER
599
-
: abort, cmpfail
600
-
#ifdef RSEQ_COMPARE_TWICE
601
-
, error1, error2
602
-
#endif
603
-
);
604
-
rseq_after_asm_goto();
605
-
return 0;
606
-
abort:
607
-
rseq_after_asm_goto();
608
-
RSEQ_INJECT_FAILED
609
-
return -1;
610
-
cmpfail:
611
-
rseq_after_asm_goto();
612
-
return 1;
613
-
#ifdef RSEQ_COMPARE_TWICE
614
-
error1:
615
-
rseq_after_asm_goto();
616
-
rseq_bug("cpu_id comparison failed");
617
-
error2:
618
-
rseq_after_asm_goto();
619
-
rseq_bug("expected value comparison failed");
620
-
#endif
621
-
}
622
-
623
-
/* x86-64 is TSO. */
624
-
static inline __attribute__((always_inline))
625
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
626
-
void *dst, void *src, size_t len,
627
-
intptr_t newv, int cpu)
628
-
{
629
-
return rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len,
630
-
newv, cpu);
631
-
}
632
-
633
-
#endif /* !RSEQ_SKIP_FASTPATH */
634
-
635
115
#elif defined(__i386__)
636
116
637
117
#define RSEQ_ASM_TP_SEGMENT %%gs
···
138
656
rseq_smp_mb(); \
139
657
RSEQ_WRITE_ONCE(*p, v); \
140
658
} while (0)
141
-
142
-
#ifdef RSEQ_SKIP_FASTPATH
143
-
#include "rseq-skip.h"
144
-
#else /* !RSEQ_SKIP_FASTPATH */
145
659
146
660
/*
147
661
* Use eax as scratch register and take memory operands as input to
···
199
721
"jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \
200
722
".popsection\n\t"
201
723
202
-
static inline __attribute__((always_inline))
203
-
int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
204
-
{
205
-
RSEQ_INJECT_C(9)
206
-
207
-
__asm__ __volatile__ goto (
208
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
209
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
210
-
#ifdef RSEQ_COMPARE_TWICE
211
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
212
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
213
-
#endif
214
-
/* Start rseq by storing table entry pointer into rseq_cs. */
215
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
216
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
217
-
RSEQ_INJECT_ASM(3)
218
-
"cmpl %[v], %[expect]\n\t"
219
-
"jnz %l[cmpfail]\n\t"
220
-
RSEQ_INJECT_ASM(4)
221
-
#ifdef RSEQ_COMPARE_TWICE
222
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
223
-
"cmpl %[v], %[expect]\n\t"
224
-
"jnz %l[error2]\n\t"
225
-
#endif
226
-
/* final store */
227
-
"movl %[newv], %[v]\n\t"
228
-
"2:\n\t"
229
-
RSEQ_INJECT_ASM(5)
230
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
231
-
: /* gcc asm goto does not allow outputs */
232
-
: [cpu_id] "r" (cpu),
233
-
[rseq_offset] "r" (rseq_offset),
234
-
[v] "m" (*v),
235
-
[expect] "r" (expect),
236
-
[newv] "r" (newv)
237
-
: "memory", "cc", "eax"
238
-
RSEQ_INJECT_CLOBBER
239
-
: abort, cmpfail
240
-
#ifdef RSEQ_COMPARE_TWICE
241
-
, error1, error2
242
-
#endif
243
-
);
244
-
rseq_after_asm_goto();
245
-
return 0;
246
-
abort:
247
-
rseq_after_asm_goto();
248
-
RSEQ_INJECT_FAILED
249
-
return -1;
250
-
cmpfail:
251
-
rseq_after_asm_goto();
252
-
return 1;
253
-
#ifdef RSEQ_COMPARE_TWICE
254
-
error1:
255
-
rseq_after_asm_goto();
256
-
rseq_bug("cpu_id comparison failed");
257
-
error2:
258
-
rseq_after_asm_goto();
259
-
rseq_bug("expected value comparison failed");
260
-
#endif
261
-
}
262
-
263
-
/*
264
-
* Compare @v against @expectnot. When it does _not_ match, load @v
265
-
* into @load, and store the content of *@v + voffp into @v.
266
-
*/
267
-
static inline __attribute__((always_inline))
268
-
int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
269
-
long voffp, intptr_t *load, int cpu)
270
-
{
271
-
RSEQ_INJECT_C(9)
272
-
273
-
__asm__ __volatile__ goto (
274
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
275
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
276
-
#ifdef RSEQ_COMPARE_TWICE
277
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
278
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
279
-
#endif
280
-
/* Start rseq by storing table entry pointer into rseq_cs. */
281
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
282
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
283
-
RSEQ_INJECT_ASM(3)
284
-
"movl %[v], %%ebx\n\t"
285
-
"cmpl %%ebx, %[expectnot]\n\t"
286
-
"je %l[cmpfail]\n\t"
287
-
RSEQ_INJECT_ASM(4)
288
-
#ifdef RSEQ_COMPARE_TWICE
289
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
290
-
"movl %[v], %%ebx\n\t"
291
-
"cmpl %%ebx, %[expectnot]\n\t"
292
-
"je %l[error2]\n\t"
293
-
#endif
294
-
"movl %%ebx, %[load]\n\t"
295
-
"addl %[voffp], %%ebx\n\t"
296
-
"movl (%%ebx), %%ebx\n\t"
297
-
/* final store */
298
-
"movl %%ebx, %[v]\n\t"
299
-
"2:\n\t"
300
-
RSEQ_INJECT_ASM(5)
301
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
302
-
: /* gcc asm goto does not allow outputs */
303
-
: [cpu_id] "r" (cpu),
304
-
[rseq_offset] "r" (rseq_offset),
305
-
/* final store input */
306
-
[v] "m" (*v),
307
-
[expectnot] "r" (expectnot),
308
-
[voffp] "ir" (voffp),
309
-
[load] "m" (*load)
310
-
: "memory", "cc", "eax", "ebx"
311
-
RSEQ_INJECT_CLOBBER
312
-
: abort, cmpfail
313
-
#ifdef RSEQ_COMPARE_TWICE
314
-
, error1, error2
315
-
#endif
316
-
);
317
-
rseq_after_asm_goto();
318
-
return 0;
319
-
abort:
320
-
rseq_after_asm_goto();
321
-
RSEQ_INJECT_FAILED
322
-
return -1;
323
-
cmpfail:
324
-
rseq_after_asm_goto();
325
-
return 1;
326
-
#ifdef RSEQ_COMPARE_TWICE
327
-
error1:
328
-
rseq_after_asm_goto();
329
-
rseq_bug("cpu_id comparison failed");
330
-
error2:
331
-
rseq_after_asm_goto();
332
-
rseq_bug("expected value comparison failed");
333
-
#endif
334
-
}
335
-
336
-
static inline __attribute__((always_inline))
337
-
int rseq_addv(intptr_t *v, intptr_t count, int cpu)
338
-
{
339
-
RSEQ_INJECT_C(9)
340
-
341
-
__asm__ __volatile__ goto (
342
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
343
-
#ifdef RSEQ_COMPARE_TWICE
344
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
345
-
#endif
346
-
/* Start rseq by storing table entry pointer into rseq_cs. */
347
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
348
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
349
-
RSEQ_INJECT_ASM(3)
350
-
#ifdef RSEQ_COMPARE_TWICE
351
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
352
-
#endif
353
-
/* final store */
354
-
"addl %[count], %[v]\n\t"
355
-
"2:\n\t"
356
-
RSEQ_INJECT_ASM(4)
357
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
358
-
: /* gcc asm goto does not allow outputs */
359
-
: [cpu_id] "r" (cpu),
360
-
[rseq_offset] "r" (rseq_offset),
361
-
/* final store input */
362
-
[v] "m" (*v),
363
-
[count] "ir" (count)
364
-
: "memory", "cc", "eax"
365
-
RSEQ_INJECT_CLOBBER
366
-
: abort
367
-
#ifdef RSEQ_COMPARE_TWICE
368
-
, error1
369
-
#endif
370
-
);
371
-
rseq_after_asm_goto();
372
-
return 0;
373
-
abort:
374
-
rseq_after_asm_goto();
375
-
RSEQ_INJECT_FAILED
376
-
return -1;
377
-
#ifdef RSEQ_COMPARE_TWICE
378
-
error1:
379
-
rseq_after_asm_goto();
380
-
rseq_bug("cpu_id comparison failed");
381
-
#endif
382
-
}
383
-
384
-
static inline __attribute__((always_inline))
385
-
int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
386
-
intptr_t *v2, intptr_t newv2,
387
-
intptr_t newv, int cpu)
388
-
{
389
-
RSEQ_INJECT_C(9)
390
-
391
-
__asm__ __volatile__ goto (
392
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
393
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
394
-
#ifdef RSEQ_COMPARE_TWICE
395
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
396
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
397
-
#endif
398
-
/* Start rseq by storing table entry pointer into rseq_cs. */
399
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
400
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
401
-
RSEQ_INJECT_ASM(3)
402
-
"cmpl %[v], %[expect]\n\t"
403
-
"jnz %l[cmpfail]\n\t"
404
-
RSEQ_INJECT_ASM(4)
405
-
#ifdef RSEQ_COMPARE_TWICE
406
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
407
-
"cmpl %[v], %[expect]\n\t"
408
-
"jnz %l[error2]\n\t"
409
-
#endif
410
-
/* try store */
411
-
"movl %[newv2], %%eax\n\t"
412
-
"movl %%eax, %[v2]\n\t"
413
-
RSEQ_INJECT_ASM(5)
414
-
/* final store */
415
-
"movl %[newv], %[v]\n\t"
416
-
"2:\n\t"
417
-
RSEQ_INJECT_ASM(6)
418
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
419
-
: /* gcc asm goto does not allow outputs */
420
-
: [cpu_id] "r" (cpu),
421
-
[rseq_offset] "r" (rseq_offset),
422
-
/* try store input */
423
-
[v2] "m" (*v2),
424
-
[newv2] "m" (newv2),
425
-
/* final store input */
426
-
[v] "m" (*v),
427
-
[expect] "r" (expect),
428
-
[newv] "r" (newv)
429
-
: "memory", "cc", "eax"
430
-
RSEQ_INJECT_CLOBBER
431
-
: abort, cmpfail
432
-
#ifdef RSEQ_COMPARE_TWICE
433
-
, error1, error2
434
-
#endif
435
-
);
436
-
rseq_after_asm_goto();
437
-
return 0;
438
-
abort:
439
-
rseq_after_asm_goto();
440
-
RSEQ_INJECT_FAILED
441
-
return -1;
442
-
cmpfail:
443
-
rseq_after_asm_goto();
444
-
return 1;
445
-
#ifdef RSEQ_COMPARE_TWICE
446
-
error1:
447
-
rseq_after_asm_goto();
448
-
rseq_bug("cpu_id comparison failed");
449
-
error2:
450
-
rseq_after_asm_goto();
451
-
rseq_bug("expected value comparison failed");
452
-
#endif
453
-
}
454
-
455
-
static inline __attribute__((always_inline))
456
-
int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
457
-
intptr_t *v2, intptr_t newv2,
458
-
intptr_t newv, int cpu)
459
-
{
460
-
RSEQ_INJECT_C(9)
461
-
462
-
__asm__ __volatile__ goto (
463
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
464
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
465
-
#ifdef RSEQ_COMPARE_TWICE
466
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
467
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
468
-
#endif
469
-
/* Start rseq by storing table entry pointer into rseq_cs. */
470
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
471
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
472
-
RSEQ_INJECT_ASM(3)
473
-
"movl %[expect], %%eax\n\t"
474
-
"cmpl %[v], %%eax\n\t"
475
-
"jnz %l[cmpfail]\n\t"
476
-
RSEQ_INJECT_ASM(4)
477
-
#ifdef RSEQ_COMPARE_TWICE
478
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
479
-
"movl %[expect], %%eax\n\t"
480
-
"cmpl %[v], %%eax\n\t"
481
-
"jnz %l[error2]\n\t"
482
-
#endif
483
-
/* try store */
484
-
"movl %[newv2], %[v2]\n\t"
485
-
RSEQ_INJECT_ASM(5)
486
-
"lock; addl $0,-128(%%esp)\n\t"
487
-
/* final store */
488
-
"movl %[newv], %[v]\n\t"
489
-
"2:\n\t"
490
-
RSEQ_INJECT_ASM(6)
491
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
492
-
: /* gcc asm goto does not allow outputs */
493
-
: [cpu_id] "r" (cpu),
494
-
[rseq_offset] "r" (rseq_offset),
495
-
/* try store input */
496
-
[v2] "m" (*v2),
497
-
[newv2] "r" (newv2),
498
-
/* final store input */
499
-
[v] "m" (*v),
500
-
[expect] "m" (expect),
501
-
[newv] "r" (newv)
502
-
: "memory", "cc", "eax"
503
-
RSEQ_INJECT_CLOBBER
504
-
: abort, cmpfail
505
-
#ifdef RSEQ_COMPARE_TWICE
506
-
, error1, error2
507
-
#endif
508
-
);
509
-
rseq_after_asm_goto();
510
-
return 0;
511
-
abort:
512
-
rseq_after_asm_goto();
513
-
RSEQ_INJECT_FAILED
514
-
return -1;
515
-
cmpfail:
516
-
rseq_after_asm_goto();
517
-
return 1;
518
-
#ifdef RSEQ_COMPARE_TWICE
519
-
error1:
520
-
rseq_after_asm_goto();
521
-
rseq_bug("cpu_id comparison failed");
522
-
error2:
523
-
rseq_after_asm_goto();
524
-
rseq_bug("expected value comparison failed");
525
724
#endif
526
725
527
-
}
726
+
/* Per-cpu-id indexing. */
528
727
529
-
static inline __attribute__((always_inline))
530
-
int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
531
-
intptr_t *v2, intptr_t expect2,
532
-
intptr_t newv, int cpu)
533
-
{
534
-
RSEQ_INJECT_C(9)
728
+
#define RSEQ_TEMPLATE_CPU_ID
729
+
#define RSEQ_TEMPLATE_MO_RELAXED
730
+
#include "rseq-x86-bits.h"
731
+
#undef RSEQ_TEMPLATE_MO_RELAXED
535
732
536
-
__asm__ __volatile__ goto (
537
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
538
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
539
-
#ifdef RSEQ_COMPARE_TWICE
540
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
541
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
542
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])
543
-
#endif
544
-
/* Start rseq by storing table entry pointer into rseq_cs. */
545
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
546
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
547
-
RSEQ_INJECT_ASM(3)
548
-
"cmpl %[v], %[expect]\n\t"
549
-
"jnz %l[cmpfail]\n\t"
550
-
RSEQ_INJECT_ASM(4)
551
-
"cmpl %[expect2], %[v2]\n\t"
552
-
"jnz %l[cmpfail]\n\t"
553
-
RSEQ_INJECT_ASM(5)
554
-
#ifdef RSEQ_COMPARE_TWICE
555
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])
556
-
"cmpl %[v], %[expect]\n\t"
557
-
"jnz %l[error2]\n\t"
558
-
"cmpl %[expect2], %[v2]\n\t"
559
-
"jnz %l[error3]\n\t"
560
-
#endif
561
-
"movl %[newv], %%eax\n\t"
562
-
/* final store */
563
-
"movl %%eax, %[v]\n\t"
564
-
"2:\n\t"
565
-
RSEQ_INJECT_ASM(6)
566
-
RSEQ_ASM_DEFINE_ABORT(4, "", abort)
567
-
: /* gcc asm goto does not allow outputs */
568
-
: [cpu_id] "r" (cpu),
569
-
[rseq_offset] "r" (rseq_offset),
570
-
/* cmp2 input */
571
-
[v2] "m" (*v2),
572
-
[expect2] "r" (expect2),
573
-
/* final store input */
574
-
[v] "m" (*v),
575
-
[expect] "r" (expect),
576
-
[newv] "m" (newv)
577
-
: "memory", "cc", "eax"
578
-
RSEQ_INJECT_CLOBBER
579
-
: abort, cmpfail
580
-
#ifdef RSEQ_COMPARE_TWICE
581
-
, error1, error2, error3
582
-
#endif
583
-
);
584
-
rseq_after_asm_goto();
585
-
return 0;
586
-
abort:
587
-
rseq_after_asm_goto();
588
-
RSEQ_INJECT_FAILED
589
-
return -1;
590
-
cmpfail:
591
-
rseq_after_asm_goto();
592
-
return 1;
593
-
#ifdef RSEQ_COMPARE_TWICE
594
-
error1:
595
-
rseq_after_asm_goto();
596
-
rseq_bug("cpu_id comparison failed");
597
-
error2:
598
-
rseq_after_asm_goto();
599
-
rseq_bug("1st expected value comparison failed");
600
-
error3:
601
-
rseq_after_asm_goto();
602
-
rseq_bug("2nd expected value comparison failed");
603
-
#endif
604
-
}
733
+
#define RSEQ_TEMPLATE_MO_RELEASE
734
+
#include "rseq-x86-bits.h"
735
+
#undef RSEQ_TEMPLATE_MO_RELEASE
736
+
#undef RSEQ_TEMPLATE_CPU_ID
605
737
606
-
/* TODO: implement a faster memcpy. */
607
-
static inline __attribute__((always_inline))
608
-
int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
609
-
void *dst, void *src, size_t len,
610
-
intptr_t newv, int cpu)
611
-
{
612
-
uint32_t rseq_scratch[3];
738
+
/* Per-mm-cid indexing. */
613
739
614
-
RSEQ_INJECT_C(9)
740
+
#define RSEQ_TEMPLATE_MM_CID
741
+
#define RSEQ_TEMPLATE_MO_RELAXED
742
+
#include "rseq-x86-bits.h"
743
+
#undef RSEQ_TEMPLATE_MO_RELAXED
615
744
616
-
__asm__ __volatile__ goto (
617
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
618
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
619
-
#ifdef RSEQ_COMPARE_TWICE
620
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
621
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
622
-
#endif
623
-
"movl %[src], %[rseq_scratch0]\n\t"
624
-
"movl %[dst], %[rseq_scratch1]\n\t"
625
-
"movl %[len], %[rseq_scratch2]\n\t"
626
-
/* Start rseq by storing table entry pointer into rseq_cs. */
627
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
628
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
629
-
RSEQ_INJECT_ASM(3)
630
-
"movl %[expect], %%eax\n\t"
631
-
"cmpl %%eax, %[v]\n\t"
632
-
"jnz 5f\n\t"
633
-
RSEQ_INJECT_ASM(4)
634
-
#ifdef RSEQ_COMPARE_TWICE
635
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)
636
-
"movl %[expect], %%eax\n\t"
637
-
"cmpl %%eax, %[v]\n\t"
638
-
"jnz 7f\n\t"
639
-
#endif
640
-
/* try memcpy */
641
-
"test %[len], %[len]\n\t" \
642
-
"jz 333f\n\t" \
643
-
"222:\n\t" \
644
-
"movb (%[src]), %%al\n\t" \
645
-
"movb %%al, (%[dst])\n\t" \
646
-
"inc %[src]\n\t" \
647
-
"inc %[dst]\n\t" \
648
-
"dec %[len]\n\t" \
649
-
"jnz 222b\n\t" \
650
-
"333:\n\t" \
651
-
RSEQ_INJECT_ASM(5)
652
-
"movl %[newv], %%eax\n\t"
653
-
/* final store */
654
-
"movl %%eax, %[v]\n\t"
655
-
"2:\n\t"
656
-
RSEQ_INJECT_ASM(6)
657
-
/* teardown */
658
-
"movl %[rseq_scratch2], %[len]\n\t"
659
-
"movl %[rseq_scratch1], %[dst]\n\t"
660
-
"movl %[rseq_scratch0], %[src]\n\t"
661
-
RSEQ_ASM_DEFINE_ABORT(4,
662
-
"movl %[rseq_scratch2], %[len]\n\t"
663
-
"movl %[rseq_scratch1], %[dst]\n\t"
664
-
"movl %[rseq_scratch0], %[src]\n\t",
665
-
abort)
666
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
667
-
"movl %[rseq_scratch2], %[len]\n\t"
668
-
"movl %[rseq_scratch1], %[dst]\n\t"
669
-
"movl %[rseq_scratch0], %[src]\n\t",
670
-
cmpfail)
671
-
#ifdef RSEQ_COMPARE_TWICE
672
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
673
-
"movl %[rseq_scratch2], %[len]\n\t"
674
-
"movl %[rseq_scratch1], %[dst]\n\t"
675
-
"movl %[rseq_scratch0], %[src]\n\t",
676
-
error1)
677
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
678
-
"movl %[rseq_scratch2], %[len]\n\t"
679
-
"movl %[rseq_scratch1], %[dst]\n\t"
680
-
"movl %[rseq_scratch0], %[src]\n\t",
681
-
error2)
682
-
#endif
683
-
: /* gcc asm goto does not allow outputs */
684
-
: [cpu_id] "r" (cpu),
685
-
[rseq_offset] "r" (rseq_offset),
686
-
/* final store input */
687
-
[v] "m" (*v),
688
-
[expect] "m" (expect),
689
-
[newv] "m" (newv),
690
-
/* try memcpy input */
691
-
[dst] "r" (dst),
692
-
[src] "r" (src),
693
-
[len] "r" (len),
694
-
[rseq_scratch0] "m" (rseq_scratch[0]),
695
-
[rseq_scratch1] "m" (rseq_scratch[1]),
696
-
[rseq_scratch2] "m" (rseq_scratch[2])
697
-
: "memory", "cc", "eax"
698
-
RSEQ_INJECT_CLOBBER
699
-
: abort, cmpfail
700
-
#ifdef RSEQ_COMPARE_TWICE
701
-
, error1, error2
702
-
#endif
703
-
);
704
-
rseq_after_asm_goto();
705
-
return 0;
706
-
abort:
707
-
rseq_after_asm_goto();
708
-
RSEQ_INJECT_FAILED
709
-
return -1;
710
-
cmpfail:
711
-
rseq_after_asm_goto();
712
-
return 1;
713
-
#ifdef RSEQ_COMPARE_TWICE
714
-
error1:
715
-
rseq_after_asm_goto();
716
-
rseq_bug("cpu_id comparison failed");
717
-
error2:
718
-
rseq_after_asm_goto();
719
-
rseq_bug("expected value comparison failed");
720
-
#endif
721
-
}
745
+
#define RSEQ_TEMPLATE_MO_RELEASE
746
+
#include "rseq-x86-bits.h"
747
+
#undef RSEQ_TEMPLATE_MO_RELEASE
748
+
#undef RSEQ_TEMPLATE_MM_CID
722
749
723
-
/* TODO: implement a faster memcpy. */
724
-
static inline __attribute__((always_inline))
725
-
int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
726
-
void *dst, void *src, size_t len,
727
-
intptr_t newv, int cpu)
728
-
{
729
-
uint32_t rseq_scratch[3];
750
+
/* APIs which are not based on cpu ids. */
730
751
731
-
RSEQ_INJECT_C(9)
732
-
733
-
__asm__ __volatile__ goto (
734
-
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
735
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
736
-
#ifdef RSEQ_COMPARE_TWICE
737
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])
738
-
RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])
739
-
#endif
740
-
"movl %[src], %[rseq_scratch0]\n\t"
741
-
"movl %[dst], %[rseq_scratch1]\n\t"
742
-
"movl %[len], %[rseq_scratch2]\n\t"
743
-
/* Start rseq by storing table entry pointer into rseq_cs. */
744
-
RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset]))
745
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f)
746
-
RSEQ_INJECT_ASM(3)
747
-
"movl %[expect], %%eax\n\t"
748
-
"cmpl %%eax, %[v]\n\t"
749
-
"jnz 5f\n\t"
750
-
RSEQ_INJECT_ASM(4)
751
-
#ifdef RSEQ_COMPARE_TWICE
752
-
RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)
753
-
"movl %[expect], %%eax\n\t"
754
-
"cmpl %%eax, %[v]\n\t"
755
-
"jnz 7f\n\t"
756
-
#endif
757
-
/* try memcpy */
758
-
"test %[len], %[len]\n\t" \
759
-
"jz 333f\n\t" \
760
-
"222:\n\t" \
761
-
"movb (%[src]), %%al\n\t" \
762
-
"movb %%al, (%[dst])\n\t" \
763
-
"inc %[src]\n\t" \
764
-
"inc %[dst]\n\t" \
765
-
"dec %[len]\n\t" \
766
-
"jnz 222b\n\t" \
767
-
"333:\n\t" \
768
-
RSEQ_INJECT_ASM(5)
769
-
"lock; addl $0,-128(%%esp)\n\t"
770
-
"movl %[newv], %%eax\n\t"
771
-
/* final store */
772
-
"movl %%eax, %[v]\n\t"
773
-
"2:\n\t"
774
-
RSEQ_INJECT_ASM(6)
775
-
/* teardown */
776
-
"movl %[rseq_scratch2], %[len]\n\t"
777
-
"movl %[rseq_scratch1], %[dst]\n\t"
778
-
"movl %[rseq_scratch0], %[src]\n\t"
779
-
RSEQ_ASM_DEFINE_ABORT(4,
780
-
"movl %[rseq_scratch2], %[len]\n\t"
781
-
"movl %[rseq_scratch1], %[dst]\n\t"
782
-
"movl %[rseq_scratch0], %[src]\n\t",
783
-
abort)
784
-
RSEQ_ASM_DEFINE_CMPFAIL(5,
785
-
"movl %[rseq_scratch2], %[len]\n\t"
786
-
"movl %[rseq_scratch1], %[dst]\n\t"
787
-
"movl %[rseq_scratch0], %[src]\n\t",
788
-
cmpfail)
789
-
#ifdef RSEQ_COMPARE_TWICE
790
-
RSEQ_ASM_DEFINE_CMPFAIL(6,
791
-
"movl %[rseq_scratch2], %[len]\n\t"
792
-
"movl %[rseq_scratch1], %[dst]\n\t"
793
-
"movl %[rseq_scratch0], %[src]\n\t",
794
-
error1)
795
-
RSEQ_ASM_DEFINE_CMPFAIL(7,
796
-
"movl %[rseq_scratch2], %[len]\n\t"
797
-
"movl %[rseq_scratch1], %[dst]\n\t"
798
-
"movl %[rseq_scratch0], %[src]\n\t",
799
-
error2)
800
-
#endif
801
-
: /* gcc asm goto does not allow outputs */
802
-
: [cpu_id] "r" (cpu),
803
-
[rseq_offset] "r" (rseq_offset),
804
-
/* final store input */
805
-
[v] "m" (*v),
806
-
[expect] "m" (expect),
807
-
[newv] "m" (newv),
808
-
/* try memcpy input */
809
-
[dst] "r" (dst),
810
-
[src] "r" (src),
811
-
[len] "r" (len),
812
-
[rseq_scratch0] "m" (rseq_scratch[0]),
813
-
[rseq_scratch1] "m" (rseq_scratch[1]),
814
-
[rseq_scratch2] "m" (rseq_scratch[2])
815
-
: "memory", "cc", "eax"
816
-
RSEQ_INJECT_CLOBBER
817
-
: abort, cmpfail
818
-
#ifdef RSEQ_COMPARE_TWICE
819
-
, error1, error2
820
-
#endif
821
-
);
822
-
rseq_after_asm_goto();
823
-
return 0;
824
-
abort:
825
-
rseq_after_asm_goto();
826
-
RSEQ_INJECT_FAILED
827
-
return -1;
828
-
cmpfail:
829
-
rseq_after_asm_goto();
830
-
return 1;
831
-
#ifdef RSEQ_COMPARE_TWICE
832
-
error1:
833
-
rseq_after_asm_goto();
834
-
rseq_bug("cpu_id comparison failed");
835
-
error2:
836
-
rseq_after_asm_goto();
837
-
rseq_bug("expected value comparison failed");
838
-
#endif
839
-
}
840
-
841
-
#endif /* !RSEQ_SKIP_FASTPATH */
842
-
843
-
#endif
752
+
#define RSEQ_TEMPLATE_CPU_ID_NONE
753
+
#define RSEQ_TEMPLATE_MO_RELAXED
754
+
#include "rseq-x86-bits.h"
755
+
#undef RSEQ_TEMPLATE_MO_RELAXED
756
+
#undef RSEQ_TEMPLATE_CPU_ID_NONE
+81
-10
tools/testing/selftests/rseq/rseq.c
+81
-10
tools/testing/selftests/rseq/rseq.c
···
28
28
#include <limits.h>
29
29
#include <dlfcn.h>
30
30
#include <stddef.h>
31
+
#include <sys/auxv.h>
32
+
#include <linux/auxvec.h>
31
33
32
34
#include "../kselftest.h"
33
35
#include "rseq.h"
···
38
36
static const unsigned int *libc_rseq_size_p;
39
37
static const unsigned int *libc_rseq_flags_p;
40
38
41
-
/* Offset from the thread pointer to the rseq area. */
39
+
/* Offset from the thread pointer to the rseq area. */
42
40
ptrdiff_t rseq_offset;
43
41
44
-
/* Size of the registered rseq area. 0 if the registration was
45
-
unsuccessful. */
42
+
/*
43
+
* Size of the registered rseq area. 0 if the registration was
44
+
* unsuccessful.
45
+
*/
46
46
unsigned int rseq_size = -1U;
47
47
48
48
/* Flags used during rseq registration. */
49
49
unsigned int rseq_flags;
50
50
51
+
/*
52
+
* rseq feature size supported by the kernel. 0 if the registration was
53
+
* unsuccessful.
54
+
*/
55
+
unsigned int rseq_feature_size = -1U;
56
+
51
57
static int rseq_ownership;
58
+
static int rseq_reg_success; /* At least one rseq registration has succeded. */
59
+
60
+
/* Allocate a large area for the TLS. */
61
+
#define RSEQ_THREAD_AREA_ALLOC_SIZE 1024
62
+
63
+
/* Original struct rseq feature size is 20 bytes. */
64
+
#define ORIG_RSEQ_FEATURE_SIZE 20
65
+
66
+
/* Original struct rseq allocation size is 32 bytes. */
67
+
#define ORIG_RSEQ_ALLOC_SIZE 32
52
68
53
69
static
54
-
__thread struct rseq_abi __rseq_abi __attribute__((tls_model("initial-exec"))) = {
70
+
__thread struct rseq_abi __rseq_abi __attribute__((tls_model("initial-exec"), aligned(RSEQ_THREAD_AREA_ALLOC_SIZE))) = {
55
71
.cpu_id = RSEQ_ABI_CPU_ID_UNINITIALIZED,
56
72
};
57
73
···
77
57
int flags, uint32_t sig)
78
58
{
79
59
return syscall(__NR_rseq, rseq_abi, rseq_len, flags, sig);
60
+
}
61
+
62
+
static int sys_getcpu(unsigned *cpu, unsigned *node)
63
+
{
64
+
return syscall(__NR_getcpu, cpu, node, NULL);
80
65
}
81
66
82
67
int rseq_available(void)
···
109
84
/* Treat libc's ownership as a successful registration. */
110
85
return 0;
111
86
}
112
-
rc = sys_rseq(&__rseq_abi, sizeof(struct rseq_abi), 0, RSEQ_SIG);
113
-
if (rc)
87
+
rc = sys_rseq(&__rseq_abi, rseq_size, 0, RSEQ_SIG);
88
+
if (rc) {
89
+
if (RSEQ_READ_ONCE(rseq_reg_success)) {
90
+
/* Incoherent success/failure within process. */
91
+
abort();
92
+
}
114
93
return -1;
94
+
}
115
95
assert(rseq_current_cpu_raw() >= 0);
96
+
RSEQ_WRITE_ONCE(rseq_reg_success, 1);
116
97
return 0;
117
98
}
118
99
···
130
99
/* Treat libc's ownership as a successful unregistration. */
131
100
return 0;
132
101
}
133
-
rc = sys_rseq(&__rseq_abi, sizeof(struct rseq_abi), RSEQ_ABI_FLAG_UNREGISTER, RSEQ_SIG);
102
+
rc = sys_rseq(&__rseq_abi, rseq_size, RSEQ_ABI_FLAG_UNREGISTER, RSEQ_SIG);
134
103
if (rc)
135
104
return -1;
136
105
return 0;
106
+
}
107
+
108
+
static
109
+
unsigned int get_rseq_feature_size(void)
110
+
{
111
+
unsigned long auxv_rseq_feature_size, auxv_rseq_align;
112
+
113
+
auxv_rseq_align = getauxval(AT_RSEQ_ALIGN);
114
+
assert(!auxv_rseq_align || auxv_rseq_align <= RSEQ_THREAD_AREA_ALLOC_SIZE);
115
+
116
+
auxv_rseq_feature_size = getauxval(AT_RSEQ_FEATURE_SIZE);
117
+
assert(!auxv_rseq_feature_size || auxv_rseq_feature_size <= RSEQ_THREAD_AREA_ALLOC_SIZE);
118
+
if (auxv_rseq_feature_size)
119
+
return auxv_rseq_feature_size;
120
+
else
121
+
return ORIG_RSEQ_FEATURE_SIZE;
137
122
}
138
123
139
124
static __attribute__((constructor))
···
164
117
rseq_offset = *libc_rseq_offset_p;
165
118
rseq_size = *libc_rseq_size_p;
166
119
rseq_flags = *libc_rseq_flags_p;
120
+
rseq_feature_size = get_rseq_feature_size();
121
+
if (rseq_feature_size > rseq_size)
122
+
rseq_feature_size = rseq_size;
167
123
return;
168
124
}
169
-
if (!rseq_available())
170
-
return;
171
125
rseq_ownership = 1;
126
+
if (!rseq_available()) {
127
+
rseq_size = 0;
128
+
rseq_feature_size = 0;
129
+
return;
130
+
}
172
131
rseq_offset = (void *)&__rseq_abi - rseq_thread_pointer();
173
-
rseq_size = sizeof(struct rseq_abi);
174
132
rseq_flags = 0;
133
+
rseq_feature_size = get_rseq_feature_size();
134
+
if (rseq_feature_size == ORIG_RSEQ_FEATURE_SIZE)
135
+
rseq_size = ORIG_RSEQ_ALLOC_SIZE;
136
+
else
137
+
rseq_size = RSEQ_THREAD_AREA_ALLOC_SIZE;
175
138
}
176
139
177
140
static __attribute__((destructor))
···
191
134
return;
192
135
rseq_offset = 0;
193
136
rseq_size = -1U;
137
+
rseq_feature_size = -1U;
194
138
rseq_ownership = 0;
195
139
}
196
140
···
205
147
abort();
206
148
}
207
149
return cpu;
150
+
}
151
+
152
+
int32_t rseq_fallback_current_node(void)
153
+
{
154
+
uint32_t cpu_id, node_id;
155
+
int ret;
156
+
157
+
ret = sys_getcpu(&cpu_id, &node_id);
158
+
if (ret) {
159
+
perror("sys_getcpu()");
160
+
return ret;
161
+
}
162
+
return (int32_t) node_id;
208
163
}
+211
-4
tools/testing/selftests/rseq/rseq.h
+211
-4
tools/testing/selftests/rseq/rseq.h
···
20
20
#include "rseq-abi.h"
21
21
#include "compiler.h"
22
22
23
+
#ifndef rseq_sizeof_field
24
+
#define rseq_sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER))
25
+
#endif
26
+
27
+
#ifndef rseq_offsetofend
28
+
#define rseq_offsetofend(TYPE, MEMBER) \
29
+
(offsetof(TYPE, MEMBER) + rseq_sizeof_field(TYPE, MEMBER))
30
+
#endif
31
+
23
32
/*
24
33
* Empty code injection macros, override when testing.
25
34
* It is important to consider that the ASM injection macros need to be
···
56
47
57
48
#include "rseq-thread-pointer.h"
58
49
59
-
/* Offset from the thread pointer to the rseq area. */
50
+
/* Offset from the thread pointer to the rseq area. */
60
51
extern ptrdiff_t rseq_offset;
61
-
/* Size of the registered rseq area. 0 if the registration was
62
-
unsuccessful. */
52
+
53
+
/*
54
+
* Size of the registered rseq area. 0 if the registration was
55
+
* unsuccessful.
56
+
*/
63
57
extern unsigned int rseq_size;
64
-
/* Flags used during rseq registration. */
58
+
59
+
/* Flags used during rseq registration. */
65
60
extern unsigned int rseq_flags;
61
+
62
+
/*
63
+
* rseq feature size supported by the kernel. 0 if the registration was
64
+
* unsuccessful.
65
+
*/
66
+
extern unsigned int rseq_feature_size;
67
+
68
+
enum rseq_mo {
69
+
RSEQ_MO_RELAXED = 0,
70
+
RSEQ_MO_CONSUME = 1, /* Unused */
71
+
RSEQ_MO_ACQUIRE = 2, /* Unused */
72
+
RSEQ_MO_RELEASE = 3,
73
+
RSEQ_MO_ACQ_REL = 4, /* Unused */
74
+
RSEQ_MO_SEQ_CST = 5, /* Unused */
75
+
};
76
+
77
+
enum rseq_percpu_mode {
78
+
RSEQ_PERCPU_CPU_ID = 0,
79
+
RSEQ_PERCPU_MM_CID = 1,
80
+
};
66
81
67
82
static inline struct rseq_abi *rseq_get_abi(void)
68
83
{
···
152
119
int32_t rseq_fallback_current_cpu(void);
153
120
154
121
/*
122
+
* Restartable sequence fallback for reading the current node number.
123
+
*/
124
+
int32_t rseq_fallback_current_node(void);
125
+
126
+
/*
155
127
* Values returned can be either the current CPU number, -1 (rseq is
156
128
* uninitialized), or -2 (rseq initialization has failed).
157
129
*/
···
191
153
return cpu;
192
154
}
193
155
156
+
static inline bool rseq_node_id_available(void)
157
+
{
158
+
return (int) rseq_feature_size >= rseq_offsetofend(struct rseq_abi, node_id);
159
+
}
160
+
161
+
/*
162
+
* Current NUMA node number.
163
+
*/
164
+
static inline uint32_t rseq_current_node_id(void)
165
+
{
166
+
assert(rseq_node_id_available());
167
+
return RSEQ_ACCESS_ONCE(rseq_get_abi()->node_id);
168
+
}
169
+
170
+
static inline bool rseq_mm_cid_available(void)
171
+
{
172
+
return (int) rseq_feature_size >= rseq_offsetofend(struct rseq_abi, mm_cid);
173
+
}
174
+
175
+
static inline uint32_t rseq_current_mm_cid(void)
176
+
{
177
+
return RSEQ_ACCESS_ONCE(rseq_get_abi()->mm_cid);
178
+
}
179
+
194
180
static inline void rseq_clear_rseq_cs(void)
195
181
{
196
182
RSEQ_WRITE_ONCE(rseq_get_abi()->rseq_cs.arch.ptr, 0);
···
234
172
static inline void rseq_prepare_unload(void)
235
173
{
236
174
rseq_clear_rseq_cs();
175
+
}
176
+
177
+
static inline __attribute__((always_inline))
178
+
int rseq_cmpeqv_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
179
+
intptr_t *v, intptr_t expect,
180
+
intptr_t newv, int cpu)
181
+
{
182
+
if (rseq_mo != RSEQ_MO_RELAXED)
183
+
return -1;
184
+
switch (percpu_mode) {
185
+
case RSEQ_PERCPU_CPU_ID:
186
+
return rseq_cmpeqv_storev_relaxed_cpu_id(v, expect, newv, cpu);
187
+
case RSEQ_PERCPU_MM_CID:
188
+
return rseq_cmpeqv_storev_relaxed_mm_cid(v, expect, newv, cpu);
189
+
}
190
+
return -1;
191
+
}
192
+
193
+
/*
194
+
* Compare @v against @expectnot. When it does _not_ match, load @v
195
+
* into @load, and store the content of *@v + voffp into @v.
196
+
*/
197
+
static inline __attribute__((always_inline))
198
+
int rseq_cmpnev_storeoffp_load(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
199
+
intptr_t *v, intptr_t expectnot, long voffp, intptr_t *load,
200
+
int cpu)
201
+
{
202
+
if (rseq_mo != RSEQ_MO_RELAXED)
203
+
return -1;
204
+
switch (percpu_mode) {
205
+
case RSEQ_PERCPU_CPU_ID:
206
+
return rseq_cmpnev_storeoffp_load_relaxed_cpu_id(v, expectnot, voffp, load, cpu);
207
+
case RSEQ_PERCPU_MM_CID:
208
+
return rseq_cmpnev_storeoffp_load_relaxed_mm_cid(v, expectnot, voffp, load, cpu);
209
+
}
210
+
return -1;
211
+
}
212
+
213
+
static inline __attribute__((always_inline))
214
+
int rseq_addv(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
215
+
intptr_t *v, intptr_t count, int cpu)
216
+
{
217
+
if (rseq_mo != RSEQ_MO_RELAXED)
218
+
return -1;
219
+
switch (percpu_mode) {
220
+
case RSEQ_PERCPU_CPU_ID:
221
+
return rseq_addv_relaxed_cpu_id(v, count, cpu);
222
+
case RSEQ_PERCPU_MM_CID:
223
+
return rseq_addv_relaxed_mm_cid(v, count, cpu);
224
+
}
225
+
return -1;
226
+
}
227
+
228
+
#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV
229
+
/*
230
+
* pval = *(ptr+off)
231
+
* *pval += inc;
232
+
*/
233
+
static inline __attribute__((always_inline))
234
+
int rseq_offset_deref_addv(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
235
+
intptr_t *ptr, long off, intptr_t inc, int cpu)
236
+
{
237
+
if (rseq_mo != RSEQ_MO_RELAXED)
238
+
return -1;
239
+
switch (percpu_mode) {
240
+
case RSEQ_PERCPU_CPU_ID:
241
+
return rseq_offset_deref_addv_relaxed_cpu_id(ptr, off, inc, cpu);
242
+
case RSEQ_PERCPU_MM_CID:
243
+
return rseq_offset_deref_addv_relaxed_mm_cid(ptr, off, inc, cpu);
244
+
}
245
+
return -1;
246
+
}
247
+
#endif
248
+
249
+
static inline __attribute__((always_inline))
250
+
int rseq_cmpeqv_trystorev_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
251
+
intptr_t *v, intptr_t expect,
252
+
intptr_t *v2, intptr_t newv2,
253
+
intptr_t newv, int cpu)
254
+
{
255
+
switch (rseq_mo) {
256
+
case RSEQ_MO_RELAXED:
257
+
switch (percpu_mode) {
258
+
case RSEQ_PERCPU_CPU_ID:
259
+
return rseq_cmpeqv_trystorev_storev_relaxed_cpu_id(v, expect, v2, newv2, newv, cpu);
260
+
case RSEQ_PERCPU_MM_CID:
261
+
return rseq_cmpeqv_trystorev_storev_relaxed_mm_cid(v, expect, v2, newv2, newv, cpu);
262
+
}
263
+
return -1;
264
+
case RSEQ_MO_RELEASE:
265
+
switch (percpu_mode) {
266
+
case RSEQ_PERCPU_CPU_ID:
267
+
return rseq_cmpeqv_trystorev_storev_release_cpu_id(v, expect, v2, newv2, newv, cpu);
268
+
case RSEQ_PERCPU_MM_CID:
269
+
return rseq_cmpeqv_trystorev_storev_release_mm_cid(v, expect, v2, newv2, newv, cpu);
270
+
}
271
+
return -1;
272
+
default:
273
+
return -1;
274
+
}
275
+
}
276
+
277
+
static inline __attribute__((always_inline))
278
+
int rseq_cmpeqv_cmpeqv_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
279
+
intptr_t *v, intptr_t expect,
280
+
intptr_t *v2, intptr_t expect2,
281
+
intptr_t newv, int cpu)
282
+
{
283
+
if (rseq_mo != RSEQ_MO_RELAXED)
284
+
return -1;
285
+
switch (percpu_mode) {
286
+
case RSEQ_PERCPU_CPU_ID:
287
+
return rseq_cmpeqv_cmpeqv_storev_relaxed_cpu_id(v, expect, v2, expect2, newv, cpu);
288
+
case RSEQ_PERCPU_MM_CID:
289
+
return rseq_cmpeqv_cmpeqv_storev_relaxed_mm_cid(v, expect, v2, expect2, newv, cpu);
290
+
}
291
+
return -1;
292
+
}
293
+
294
+
static inline __attribute__((always_inline))
295
+
int rseq_cmpeqv_trymemcpy_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode,
296
+
intptr_t *v, intptr_t expect,
297
+
void *dst, void *src, size_t len,
298
+
intptr_t newv, int cpu)
299
+
{
300
+
switch (rseq_mo) {
301
+
case RSEQ_MO_RELAXED:
302
+
switch (percpu_mode) {
303
+
case RSEQ_PERCPU_CPU_ID:
304
+
return rseq_cmpeqv_trymemcpy_storev_relaxed_cpu_id(v, expect, dst, src, len, newv, cpu);
305
+
case RSEQ_PERCPU_MM_CID:
306
+
return rseq_cmpeqv_trymemcpy_storev_relaxed_mm_cid(v, expect, dst, src, len, newv, cpu);
307
+
}
308
+
return -1;
309
+
case RSEQ_MO_RELEASE:
310
+
switch (percpu_mode) {
311
+
case RSEQ_PERCPU_CPU_ID:
312
+
return rseq_cmpeqv_trymemcpy_storev_release_cpu_id(v, expect, dst, src, len, newv, cpu);
313
+
case RSEQ_PERCPU_MM_CID:
314
+
return rseq_cmpeqv_trymemcpy_storev_release_mm_cid(v, expect, dst, src, len, newv, cpu);
315
+
}
316
+
return -1;
317
+
default:
318
+
return -1;
319
+
}
237
320
}
238
321
239
322
#endif /* RSEQ_H_ */
+5
tools/testing/selftests/rseq/run_param_test.sh
+5
tools/testing/selftests/rseq/run_param_test.sh
···
42
42
./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1
43
43
echo "Running compare-twice test ${TEST_NAME[$i]}"
44
44
./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1
45
+
46
+
echo "Running mm_cid test ${TEST_NAME[$i]}"
47
+
./param_test_mm_cid ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1
48
+
echo "Running mm_cid compare-twice test ${TEST_NAME[$i]}"
49
+
./param_test_mm_cid_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1
45
50
let "i++"
46
51
done
47
52
}