Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'riscv-for-linus-7.0-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
Pull RISC-V updates from Paul Walmsley:
- Add support for control flow integrity for userspace processes.
This is based on the standard RISC-V ISA extensions Zicfiss and
Zicfilp
- Improve ptrace behavior regarding vector registers, and add some
selftests
- Optimize our strlen() assembly
- Enable the ISO-8859-1 code page as built-in, similar to ARM64, for
EFI volume mounting
- Clean up some code slightly, including defining copy_user_page() as
copy_page() rather than memcpy(), aligning us with other
architectures; and using max3() to slightly simplify an expression
in riscv_iommu_init_check()
* tag 'riscv-for-linus-7.0-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (42 commits)
riscv: lib: optimize strlen loop efficiency
selftests: riscv: vstate_exec_nolibc: Use the regular prctl() function
selftests: riscv: verify ptrace accepts valid vector csr values
selftests: riscv: verify ptrace rejects invalid vector csr inputs
selftests: riscv: verify syscalls discard vector context
selftests: riscv: verify initial vector state with ptrace
selftests: riscv: test ptrace vector interface
riscv: ptrace: validate input vector csr registers
riscv: csr: define vtype register elements
riscv: vector: init vector context with proper vlenb
riscv: ptrace: return ENODATA for inactive vector extension
kselftest/riscv: add kselftest for user mode CFI
riscv: add documentation for shadow stack
riscv: add documentation for landing pad / indirect branch tracking
riscv: create a Kconfig fragment for shadow stack and landing pad support
arch/riscv: add dual vdso creation logic and select vdso based on hw
arch/riscv: compile vdso with landing pad and shadow stack note
riscv: enable kernel access to shadow stack memory via the FWFT SBI call
riscv: add kernel command line option to opt out of user CFI
riscv/hwprobe: add zicfilp / zicfiss enumeration in hwprobe
...
+3655
-122
+8
Documentation/admin-guide/kernel-parameters.txt
+8
Documentation/admin-guide/kernel-parameters.txt
···
6641
6641
replacement properties are not found. See the Kconfig
6642
6642
entry for RISCV_ISA_FALLBACK.
6643
6643
6644
+
riscv_nousercfi=
6645
+
all Disable user CFI ABI to userspace even if cpu extension
6646
+
are available.
6647
+
bcfi Disable user backward CFI ABI to userspace even if
6648
+
the shadow stack extension is available.
6649
+
fcfi Disable user forward CFI ABI to userspace even if the
6650
+
landing pad extension is available.
6651
+
6644
6652
ro [KNL] Mount root device read-only on boot
6645
6653
6646
6654
rodata= [KNL,EARLY]
+5
-1
Documentation/arch/riscv/hwprobe.rst
+5
-1
Documentation/arch/riscv/hwprobe.rst
···
67
67
programs (it may still be executed in userspace via a
68
68
kernel-controlled mechanism such as the vDSO).
69
69
70
-
* :c:macro:`RISCV_HWPROBE_KEY_IMA_EXT_0`: A bitmask containing the extensions
70
+
* :c:macro:`RISCV_HWPROBE_KEY_IMA_EXT_0`: A bitmask containing extensions
71
71
that are compatible with the :c:macro:`RISCV_HWPROBE_BASE_BEHAVIOR_IMA`:
72
72
base system behavior.
73
73
···
387
387
388
388
* :c:macro:`RISCV_HWPROBE_KEY_ZICBOP_BLOCK_SIZE`: An unsigned int which
389
389
represents the size of the Zicbop block in bytes.
390
+
391
+
* :c:macro:`RISCV_HWPROBE_KEY_IMA_EXT_1`: A bitmask containing additional
392
+
extensions that are compatible with the
393
+
:c:macro:`RISCV_HWPROBE_BASE_BEHAVIOR_IMA`: base system behavior.
+2
Documentation/arch/riscv/index.rst
+2
Documentation/arch/riscv/index.rst
+122
Documentation/arch/riscv/zicfilp.rst
+122
Documentation/arch/riscv/zicfilp.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
:Author: Deepak Gupta <debug@rivosinc.com>
4
+
:Date: 12 January 2024
5
+
6
+
====================================================
7
+
Tracking indirect control transfers on RISC-V Linux
8
+
====================================================
9
+
10
+
This document briefly describes the interface provided to userspace by Linux
11
+
to enable indirect branch tracking for user mode applications on RISC-V.
12
+
13
+
1. Feature Overview
14
+
--------------------
15
+
16
+
Memory corruption issues usually result in crashes. However, in the
17
+
hands of a creative adversary, these can result in a variety of
18
+
security issues.
19
+
20
+
Some of those security issues can be code re-use attacks, where an
21
+
adversary can use corrupt function pointers, chaining them together to
22
+
perform jump oriented programming (JOP) or call oriented programming
23
+
(COP) and thus compromise control flow integrity (CFI) of the program.
24
+
25
+
Function pointers live in read-write memory and thus are susceptible
26
+
to corruption. This can allow an adversary to control the program
27
+
counter (PC) value. On RISC-V, the zicfilp extension enforces a
28
+
restriction on such indirect control transfers:
29
+
30
+
- Indirect control transfers must land on a landing pad instruction ``lpad``.
31
+
There are two exceptions to this rule:
32
+
33
+
- rs1 = x1 or rs1 = x5, i.e. a return from a function and returns are
34
+
protected using shadow stack (see zicfiss.rst)
35
+
36
+
- rs1 = x7. On RISC-V, the compiler usually does the following to reach a
37
+
function which is beyond the offset of possible J-type instruction::
38
+
39
+
auipc x7, <imm>
40
+
jalr (x7)
41
+
42
+
This form of indirect control transfer is immutable and doesn't
43
+
rely on memory. Thus rs1=x7 is exempted from tracking and
44
+
these are considered software guarded jumps.
45
+
46
+
The ``lpad`` instruction is a pseudo-op of ``auipc rd, <imm_20bit>``
47
+
with ``rd=x0``. This is a HINT op. The ``lpad`` instruction must be
48
+
aligned on a 4 byte boundary. It compares the 20 bit immediate with
49
+
x7. If ``imm_20bit`` == 0, the CPU doesn't perform any comparison with
50
+
``x7``. If ``imm_20bit`` != 0, then ``imm_20bit`` must match ``x7``
51
+
else CPU will raise ``software check exception`` (``cause=18``) with
52
+
``*tval = 2``.
53
+
54
+
The compiler can generate a hash over function signatures and set them
55
+
up (truncated to 20 bits) in x7 at callsites. Function prologues can
56
+
have ``lpad`` instructions encoded with the same function hash. This
57
+
further reduces the number of valid program counter addresses a call
58
+
site can reach.
59
+
60
+
2. ELF and psABI
61
+
-----------------
62
+
63
+
The toolchain sets up :c:macro:`GNU_PROPERTY_RISCV_FEATURE_1_FCFI` for
64
+
property :c:macro:`GNU_PROPERTY_RISCV_FEATURE_1_AND` in the notes
65
+
section of the object file.
66
+
67
+
3. Linux enabling
68
+
------------------
69
+
70
+
User space programs can have multiple shared objects loaded in their
71
+
address spaces. It's a difficult task to make sure all the
72
+
dependencies have been compiled with indirect branch support. Thus
73
+
it's left to the dynamic loader to enable indirect branch tracking for
74
+
the program.
75
+
76
+
4. prctl() enabling
77
+
--------------------
78
+
79
+
:c:macro:`PR_SET_INDIR_BR_LP_STATUS` / :c:macro:`PR_GET_INDIR_BR_LP_STATUS` /
80
+
:c:macro:`PR_LOCK_INDIR_BR_LP_STATUS` are three prctls added to manage indirect
81
+
branch tracking. These prctls are architecture-agnostic and return -EINVAL if
82
+
the underlying functionality is not supported.
83
+
84
+
* prctl(PR_SET_INDIR_BR_LP_STATUS, unsigned long arg)
85
+
86
+
If arg1 is :c:macro:`PR_INDIR_BR_LP_ENABLE` and if CPU supports
87
+
``zicfilp`` then the kernel will enable indirect branch tracking for the
88
+
task. The dynamic loader can issue this :c:macro:`prctl` once it has
89
+
determined that all the objects loaded in the address space support
90
+
indirect branch tracking. Additionally, if there is a `dlopen` to an
91
+
object which wasn't compiled with ``zicfilp``, the dynamic loader can
92
+
issue this prctl with arg1 set to 0 (i.e. :c:macro:`PR_INDIR_BR_LP_ENABLE`
93
+
cleared).
94
+
95
+
* prctl(PR_GET_INDIR_BR_LP_STATUS, unsigned long * arg)
96
+
97
+
Returns the current status of indirect branch tracking. If enabled
98
+
it'll return :c:macro:`PR_INDIR_BR_LP_ENABLE`
99
+
100
+
* prctl(PR_LOCK_INDIR_BR_LP_STATUS, unsigned long arg)
101
+
102
+
Locks the current status of indirect branch tracking on the task. User
103
+
space may want to run with a strict security posture and wouldn't want
104
+
loading of objects without ``zicfilp`` support in them, to disallow
105
+
disabling of indirect branch tracking. In this case, user space can
106
+
use this prctl to lock the current settings.
107
+
108
+
5. violations related to indirect branch tracking
109
+
--------------------------------------------------
110
+
111
+
Pertaining to indirect branch tracking, the CPU raises a software
112
+
check exception in the following conditions:
113
+
114
+
- missing ``lpad`` after indirect call / jmp
115
+
- ``lpad`` not on 4 byte boundary
116
+
- ``imm_20bit`` embedded in ``lpad`` instruction doesn't match with ``x7``
117
+
118
+
In all 3 cases, ``*tval = 2`` is captured and software check exception is
119
+
raised (``cause=18``).
120
+
121
+
The kernel will treat this as :c:macro:`SIGSEGV` with code =
122
+
:c:macro:`SEGV_CPERR` and follow the normal course of signal delivery.
+194
Documentation/arch/riscv/zicfiss.rst
+194
Documentation/arch/riscv/zicfiss.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
:Author: Deepak Gupta <debug@rivosinc.com>
4
+
:Date: 12 January 2024
5
+
6
+
=========================================================
7
+
Shadow stack to protect function returns on RISC-V Linux
8
+
=========================================================
9
+
10
+
This document briefly describes the interface provided to userspace by Linux
11
+
to enable shadow stacks for user mode applications on RISC-V.
12
+
13
+
1. Feature Overview
14
+
--------------------
15
+
16
+
Memory corruption issues usually result in crashes. However, in the
17
+
hands of a creative adversary, these issues can result in a variety of
18
+
security problems.
19
+
20
+
Some of those security issues can be code re-use attacks on programs
21
+
where an adversary can use corrupt return addresses present on the
22
+
stack. chaining them together to perform return oriented programming
23
+
(ROP) and thus compromising the control flow integrity (CFI) of the
24
+
program.
25
+
26
+
Return addresses live on the stack in read-write memory. Therefore
27
+
they are susceptible to corruption, which allows an adversary to
28
+
control the program counter. On RISC-V, the ``zicfiss`` extension
29
+
provides an alternate stack (the "shadow stack") on which return
30
+
addresses can be safely placed in the prologue of the function and
31
+
retrieved in the epilogue. The ``zicfiss`` extension makes the
32
+
following changes:
33
+
34
+
- PTE encodings for shadow stack virtual memory
35
+
An earlier reserved encoding in first stage translation i.e.
36
+
PTE.R=0, PTE.W=1, PTE.X=0 becomes the PTE encoding for shadow stack pages.
37
+
38
+
- The ``sspush x1/x5`` instruction pushes (stores) ``x1/x5`` to shadow stack.
39
+
40
+
- The ``sspopchk x1/x5`` instruction pops (loads) from shadow stack and compares
41
+
with ``x1/x5`` and if not equal, the CPU raises a ``software check exception``
42
+
with ``*tval = 3``
43
+
44
+
The compiler toolchain ensures that function prologues have ``sspush
45
+
x1/x5`` to save the return address on shadow stack in addition to the
46
+
regular stack. Similarly, function epilogues have ``ld x5,
47
+
offset(x2)`` followed by ``sspopchk x5`` to ensure that a popped value
48
+
from the regular stack matches with the popped value from the shadow
49
+
stack.
50
+
51
+
2. Shadow stack protections and linux memory manager
52
+
-----------------------------------------------------
53
+
54
+
As mentioned earlier, shadow stacks get new page table encodings that
55
+
have some special properties assigned to them, along with instructions
56
+
that operate on the shadow stacks:
57
+
58
+
- Regular stores to shadow stack memory raise store access faults. This
59
+
protects shadow stack memory from stray writes.
60
+
61
+
- Regular loads from shadow stack memory are allowed. This allows
62
+
stack trace utilities or backtrace functions to read the true call
63
+
stack and ensure that it has not been tampered with.
64
+
65
+
- Only shadow stack instructions can generate shadow stack loads or
66
+
shadow stack stores.
67
+
68
+
- Shadow stack loads and stores on read-only memory raise AMO/store
69
+
page faults. Thus both ``sspush x1/x5`` and ``sspopchk x1/x5`` will
70
+
raise AMO/store page fault. This simplies COW handling in kernel
71
+
during fork(). The kernel can convert shadow stack pages into
72
+
read-only memory (as it does for regular read-write memory). As
73
+
soon as subsequent ``sspush`` or ``sspopchk`` instructions in
74
+
userspace are encountered, the kernel can perform COW.
75
+
76
+
- Shadow stack loads and stores on read-write or read-write-execute
77
+
memory raise an access fault. This is a fatal condition because
78
+
shadow stack loads and stores should never be operating on
79
+
read-write or read-write-execute memory.
80
+
81
+
3. ELF and psABI
82
+
-----------------
83
+
84
+
The toolchain sets up :c:macro:`GNU_PROPERTY_RISCV_FEATURE_1_BCFI` for
85
+
property :c:macro:`GNU_PROPERTY_RISCV_FEATURE_1_AND` in the notes
86
+
section of the object file.
87
+
88
+
4. Linux enabling
89
+
------------------
90
+
91
+
User space programs can have multiple shared objects loaded in their
92
+
address space. It's a difficult task to make sure all the
93
+
dependencies have been compiled with shadow stack support. Thus
94
+
it's left to the dynamic loader to enable shadow stacks for the
95
+
program.
96
+
97
+
5. prctl() enabling
98
+
--------------------
99
+
100
+
:c:macro:`PR_SET_SHADOW_STACK_STATUS` / :c:macro:`PR_GET_SHADOW_STACK_STATUS` /
101
+
:c:macro:`PR_LOCK_SHADOW_STACK_STATUS` are three prctls added to manage shadow
102
+
stack enabling for tasks. These prctls are architecture-agnostic and return
103
+
-EINVAL if not implemented.
104
+
105
+
* prctl(PR_SET_SHADOW_STACK_STATUS, unsigned long arg)
106
+
107
+
If arg = :c:macro:`PR_SHADOW_STACK_ENABLE` and if CPU supports
108
+
``zicfiss`` then the kernel will enable shadow stacks for the task.
109
+
The dynamic loader can issue this :c:macro:`prctl` once it has
110
+
determined that all the objects loaded in address space have support
111
+
for shadow stacks. Additionally, if there is a :c:macro:`dlopen` to
112
+
an object which wasn't compiled with ``zicfiss``, the dynamic loader
113
+
can issue this prctl with arg set to 0 (i.e.
114
+
:c:macro:`PR_SHADOW_STACK_ENABLE` being clear)
115
+
116
+
* prctl(PR_GET_SHADOW_STACK_STATUS, unsigned long * arg)
117
+
118
+
Returns the current status of indirect branch tracking. If enabled
119
+
it'll return :c:macro:`PR_SHADOW_STACK_ENABLE`.
120
+
121
+
* prctl(PR_LOCK_SHADOW_STACK_STATUS, unsigned long arg)
122
+
123
+
Locks the current status of shadow stack enabling on the
124
+
task. Userspace may want to run with a strict security posture and
125
+
wouldn't want loading of objects without ``zicfiss`` support. In this
126
+
case userspace can use this prctl to disallow disabling of shadow
127
+
stacks on the current task.
128
+
129
+
5. violations related to returns with shadow stack enabled
130
+
-----------------------------------------------------------
131
+
132
+
Pertaining to shadow stacks, the CPU raises a ``software check
133
+
exception`` upon executing ``sspopchk x1/x5`` if ``x1/x5`` doesn't
134
+
match the top of shadow stack. If a mismatch happens, then the CPU
135
+
sets ``*tval = 3`` and raises the exception.
136
+
137
+
The Linux kernel will treat this as a :c:macro:`SIGSEGV` with code =
138
+
:c:macro:`SEGV_CPERR` and follow the normal course of signal delivery.
139
+
140
+
6. Shadow stack tokens
141
+
-----------------------
142
+
143
+
Regular stores on shadow stacks are not allowed and thus can't be
144
+
tampered with via arbitrary stray writes. However, one method of
145
+
pivoting / switching to a shadow stack is simply writing to the CSR
146
+
``CSR_SSP``. This will change the active shadow stack for the
147
+
program. Writes to ``CSR_SSP`` in the program should be mostly
148
+
limited to context switches, stack unwinds, or longjmp or similar
149
+
mechanisms (like context switching of Green Threads) in languages like
150
+
Go and Rust. CSR_SSP writes can be problematic because an attacker can
151
+
use memory corruption bugs and leverage context switching routines to
152
+
pivot to any shadow stack. Shadow stack tokens can help mitigate this
153
+
problem by making sure that:
154
+
155
+
- When software is switching away from a shadow stack, the shadow
156
+
stack pointer should be saved on the shadow stack itself (this is
157
+
called the ``shadow stack token``).
158
+
159
+
- When software is switching to a shadow stack, it should read the
160
+
``shadow stack token`` from the shadow stack pointer and verify that
161
+
the ``shadow stack token`` itself is a pointer to the shadow stack
162
+
itself.
163
+
164
+
- Once the token verification is done, software can perform the write
165
+
to ``CSR_SSP`` to switch shadow stacks.
166
+
167
+
Here "software" could refer to the user mode task runtime itself,
168
+
managing various contexts as part of a single thread. Or "software"
169
+
could refer to the kernel, when the kernel has to deliver a signal to
170
+
a user task and must save the shadow stack pointer. The kernel can
171
+
perform similar procedure itself by saving a token on the user mode
172
+
task's shadow stack. This way, whenever :c:macro:`sigreturn` happens,
173
+
the kernel can read and verify the token and then switch to the shadow
174
+
stack. Using this mechanism, the kernel helps the user task so that
175
+
any corruption issue in the user task is not exploited by adversaries
176
+
arbitrarily using :c:macro:`sigreturn`. Adversaries will have to make
177
+
sure that there is a valid ``shadow stack token`` in addition to
178
+
invoking :c:macro:`sigreturn`.
179
+
180
+
7. Signal shadow stack
181
+
-----------------------
182
+
The following structure has been added to sigcontext for RISC-V::
183
+
184
+
struct __sc_riscv_cfi_state {
185
+
unsigned long ss_ptr;
186
+
};
187
+
188
+
As part of signal delivery, the shadow stack token is saved on the
189
+
current shadow stack itself. The updated pointer is saved away in the
190
+
:c:macro:`ss_ptr` field in :c:macro:`__sc_riscv_cfi_state` under
191
+
:c:macro:`sigcontext`. The existing shadow stack allocation is used
192
+
for signal delivery. During :c:macro:`sigreturn`, kernel will obtain
193
+
:c:macro:`ss_ptr` from :c:macro:`sigcontext`, verify the saved
194
+
token on the shadow stack, and switch the shadow stack.
+14
Documentation/devicetree/bindings/riscv/extensions.yaml
+14
Documentation/devicetree/bindings/riscv/extensions.yaml
···
589
589
The standard Zicboz extension for cache-block zeroing as ratified
590
590
in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
591
591
592
+
- const: zicfilp
593
+
description: |
594
+
The standard Zicfilp extension for enforcing forward edge
595
+
control-flow integrity as ratified in commit 3f8e450 ("merge
596
+
pull request #227 from ved-rivos/0709") of riscv-cfi
597
+
github repo.
598
+
599
+
- const: zicfiss
600
+
description: |
601
+
The standard Zicfiss extension for enforcing backward edge
602
+
control-flow integrity as ratified in commit 3f8e450 ("merge
603
+
pull request #227 from ved-rivos/0709") of riscv-cfi
604
+
github repo.
605
+
592
606
- const: zicntr
593
607
description:
594
608
The standard Zicntr extension for base counters and timers, as
+22
arch/riscv/Kconfig
+22
arch/riscv/Kconfig
···
1163
1163
1164
1164
If unsure, say N.
1165
1165
1166
+
config RISCV_USER_CFI
1167
+
def_bool y
1168
+
bool "riscv userspace control flow integrity"
1169
+
depends on 64BIT && MMU && \
1170
+
$(cc-option,-mabi=lp64 -march=rv64ima_zicfiss_zicfilp -fcf-protection=full)
1171
+
depends on RISCV_ALTERNATIVE
1172
+
select RISCV_SBI
1173
+
select ARCH_HAS_USER_SHADOW_STACK
1174
+
select ARCH_USES_HIGH_VMA_FLAGS
1175
+
select DYNAMIC_SIGFRAME
1176
+
help
1177
+
Provides CPU-assisted control flow integrity to userspace tasks.
1178
+
Control flow integrity is provided by implementing shadow stack for
1179
+
backward edge and indirect branch tracking for forward edge.
1180
+
Shadow stack protection is a hardware feature that detects function
1181
+
return address corruption. This helps mitigate ROP attacks.
1182
+
Indirect branch tracking enforces that all indirect branches must land
1183
+
on a landing pad instruction else CPU will fault. This mitigates against
1184
+
JOP / COP attacks. Applications must be enabled to use it, and old userspace
1185
+
does not get protection "for free".
1186
+
default y.
1187
+
1166
1188
endmenu # "Kernel features"
1167
1189
1168
1190
menu "Boot options"
+7
-1
arch/riscv/Makefile
+7
-1
arch/riscv/Makefile
···
81
81
# Check if the toolchain supports Zabha
82
82
riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha
83
83
84
+
KBUILD_BASE_ISA = -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')
85
+
export KBUILD_BASE_ISA
86
+
84
87
# Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
85
88
# matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
86
-
KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')
89
+
KBUILD_CFLAGS += $(KBUILD_BASE_ISA)
87
90
88
91
KBUILD_AFLAGS += -march=$(riscv-march-y)
89
92
···
161
158
prepare: vdso_prepare
162
159
vdso_prepare: prepare0
163
160
$(Q)$(MAKE) $(build)=arch/riscv/kernel/vdso include/generated/vdso-offsets.h
161
+
$(if $(CONFIG_RISCV_USER_CFI),$(Q)$(MAKE) \
162
+
$(build)=arch/riscv/kernel/vdso_cfi include/generated/vdso-cfi-offsets.h)
164
163
$(if $(CONFIG_COMPAT),$(Q)$(MAKE) \
165
164
$(build)=arch/riscv/kernel/compat_vdso include/generated/compat_vdso-offsets.h)
166
165
···
170
165
endif
171
166
172
167
vdso-install-y += arch/riscv/kernel/vdso/vdso.so.dbg
168
+
vdso-install-$(CONFIG_RISCV_USER_CFI) += arch/riscv/kernel/vdso_cfi/vdso-cfi.so.dbg
173
169
vdso-install-$(CONFIG_COMPAT) += arch/riscv/kernel/compat_vdso/compat_vdso.so.dbg
174
170
175
171
BOOT_TARGETS := Image Image.gz Image.bz2 Image.lz4 Image.lzma Image.lzo Image.zst Image.xz loader loader.bin xipImage vmlinuz.efi
+1
-1
arch/riscv/configs/defconfig
+1
-1
arch/riscv/configs/defconfig
+4
arch/riscv/configs/hardening.config
+4
arch/riscv/configs/hardening.config
+1
arch/riscv/include/asm/asm-prototypes.h
+1
arch/riscv/include/asm/asm-prototypes.h
···
51
51
DECLARE_DO_ERROR_INFO(do_trap_ecall_s);
52
52
DECLARE_DO_ERROR_INFO(do_trap_ecall_m);
53
53
DECLARE_DO_ERROR_INFO(do_trap_break);
54
+
DECLARE_DO_ERROR_INFO(do_trap_software_check);
54
55
55
56
asmlinkage void ret_from_fork_kernel(void *fn_arg, int (*fn)(void *), struct pt_regs *regs);
56
57
asmlinkage void ret_from_fork_user(struct pt_regs *regs);
+44
arch/riscv/include/asm/assembler.h
+44
arch/riscv/include/asm/assembler.h
···
80
80
.endm
81
81
82
82
#endif /* __ASM_ASSEMBLER_H */
83
+
84
+
#if defined(VDSO_CFI) && (__riscv_xlen == 64)
85
+
.macro vdso_lpad, label = 0
86
+
lpad \label
87
+
.endm
88
+
#else
89
+
.macro vdso_lpad, label = 0
90
+
.endm
91
+
#endif
92
+
93
+
/*
94
+
* This macro emits a program property note section identifying
95
+
* architecture features which require special handling, mainly for
96
+
* use in assembly files included in the VDSO.
97
+
*/
98
+
#define NT_GNU_PROPERTY_TYPE_0 5
99
+
#define GNU_PROPERTY_RISCV_FEATURE_1_AND 0xc0000000
100
+
101
+
#define GNU_PROPERTY_RISCV_FEATURE_1_ZICFILP BIT(0)
102
+
#define GNU_PROPERTY_RISCV_FEATURE_1_ZICFISS BIT(1)
103
+
104
+
#if defined(VDSO_CFI) && (__riscv_xlen == 64)
105
+
#define GNU_PROPERTY_RISCV_FEATURE_1_DEFAULT \
106
+
(GNU_PROPERTY_RISCV_FEATURE_1_ZICFILP | GNU_PROPERTY_RISCV_FEATURE_1_ZICFISS)
107
+
#endif
108
+
109
+
#ifdef GNU_PROPERTY_RISCV_FEATURE_1_DEFAULT
110
+
.macro emit_riscv_feature_1_and, feat = GNU_PROPERTY_RISCV_FEATURE_1_DEFAULT
111
+
.pushsection .note.gnu.property, "a"
112
+
.p2align 3
113
+
.word 4
114
+
.word 16
115
+
.word NT_GNU_PROPERTY_TYPE_0
116
+
.asciz "GNU"
117
+
.word GNU_PROPERTY_RISCV_FEATURE_1_AND
118
+
.word 4
119
+
.word \feat
120
+
.word 0
121
+
.popsection
122
+
.endm
123
+
#else
124
+
.macro emit_riscv_feature_1_and, feat = 0
125
+
.endm
126
+
#endif
+12
arch/riscv/include/asm/cpufeature.h
+12
arch/riscv/include/asm/cpufeature.h
···
152
152
return __riscv_isa_extension_available(hart_isa[cpu].isa, ext);
153
153
}
154
154
155
+
static inline bool cpu_supports_shadow_stack(void)
156
+
{
157
+
return (IS_ENABLED(CONFIG_RISCV_USER_CFI) &&
158
+
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFISS));
159
+
}
160
+
161
+
static inline bool cpu_supports_indirect_br_lp_instr(void)
162
+
{
163
+
return (IS_ENABLED(CONFIG_RISCV_USER_CFI) &&
164
+
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFILP));
165
+
}
166
+
155
167
#endif
+31
arch/riscv/include/asm/csr.h
+31
arch/riscv/include/asm/csr.h
···
18
18
#define SR_MPP _AC(0x00001800, UL) /* Previously Machine */
19
19
#define SR_SUM _AC(0x00040000, UL) /* Supervisor User Memory Access */
20
20
21
+
/* zicfilp landing pad status bit */
22
+
#define SR_SPELP _AC(0x00800000, UL)
23
+
#define SR_MPELP _AC(0x020000000000, UL)
24
+
#ifdef CONFIG_RISCV_M_MODE
25
+
#define SR_ELP SR_MPELP
26
+
#else
27
+
#define SR_ELP SR_SPELP
28
+
#endif
29
+
21
30
#define SR_FS _AC(0x00006000, UL) /* Floating-point Status */
22
31
#define SR_FS_OFF _AC(0x00000000, UL)
23
32
#define SR_FS_INITIAL _AC(0x00002000, UL)
···
221
212
#define ENVCFG_PMM_PMLEN_16 (_AC(0x3, ULL) << 32)
222
213
#define ENVCFG_CBZE (_AC(1, UL) << 7)
223
214
#define ENVCFG_CBCFE (_AC(1, UL) << 6)
215
+
#define ENVCFG_LPE (_AC(1, UL) << 2)
216
+
#define ENVCFG_SSE (_AC(1, UL) << 3)
224
217
#define ENVCFG_CBIE_SHIFT 4
225
218
#define ENVCFG_CBIE (_AC(0x3, UL) << ENVCFG_CBIE_SHIFT)
226
219
#define ENVCFG_CBIE_ILL _AC(0x0, UL)
···
331
320
332
321
#define CSR_STIMECMP 0x14D
333
322
#define CSR_STIMECMPH 0x15D
323
+
324
+
/* zicfiss user mode csr. CSR_SSP holds current shadow stack pointer */
325
+
#define CSR_SSP 0x011
334
326
335
327
/* xtheadvector symbolic CSR names */
336
328
#define CSR_VXSAT 0x9
···
457
443
#define CSR_VL 0xc20
458
444
#define CSR_VTYPE 0xc21
459
445
#define CSR_VLENB 0xc22
446
+
447
+
#define VTYPE_VLMUL _AC(7, UL)
448
+
#define VTYPE_VLMUL_FRAC _AC(4, UL)
449
+
#define VTYPE_VSEW_SHIFT 3
450
+
#define VTYPE_VSEW (_AC(7, UL) << VTYPE_VSEW_SHIFT)
451
+
#define VTYPE_VTA_SHIFT 6
452
+
#define VTYPE_VTA (_AC(1, UL) << VTYPE_VTA_SHIFT)
453
+
#define VTYPE_VMA_SHIFT 7
454
+
#define VTYPE_VMA (_AC(1, UL) << VTYPE_VMA_SHIFT)
455
+
#define VTYPE_VILL_SHIFT (__riscv_xlen - 1)
456
+
#define VTYPE_VILL (_AC(1, UL) << VTYPE_VILL_SHIFT)
457
+
458
+
#define VTYPE_VLMUL_THEAD _AC(3, UL)
459
+
#define VTYPE_VSEW_THEAD_SHIFT 2
460
+
#define VTYPE_VSEW_THEAD (_AC(7, UL) << VTYPE_VSEW_THEAD_SHIFT)
461
+
#define VTYPE_VEDIV_THEAD_SHIFT 5
462
+
#define VTYPE_VEDIV_THEAD (_AC(3, UL) << VTYPE_VEDIV_THEAD_SHIFT)
460
463
461
464
/* Scalar Crypto Extension - Entropy */
462
465
#define CSR_SEED 0x015
+2
arch/riscv/include/asm/entry-common.h
+2
arch/riscv/include/asm/entry-common.h
+2
arch/riscv/include/asm/hwcap.h
+2
arch/riscv/include/asm/hwcap.h
+2
-1
arch/riscv/include/asm/hwprobe.h
+2
-1
arch/riscv/include/asm/hwprobe.h
···
8
8
9
9
#include <uapi/asm/hwprobe.h>
10
10
11
-
#define RISCV_HWPROBE_MAX_KEY 15
11
+
#define RISCV_HWPROBE_MAX_KEY 16
12
12
13
13
static inline bool riscv_hwprobe_key_is_valid(__s64 key)
14
14
{
···
20
20
switch (key) {
21
21
case RISCV_HWPROBE_KEY_BASE_BEHAVIOR:
22
22
case RISCV_HWPROBE_KEY_IMA_EXT_0:
23
+
case RISCV_HWPROBE_KEY_IMA_EXT_1:
23
24
case RISCV_HWPROBE_KEY_CPUPERF_0:
24
25
case RISCV_HWPROBE_KEY_VENDOR_EXT_THEAD_0:
25
26
case RISCV_HWPROBE_KEY_VENDOR_EXT_MIPS_0:
+26
arch/riscv/include/asm/mman.h
+26
arch/riscv/include/asm/mman.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0 */
2
+
#ifndef __ASM_MMAN_H__
3
+
#define __ASM_MMAN_H__
4
+
5
+
#include <linux/compiler.h>
6
+
#include <linux/types.h>
7
+
#include <linux/mm.h>
8
+
#include <uapi/asm/mman.h>
9
+
10
+
static inline unsigned long arch_calc_vm_prot_bits(unsigned long prot,
11
+
unsigned long pkey __always_unused)
12
+
{
13
+
unsigned long ret = 0;
14
+
15
+
/*
16
+
* If PROT_WRITE was specified, force it to VM_READ | VM_WRITE.
17
+
* Only VM_WRITE means shadow stack.
18
+
*/
19
+
if (prot & PROT_WRITE)
20
+
ret = (VM_READ | VM_WRITE);
21
+
return ret;
22
+
}
23
+
24
+
#define arch_calc_vm_prot_bits(prot, pkey) arch_calc_vm_prot_bits(prot, pkey)
25
+
26
+
#endif /* ! __ASM_MMAN_H__ */
+7
arch/riscv/include/asm/mmu_context.h
+7
arch/riscv/include/asm/mmu_context.h
+1
-2
arch/riscv/include/asm/page.h
+1
-2
arch/riscv/include/asm/page.h
···
50
50
#endif
51
51
#define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)
52
52
53
-
#define copy_user_page(vto, vfrom, vaddr, topg) \
54
-
memcpy((vto), (vfrom), PAGE_SIZE)
53
+
#define copy_user_page(vto, vfrom, vaddr, topg) copy_page(vto, vfrom)
55
54
56
55
/*
57
56
* Use struct definitions to apply C type checking
+28
-2
arch/riscv/include/asm/pgtable.h
+28
-2
arch/riscv/include/asm/pgtable.h
···
178
178
#define PAGE_READ_EXEC __pgprot(_PAGE_BASE | _PAGE_READ | _PAGE_EXEC)
179
179
#define PAGE_WRITE_EXEC __pgprot(_PAGE_BASE | _PAGE_READ | \
180
180
_PAGE_EXEC | _PAGE_WRITE)
181
+
#define PAGE_SHADOWSTACK __pgprot(_PAGE_BASE | _PAGE_WRITE)
181
182
182
183
#define PAGE_COPY PAGE_READ
183
184
#define PAGE_COPY_EXEC PAGE_READ_EXEC
···
411
410
412
411
static inline pte_t pte_wrprotect(pte_t pte)
413
412
{
414
-
return __pte(pte_val(pte) & ~(_PAGE_WRITE));
413
+
return __pte((pte_val(pte) & ~(_PAGE_WRITE)) | (_PAGE_READ));
415
414
}
416
415
417
416
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
···
451
450
452
451
/* static inline pte_t pte_mkread(pte_t pte) */
453
452
453
+
struct vm_area_struct;
454
+
pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma);
455
+
#define pte_mkwrite pte_mkwrite
456
+
454
457
static inline pte_t pte_mkwrite_novma(pte_t pte)
455
458
{
456
459
return __pte(pte_val(pte) | _PAGE_WRITE);
460
+
}
461
+
462
+
static inline pte_t pte_mkwrite_shstk(pte_t pte)
463
+
{
464
+
return __pte((pte_val(pte) & ~(_PAGE_LEAF)) | _PAGE_WRITE);
457
465
}
458
466
459
467
/* static inline pte_t pte_mkexec(pte_t pte) */
···
683
673
static inline void ptep_set_wrprotect(struct mm_struct *mm,
684
674
unsigned long address, pte_t *ptep)
685
675
{
686
-
atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep);
676
+
pte_t read_pte = READ_ONCE(*ptep);
677
+
/*
678
+
* ptep_set_wrprotect can be called for shadow stack ranges too.
679
+
* shadow stack memory is XWR = 010 and thus clearing _PAGE_WRITE will lead to
680
+
* encoding 000b which is wrong encoding with V = 1. This should lead to page fault
681
+
* but we dont want this wrong configuration to be set in page tables.
682
+
*/
683
+
atomic_long_set((atomic_long_t *)ptep,
684
+
((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
687
685
}
688
686
689
687
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
···
851
833
return pte_pmd(pte_mkyoung(pmd_pte(pmd)));
852
834
}
853
835
836
+
pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
837
+
#define pmd_mkwrite pmd_mkwrite
838
+
854
839
static inline pmd_t pmd_mkwrite_novma(pmd_t pmd)
855
840
{
856
841
return pte_pmd(pte_mkwrite_novma(pmd_pte(pmd)));
842
+
}
843
+
844
+
static inline pmd_t pmd_mkwrite_shstk(pmd_t pte)
845
+
{
846
+
return __pmd((pmd_val(pte) & ~(_PAGE_LEAF)) | _PAGE_WRITE);
857
847
}
858
848
859
849
static inline pmd_t pmd_wrprotect(pmd_t pmd)
+1
arch/riscv/include/asm/processor.h
+1
arch/riscv/include/asm/processor.h
+3
arch/riscv/include/asm/thread_info.h
+3
arch/riscv/include/asm/thread_info.h
+97
arch/riscv/include/asm/usercfi.h
+97
arch/riscv/include/asm/usercfi.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0
2
+
* Copyright (C) 2024 Rivos, Inc.
3
+
* Deepak Gupta <debug@rivosinc.com>
4
+
*/
5
+
#ifndef _ASM_RISCV_USERCFI_H
6
+
#define _ASM_RISCV_USERCFI_H
7
+
8
+
#define CMDLINE_DISABLE_RISCV_USERCFI_FCFI 1
9
+
#define CMDLINE_DISABLE_RISCV_USERCFI_BCFI 2
10
+
#define CMDLINE_DISABLE_RISCV_USERCFI 3
11
+
12
+
#ifndef __ASSEMBLER__
13
+
#include <linux/types.h>
14
+
#include <linux/prctl.h>
15
+
#include <linux/errno.h>
16
+
17
+
struct task_struct;
18
+
struct kernel_clone_args;
19
+
20
+
extern unsigned long riscv_nousercfi;
21
+
22
+
#ifdef CONFIG_RISCV_USER_CFI
23
+
struct cfi_state {
24
+
unsigned long ubcfi_en : 1; /* Enable for backward cfi. */
25
+
unsigned long ubcfi_locked : 1;
26
+
unsigned long ufcfi_en : 1; /* Enable for forward cfi. Note that ELP goes in sstatus */
27
+
unsigned long ufcfi_locked : 1;
28
+
unsigned long user_shdw_stk; /* Current user shadow stack pointer */
29
+
unsigned long shdw_stk_base; /* Base address of shadow stack */
30
+
unsigned long shdw_stk_size; /* size of shadow stack */
31
+
};
32
+
33
+
unsigned long shstk_alloc_thread_stack(struct task_struct *tsk,
34
+
const struct kernel_clone_args *args);
35
+
void shstk_release(struct task_struct *tsk);
36
+
void set_shstk_base(struct task_struct *task, unsigned long shstk_addr, unsigned long size);
37
+
unsigned long get_shstk_base(struct task_struct *task, unsigned long *size);
38
+
void set_active_shstk(struct task_struct *task, unsigned long shstk_addr);
39
+
bool is_shstk_enabled(struct task_struct *task);
40
+
bool is_shstk_locked(struct task_struct *task);
41
+
bool is_shstk_allocated(struct task_struct *task);
42
+
void set_shstk_lock(struct task_struct *task);
43
+
void set_shstk_status(struct task_struct *task, bool enable);
44
+
unsigned long get_active_shstk(struct task_struct *task);
45
+
int restore_user_shstk(struct task_struct *tsk, unsigned long shstk_ptr);
46
+
int save_user_shstk(struct task_struct *tsk, unsigned long *saved_shstk_ptr);
47
+
bool is_indir_lp_enabled(struct task_struct *task);
48
+
bool is_indir_lp_locked(struct task_struct *task);
49
+
void set_indir_lp_status(struct task_struct *task, bool enable);
50
+
void set_indir_lp_lock(struct task_struct *task);
51
+
52
+
#define PR_SHADOW_STACK_SUPPORTED_STATUS_MASK (PR_SHADOW_STACK_ENABLE)
53
+
54
+
#else
55
+
56
+
#define shstk_alloc_thread_stack(tsk, args) 0
57
+
58
+
#define shstk_release(tsk)
59
+
60
+
#define get_shstk_base(task, size) 0UL
61
+
62
+
#define set_shstk_base(task, shstk_addr, size) do {} while (0)
63
+
64
+
#define set_active_shstk(task, shstk_addr) do {} while (0)
65
+
66
+
#define is_shstk_enabled(task) false
67
+
68
+
#define is_shstk_locked(task) false
69
+
70
+
#define is_shstk_allocated(task) false
71
+
72
+
#define set_shstk_lock(task) do {} while (0)
73
+
74
+
#define set_shstk_status(task, enable) do {} while (0)
75
+
76
+
#define is_indir_lp_enabled(task) false
77
+
78
+
#define is_indir_lp_locked(task) false
79
+
80
+
#define set_indir_lp_status(task, enable) do {} while (0)
81
+
82
+
#define set_indir_lp_lock(task) do {} while (0)
83
+
84
+
#define restore_user_shstk(tsk, shstk_ptr) -EINVAL
85
+
86
+
#define save_user_shstk(tsk, saved_shstk_ptr) -EINVAL
87
+
88
+
#define get_active_shstk(task) 0UL
89
+
90
+
#endif /* CONFIG_RISCV_USER_CFI */
91
+
92
+
bool is_user_shstk_enabled(void);
93
+
bool is_user_lpad_enabled(void);
94
+
95
+
#endif /* __ASSEMBLER__ */
96
+
97
+
#endif /* _ASM_RISCV_USERCFI_H */
+12
-1
arch/riscv/include/asm/vdso.h
+12
-1
arch/riscv/include/asm/vdso.h
···
18
18
19
19
#ifndef __ASSEMBLER__
20
20
#include <generated/vdso-offsets.h>
21
+
#ifdef CONFIG_RISCV_USER_CFI
22
+
#include <generated/vdso-cfi-offsets.h>
23
+
#endif
21
24
25
+
#ifdef CONFIG_RISCV_USER_CFI
22
26
#define VDSO_SYMBOL(base, name) \
23
-
(void __user *)((unsigned long)(base) + __vdso_##name##_offset)
27
+
(riscv_has_extension_unlikely(RISCV_ISA_EXT_ZIMOP) ? \
28
+
(void __user *)((unsigned long)(base) + __vdso_##name##_cfi_offset) : \
29
+
(void __user *)((unsigned long)(base) + __vdso_##name##_offset))
30
+
#else
31
+
#define VDSO_SYMBOL(base, name) \
32
+
((void __user *)((unsigned long)(base) + __vdso_##name##_offset))
33
+
#endif
24
34
25
35
#ifdef CONFIG_COMPAT
26
36
#include <generated/compat_vdso-offsets.h>
···
43
33
#endif /* CONFIG_COMPAT */
44
34
45
35
extern char vdso_start[], vdso_end[];
36
+
extern char vdso_cfi_start[], vdso_cfi_end[];
46
37
47
38
#endif /* !__ASSEMBLER__ */
48
39
+4
arch/riscv/include/uapi/asm/hwprobe.h
+4
arch/riscv/include/uapi/asm/hwprobe.h
···
86
86
#define RISCV_HWPROBE_EXT_ZICBOP (1ULL << 60)
87
87
#define RISCV_HWPROBE_EXT_ZILSD (1ULL << 61)
88
88
#define RISCV_HWPROBE_EXT_ZCLSD (1ULL << 62)
89
+
#define RISCV_HWPROBE_EXT_ZICFILP (1ULL << 63)
89
90
90
91
#define RISCV_HWPROBE_KEY_CPUPERF_0 5
91
92
#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)
···
114
113
#define RISCV_HWPROBE_KEY_VENDOR_EXT_SIFIVE_0 13
115
114
#define RISCV_HWPROBE_KEY_VENDOR_EXT_MIPS_0 14
116
115
#define RISCV_HWPROBE_KEY_ZICBOP_BLOCK_SIZE 15
116
+
#define RISCV_HWPROBE_KEY_IMA_EXT_1 16
117
+
#define RISCV_HWPROBE_EXT_ZICFISS (1ULL << 0)
118
+
117
119
/* Increase RISCV_HWPROBE_MAX_KEY when adding items. */
118
120
119
121
/* Flags */
+34
arch/riscv/include/uapi/asm/ptrace.h
+34
arch/riscv/include/uapi/asm/ptrace.h
···
127
127
*/
128
128
#define RISCV_MAX_VLENB (8192)
129
129
130
+
struct __sc_riscv_cfi_state {
131
+
unsigned long ss_ptr; /* shadow stack pointer */
132
+
};
133
+
134
+
#define PTRACE_CFI_LP_EN_BIT 0
135
+
#define PTRACE_CFI_LP_LOCK_BIT 1
136
+
#define PTRACE_CFI_ELP_BIT 2
137
+
#define PTRACE_CFI_SS_EN_BIT 3
138
+
#define PTRACE_CFI_SS_LOCK_BIT 4
139
+
#define PTRACE_CFI_SS_PTR_BIT 5
140
+
141
+
#define PTRACE_CFI_LP_EN_STATE BIT(PTRACE_CFI_LP_EN_BIT)
142
+
#define PTRACE_CFI_LP_LOCK_STATE BIT(PTRACE_CFI_LP_LOCK_BIT)
143
+
#define PTRACE_CFI_ELP_STATE BIT(PTRACE_CFI_ELP_BIT)
144
+
#define PTRACE_CFI_SS_EN_STATE BIT(PTRACE_CFI_SS_EN_BIT)
145
+
#define PTRACE_CFI_SS_LOCK_STATE BIT(PTRACE_CFI_SS_LOCK_BIT)
146
+
#define PTRACE_CFI_SS_PTR_STATE BIT(PTRACE_CFI_SS_PTR_BIT)
147
+
148
+
#define PRACE_CFI_STATE_INVALID_MASK ~(PTRACE_CFI_LP_EN_STATE | \
149
+
PTRACE_CFI_LP_LOCK_STATE | \
150
+
PTRACE_CFI_ELP_STATE | \
151
+
PTRACE_CFI_SS_EN_STATE | \
152
+
PTRACE_CFI_SS_LOCK_STATE | \
153
+
PTRACE_CFI_SS_PTR_STATE)
154
+
155
+
struct __cfi_status {
156
+
__u64 cfi_state;
157
+
};
158
+
159
+
struct user_cfi_state {
160
+
struct __cfi_status cfi_status;
161
+
__u64 shstk_ptr;
162
+
};
163
+
130
164
#endif /* __ASSEMBLER__ */
131
165
132
166
#endif /* _UAPI_ASM_RISCV_PTRACE_H */
+1
arch/riscv/include/uapi/asm/sigcontext.h
+1
arch/riscv/include/uapi/asm/sigcontext.h
+2
arch/riscv/kernel/Makefile
+2
arch/riscv/kernel/Makefile
···
73
73
obj-y += probes/
74
74
obj-y += tests/
75
75
obj-$(CONFIG_MMU) += vdso.o vdso/
76
+
obj-$(CONFIG_RISCV_USER_CFI) += vdso_cfi/
76
77
77
78
obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o
78
79
obj-$(CONFIG_RISCV_MISALIGNED) += unaligned_access_speed.o
···
127
126
obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o
128
127
129
128
obj-$(CONFIG_GENERIC_CPU_VULNERABILITIES) += bugs.o
129
+
obj-$(CONFIG_RISCV_USER_CFI) += usercfi.o
+10
arch/riscv/kernel/asm-offsets.c
+10
arch/riscv/kernel/asm-offsets.c
···
51
51
#endif
52
52
53
53
OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu);
54
+
#ifdef CONFIG_RISCV_USER_CFI
55
+
OFFSET(TASK_TI_CFI_STATE, task_struct, thread_info.user_cfi_state);
56
+
OFFSET(TASK_TI_USER_SSP, task_struct, thread_info.user_cfi_state.user_shdw_stk);
57
+
#endif
54
58
OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]);
55
59
OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]);
56
60
OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]);
···
532
528
DEFINE(FREGS_A5, offsetof(struct __arch_ftrace_regs, a5));
533
529
DEFINE(FREGS_A6, offsetof(struct __arch_ftrace_regs, a6));
534
530
DEFINE(FREGS_A7, offsetof(struct __arch_ftrace_regs, a7));
531
+
#endif
532
+
#ifdef CONFIG_RISCV_SBI
533
+
DEFINE(SBI_EXT_FWFT, SBI_EXT_FWFT);
534
+
DEFINE(SBI_EXT_FWFT_SET, SBI_EXT_FWFT_SET);
535
+
DEFINE(SBI_FWFT_SHADOW_STACK, SBI_FWFT_SHADOW_STACK);
536
+
DEFINE(SBI_FWFT_SET_FLAG_LOCK, SBI_FWFT_SET_FLAG_LOCK);
535
537
#endif
536
538
}
+25
arch/riscv/kernel/cpufeature.c
+25
arch/riscv/kernel/cpufeature.c
···
28
28
#include <asm/vector.h>
29
29
#include <asm/vendor_extensions.h>
30
30
#include <asm/vendor_extensions/thead.h>
31
+
#include <asm/usercfi.h>
31
32
32
33
#define NUM_ALPHA_EXTS ('z' - 'a' + 1)
33
34
···
297
296
return 0;
298
297
}
299
298
299
+
static int riscv_cfilp_validate(const struct riscv_isa_ext_data *data,
300
+
const unsigned long *isa_bitmap)
301
+
{
302
+
if (!IS_ENABLED(CONFIG_RISCV_USER_CFI) ||
303
+
(riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_FCFI))
304
+
return -EINVAL;
305
+
306
+
return 0;
307
+
}
308
+
309
+
static int riscv_cfiss_validate(const struct riscv_isa_ext_data *data,
310
+
const unsigned long *isa_bitmap)
311
+
{
312
+
if (!IS_ENABLED(CONFIG_RISCV_USER_CFI) ||
313
+
(riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_BCFI))
314
+
return -EINVAL;
315
+
316
+
return 0;
317
+
}
318
+
300
319
static const unsigned int riscv_a_exts[] = {
301
320
RISCV_ISA_EXT_ZAAMO,
302
321
RISCV_ISA_EXT_ZALRSC,
···
503
482
__RISCV_ISA_EXT_DATA_VALIDATE(zicbop, RISCV_ISA_EXT_ZICBOP, riscv_ext_zicbop_validate),
504
483
__RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, riscv_ext_zicboz_validate),
505
484
__RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE),
485
+
__RISCV_ISA_EXT_SUPERSET_VALIDATE(zicfilp, RISCV_ISA_EXT_ZICFILP, riscv_xlinuxenvcfg_exts,
486
+
riscv_cfilp_validate),
487
+
__RISCV_ISA_EXT_SUPERSET_VALIDATE(zicfiss, RISCV_ISA_EXT_ZICFISS, riscv_xlinuxenvcfg_exts,
488
+
riscv_cfiss_validate),
506
489
__RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR),
507
490
__RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND),
508
491
__RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR),
+38
arch/riscv/kernel/entry.S
+38
arch/riscv/kernel/entry.S
···
92
92
REG_L a0, TASK_TI_A0(tp)
93
93
.endm
94
94
95
+
/*
96
+
* If previous mode was U, capture shadow stack pointer and save it away
97
+
* Zero CSR_SSP at the same time for sanitization.
98
+
*/
99
+
.macro save_userssp tmp, status
100
+
ALTERNATIVE("nops(4)",
101
+
__stringify( \
102
+
andi \tmp, \status, SR_SPP; \
103
+
bnez \tmp, skip_ssp_save; \
104
+
csrrw \tmp, CSR_SSP, x0; \
105
+
REG_S \tmp, TASK_TI_USER_SSP(tp); \
106
+
skip_ssp_save:),
107
+
0,
108
+
RISCV_ISA_EXT_ZICFISS,
109
+
CONFIG_RISCV_USER_CFI)
110
+
.endm
111
+
112
+
.macro restore_userssp tmp, status
113
+
ALTERNATIVE("nops(4)",
114
+
__stringify( \
115
+
andi \tmp, \status, SR_SPP; \
116
+
bnez \tmp, skip_ssp_restore; \
117
+
REG_L \tmp, TASK_TI_USER_SSP(tp); \
118
+
csrw CSR_SSP, \tmp; \
119
+
skip_ssp_restore:),
120
+
0,
121
+
RISCV_ISA_EXT_ZICFISS,
122
+
CONFIG_RISCV_USER_CFI)
123
+
.endm
95
124
96
125
SYM_CODE_START(handle_exception)
97
126
/*
···
174
145
* or vector in kernel space.
175
146
*/
176
147
li t0, SR_SUM | SR_FS_VS
148
+
#ifdef CONFIG_64BIT
149
+
li t1, SR_ELP
150
+
or t0, t0, t1
151
+
#endif
177
152
178
153
REG_L s0, TASK_TI_USER_SP(tp)
179
154
csrrc s1, CSR_STATUS, t0
155
+
save_userssp s2, s1
180
156
csrr s2, CSR_EPC
181
157
csrr s3, CSR_TVAL
182
158
csrr s4, CSR_CAUSE
···
277
243
call riscv_v_context_nesting_end
278
244
#endif
279
245
REG_L a0, PT_STATUS(sp)
246
+
restore_userssp s3, a0
280
247
/*
281
248
* The current load reservation is effectively part of the processor's
282
249
* state, in the sense that load reservations cannot be shared between
···
495
460
RISCV_PTR do_page_fault /* load page fault */
496
461
RISCV_PTR do_trap_unknown
497
462
RISCV_PTR do_page_fault /* store page fault */
463
+
RISCV_PTR do_trap_unknown /* cause=16 */
464
+
RISCV_PTR do_trap_unknown /* cause=17 */
465
+
RISCV_PTR do_trap_software_check /* cause=18 is sw check exception */
498
466
SYM_DATA_END_LABEL(excp_vect_table, SYM_L_LOCAL, excp_vect_table_end)
499
467
500
468
#ifndef CONFIG_MMU
+27
arch/riscv/kernel/head.S
+27
arch/riscv/kernel/head.S
···
15
15
#include <asm/image.h>
16
16
#include <asm/scs.h>
17
17
#include <asm/xip_fixup.h>
18
+
#include <asm/usercfi.h>
18
19
#include "efi-header.S"
19
20
20
21
__HEAD
···
171
170
call relocate_enable_mmu
172
171
#endif
173
172
call .Lsetup_trap_vector
173
+
#if defined(CONFIG_RISCV_SBI) && defined(CONFIG_RISCV_USER_CFI)
174
+
li a7, SBI_EXT_FWFT
175
+
li a6, SBI_EXT_FWFT_SET
176
+
li a0, SBI_FWFT_SHADOW_STACK
177
+
li a1, 1 /* enable supervisor to access shadow stack access */
178
+
li a2, SBI_FWFT_SET_FLAG_LOCK
179
+
ecall
180
+
beqz a0, 1f
181
+
la a1, riscv_nousercfi
182
+
li a0, CMDLINE_DISABLE_RISCV_USERCFI_BCFI
183
+
REG_S a0, (a1)
184
+
1:
185
+
#endif
174
186
scs_load_current
175
187
call smp_callin
176
188
#endif /* CONFIG_SMP */
···
344
330
la tp, init_task
345
331
la sp, init_thread_union + THREAD_SIZE
346
332
addi sp, sp, -PT_SIZE_ON_STACK
333
+
#if defined(CONFIG_RISCV_SBI) && defined(CONFIG_RISCV_USER_CFI)
334
+
li a7, SBI_EXT_FWFT
335
+
li a6, SBI_EXT_FWFT_SET
336
+
li a0, SBI_FWFT_SHADOW_STACK
337
+
li a1, 1 /* enable supervisor to access shadow stack access */
338
+
li a2, SBI_FWFT_SET_FLAG_LOCK
339
+
ecall
340
+
beqz a0, 1f
341
+
la a1, riscv_nousercfi
342
+
li a0, CMDLINE_DISABLE_RISCV_USERCFI_BCFI
343
+
REG_S a0, (a1)
344
+
1:
345
+
#endif
347
346
scs_load_current
348
347
349
348
#ifdef CONFIG_KASAN
+25
-2
arch/riscv/kernel/process.c
+25
-2
arch/riscv/kernel/process.c
···
31
31
#include <asm/vector.h>
32
32
#include <asm/cpufeature.h>
33
33
#include <asm/exec.h>
34
+
#include <asm/usercfi.h>
34
35
35
36
#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK)
36
37
#include <linux/stackprotector.h>
···
93
92
regs->s8, regs->s9, regs->s10);
94
93
pr_cont(" s11: " REG_FMT " t3 : " REG_FMT " t4 : " REG_FMT "\n",
95
94
regs->s11, regs->t3, regs->t4);
96
-
pr_cont(" t5 : " REG_FMT " t6 : " REG_FMT "\n",
97
-
regs->t5, regs->t6);
95
+
pr_cont(" t5 : " REG_FMT " t6 : " REG_FMT " ssp : " REG_FMT "\n",
96
+
regs->t5, regs->t6, get_active_shstk(current));
98
97
99
98
pr_cont("status: " REG_FMT " badaddr: " REG_FMT " cause: " REG_FMT "\n",
100
99
regs->status, regs->badaddr, regs->cause);
···
155
154
}
156
155
regs->epc = pc;
157
156
regs->sp = sp;
157
+
158
+
/*
159
+
* clear shadow stack state on exec.
160
+
* libc will set it later via prctl.
161
+
*/
162
+
set_shstk_status(current, false);
163
+
set_shstk_base(current, 0, 0);
164
+
set_active_shstk(current, 0);
165
+
/*
166
+
* disable indirect branch tracking on exec.
167
+
* libc will enable it later via prctl.
168
+
*/
169
+
set_indir_lp_status(current, false);
158
170
159
171
#ifdef CONFIG_64BIT
160
172
regs->status &= ~SR_UXL;
···
240
226
u64 clone_flags = args->flags;
241
227
unsigned long usp = args->stack;
242
228
unsigned long tls = args->tls;
229
+
unsigned long ssp = 0;
243
230
struct pt_regs *childregs = task_pt_regs(p);
244
231
245
232
/* Ensure all threads in this mm have the same pointer masking mode. */
···
260
245
p->thread.s[1] = (unsigned long)args->fn_arg;
261
246
p->thread.ra = (unsigned long)ret_from_fork_kernel_asm;
262
247
} else {
248
+
/* allocate new shadow stack if needed. In case of CLONE_VM we have to */
249
+
ssp = shstk_alloc_thread_stack(p, args);
250
+
if (IS_ERR_VALUE(ssp))
251
+
return PTR_ERR((void *)ssp);
252
+
263
253
*childregs = *(current_pt_regs());
264
254
/* Turn off status.VS */
265
255
riscv_v_vstate_off(childregs);
266
256
if (usp) /* User fork */
267
257
childregs->sp = usp;
258
+
/* if needed, set new ssp */
259
+
if (ssp)
260
+
set_active_shstk(p, ssp);
268
261
if (clone_flags & CLONE_SETTLS)
269
262
childregs->tp = tls;
270
263
childregs->a0 = 0; /* Return value of fork() */
+190
-3
arch/riscv/kernel/ptrace.c
+190
-3
arch/riscv/kernel/ptrace.c
···
19
19
#include <linux/regset.h>
20
20
#include <linux/sched.h>
21
21
#include <linux/sched/task_stack.h>
22
+
#include <asm/usercfi.h>
22
23
23
24
enum riscv_regset {
24
25
REGSET_X,
···
31
30
#endif
32
31
#ifdef CONFIG_RISCV_ISA_SUPM
33
32
REGSET_TAGGED_ADDR_CTRL,
33
+
#endif
34
+
#ifdef CONFIG_RISCV_USER_CFI
35
+
REGSET_CFI,
34
36
#endif
35
37
};
36
38
···
99
95
struct __riscv_v_ext_state *vstate = &target->thread.vstate;
100
96
struct __riscv_v_regset_state ptrace_vstate;
101
97
102
-
if (!riscv_v_vstate_query(task_pt_regs(target)))
98
+
if (!(has_vector() || has_xtheadvector()))
103
99
return -EINVAL;
100
+
101
+
if (!riscv_v_vstate_query(task_pt_regs(target)))
102
+
return -ENODATA;
104
103
105
104
/*
106
105
* Ensure the vector registers have been saved to the memory before
···
128
121
return membuf_write(&to, vstate->datap, riscv_v_vsize);
129
122
}
130
123
124
+
static int invalid_ptrace_v_csr(struct __riscv_v_ext_state *vstate,
125
+
struct __riscv_v_regset_state *ptrace)
126
+
{
127
+
unsigned long vsew, vlmul, vfrac, vl;
128
+
unsigned long elen, vlen;
129
+
unsigned long sew, lmul;
130
+
unsigned long reserved;
131
+
132
+
vlen = vstate->vlenb * 8;
133
+
if (vstate->vlenb != ptrace->vlenb)
134
+
return 1;
135
+
136
+
/* do not allow to set vcsr/vxrm/vxsat reserved bits */
137
+
reserved = ~(CSR_VXSAT_MASK | (CSR_VXRM_MASK << CSR_VXRM_SHIFT));
138
+
if (ptrace->vcsr & reserved)
139
+
return 1;
140
+
141
+
if (has_vector()) {
142
+
/* do not allow to set vtype reserved bits and vill bit */
143
+
reserved = ~(VTYPE_VSEW | VTYPE_VLMUL | VTYPE_VMA | VTYPE_VTA);
144
+
if (ptrace->vtype & reserved)
145
+
return 1;
146
+
147
+
elen = riscv_has_extension_unlikely(RISCV_ISA_EXT_ZVE64X) ? 64 : 32;
148
+
vsew = (ptrace->vtype & VTYPE_VSEW) >> VTYPE_VSEW_SHIFT;
149
+
sew = 8 << vsew;
150
+
151
+
if (sew > elen)
152
+
return 1;
153
+
154
+
vfrac = (ptrace->vtype & VTYPE_VLMUL_FRAC);
155
+
vlmul = (ptrace->vtype & VTYPE_VLMUL);
156
+
157
+
/* RVV 1.0 spec 3.4.2: VLMUL(0x4) reserved */
158
+
if (vlmul == 4)
159
+
return 1;
160
+
161
+
/* RVV 1.0 spec 3.4.2: (LMUL < SEW_min / ELEN) reserved */
162
+
if (vlmul == 5 && elen == 32)
163
+
return 1;
164
+
165
+
/* for zero vl verify that at least one element is possible */
166
+
vl = ptrace->vl ? ptrace->vl : 1;
167
+
168
+
if (vfrac) {
169
+
/* integer 1/LMUL: VL =< VLMAX = VLEN / SEW / LMUL */
170
+
lmul = 2 << (3 - (vlmul - vfrac));
171
+
if (vlen < vl * sew * lmul)
172
+
return 1;
173
+
} else {
174
+
/* integer LMUL: VL =< VLMAX = LMUL * VLEN / SEW */
175
+
lmul = 1 << vlmul;
176
+
if (vl * sew > lmul * vlen)
177
+
return 1;
178
+
}
179
+
}
180
+
181
+
if (has_xtheadvector()) {
182
+
/* do not allow to set vtype reserved bits and vill bit */
183
+
reserved = ~(VTYPE_VSEW_THEAD | VTYPE_VLMUL_THEAD | VTYPE_VEDIV_THEAD);
184
+
if (ptrace->vtype & reserved)
185
+
return 1;
186
+
187
+
/*
188
+
* THead ISA Extension spec chapter 16:
189
+
* divided element extension ('Zvediv') is not part of XTheadVector
190
+
*/
191
+
if (ptrace->vtype & VTYPE_VEDIV_THEAD)
192
+
return 1;
193
+
194
+
vsew = (ptrace->vtype & VTYPE_VSEW_THEAD) >> VTYPE_VSEW_THEAD_SHIFT;
195
+
sew = 8 << vsew;
196
+
197
+
vlmul = (ptrace->vtype & VTYPE_VLMUL_THEAD);
198
+
lmul = 1 << vlmul;
199
+
200
+
/* for zero vl verify that at least one element is possible */
201
+
vl = ptrace->vl ? ptrace->vl : 1;
202
+
203
+
if (vl * sew > lmul * vlen)
204
+
return 1;
205
+
}
206
+
207
+
return 0;
208
+
}
209
+
131
210
static int riscv_vr_set(struct task_struct *target,
132
211
const struct user_regset *regset,
133
212
unsigned int pos, unsigned int count,
···
223
130
struct __riscv_v_ext_state *vstate = &target->thread.vstate;
224
131
struct __riscv_v_regset_state ptrace_vstate;
225
132
226
-
if (!riscv_v_vstate_query(task_pt_regs(target)))
133
+
if (!(has_vector() || has_xtheadvector()))
227
134
return -EINVAL;
135
+
136
+
if (!riscv_v_vstate_query(task_pt_regs(target)))
137
+
return -ENODATA;
228
138
229
139
/* Copy rest of the vstate except datap */
230
140
ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &ptrace_vstate, 0,
···
235
139
if (unlikely(ret))
236
140
return ret;
237
141
238
-
if (vstate->vlenb != ptrace_vstate.vlenb)
142
+
if (invalid_ptrace_v_csr(vstate, &ptrace_vstate))
239
143
return -EINVAL;
240
144
241
145
vstate->vstart = ptrace_vstate.vstart;
···
291
195
}
292
196
#endif
293
197
198
+
#ifdef CONFIG_RISCV_USER_CFI
199
+
static int riscv_cfi_get(struct task_struct *target,
200
+
const struct user_regset *regset,
201
+
struct membuf to)
202
+
{
203
+
struct user_cfi_state user_cfi;
204
+
struct pt_regs *regs;
205
+
206
+
memset(&user_cfi, 0, sizeof(user_cfi));
207
+
regs = task_pt_regs(target);
208
+
209
+
if (is_indir_lp_enabled(target)) {
210
+
user_cfi.cfi_status.cfi_state |= PTRACE_CFI_LP_EN_STATE;
211
+
user_cfi.cfi_status.cfi_state |= is_indir_lp_locked(target) ?
212
+
PTRACE_CFI_LP_LOCK_STATE : 0;
213
+
user_cfi.cfi_status.cfi_state |= (regs->status & SR_ELP) ?
214
+
PTRACE_CFI_ELP_STATE : 0;
215
+
}
216
+
217
+
if (is_shstk_enabled(target)) {
218
+
user_cfi.cfi_status.cfi_state |= (PTRACE_CFI_SS_EN_STATE |
219
+
PTRACE_CFI_SS_PTR_STATE);
220
+
user_cfi.cfi_status.cfi_state |= is_shstk_locked(target) ?
221
+
PTRACE_CFI_SS_LOCK_STATE : 0;
222
+
user_cfi.shstk_ptr = get_active_shstk(target);
223
+
}
224
+
225
+
return membuf_write(&to, &user_cfi, sizeof(user_cfi));
226
+
}
227
+
228
+
/*
229
+
* Does it make sense to allow enable / disable of cfi via ptrace?
230
+
* We don't allow enable / disable / locking control via ptrace for now.
231
+
* Setting the shadow stack pointer is allowed. GDB might use it to unwind or
232
+
* some other fixup. Similarly gdb might want to suppress elp and may want
233
+
* to reset elp state.
234
+
*/
235
+
static int riscv_cfi_set(struct task_struct *target,
236
+
const struct user_regset *regset,
237
+
unsigned int pos, unsigned int count,
238
+
const void *kbuf, const void __user *ubuf)
239
+
{
240
+
int ret;
241
+
struct user_cfi_state user_cfi;
242
+
struct pt_regs *regs;
243
+
244
+
regs = task_pt_regs(target);
245
+
246
+
ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &user_cfi, 0, -1);
247
+
if (ret)
248
+
return ret;
249
+
250
+
/*
251
+
* Not allowing enabling or locking shadow stack or landing pad
252
+
* There is no disabling of shadow stack or landing pad via ptrace
253
+
* rsvd field should be set to zero so that if those fields are needed in future
254
+
*/
255
+
if ((user_cfi.cfi_status.cfi_state &
256
+
(PTRACE_CFI_LP_EN_STATE | PTRACE_CFI_LP_LOCK_STATE |
257
+
PTRACE_CFI_SS_EN_STATE | PTRACE_CFI_SS_LOCK_STATE)) ||
258
+
(user_cfi.cfi_status.cfi_state & PRACE_CFI_STATE_INVALID_MASK))
259
+
return -EINVAL;
260
+
261
+
/* If lpad is enabled on target and ptrace requests to set / clear elp, do that */
262
+
if (is_indir_lp_enabled(target)) {
263
+
if (user_cfi.cfi_status.cfi_state &
264
+
PTRACE_CFI_ELP_STATE) /* set elp state */
265
+
regs->status |= SR_ELP;
266
+
else
267
+
regs->status &= ~SR_ELP; /* clear elp state */
268
+
}
269
+
270
+
/* If shadow stack enabled on target, set new shadow stack pointer */
271
+
if (is_shstk_enabled(target) &&
272
+
(user_cfi.cfi_status.cfi_state & PTRACE_CFI_SS_PTR_STATE))
273
+
set_active_shstk(target, user_cfi.shstk_ptr);
274
+
275
+
return 0;
276
+
}
277
+
#endif
278
+
294
279
static struct user_regset riscv_user_regset[] __ro_after_init = {
295
280
[REGSET_X] = {
296
281
USER_REGSET_NOTE_TYPE(PRSTATUS),
···
409
232
.align = sizeof(long),
410
233
.regset_get = tagged_addr_ctrl_get,
411
234
.set = tagged_addr_ctrl_set,
235
+
},
236
+
#endif
237
+
#ifdef CONFIG_RISCV_USER_CFI
238
+
[REGSET_CFI] = {
239
+
.core_note_type = NT_RISCV_USER_CFI,
240
+
.align = sizeof(__u64),
241
+
.n = sizeof(struct user_cfi_state) / sizeof(__u64),
242
+
.size = sizeof(__u64),
243
+
.regset_get = riscv_cfi_get,
244
+
.set = riscv_cfi_set,
412
245
},
413
246
#endif
414
247
};
+86
arch/riscv/kernel/signal.c
+86
arch/riscv/kernel/signal.c
···
22
22
#include <asm/vector.h>
23
23
#include <asm/csr.h>
24
24
#include <asm/cacheflush.h>
25
+
#include <asm/usercfi.h>
25
26
26
27
unsigned long signal_minsigstksz __ro_after_init;
27
28
28
29
extern u32 __user_rt_sigreturn[2];
29
30
static size_t riscv_v_sc_size __ro_after_init;
31
+
static size_t riscv_zicfiss_sc_size __ro_after_init;
30
32
31
33
#define DEBUG_SIG 0
32
34
···
142
140
return copy_from_user(current->thread.vstate.datap, datap, riscv_v_vsize);
143
141
}
144
142
143
+
static long save_cfiss_state(struct pt_regs *regs, void __user *sc_cfi)
144
+
{
145
+
struct __sc_riscv_cfi_state __user *state = sc_cfi;
146
+
unsigned long ss_ptr = 0;
147
+
long err = 0;
148
+
149
+
if (!is_shstk_enabled(current))
150
+
return 0;
151
+
152
+
/*
153
+
* Save a pointer to the shadow stack itself on shadow stack as a form of token.
154
+
* A token on the shadow stack gives the following properties:
155
+
* - Safe save and restore for shadow stack switching. Any save of a shadow stack
156
+
* must have saved a token on the shadow stack. Similarly any restore of shadow
157
+
* stack must check the token before restore. Since writing to the shadow stack with
158
+
* address of the shadow stack itself is not easily allowed, a restore without a save
159
+
* is quite difficult for an attacker to perform.
160
+
* - A natural break. A token in shadow stack provides a natural break in shadow stack
161
+
* So a single linear range can be bucketed into different shadow stack segments. Any
162
+
* sspopchk will detect the condition and fault to kernel as a sw check exception.
163
+
*/
164
+
err |= save_user_shstk(current, &ss_ptr);
165
+
err |= __put_user(ss_ptr, &state->ss_ptr);
166
+
if (unlikely(err))
167
+
return -EFAULT;
168
+
169
+
return riscv_zicfiss_sc_size;
170
+
}
171
+
172
+
static long __restore_cfiss_state(struct pt_regs *regs, void __user *sc_cfi)
173
+
{
174
+
struct __sc_riscv_cfi_state __user *state = sc_cfi;
175
+
unsigned long ss_ptr = 0;
176
+
long err;
177
+
178
+
/*
179
+
* Restore shadow stack as a form of token stored on the shadow stack itself as a safe
180
+
* way to restore.
181
+
* A token on the shadow stack gives the following properties:
182
+
* - Safe save and restore for shadow stack switching. Any save of shadow stack
183
+
* must have saved a token on shadow stack. Similarly any restore of shadow
184
+
* stack must check the token before restore. Since writing to a shadow stack with
185
+
* the address of shadow stack itself is not easily allowed, a restore without a save
186
+
* is quite difficult for an attacker to perform.
187
+
* - A natural break. A token in the shadow stack provides a natural break in shadow stack
188
+
* So a single linear range can be bucketed into different shadow stack segments.
189
+
* sspopchk will detect the condition and fault to kernel as a sw check exception.
190
+
*/
191
+
err = __copy_from_user(&ss_ptr, &state->ss_ptr, sizeof(unsigned long));
192
+
193
+
if (unlikely(err))
194
+
return err;
195
+
196
+
return restore_user_shstk(current, ss_ptr);
197
+
}
198
+
145
199
struct arch_ext_priv {
146
200
__u32 magic;
147
201
long (*save)(struct pt_regs *regs, void __user *sc_vec);
···
207
149
{
208
150
.magic = RISCV_V_MAGIC,
209
151
.save = &save_v_state,
152
+
},
153
+
{
154
+
.magic = RISCV_ZICFISS_MAGIC,
155
+
.save = &save_cfiss_state,
210
156
},
211
157
};
212
158
···
264
202
265
203
err = __restore_v_state(regs, sc_ext_ptr);
266
204
break;
205
+
case RISCV_ZICFISS_MAGIC:
206
+
if (!is_shstk_enabled(current) || size != riscv_zicfiss_sc_size)
207
+
return -EINVAL;
208
+
209
+
err = __restore_cfiss_state(regs, sc_ext_ptr);
210
+
break;
267
211
default:
268
212
return -EINVAL;
269
213
}
···
290
222
if (cal_all || riscv_v_vstate_query(task_pt_regs(current)))
291
223
total_context_size += riscv_v_sc_size;
292
224
}
225
+
226
+
if (is_shstk_enabled(current))
227
+
total_context_size += riscv_zicfiss_sc_size;
228
+
229
+
/*
230
+
* Preserved a __riscv_ctx_hdr for END signal context header if an
231
+
* extension uses __riscv_extra_ext_header
232
+
*/
233
+
if (total_context_size)
234
+
total_context_size += sizeof(struct __riscv_ctx_hdr);
293
235
294
236
frame_size += total_context_size;
295
237
···
437
359
#ifdef CONFIG_MMU
438
360
regs->ra = (unsigned long)VDSO_SYMBOL(
439
361
current->mm->context.vdso, rt_sigreturn);
362
+
363
+
/* if bcfi is enabled x1 (ra) and x5 (t0) must match. not sure if we need this? */
364
+
if (is_shstk_enabled(current))
365
+
regs->t0 = regs->ra;
366
+
440
367
#else
441
368
/*
442
369
* For the nommu case we don't have a VDSO. Instead we push two
···
570
487
{
571
488
riscv_v_sc_size = sizeof(struct __riscv_ctx_hdr) +
572
489
sizeof(struct __sc_riscv_v_state) + riscv_v_vsize;
490
+
491
+
riscv_zicfiss_sc_size = sizeof(struct __riscv_ctx_hdr) +
492
+
sizeof(struct __sc_riscv_cfi_state);
573
493
/*
574
494
* Determine the stack space required for guaranteed signal delivery.
575
495
* The signal_minsigstksz will be populated into the AT_MINSIGSTKSZ entry
+101
-69
arch/riscv/kernel/sys_hwprobe.c
+101
-69
arch/riscv/kernel/sys_hwprobe.c
···
24
24
#include <vdso/vsyscall.h>
25
25
26
26
27
+
#define EXT_KEY(isa_arg, ext, pv, missing) \
28
+
do { \
29
+
if (__riscv_isa_extension_available(isa_arg, RISCV_ISA_EXT_##ext)) \
30
+
pv |= RISCV_HWPROBE_EXT_##ext; \
31
+
else \
32
+
missing |= RISCV_HWPROBE_EXT_##ext; \
33
+
} while (false)
34
+
27
35
static void hwprobe_arch_id(struct riscv_hwprobe *pair,
28
36
const struct cpumask *cpus)
29
37
{
···
101
93
for_each_cpu(cpu, cpus) {
102
94
struct riscv_isainfo *isainfo = &hart_isa[cpu];
103
95
104
-
#define EXT_KEY(ext) \
105
-
do { \
106
-
if (__riscv_isa_extension_available(isainfo->isa, RISCV_ISA_EXT_##ext)) \
107
-
pair->value |= RISCV_HWPROBE_EXT_##ext; \
108
-
else \
109
-
missing |= RISCV_HWPROBE_EXT_##ext; \
110
-
} while (false)
111
-
112
96
/*
113
97
* Only use EXT_KEY() for extensions which can be exposed to userspace,
114
98
* regardless of the kernel's configuration, as no other checks, besides
115
99
* presence in the hart_isa bitmap, are made.
116
100
*/
117
-
EXT_KEY(ZAAMO);
118
-
EXT_KEY(ZABHA);
119
-
EXT_KEY(ZACAS);
120
-
EXT_KEY(ZALASR);
121
-
EXT_KEY(ZALRSC);
122
-
EXT_KEY(ZAWRS);
123
-
EXT_KEY(ZBA);
124
-
EXT_KEY(ZBB);
125
-
EXT_KEY(ZBC);
126
-
EXT_KEY(ZBKB);
127
-
EXT_KEY(ZBKC);
128
-
EXT_KEY(ZBKX);
129
-
EXT_KEY(ZBS);
130
-
EXT_KEY(ZCA);
131
-
EXT_KEY(ZCB);
132
-
EXT_KEY(ZCLSD);
133
-
EXT_KEY(ZCMOP);
134
-
EXT_KEY(ZICBOM);
135
-
EXT_KEY(ZICBOP);
136
-
EXT_KEY(ZICBOZ);
137
-
EXT_KEY(ZICNTR);
138
-
EXT_KEY(ZICOND);
139
-
EXT_KEY(ZIHINTNTL);
140
-
EXT_KEY(ZIHINTPAUSE);
141
-
EXT_KEY(ZIHPM);
142
-
EXT_KEY(ZILSD);
143
-
EXT_KEY(ZIMOP);
144
-
EXT_KEY(ZKND);
145
-
EXT_KEY(ZKNE);
146
-
EXT_KEY(ZKNH);
147
-
EXT_KEY(ZKSED);
148
-
EXT_KEY(ZKSH);
149
-
EXT_KEY(ZKT);
150
-
EXT_KEY(ZTSO);
101
+
EXT_KEY(isainfo->isa, ZAAMO, pair->value, missing);
102
+
EXT_KEY(isainfo->isa, ZABHA, pair->value, missing);
103
+
EXT_KEY(isainfo->isa, ZACAS, pair->value, missing);
104
+
EXT_KEY(isainfo->isa, ZALASR, pair->value, missing);
105
+
EXT_KEY(isainfo->isa, ZALRSC, pair->value, missing);
106
+
EXT_KEY(isainfo->isa, ZAWRS, pair->value, missing);
107
+
EXT_KEY(isainfo->isa, ZBA, pair->value, missing);
108
+
EXT_KEY(isainfo->isa, ZBB, pair->value, missing);
109
+
EXT_KEY(isainfo->isa, ZBC, pair->value, missing);
110
+
EXT_KEY(isainfo->isa, ZBKB, pair->value, missing);
111
+
EXT_KEY(isainfo->isa, ZBKC, pair->value, missing);
112
+
EXT_KEY(isainfo->isa, ZBKX, pair->value, missing);
113
+
EXT_KEY(isainfo->isa, ZBS, pair->value, missing);
114
+
EXT_KEY(isainfo->isa, ZCA, pair->value, missing);
115
+
EXT_KEY(isainfo->isa, ZCB, pair->value, missing);
116
+
EXT_KEY(isainfo->isa, ZCLSD, pair->value, missing);
117
+
EXT_KEY(isainfo->isa, ZCMOP, pair->value, missing);
118
+
EXT_KEY(isainfo->isa, ZICBOM, pair->value, missing);
119
+
EXT_KEY(isainfo->isa, ZICBOP, pair->value, missing);
120
+
EXT_KEY(isainfo->isa, ZICBOZ, pair->value, missing);
121
+
EXT_KEY(isainfo->isa, ZICFILP, pair->value, missing);
122
+
EXT_KEY(isainfo->isa, ZICNTR, pair->value, missing);
123
+
EXT_KEY(isainfo->isa, ZICOND, pair->value, missing);
124
+
EXT_KEY(isainfo->isa, ZIHINTNTL, pair->value, missing);
125
+
EXT_KEY(isainfo->isa, ZIHINTPAUSE, pair->value, missing);
126
+
EXT_KEY(isainfo->isa, ZIHPM, pair->value, missing);
127
+
EXT_KEY(isainfo->isa, ZILSD, pair->value, missing);
128
+
EXT_KEY(isainfo->isa, ZIMOP, pair->value, missing);
129
+
EXT_KEY(isainfo->isa, ZKND, pair->value, missing);
130
+
EXT_KEY(isainfo->isa, ZKNE, pair->value, missing);
131
+
EXT_KEY(isainfo->isa, ZKNH, pair->value, missing);
132
+
EXT_KEY(isainfo->isa, ZKSED, pair->value, missing);
133
+
EXT_KEY(isainfo->isa, ZKSH, pair->value, missing);
134
+
EXT_KEY(isainfo->isa, ZKT, pair->value, missing);
135
+
EXT_KEY(isainfo->isa, ZTSO, pair->value, missing);
151
136
152
137
/*
153
138
* All the following extensions must depend on the kernel
154
139
* support of V.
155
140
*/
156
141
if (has_vector()) {
157
-
EXT_KEY(ZVBB);
158
-
EXT_KEY(ZVBC);
159
-
EXT_KEY(ZVE32F);
160
-
EXT_KEY(ZVE32X);
161
-
EXT_KEY(ZVE64D);
162
-
EXT_KEY(ZVE64F);
163
-
EXT_KEY(ZVE64X);
164
-
EXT_KEY(ZVFBFMIN);
165
-
EXT_KEY(ZVFBFWMA);
166
-
EXT_KEY(ZVFH);
167
-
EXT_KEY(ZVFHMIN);
168
-
EXT_KEY(ZVKB);
169
-
EXT_KEY(ZVKG);
170
-
EXT_KEY(ZVKNED);
171
-
EXT_KEY(ZVKNHA);
172
-
EXT_KEY(ZVKNHB);
173
-
EXT_KEY(ZVKSED);
174
-
EXT_KEY(ZVKSH);
175
-
EXT_KEY(ZVKT);
142
+
EXT_KEY(isainfo->isa, ZVBB, pair->value, missing);
143
+
EXT_KEY(isainfo->isa, ZVBC, pair->value, missing);
144
+
EXT_KEY(isainfo->isa, ZVE32F, pair->value, missing);
145
+
EXT_KEY(isainfo->isa, ZVE32X, pair->value, missing);
146
+
EXT_KEY(isainfo->isa, ZVE64D, pair->value, missing);
147
+
EXT_KEY(isainfo->isa, ZVE64F, pair->value, missing);
148
+
EXT_KEY(isainfo->isa, ZVE64X, pair->value, missing);
149
+
EXT_KEY(isainfo->isa, ZVFBFMIN, pair->value, missing);
150
+
EXT_KEY(isainfo->isa, ZVFBFWMA, pair->value, missing);
151
+
EXT_KEY(isainfo->isa, ZVFH, pair->value, missing);
152
+
EXT_KEY(isainfo->isa, ZVFHMIN, pair->value, missing);
153
+
EXT_KEY(isainfo->isa, ZVKB, pair->value, missing);
154
+
EXT_KEY(isainfo->isa, ZVKG, pair->value, missing);
155
+
EXT_KEY(isainfo->isa, ZVKNED, pair->value, missing);
156
+
EXT_KEY(isainfo->isa, ZVKNHA, pair->value, missing);
157
+
EXT_KEY(isainfo->isa, ZVKNHB, pair->value, missing);
158
+
EXT_KEY(isainfo->isa, ZVKSED, pair->value, missing);
159
+
EXT_KEY(isainfo->isa, ZVKSH, pair->value, missing);
160
+
EXT_KEY(isainfo->isa, ZVKT, pair->value, missing);
176
161
}
177
162
178
-
EXT_KEY(ZCD);
179
-
EXT_KEY(ZCF);
180
-
EXT_KEY(ZFA);
181
-
EXT_KEY(ZFBFMIN);
182
-
EXT_KEY(ZFH);
183
-
EXT_KEY(ZFHMIN);
163
+
EXT_KEY(isainfo->isa, ZCD, pair->value, missing);
164
+
EXT_KEY(isainfo->isa, ZCF, pair->value, missing);
165
+
EXT_KEY(isainfo->isa, ZFA, pair->value, missing);
166
+
EXT_KEY(isainfo->isa, ZFBFMIN, pair->value, missing);
167
+
EXT_KEY(isainfo->isa, ZFH, pair->value, missing);
168
+
EXT_KEY(isainfo->isa, ZFHMIN, pair->value, missing);
184
169
185
170
if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM))
186
-
EXT_KEY(SUPM);
187
-
#undef EXT_KEY
171
+
EXT_KEY(isainfo->isa, SUPM, pair->value, missing);
172
+
}
173
+
174
+
/* Now turn off reporting features if any CPU is missing it. */
175
+
pair->value &= ~missing;
176
+
}
177
+
178
+
static void hwprobe_isa_ext1(struct riscv_hwprobe *pair,
179
+
const struct cpumask *cpus)
180
+
{
181
+
int cpu;
182
+
u64 missing = 0;
183
+
184
+
pair->value = 0;
185
+
186
+
/*
187
+
* Loop through and record extensions that 1) anyone has, and 2) anyone
188
+
* doesn't have.
189
+
*/
190
+
for_each_cpu(cpu, cpus) {
191
+
struct riscv_isainfo *isainfo = &hart_isa[cpu];
192
+
193
+
/*
194
+
* Only use EXT_KEY() for extensions which can be
195
+
* exposed to userspace, regardless of the kernel's
196
+
* configuration, as no other checks, besides presence
197
+
* in the hart_isa bitmap, are made.
198
+
*/
199
+
EXT_KEY(isainfo->isa, ZICFISS, pair->value, missing);
188
200
}
189
201
190
202
/* Now turn off reporting features if any CPU is missing it. */
···
313
285
314
286
case RISCV_HWPROBE_KEY_IMA_EXT_0:
315
287
hwprobe_isa_ext0(pair, cpus);
288
+
break;
289
+
290
+
case RISCV_HWPROBE_KEY_IMA_EXT_1:
291
+
hwprobe_isa_ext1(pair, cpus);
316
292
break;
317
293
318
294
case RISCV_HWPROBE_KEY_CPUPERF_0:
+10
arch/riscv/kernel/sys_riscv.c
+10
arch/riscv/kernel/sys_riscv.c
···
7
7
8
8
#include <linux/syscalls.h>
9
9
#include <asm/cacheflush.h>
10
+
#include <asm-generic/mman-common.h>
10
11
11
12
static long riscv_sys_mmap(unsigned long addr, unsigned long len,
12
13
unsigned long prot, unsigned long flags,
···
16
15
{
17
16
if (unlikely(offset & (~PAGE_MASK >> page_shift_offset)))
18
17
return -EINVAL;
18
+
19
+
/*
20
+
* If PROT_WRITE is specified then extend that to PROT_READ
21
+
* protection_map[VM_WRITE] is now going to select shadow stack encodings.
22
+
* So specifying PROT_WRITE actually should select protection_map [VM_WRITE | VM_READ]
23
+
* If user wants to create shadow stack then they should use `map_shadow_stack` syscall.
24
+
*/
25
+
if (unlikely((prot & PROT_WRITE) && !(prot & PROT_READ)))
26
+
prot |= PROT_READ;
19
27
20
28
return ksys_mmap_pgoff(addr, len, prot, flags, fd,
21
29
offset >> (PAGE_SHIFT - page_shift_offset));
+54
arch/riscv/kernel/traps.c
+54
arch/riscv/kernel/traps.c
···
368
368
369
369
}
370
370
371
+
#define CFI_TVAL_FCFI_CODE 2
372
+
#define CFI_TVAL_BCFI_CODE 3
373
+
/* handle cfi violations */
374
+
bool handle_user_cfi_violation(struct pt_regs *regs)
375
+
{
376
+
unsigned long tval = csr_read(CSR_TVAL);
377
+
bool is_fcfi = (tval == CFI_TVAL_FCFI_CODE && cpu_supports_indirect_br_lp_instr());
378
+
bool is_bcfi = (tval == CFI_TVAL_BCFI_CODE && cpu_supports_shadow_stack());
379
+
380
+
/*
381
+
* Handle uprobe event first. The probe point can be a valid target
382
+
* of indirect jumps or calls, in this case, forward cfi violation
383
+
* will be triggered instead of breakpoint exception. Clear ELP flag
384
+
* on sstatus image as well to avoid recurring fault.
385
+
*/
386
+
if (is_fcfi && probe_breakpoint_handler(regs)) {
387
+
regs->status &= ~SR_ELP;
388
+
return true;
389
+
}
390
+
391
+
if (is_fcfi || is_bcfi) {
392
+
do_trap_error(regs, SIGSEGV, SEGV_CPERR, regs->epc,
393
+
"Oops - control flow violation");
394
+
return true;
395
+
}
396
+
397
+
return false;
398
+
}
399
+
400
+
/*
401
+
* software check exception is defined with risc-v cfi spec. Software check
402
+
* exception is raised when:
403
+
* a) An indirect branch doesn't land on 4 byte aligned PC or `lpad`
404
+
* instruction or `label` value programmed in `lpad` instr doesn't
405
+
* match with value setup in `x7`. reported code in `xtval` is 2.
406
+
* b) `sspopchk` instruction finds a mismatch between top of shadow stack (ssp)
407
+
* and x1/x5. reported code in `xtval` is 3.
408
+
*/
409
+
asmlinkage __visible __trap_section void do_trap_software_check(struct pt_regs *regs)
410
+
{
411
+
if (user_mode(regs)) {
412
+
irqentry_enter_from_user_mode(regs);
413
+
414
+
/* not a cfi violation, then merge into flow of unknown trap handler */
415
+
if (!handle_user_cfi_violation(regs))
416
+
do_trap_unknown(regs);
417
+
418
+
irqentry_exit_to_user_mode(regs);
419
+
} else {
420
+
/* sw check exception coming from kernel is a bug in kernel */
421
+
die(regs, "Kernel BUG");
422
+
}
423
+
}
424
+
371
425
#ifdef CONFIG_MMU
372
426
asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
373
427
{
+542
arch/riscv/kernel/usercfi.c
+542
arch/riscv/kernel/usercfi.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/*
3
+
* Copyright (C) 2024 Rivos, Inc.
4
+
* Deepak Gupta <debug@rivosinc.com>
5
+
*/
6
+
7
+
#include <linux/sched.h>
8
+
#include <linux/bitops.h>
9
+
#include <linux/types.h>
10
+
#include <linux/mm.h>
11
+
#include <linux/mman.h>
12
+
#include <linux/uaccess.h>
13
+
#include <linux/sizes.h>
14
+
#include <linux/user.h>
15
+
#include <linux/syscalls.h>
16
+
#include <linux/prctl.h>
17
+
#include <asm/csr.h>
18
+
#include <asm/usercfi.h>
19
+
20
+
unsigned long riscv_nousercfi __read_mostly;
21
+
22
+
#define SHSTK_ENTRY_SIZE sizeof(void *)
23
+
24
+
bool is_shstk_enabled(struct task_struct *task)
25
+
{
26
+
return task->thread_info.user_cfi_state.ubcfi_en;
27
+
}
28
+
29
+
bool is_shstk_allocated(struct task_struct *task)
30
+
{
31
+
return task->thread_info.user_cfi_state.shdw_stk_base;
32
+
}
33
+
34
+
bool is_shstk_locked(struct task_struct *task)
35
+
{
36
+
return task->thread_info.user_cfi_state.ubcfi_locked;
37
+
}
38
+
39
+
void set_shstk_base(struct task_struct *task, unsigned long shstk_addr, unsigned long size)
40
+
{
41
+
task->thread_info.user_cfi_state.shdw_stk_base = shstk_addr;
42
+
task->thread_info.user_cfi_state.shdw_stk_size = size;
43
+
}
44
+
45
+
unsigned long get_shstk_base(struct task_struct *task, unsigned long *size)
46
+
{
47
+
if (size)
48
+
*size = task->thread_info.user_cfi_state.shdw_stk_size;
49
+
return task->thread_info.user_cfi_state.shdw_stk_base;
50
+
}
51
+
52
+
void set_active_shstk(struct task_struct *task, unsigned long shstk_addr)
53
+
{
54
+
task->thread_info.user_cfi_state.user_shdw_stk = shstk_addr;
55
+
}
56
+
57
+
unsigned long get_active_shstk(struct task_struct *task)
58
+
{
59
+
return task->thread_info.user_cfi_state.user_shdw_stk;
60
+
}
61
+
62
+
void set_shstk_status(struct task_struct *task, bool enable)
63
+
{
64
+
if (!is_user_shstk_enabled())
65
+
return;
66
+
67
+
task->thread_info.user_cfi_state.ubcfi_en = enable ? 1 : 0;
68
+
69
+
if (enable)
70
+
task->thread.envcfg |= ENVCFG_SSE;
71
+
else
72
+
task->thread.envcfg &= ~ENVCFG_SSE;
73
+
74
+
csr_write(CSR_ENVCFG, task->thread.envcfg);
75
+
}
76
+
77
+
void set_shstk_lock(struct task_struct *task)
78
+
{
79
+
task->thread_info.user_cfi_state.ubcfi_locked = 1;
80
+
}
81
+
82
+
bool is_indir_lp_enabled(struct task_struct *task)
83
+
{
84
+
return task->thread_info.user_cfi_state.ufcfi_en;
85
+
}
86
+
87
+
bool is_indir_lp_locked(struct task_struct *task)
88
+
{
89
+
return task->thread_info.user_cfi_state.ufcfi_locked;
90
+
}
91
+
92
+
void set_indir_lp_status(struct task_struct *task, bool enable)
93
+
{
94
+
if (!is_user_lpad_enabled())
95
+
return;
96
+
97
+
task->thread_info.user_cfi_state.ufcfi_en = enable ? 1 : 0;
98
+
99
+
if (enable)
100
+
task->thread.envcfg |= ENVCFG_LPE;
101
+
else
102
+
task->thread.envcfg &= ~ENVCFG_LPE;
103
+
104
+
csr_write(CSR_ENVCFG, task->thread.envcfg);
105
+
}
106
+
107
+
void set_indir_lp_lock(struct task_struct *task)
108
+
{
109
+
task->thread_info.user_cfi_state.ufcfi_locked = 1;
110
+
}
111
+
/*
112
+
* If size is 0, then to be compatible with regular stack we want it to be as big as
113
+
* regular stack. Else PAGE_ALIGN it and return back
114
+
*/
115
+
static unsigned long calc_shstk_size(unsigned long size)
116
+
{
117
+
if (size)
118
+
return PAGE_ALIGN(size);
119
+
120
+
return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G));
121
+
}
122
+
123
+
/*
124
+
* Writes on shadow stack can either be `sspush` or `ssamoswap`. `sspush` can happen
125
+
* implicitly on current shadow stack pointed to by CSR_SSP. `ssamoswap` takes pointer to
126
+
* shadow stack. To keep it simple, we plan to use `ssamoswap` to perform writes on shadow
127
+
* stack.
128
+
*/
129
+
static noinline unsigned long amo_user_shstk(unsigned long __user *addr, unsigned long val)
130
+
{
131
+
/*
132
+
* Never expect -1 on shadow stack. Expect return addresses and zero
133
+
*/
134
+
unsigned long swap = -1;
135
+
136
+
__enable_user_access();
137
+
asm goto(".option push\n"
138
+
".option arch, +zicfiss\n"
139
+
"1: ssamoswap.d %[swap], %[val], %[addr]\n"
140
+
_ASM_EXTABLE(1b, %l[fault])
141
+
".option pop\n"
142
+
: [swap] "=r" (swap), [addr] "+A" (*(__force unsigned long *)addr)
143
+
: [val] "r" (val)
144
+
: "memory"
145
+
: fault
146
+
);
147
+
__disable_user_access();
148
+
return swap;
149
+
fault:
150
+
__disable_user_access();
151
+
return -1;
152
+
}
153
+
154
+
/*
155
+
* Create a restore token on the shadow stack. A token is always XLEN wide
156
+
* and aligned to XLEN.
157
+
*/
158
+
static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)
159
+
{
160
+
unsigned long addr;
161
+
162
+
/* Token must be aligned */
163
+
if (!IS_ALIGNED(ssp, SHSTK_ENTRY_SIZE))
164
+
return -EINVAL;
165
+
166
+
/* On RISC-V we're constructing token to be function of address itself */
167
+
addr = ssp - SHSTK_ENTRY_SIZE;
168
+
169
+
if (amo_user_shstk((unsigned long __user *)addr, (unsigned long)ssp) == -1)
170
+
return -EFAULT;
171
+
172
+
if (token_addr)
173
+
*token_addr = addr;
174
+
175
+
return 0;
176
+
}
177
+
178
+
/*
179
+
* Save user shadow stack pointer on the shadow stack itself and return a pointer to saved location.
180
+
* Returns -EFAULT if unsuccessful.
181
+
*/
182
+
int save_user_shstk(struct task_struct *tsk, unsigned long *saved_shstk_ptr)
183
+
{
184
+
unsigned long ss_ptr = 0;
185
+
unsigned long token_loc = 0;
186
+
int ret = 0;
187
+
188
+
if (!saved_shstk_ptr)
189
+
return -EINVAL;
190
+
191
+
ss_ptr = get_active_shstk(tsk);
192
+
ret = create_rstor_token(ss_ptr, &token_loc);
193
+
194
+
if (!ret) {
195
+
*saved_shstk_ptr = token_loc;
196
+
set_active_shstk(tsk, token_loc);
197
+
}
198
+
199
+
return ret;
200
+
}
201
+
202
+
/*
203
+
* Restores the user shadow stack pointer from the token on the shadow stack for task 'tsk'.
204
+
* Returns -EFAULT if unsuccessful.
205
+
*/
206
+
int restore_user_shstk(struct task_struct *tsk, unsigned long shstk_ptr)
207
+
{
208
+
unsigned long token = 0;
209
+
210
+
token = amo_user_shstk((unsigned long __user *)shstk_ptr, 0);
211
+
212
+
if (token == -1)
213
+
return -EFAULT;
214
+
215
+
/* invalid token, return EINVAL */
216
+
if ((token - shstk_ptr) != SHSTK_ENTRY_SIZE) {
217
+
pr_info_ratelimited("%s[%d]: bad restore token in %s: pc=%p sp=%p, token=%p, shstk_ptr=%p\n",
218
+
tsk->comm, task_pid_nr(tsk), __func__,
219
+
(void *)(task_pt_regs(tsk)->epc),
220
+
(void *)(task_pt_regs(tsk)->sp),
221
+
(void *)token, (void *)shstk_ptr);
222
+
return -EINVAL;
223
+
}
224
+
225
+
/* all checks passed, set active shstk and return success */
226
+
set_active_shstk(tsk, token);
227
+
return 0;
228
+
}
229
+
230
+
static unsigned long allocate_shadow_stack(unsigned long addr, unsigned long size,
231
+
unsigned long token_offset, bool set_tok)
232
+
{
233
+
int flags = MAP_ANONYMOUS | MAP_PRIVATE;
234
+
struct mm_struct *mm = current->mm;
235
+
unsigned long populate;
236
+
237
+
if (addr)
238
+
flags |= MAP_FIXED_NOREPLACE;
239
+
240
+
mmap_write_lock(mm);
241
+
addr = do_mmap(NULL, addr, size, PROT_READ, flags,
242
+
VM_SHADOW_STACK | VM_WRITE, 0, &populate, NULL);
243
+
mmap_write_unlock(mm);
244
+
245
+
if (!set_tok || IS_ERR_VALUE(addr))
246
+
goto out;
247
+
248
+
if (create_rstor_token(addr + token_offset, NULL)) {
249
+
vm_munmap(addr, size);
250
+
return -EINVAL;
251
+
}
252
+
253
+
out:
254
+
return addr;
255
+
}
256
+
257
+
SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags)
258
+
{
259
+
bool set_tok = flags & SHADOW_STACK_SET_TOKEN;
260
+
unsigned long aligned_size = 0;
261
+
262
+
if (!is_user_shstk_enabled())
263
+
return -EOPNOTSUPP;
264
+
265
+
/* Anything other than set token should result in invalid param */
266
+
if (flags & ~SHADOW_STACK_SET_TOKEN)
267
+
return -EINVAL;
268
+
269
+
/*
270
+
* Unlike other architectures, on RISC-V, SSP pointer is held in CSR_SSP and is an available
271
+
* CSR in all modes. CSR accesses are performed using 12bit index programmed in instruction
272
+
* itself. This provides static property on register programming and writes to CSR can't
273
+
* be unintentional from programmer's perspective. As long as programmer has guarded areas
274
+
* which perform writes to CSR_SSP properly, shadow stack pivoting is not possible. Since
275
+
* CSR_SSP is writable by user mode, it itself can setup a shadow stack token subsequent
276
+
* to allocation. Although in order to provide portablity with other architectures (because
277
+
* `map_shadow_stack` is arch agnostic syscall), RISC-V will follow expectation of a token
278
+
* flag in flags and if provided in flags, will setup a token at the base.
279
+
*/
280
+
281
+
/* If there isn't space for a token */
282
+
if (set_tok && size < SHSTK_ENTRY_SIZE)
283
+
return -ENOSPC;
284
+
285
+
if (addr && (addr & (PAGE_SIZE - 1)))
286
+
return -EINVAL;
287
+
288
+
aligned_size = PAGE_ALIGN(size);
289
+
if (aligned_size < size)
290
+
return -EOVERFLOW;
291
+
292
+
return allocate_shadow_stack(addr, aligned_size, size, set_tok);
293
+
}
294
+
295
+
/*
296
+
* This gets called during clone/clone3/fork. And is needed to allocate a shadow stack for
297
+
* cases where CLONE_VM is specified and thus a different stack is specified by user. We
298
+
* thus need a separate shadow stack too. How a separate shadow stack is specified by
299
+
* user is still being debated. Once that's settled, remove this part of the comment.
300
+
* This function simply returns 0 if shadow stacks are not supported or if separate shadow
301
+
* stack allocation is not needed (like in case of !CLONE_VM)
302
+
*/
303
+
unsigned long shstk_alloc_thread_stack(struct task_struct *tsk,
304
+
const struct kernel_clone_args *args)
305
+
{
306
+
unsigned long addr, size;
307
+
308
+
/* If shadow stack is not supported, return 0 */
309
+
if (!is_user_shstk_enabled())
310
+
return 0;
311
+
312
+
/*
313
+
* If shadow stack is not enabled on the new thread, skip any
314
+
* switch to a new shadow stack.
315
+
*/
316
+
if (!is_shstk_enabled(tsk))
317
+
return 0;
318
+
319
+
/*
320
+
* For CLONE_VFORK the child will share the parents shadow stack.
321
+
* Set base = 0 and size = 0, this is special means to track this state
322
+
* so the freeing logic run for child knows to leave it alone.
323
+
*/
324
+
if (args->flags & CLONE_VFORK) {
325
+
set_shstk_base(tsk, 0, 0);
326
+
return 0;
327
+
}
328
+
329
+
/*
330
+
* For !CLONE_VM the child will use a copy of the parents shadow
331
+
* stack.
332
+
*/
333
+
if (!(args->flags & CLONE_VM))
334
+
return 0;
335
+
336
+
/*
337
+
* reaching here means, CLONE_VM was specified and thus a separate shadow
338
+
* stack is needed for new cloned thread. Note: below allocation is happening
339
+
* using current mm.
340
+
*/
341
+
size = calc_shstk_size(args->stack_size);
342
+
addr = allocate_shadow_stack(0, size, 0, false);
343
+
if (IS_ERR_VALUE(addr))
344
+
return addr;
345
+
346
+
set_shstk_base(tsk, addr, size);
347
+
348
+
return addr + size;
349
+
}
350
+
351
+
void shstk_release(struct task_struct *tsk)
352
+
{
353
+
unsigned long base = 0, size = 0;
354
+
/* If shadow stack is not supported or not enabled, nothing to release */
355
+
if (!is_user_shstk_enabled() || !is_shstk_enabled(tsk))
356
+
return;
357
+
358
+
/*
359
+
* When fork() with CLONE_VM fails, the child (tsk) already has a
360
+
* shadow stack allocated, and exit_thread() calls this function to
361
+
* free it. In this case the parent (current) and the child share
362
+
* the same mm struct. Move forward only when they're same.
363
+
*/
364
+
if (!tsk->mm || tsk->mm != current->mm)
365
+
return;
366
+
367
+
/*
368
+
* We know shadow stack is enabled but if base is NULL, then
369
+
* this task is not managing its own shadow stack (CLONE_VFORK). So
370
+
* skip freeing it.
371
+
*/
372
+
base = get_shstk_base(tsk, &size);
373
+
if (!base)
374
+
return;
375
+
376
+
vm_munmap(base, size);
377
+
set_shstk_base(tsk, 0, 0);
378
+
}
379
+
380
+
int arch_get_shadow_stack_status(struct task_struct *t, unsigned long __user *status)
381
+
{
382
+
unsigned long bcfi_status = 0;
383
+
384
+
if (!is_user_shstk_enabled())
385
+
return -EINVAL;
386
+
387
+
/* this means shadow stack is enabled on the task */
388
+
bcfi_status |= (is_shstk_enabled(t) ? PR_SHADOW_STACK_ENABLE : 0);
389
+
390
+
return copy_to_user(status, &bcfi_status, sizeof(bcfi_status)) ? -EFAULT : 0;
391
+
}
392
+
393
+
int arch_set_shadow_stack_status(struct task_struct *t, unsigned long status)
394
+
{
395
+
unsigned long size = 0, addr = 0;
396
+
bool enable_shstk = false;
397
+
398
+
if (!is_user_shstk_enabled())
399
+
return -EINVAL;
400
+
401
+
/* Reject unknown flags */
402
+
if (status & ~PR_SHADOW_STACK_SUPPORTED_STATUS_MASK)
403
+
return -EINVAL;
404
+
405
+
/* bcfi status is locked and further can't be modified by user */
406
+
if (is_shstk_locked(t))
407
+
return -EINVAL;
408
+
409
+
enable_shstk = status & PR_SHADOW_STACK_ENABLE;
410
+
/* Request is to enable shadow stack and shadow stack is not enabled already */
411
+
if (enable_shstk && !is_shstk_enabled(t)) {
412
+
/* shadow stack was allocated and enable request again
413
+
* no need to support such usecase and return EINVAL.
414
+
*/
415
+
if (is_shstk_allocated(t))
416
+
return -EINVAL;
417
+
418
+
size = calc_shstk_size(0);
419
+
addr = allocate_shadow_stack(0, size, 0, false);
420
+
if (IS_ERR_VALUE(addr))
421
+
return -ENOMEM;
422
+
set_shstk_base(t, addr, size);
423
+
set_active_shstk(t, addr + size);
424
+
}
425
+
426
+
/*
427
+
* If a request to disable shadow stack happens, let's go ahead and release it
428
+
* Although, if CLONE_VFORKed child did this, then in that case we will end up
429
+
* not releasing the shadow stack (because it might be needed in parent). Although
430
+
* we will disable it for VFORKed child. And if VFORKed child tries to enable again
431
+
* then in that case, it'll get entirely new shadow stack because following condition
432
+
* are true
433
+
* - shadow stack was not enabled for vforked child
434
+
* - shadow stack base was anyways pointing to 0
435
+
* This shouldn't be a big issue because we want parent to have availability of shadow
436
+
* stack whenever VFORKed child releases resources via exit or exec but at the same
437
+
* time we want VFORKed child to break away and establish new shadow stack if it desires
438
+
*
439
+
*/
440
+
if (!enable_shstk)
441
+
shstk_release(t);
442
+
443
+
set_shstk_status(t, enable_shstk);
444
+
return 0;
445
+
}
446
+
447
+
int arch_lock_shadow_stack_status(struct task_struct *task,
448
+
unsigned long arg)
449
+
{
450
+
/* If shtstk not supported or not enabled on task, nothing to lock here */
451
+
if (!is_user_shstk_enabled() ||
452
+
!is_shstk_enabled(task) || arg != 0)
453
+
return -EINVAL;
454
+
455
+
set_shstk_lock(task);
456
+
457
+
return 0;
458
+
}
459
+
460
+
int arch_get_indir_br_lp_status(struct task_struct *t, unsigned long __user *status)
461
+
{
462
+
unsigned long fcfi_status = 0;
463
+
464
+
if (!is_user_lpad_enabled())
465
+
return -EINVAL;
466
+
467
+
/* indirect branch tracking is enabled on the task or not */
468
+
fcfi_status |= (is_indir_lp_enabled(t) ? PR_INDIR_BR_LP_ENABLE : 0);
469
+
470
+
return copy_to_user(status, &fcfi_status, sizeof(fcfi_status)) ? -EFAULT : 0;
471
+
}
472
+
473
+
int arch_set_indir_br_lp_status(struct task_struct *t, unsigned long status)
474
+
{
475
+
bool enable_indir_lp = false;
476
+
477
+
if (!is_user_lpad_enabled())
478
+
return -EINVAL;
479
+
480
+
/* indirect branch tracking is locked and further can't be modified by user */
481
+
if (is_indir_lp_locked(t))
482
+
return -EINVAL;
483
+
484
+
/* Reject unknown flags */
485
+
if (status & ~PR_INDIR_BR_LP_ENABLE)
486
+
return -EINVAL;
487
+
488
+
enable_indir_lp = (status & PR_INDIR_BR_LP_ENABLE);
489
+
set_indir_lp_status(t, enable_indir_lp);
490
+
491
+
return 0;
492
+
}
493
+
494
+
int arch_lock_indir_br_lp_status(struct task_struct *task,
495
+
unsigned long arg)
496
+
{
497
+
/*
498
+
* If indirect branch tracking is not supported or not enabled on task,
499
+
* nothing to lock here
500
+
*/
501
+
if (!is_user_lpad_enabled() ||
502
+
!is_indir_lp_enabled(task) || arg != 0)
503
+
return -EINVAL;
504
+
505
+
set_indir_lp_lock(task);
506
+
507
+
return 0;
508
+
}
509
+
510
+
bool is_user_shstk_enabled(void)
511
+
{
512
+
return (cpu_supports_shadow_stack() &&
513
+
!(riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_BCFI));
514
+
}
515
+
516
+
bool is_user_lpad_enabled(void)
517
+
{
518
+
return (cpu_supports_indirect_br_lp_instr() &&
519
+
!(riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_FCFI));
520
+
}
521
+
522
+
static int __init setup_global_riscv_enable(char *str)
523
+
{
524
+
if (strcmp(str, "all") == 0)
525
+
riscv_nousercfi = CMDLINE_DISABLE_RISCV_USERCFI;
526
+
527
+
if (strcmp(str, "fcfi") == 0)
528
+
riscv_nousercfi |= CMDLINE_DISABLE_RISCV_USERCFI_FCFI;
529
+
530
+
if (strcmp(str, "bcfi") == 0)
531
+
riscv_nousercfi |= CMDLINE_DISABLE_RISCV_USERCFI_BCFI;
532
+
533
+
if (riscv_nousercfi)
534
+
pr_info("RISC-V user CFI disabled via cmdline - shadow stack status : %s, landing pad status : %s\n",
535
+
(riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_BCFI) ? "disabled" :
536
+
"enabled", (riscv_nousercfi & CMDLINE_DISABLE_RISCV_USERCFI_FCFI) ?
537
+
"disabled" : "enabled");
538
+
539
+
return 1;
540
+
}
541
+
542
+
__setup("riscv_nousercfi=", setup_global_riscv_enable);
+7
arch/riscv/kernel/vdso.c
+7
arch/riscv/kernel/vdso.c
···
98
98
99
99
static int __init vdso_init(void)
100
100
{
101
+
/* Hart implements zimop, expose cfi compiled vdso */
102
+
if (IS_ENABLED(CONFIG_RISCV_USER_CFI) &&
103
+
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZIMOP)) {
104
+
vdso_info.vdso_code_start = vdso_cfi_start;
105
+
vdso_info.vdso_code_end = vdso_cfi_end;
106
+
}
107
+
101
108
__vdso_init(&vdso_info);
102
109
#ifdef CONFIG_COMPAT
103
110
__vdso_init(&compat_vdso_info);
+30
-10
arch/riscv/kernel/vdso/Makefile
+30
-10
arch/riscv/kernel/vdso/Makefile
···
17
17
vdso-syms += getrandom
18
18
endif
19
19
20
+
ifdef VDSO_CFI_BUILD
21
+
CFI_MARCH = _zicfilp_zicfiss
22
+
CFI_FULL = -fcf-protection=full
23
+
CFI_SUFFIX = -cfi
24
+
OFFSET_SUFFIX = _cfi
25
+
ccflags-y += -DVDSO_CFI=1
26
+
asflags-y += -DVDSO_CFI=1
27
+
endif
28
+
20
29
# Files to link into the vdso
21
30
obj-vdso = $(patsubst %, %.o, $(vdso-syms)) note.o
22
31
···
36
27
ccflags-y := -fno-stack-protector
37
28
ccflags-y += -DDISABLE_BRANCH_PROFILING
38
29
ccflags-y += -fno-builtin
30
+
ccflags-y += $(KBUILD_BASE_ISA)$(CFI_MARCH)
31
+
ccflags-y += $(CFI_FULL)
32
+
asflags-y += $(KBUILD_BASE_ISA)$(CFI_MARCH)
33
+
asflags-y += $(CFI_FULL)
39
34
40
35
ifneq ($(c-gettimeofday-y),)
41
36
CFLAGS_vgettimeofday.o += -fPIC -include $(c-gettimeofday-y)
···
52
39
CFLAGS_hwprobe.o += -fPIC
53
40
54
41
# Build rules
55
-
targets := $(obj-vdso) vdso.so vdso.so.dbg vdso.lds
42
+
vdso_offsets := vdso$(if $(VDSO_CFI_BUILD),$(CFI_SUFFIX),)-offsets.h
43
+
vdso_o := vdso$(if $(VDSO_CFI_BUILD),$(CFI_SUFFIX),).o
44
+
vdso_so := vdso$(if $(VDSO_CFI_BUILD),$(CFI_SUFFIX),).so
45
+
vdso_so_dbg := vdso$(if $(VDSO_CFI_BUILD),$(CFI_SUFFIX),).so.dbg
46
+
vdso_lds := vdso.lds
47
+
48
+
targets := $(obj-vdso) $(vdso_so) $(vdso_so_dbg) $(vdso_lds)
49
+
56
50
obj-vdso := $(addprefix $(obj)/, $(obj-vdso))
57
51
58
-
obj-y += vdso.o
59
-
CPPFLAGS_vdso.lds += -P -C -U$(ARCH)
52
+
obj-y += vdso$(if $(VDSO_CFI_BUILD),$(CFI_SUFFIX),).o
53
+
CPPFLAGS_$(vdso_lds) += -P -C -U$(ARCH)
60
54
ifneq ($(filter vgettimeofday, $(vdso-syms)),)
61
-
CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY
55
+
CPPFLAGS_$(vdso_lds) += -DHAS_VGETTIMEOFDAY
62
56
endif
63
57
64
58
# Disable -pg to prevent insert call site
···
74
54
CFLAGS_REMOVE_hwprobe.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
75
55
76
56
# Force dependency
77
-
$(obj)/vdso.o: $(obj)/vdso.so
57
+
$(obj)/$(vdso_o): $(obj)/$(vdso_so)
78
58
79
59
# link rule for the .so file, .lds has to be first
80
-
$(obj)/vdso.so.dbg: $(obj)/vdso.lds $(obj-vdso) FORCE
60
+
$(obj)/$(vdso_so_dbg): $(obj)/$(vdso_lds) $(obj-vdso) FORCE
81
61
$(call if_changed,vdsold_and_check)
82
-
LDFLAGS_vdso.so.dbg = -shared -soname=linux-vdso.so.1 \
62
+
LDFLAGS_$(vdso_so_dbg) = -shared -soname=linux-vdso.so.1 \
83
63
--build-id=sha1 --eh-frame-hdr
84
64
85
65
# strip rule for the .so file
···
90
70
# Generate VDSO offsets using helper script
91
71
gen-vdsosym := $(src)/gen_vdso_offsets.sh
92
72
quiet_cmd_vdsosym = VDSOSYM $@
93
-
cmd_vdsosym = $(NM) $< | $(gen-vdsosym) | LC_ALL=C sort > $@
73
+
cmd_vdsosym = $(NM) $< | $(gen-vdsosym) $(OFFSET_SUFFIX) | LC_ALL=C sort > $@
94
74
95
-
include/generated/vdso-offsets.h: $(obj)/vdso.so.dbg FORCE
75
+
include/generated/$(vdso_offsets): $(obj)/$(vdso_so_dbg) FORCE
96
76
$(call if_changed,vdsosym)
97
77
98
78
# actual build commands
99
79
# The DSO images are built using a special linker script
100
80
# Make sure only to export the intended __vdso_xxx symbol offsets.
101
81
quiet_cmd_vdsold_and_check = VDSOLD $@
102
-
cmd_vdsold_and_check = $(LD) $(ld_flags) -T $(filter-out FORCE,$^) -o $@.tmp && \
82
+
cmd_vdsold_and_check = $(LD) $(CFI_FULL) $(ld_flags) -T $(filter-out FORCE,$^) -o $@.tmp && \
103
83
$(OBJCOPY) $(patsubst %, -G __vdso_%, $(vdso-syms)) $@.tmp $@ && \
104
84
rm $@.tmp && \
105
85
$(cmd_vdso_check)
+4
arch/riscv/kernel/vdso/flush_icache.S
+4
arch/riscv/kernel/vdso/flush_icache.S
···
5
5
6
6
#include <linux/linkage.h>
7
7
#include <asm/unistd.h>
8
+
#include <asm/assembler.h>
8
9
9
10
.text
10
11
/* int __vdso_flush_icache(void *start, void *end, unsigned long flags); */
11
12
SYM_FUNC_START(__vdso_flush_icache)
12
13
.cfi_startproc
14
+
vdso_lpad
13
15
#ifdef CONFIG_SMP
14
16
li a7, __NR_riscv_flush_icache
15
17
ecall
···
22
20
ret
23
21
.cfi_endproc
24
22
SYM_FUNC_END(__vdso_flush_icache)
23
+
24
+
emit_riscv_feature_1_and
+3
-1
arch/riscv/kernel/vdso/gen_vdso_offsets.sh
+3
-1
arch/riscv/kernel/vdso/gen_vdso_offsets.sh
+4
arch/riscv/kernel/vdso/getcpu.S
+4
arch/riscv/kernel/vdso/getcpu.S
···
5
5
6
6
#include <linux/linkage.h>
7
7
#include <asm/unistd.h>
8
+
#include <asm/assembler.h>
8
9
9
10
.text
10
11
/* int __vdso_getcpu(unsigned *cpu, unsigned *node, void *unused); */
11
12
SYM_FUNC_START(__vdso_getcpu)
12
13
.cfi_startproc
14
+
vdso_lpad
13
15
/* For now, just do the syscall. */
14
16
li a7, __NR_getcpu
15
17
ecall
16
18
ret
17
19
.cfi_endproc
18
20
SYM_FUNC_END(__vdso_getcpu)
21
+
22
+
emit_riscv_feature_1_and
+3
arch/riscv/kernel/vdso/note.S
+3
arch/riscv/kernel/vdso/note.S
+4
arch/riscv/kernel/vdso/rt_sigreturn.S
+4
arch/riscv/kernel/vdso/rt_sigreturn.S
···
5
5
6
6
#include <linux/linkage.h>
7
7
#include <asm/unistd.h>
8
+
#include <asm/assembler.h>
8
9
9
10
.text
10
11
SYM_FUNC_START(__vdso_rt_sigreturn)
11
12
.cfi_startproc
12
13
.cfi_signal_frame
14
+
vdso_lpad
13
15
li a7, __NR_rt_sigreturn
14
16
ecall
15
17
.cfi_endproc
16
18
SYM_FUNC_END(__vdso_rt_sigreturn)
19
+
20
+
emit_riscv_feature_1_and
+4
arch/riscv/kernel/vdso/sys_hwprobe.S
+4
arch/riscv/kernel/vdso/sys_hwprobe.S
···
3
3
4
4
#include <linux/linkage.h>
5
5
#include <asm/unistd.h>
6
+
#include <asm/assembler.h>
6
7
7
8
.text
8
9
SYM_FUNC_START(riscv_hwprobe)
9
10
.cfi_startproc
11
+
vdso_lpad
10
12
li a7, __NR_riscv_hwprobe
11
13
ecall
12
14
ret
13
15
14
16
.cfi_endproc
15
17
SYM_FUNC_END(riscv_hwprobe)
18
+
19
+
emit_riscv_feature_1_and
+4
-1
arch/riscv/kernel/vdso/vgetrandom-chacha.S
+4
-1
arch/riscv/kernel/vdso/vgetrandom-chacha.S
···
7
7
8
8
#include <asm/asm.h>
9
9
#include <linux/linkage.h>
10
+
#include <asm/assembler.h>
10
11
11
12
.text
12
13
···
75
74
#define _20 20, 20, 20, 20
76
75
#define _24 24, 24, 24, 24
77
76
#define _25 25, 25, 25, 25
78
-
77
+
vdso_lpad
79
78
/*
80
79
* The ABI requires s0-s9 saved.
81
80
* This does not violate the stack-less requirement: no sensitive data
···
248
247
249
248
ret
250
249
SYM_FUNC_END(__arch_chacha20_blocks_nostack)
250
+
251
+
emit_riscv_feature_1_and
+25
arch/riscv/kernel/vdso_cfi/Makefile
+25
arch/riscv/kernel/vdso_cfi/Makefile
···
1
+
# SPDX-License-Identifier: GPL-2.0-only
2
+
# RISC-V VDSO CFI Makefile
3
+
# This Makefile builds the VDSO with CFI support when CONFIG_RISCV_USER_CFI is enabled
4
+
5
+
# setting VDSO_CFI_BUILD triggers build for vdso differently
6
+
VDSO_CFI_BUILD := 1
7
+
8
+
# Set the source directory to the main vdso directory
9
+
src := $(srctree)/arch/riscv/kernel/vdso
10
+
11
+
# Copy all .S and .c files from vdso directory to vdso_cfi object build directory
12
+
vdso_c_sources := $(wildcard $(src)/*.c)
13
+
vdso_S_sources := $(wildcard $(src)/*.S)
14
+
vdso_c_objects := $(addprefix $(obj)/, $(notdir $(vdso_c_sources)))
15
+
vdso_S_objects := $(addprefix $(obj)/, $(notdir $(vdso_S_sources)))
16
+
17
+
$(vdso_S_objects): $(obj)/%.S: $(src)/%.S
18
+
$(Q)cp $< $@
19
+
20
+
$(vdso_c_objects): $(obj)/%.c: $(src)/%.c
21
+
$(Q)cp $< $@
22
+
23
+
# Include the main VDSO Makefile which contains all the build rules and sources
24
+
# The VDSO_CFI_BUILD variable will be passed to it to enable CFI compilation
25
+
include $(src)/Makefile
+11
arch/riscv/kernel/vdso_cfi/vdso-cfi.S
+11
arch/riscv/kernel/vdso_cfi/vdso-cfi.S
+8
-4
arch/riscv/kernel/vector.c
+8
-4
arch/riscv/kernel/vector.c
···
111
111
return false;
112
112
}
113
113
114
-
static int riscv_v_thread_zalloc(struct kmem_cache *cache,
115
-
struct __riscv_v_ext_state *ctx)
114
+
static int riscv_v_thread_ctx_alloc(struct kmem_cache *cache,
115
+
struct __riscv_v_ext_state *ctx)
116
116
{
117
117
void *datap;
118
118
···
122
122
123
123
ctx->datap = datap;
124
124
memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap));
125
+
ctx->vlenb = riscv_v_vsize / 32;
126
+
125
127
return 0;
126
128
}
127
129
128
130
void riscv_v_thread_alloc(struct task_struct *tsk)
129
131
{
130
132
#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
131
-
riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate);
133
+
riscv_v_thread_ctx_alloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate);
132
134
#endif
133
135
}
134
136
···
216
214
* context where VS has been off. So, try to allocate the user's V
217
215
* context and resume execution.
218
216
*/
219
-
if (riscv_v_thread_zalloc(riscv_v_user_cachep, ¤t->thread.vstate)) {
217
+
if (riscv_v_thread_ctx_alloc(riscv_v_user_cachep, ¤t->thread.vstate)) {
220
218
force_sig(SIGBUS);
221
219
return true;
222
220
}
221
+
223
222
riscv_v_vstate_on(regs);
224
223
riscv_v_vstate_set_restore(current, regs);
224
+
225
225
return true;
226
226
}
227
227
+3
-5
arch/riscv/lib/strlen.S
+3
-5
arch/riscv/lib/strlen.S
+1
-1
arch/riscv/mm/init.c
+1
-1
arch/riscv/mm/init.c
···
370
370
static const pgprot_t protection_map[16] = {
371
371
[VM_NONE] = PAGE_NONE,
372
372
[VM_READ] = PAGE_READ,
373
-
[VM_WRITE] = PAGE_COPY,
373
+
[VM_WRITE] = PAGE_SHADOWSTACK,
374
374
[VM_WRITE | VM_READ] = PAGE_COPY,
375
375
[VM_EXEC] = PAGE_EXEC,
376
376
[VM_EXEC | VM_READ] = PAGE_READ_EXEC,
+16
arch/riscv/mm/pgtable.c
+16
arch/riscv/mm/pgtable.c
···
163
163
return old;
164
164
}
165
165
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
166
+
167
+
pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
168
+
{
169
+
if (vma->vm_flags & VM_SHADOW_STACK)
170
+
return pte_mkwrite_shstk(pte);
171
+
172
+
return pte_mkwrite_novma(pte);
173
+
}
174
+
175
+
pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
176
+
{
177
+
if (vma->vm_flags & VM_SHADOW_STACK)
178
+
return pmd_mkwrite_shstk(pmd);
179
+
180
+
return pmd_mkwrite_novma(pmd);
181
+
}
+4
-4
drivers/iommu/riscv/iommu.c
+4
-4
drivers/iommu/riscv/iommu.c
···
1593
1593
FIELD_PREP(RISCV_IOMMU_ICVEC_PMIV, 3 % iommu->irqs_count);
1594
1594
riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_ICVEC, iommu->icvec);
1595
1595
iommu->icvec = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_ICVEC);
1596
-
if (max(max(FIELD_GET(RISCV_IOMMU_ICVEC_CIV, iommu->icvec),
1597
-
FIELD_GET(RISCV_IOMMU_ICVEC_FIV, iommu->icvec)),
1598
-
max(FIELD_GET(RISCV_IOMMU_ICVEC_PIV, iommu->icvec),
1599
-
FIELD_GET(RISCV_IOMMU_ICVEC_PMIV, iommu->icvec))) >= iommu->irqs_count)
1596
+
if (max3(FIELD_GET(RISCV_IOMMU_ICVEC_CIV, iommu->icvec),
1597
+
FIELD_GET(RISCV_IOMMU_ICVEC_FIV, iommu->icvec),
1598
+
max(FIELD_GET(RISCV_IOMMU_ICVEC_PIV, iommu->icvec),
1599
+
FIELD_GET(RISCV_IOMMU_ICVEC_PMIV, iommu->icvec))) >= iommu->irqs_count)
1600
1600
return -EINVAL;
1601
1601
1602
1602
return 0;
+4
include/linux/cpu.h
+4
include/linux/cpu.h
···
229
229
#define smt_mitigations SMT_MITIGATIONS_OFF
230
230
#endif
231
231
232
+
int arch_get_indir_br_lp_status(struct task_struct *t, unsigned long __user *status);
233
+
int arch_set_indir_br_lp_status(struct task_struct *t, unsigned long status);
234
+
int arch_lock_indir_br_lp_status(struct task_struct *t, unsigned long status);
235
+
232
236
#endif /* _LINUX_CPU_H_ */
+3
-2
include/linux/mm.h
+3
-2
include/linux/mm.h
···
361
361
DECLARE_VMA_BIT_ALIAS(PKEY_BIT2, HIGH_ARCH_2),
362
362
DECLARE_VMA_BIT_ALIAS(PKEY_BIT3, HIGH_ARCH_3),
363
363
DECLARE_VMA_BIT_ALIAS(PKEY_BIT4, HIGH_ARCH_4),
364
-
#if defined(CONFIG_X86_USER_SHADOW_STACK)
364
+
#if defined(CONFIG_X86_USER_SHADOW_STACK) || defined(CONFIG_RISCV_USER_CFI)
365
365
/*
366
366
* VM_SHADOW_STACK should not be set with VM_SHARED because of lack of
367
367
* support core mm.
···
462
462
#define VM_PKEY_BIT4 VM_NONE
463
463
#endif /* CONFIG_ARCH_PKEY_BITS > 4 */
464
464
#endif /* CONFIG_ARCH_HAS_PKEYS */
465
-
#if defined(CONFIG_X86_USER_SHADOW_STACK) || defined(CONFIG_ARM64_GCS)
465
+
#if defined(CONFIG_X86_USER_SHADOW_STACK) || defined(CONFIG_ARM64_GCS) || \
466
+
defined(CONFIG_RISCV_USER_CFI)
466
467
#define VM_SHADOW_STACK INIT_VM_FLAG(SHADOW_STACK)
467
468
#else
468
469
#define VM_SHADOW_STACK VM_NONE
+2
include/uapi/linux/elf.h
+2
include/uapi/linux/elf.h
···
545
545
#define NT_RISCV_VECTOR 0x901 /* RISC-V vector registers */
546
546
#define NN_RISCV_TAGGED_ADDR_CTRL "LINUX"
547
547
#define NT_RISCV_TAGGED_ADDR_CTRL 0x902 /* RISC-V tagged address control (prctl()) */
548
+
#define NN_RISCV_USER_CFI "LINUX"
549
+
#define NT_RISCV_USER_CFI 0x903 /* RISC-V shadow stack state */
548
550
#define NN_LOONGARCH_CPUCFG "LINUX"
549
551
#define NT_LOONGARCH_CPUCFG 0xa00 /* LoongArch CPU config registers */
550
552
#define NN_LOONGARCH_CSR "LINUX"
+27
include/uapi/linux/prctl.h
+27
include/uapi/linux/prctl.h
···
396
396
*/
397
397
# define PR_RSEQ_SLICE_EXT_ENABLE 0x01
398
398
399
+
/*
400
+
* Get the current indirect branch tracking configuration for the current
401
+
* thread, this will be the value configured via PR_SET_INDIR_BR_LP_STATUS.
402
+
*/
403
+
#define PR_GET_INDIR_BR_LP_STATUS 80
404
+
405
+
/*
406
+
* Set the indirect branch tracking configuration. PR_INDIR_BR_LP_ENABLE will
407
+
* enable cpu feature for user thread, to track all indirect branches and ensure
408
+
* they land on arch defined landing pad instruction.
409
+
* x86 - If enabled, an indirect branch must land on an ENDBRANCH instruction.
410
+
* arch64 - If enabled, an indirect branch must land on a BTI instruction.
411
+
* riscv - If enabled, an indirect branch must land on an lpad instruction.
412
+
* PR_INDIR_BR_LP_DISABLE will disable feature for user thread and indirect
413
+
* branches will no more be tracked by cpu to land on arch defined landing pad
414
+
* instruction.
415
+
*/
416
+
#define PR_SET_INDIR_BR_LP_STATUS 81
417
+
# define PR_INDIR_BR_LP_ENABLE (1UL << 0)
418
+
419
+
/*
420
+
* Prevent further changes to the specified indirect branch tracking
421
+
* configuration. All bits may be locked via this call, including
422
+
* undefined bits.
423
+
*/
424
+
#define PR_LOCK_INDIR_BR_LP_STATUS 82
425
+
399
426
#endif /* _LINUX_PRCTL_H */
+30
kernel/sys.c
+30
kernel/sys.c
···
2388
2388
return -EINVAL;
2389
2389
}
2390
2390
2391
+
int __weak arch_get_indir_br_lp_status(struct task_struct *t, unsigned long __user *status)
2392
+
{
2393
+
return -EINVAL;
2394
+
}
2395
+
2396
+
int __weak arch_set_indir_br_lp_status(struct task_struct *t, unsigned long status)
2397
+
{
2398
+
return -EINVAL;
2399
+
}
2400
+
2401
+
int __weak arch_lock_indir_br_lp_status(struct task_struct *t, unsigned long status)
2402
+
{
2403
+
return -EINVAL;
2404
+
}
2405
+
2391
2406
#define PR_IO_FLUSHER (PF_MEMALLOC_NOIO | PF_LOCAL_THROTTLE)
2392
2407
2393
2408
static int prctl_set_vma(unsigned long opt, unsigned long addr,
···
2887
2872
if (arg4 || arg5)
2888
2873
return -EINVAL;
2889
2874
error = rseq_slice_extension_prctl(arg2, arg3);
2875
+
break;
2876
+
case PR_GET_INDIR_BR_LP_STATUS:
2877
+
if (arg3 || arg4 || arg5)
2878
+
return -EINVAL;
2879
+
error = arch_get_indir_br_lp_status(me, (unsigned long __user *)arg2);
2880
+
break;
2881
+
case PR_SET_INDIR_BR_LP_STATUS:
2882
+
if (arg3 || arg4 || arg5)
2883
+
return -EINVAL;
2884
+
error = arch_set_indir_br_lp_status(me, arg2);
2885
+
break;
2886
+
case PR_LOCK_INDIR_BR_LP_STATUS:
2887
+
if (arg3 || arg4 || arg5)
2888
+
return -EINVAL;
2889
+
error = arch_lock_indir_br_lp_status(me, arg2);
2890
2890
break;
2891
2891
default:
2892
2892
trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5);
+1
-1
tools/testing/selftests/riscv/Makefile
+1
-1
tools/testing/selftests/riscv/Makefile
+23
tools/testing/selftests/riscv/cfi/Makefile
+23
tools/testing/selftests/riscv/cfi/Makefile
···
1
+
CFLAGS += $(KHDR_INCLUDES)
2
+
CFLAGS += -I$(top_srcdir)/tools/include
3
+
4
+
CFLAGS += -march=rv64gc_zicfilp_zicfiss -fcf-protection=full
5
+
6
+
# Check for zicfi* extensions needs cross compiler
7
+
# which is not set until lib.mk is included
8
+
ifeq ($(LLVM)$(CC),cc)
9
+
CC := $(CROSS_COMPILE)gcc
10
+
endif
11
+
12
+
13
+
ifeq ($(shell $(CC) $(CFLAGS) -nostdlib -xc /dev/null -o /dev/null > /dev/null 2>&1; echo $$?),0)
14
+
TEST_GEN_PROGS := cfitests
15
+
16
+
$(OUTPUT)/cfitests: cfitests.c shadowstack.c
17
+
$(CC) -o$@ $(CFLAGS) $(LDFLAGS) $^
18
+
else
19
+
20
+
$(shell echo "Toolchain doesn't support CFI, skipping CFI kselftest." >&2)
21
+
endif
22
+
23
+
include ../../lib.mk
+82
tools/testing/selftests/riscv/cfi/cfi_rv_test.h
+82
tools/testing/selftests/riscv/cfi/cfi_rv_test.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
3
+
#ifndef SELFTEST_RISCV_CFI_H
4
+
#define SELFTEST_RISCV_CFI_H
5
+
#include <stddef.h>
6
+
#include <sys/types.h>
7
+
#include "shadowstack.h"
8
+
9
+
#define CHILD_EXIT_CODE_SSWRITE 10
10
+
#define CHILD_EXIT_CODE_SIG_TEST 11
11
+
12
+
#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
13
+
({ \
14
+
register long _num __asm__ ("a7") = (num); \
15
+
register long _arg1 __asm__ ("a0") = (long)(arg1); \
16
+
register long _arg2 __asm__ ("a1") = (long)(arg2); \
17
+
register long _arg3 __asm__ ("a2") = (long)(arg3); \
18
+
register long _arg4 __asm__ ("a3") = (long)(arg4); \
19
+
register long _arg5 __asm__ ("a4") = (long)(arg5); \
20
+
\
21
+
__asm__ volatile( \
22
+
"ecall\n" \
23
+
: "+r" \
24
+
(_arg1) \
25
+
: "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \
26
+
"r"(_num) \
27
+
: "memory", "cc" \
28
+
); \
29
+
_arg1; \
30
+
})
31
+
32
+
#define my_syscall3(num, arg1, arg2, arg3) \
33
+
({ \
34
+
register long _num __asm__ ("a7") = (num); \
35
+
register long _arg1 __asm__ ("a0") = (long)(arg1); \
36
+
register long _arg2 __asm__ ("a1") = (long)(arg2); \
37
+
register long _arg3 __asm__ ("a2") = (long)(arg3); \
38
+
\
39
+
__asm__ volatile( \
40
+
"ecall\n" \
41
+
: "+r" (_arg1) \
42
+
: "r"(_arg2), "r"(_arg3), \
43
+
"r"(_num) \
44
+
: "memory", "cc" \
45
+
); \
46
+
_arg1; \
47
+
})
48
+
49
+
#ifndef __NR_prctl
50
+
#define __NR_prctl 167
51
+
#endif
52
+
53
+
#ifndef __NR_map_shadow_stack
54
+
#define __NR_map_shadow_stack 453
55
+
#endif
56
+
57
+
#define CSR_SSP 0x011
58
+
59
+
#ifdef __ASSEMBLY__
60
+
#define __ASM_STR(x) x
61
+
#else
62
+
#define __ASM_STR(x) #x
63
+
#endif
64
+
65
+
#define csr_read(csr) \
66
+
({ \
67
+
register unsigned long __v; \
68
+
__asm__ __volatile__ ("csrr %0, " __ASM_STR(csr) \
69
+
: "=r" (__v) : \
70
+
: "memory"); \
71
+
__v; \
72
+
})
73
+
74
+
#define csr_write(csr, val) \
75
+
({ \
76
+
unsigned long __v = (unsigned long)(val); \
77
+
__asm__ __volatile__ ("csrw " __ASM_STR(csr) ", %0" \
78
+
: : "rK" (__v) \
79
+
: "memory"); \
80
+
})
81
+
82
+
#endif
+173
tools/testing/selftests/riscv/cfi/cfitests.c
+173
tools/testing/selftests/riscv/cfi/cfitests.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
3
+
#include "../../kselftest.h"
4
+
#include <sys/signal.h>
5
+
#include <asm/ucontext.h>
6
+
#include <linux/prctl.h>
7
+
#include <errno.h>
8
+
#include <linux/ptrace.h>
9
+
#include <sys/wait.h>
10
+
#include <linux/elf.h>
11
+
#include <sys/uio.h>
12
+
#include <asm-generic/unistd.h>
13
+
14
+
#include "cfi_rv_test.h"
15
+
16
+
/* do not optimize cfi related test functions */
17
+
#pragma GCC push_options
18
+
#pragma GCC optimize("O0")
19
+
20
+
void sigsegv_handler(int signum, siginfo_t *si, void *uc)
21
+
{
22
+
struct ucontext *ctx = (struct ucontext *)uc;
23
+
24
+
if (si->si_code == SEGV_CPERR) {
25
+
ksft_print_msg("Control flow violation happened somewhere\n");
26
+
ksft_print_msg("PC where violation happened %lx\n", ctx->uc_mcontext.gregs[0]);
27
+
exit(-1);
28
+
}
29
+
30
+
/* all other cases are expected to be of shadow stack write case */
31
+
exit(CHILD_EXIT_CODE_SSWRITE);
32
+
}
33
+
34
+
bool register_signal_handler(void)
35
+
{
36
+
struct sigaction sa = {};
37
+
38
+
sa.sa_sigaction = sigsegv_handler;
39
+
sa.sa_flags = SA_SIGINFO;
40
+
if (sigaction(SIGSEGV, &sa, NULL)) {
41
+
ksft_print_msg("Registering signal handler for landing pad violation failed\n");
42
+
return false;
43
+
}
44
+
45
+
return true;
46
+
}
47
+
48
+
long ptrace(int request, pid_t pid, void *addr, void *data);
49
+
50
+
bool cfi_ptrace_test(void)
51
+
{
52
+
pid_t pid;
53
+
int status, ret = 0;
54
+
unsigned long ptrace_test_num = 0, total_ptrace_tests = 2;
55
+
56
+
struct user_cfi_state cfi_reg;
57
+
struct iovec iov;
58
+
59
+
pid = fork();
60
+
61
+
if (pid == -1) {
62
+
ksft_exit_fail_msg("%s: fork failed\n", __func__);
63
+
exit(1);
64
+
}
65
+
66
+
if (pid == 0) {
67
+
/* allow to be traced */
68
+
ptrace(PTRACE_TRACEME, 0, NULL, NULL);
69
+
raise(SIGSTOP);
70
+
asm volatile ("la a5, 1f\n"
71
+
"jalr a5\n"
72
+
"nop\n"
73
+
"nop\n"
74
+
"1: nop\n"
75
+
: : : "a5");
76
+
exit(11);
77
+
/* child shouldn't go beyond here */
78
+
}
79
+
80
+
/* parent's code goes here */
81
+
iov.iov_base = &cfi_reg;
82
+
iov.iov_len = sizeof(cfi_reg);
83
+
84
+
while (ptrace_test_num < total_ptrace_tests) {
85
+
memset(&cfi_reg, 0, sizeof(cfi_reg));
86
+
waitpid(pid, &status, 0);
87
+
if (WIFSTOPPED(status)) {
88
+
errno = 0;
89
+
ret = ptrace(PTRACE_GETREGSET, pid, (void *)NT_RISCV_USER_CFI, &iov);
90
+
if (ret == -1 && errno)
91
+
ksft_exit_fail_msg("%s: PTRACE_GETREGSET failed\n", __func__);
92
+
} else {
93
+
ksft_exit_fail_msg("%s: child didn't stop, failed\n", __func__);
94
+
}
95
+
96
+
switch (ptrace_test_num) {
97
+
#define CFI_ENABLE_MASK (PTRACE_CFI_LP_EN_STATE | \
98
+
PTRACE_CFI_SS_EN_STATE | \
99
+
PTRACE_CFI_SS_PTR_STATE)
100
+
case 0:
101
+
if ((cfi_reg.cfi_status.cfi_state & CFI_ENABLE_MASK) != CFI_ENABLE_MASK)
102
+
ksft_exit_fail_msg("%s: ptrace_getregset failed, %llu\n", __func__,
103
+
cfi_reg.cfi_status.cfi_state);
104
+
if (!cfi_reg.shstk_ptr)
105
+
ksft_exit_fail_msg("%s: NULL shadow stack pointer, test failed\n",
106
+
__func__);
107
+
break;
108
+
case 1:
109
+
if (!(cfi_reg.cfi_status.cfi_state & PTRACE_CFI_ELP_STATE))
110
+
ksft_exit_fail_msg("%s: elp must have been set\n", __func__);
111
+
/* clear elp state. not interested in anything else */
112
+
cfi_reg.cfi_status.cfi_state = 0;
113
+
114
+
ret = ptrace(PTRACE_SETREGSET, pid, (void *)NT_RISCV_USER_CFI, &iov);
115
+
if (ret == -1 && errno)
116
+
ksft_exit_fail_msg("%s: PTRACE_GETREGSET failed\n", __func__);
117
+
break;
118
+
default:
119
+
ksft_exit_fail_msg("%s: unreachable switch case\n", __func__);
120
+
break;
121
+
}
122
+
ptrace(PTRACE_CONT, pid, NULL, NULL);
123
+
ptrace_test_num++;
124
+
}
125
+
126
+
waitpid(pid, &status, 0);
127
+
if (WEXITSTATUS(status) != 11)
128
+
ksft_print_msg("%s, bad return code from child\n", __func__);
129
+
130
+
ksft_print_msg("%s, ptrace test succeeded\n", __func__);
131
+
return true;
132
+
}
133
+
134
+
int main(int argc, char *argv[])
135
+
{
136
+
int ret = 0;
137
+
unsigned long lpad_status = 0, ss_status = 0;
138
+
139
+
ksft_print_header();
140
+
141
+
ksft_print_msg("Starting risc-v tests\n");
142
+
143
+
/*
144
+
* Landing pad test. Not a lot of kernel changes to support landing
145
+
* pads for user mode except lighting up a bit in senvcfg via a prctl.
146
+
* Enable landing pad support throughout the execution of the test binary.
147
+
*/
148
+
ret = my_syscall5(__NR_prctl, PR_GET_INDIR_BR_LP_STATUS, &lpad_status, 0, 0, 0);
149
+
if (ret)
150
+
ksft_exit_fail_msg("Get landing pad status failed with %d\n", ret);
151
+
152
+
if (!(lpad_status & PR_INDIR_BR_LP_ENABLE))
153
+
ksft_exit_fail_msg("Landing pad is not enabled, should be enabled via glibc\n");
154
+
155
+
ret = my_syscall5(__NR_prctl, PR_GET_SHADOW_STACK_STATUS, &ss_status, 0, 0, 0);
156
+
if (ret)
157
+
ksft_exit_fail_msg("Get shadow stack failed with %d\n", ret);
158
+
159
+
if (!(ss_status & PR_SHADOW_STACK_ENABLE))
160
+
ksft_exit_fail_msg("Shadow stack is not enabled, should be enabled via glibc\n");
161
+
162
+
if (!register_signal_handler())
163
+
ksft_exit_fail_msg("Registering signal handler for SIGSEGV failed\n");
164
+
165
+
ksft_print_msg("Landing pad and shadow stack are enabled for binary\n");
166
+
cfi_ptrace_test();
167
+
168
+
execute_shadow_stack_tests();
169
+
170
+
return 0;
171
+
}
172
+
173
+
#pragma GCC pop_options
+385
tools/testing/selftests/riscv/cfi/shadowstack.c
+385
tools/testing/selftests/riscv/cfi/shadowstack.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
3
+
#include "../../kselftest.h"
4
+
#include <sys/wait.h>
5
+
#include <signal.h>
6
+
#include <fcntl.h>
7
+
#include <asm-generic/unistd.h>
8
+
#include <sys/mman.h>
9
+
#include "shadowstack.h"
10
+
#include "cfi_rv_test.h"
11
+
12
+
static struct shadow_stack_tests shstk_tests[] = {
13
+
{ "shstk fork test\n", shadow_stack_fork_test },
14
+
{ "map shadow stack syscall\n", shadow_stack_map_test },
15
+
{ "shadow stack gup tests\n", shadow_stack_gup_tests },
16
+
{ "shadow stack signal tests\n", shadow_stack_signal_test},
17
+
{ "memory protections of shadow stack memory\n", shadow_stack_protection_test }
18
+
};
19
+
20
+
#define RISCV_SHADOW_STACK_TESTS ARRAY_SIZE(shstk_tests)
21
+
22
+
/* do not optimize shadow stack related test functions */
23
+
#pragma GCC push_options
24
+
#pragma GCC optimize("O0")
25
+
26
+
void zar(void)
27
+
{
28
+
unsigned long ssp = 0;
29
+
30
+
ssp = csr_read(CSR_SSP);
31
+
ksft_print_msg("Spewing out shadow stack ptr: %lx\n"
32
+
" This is to ensure shadow stack is indeed enabled and working\n",
33
+
ssp);
34
+
}
35
+
36
+
void bar(void)
37
+
{
38
+
zar();
39
+
}
40
+
41
+
void foo(void)
42
+
{
43
+
bar();
44
+
}
45
+
46
+
void zar_child(void)
47
+
{
48
+
unsigned long ssp = 0;
49
+
50
+
ssp = csr_read(CSR_SSP);
51
+
ksft_print_msg("Spewing out shadow stack ptr: %lx\n"
52
+
" This is to ensure shadow stack is indeed enabled and working\n",
53
+
ssp);
54
+
}
55
+
56
+
void bar_child(void)
57
+
{
58
+
zar_child();
59
+
}
60
+
61
+
void foo_child(void)
62
+
{
63
+
bar_child();
64
+
}
65
+
66
+
typedef void (call_func_ptr)(void);
67
+
/*
68
+
* call couple of functions to test push/pop.
69
+
*/
70
+
int shadow_stack_call_tests(call_func_ptr fn_ptr, bool parent)
71
+
{
72
+
ksft_print_msg("dummy calls for sspush and sspopchk in context of %s\n",
73
+
parent ? "parent" : "child");
74
+
75
+
(fn_ptr)();
76
+
77
+
return 0;
78
+
}
79
+
80
+
/* forks a thread, and ensure shadow stacks fork out */
81
+
bool shadow_stack_fork_test(unsigned long test_num, void *ctx)
82
+
{
83
+
int pid = 0, child_status = 0, parent_pid = 0, ret = 0;
84
+
unsigned long ss_status = 0;
85
+
86
+
ksft_print_msg("Exercising shadow stack fork test\n");
87
+
88
+
ret = my_syscall5(__NR_prctl, PR_GET_SHADOW_STACK_STATUS, &ss_status, 0, 0, 0);
89
+
if (ret) {
90
+
ksft_exit_skip("Shadow stack get status prctl failed with errorcode %d\n", ret);
91
+
return false;
92
+
}
93
+
94
+
if (!(ss_status & PR_SHADOW_STACK_ENABLE))
95
+
ksft_exit_skip("Shadow stack is not enabled, should be enabled via glibc\n");
96
+
97
+
parent_pid = getpid();
98
+
pid = fork();
99
+
100
+
if (pid) {
101
+
ksft_print_msg("Parent pid %d and child pid %d\n", parent_pid, pid);
102
+
shadow_stack_call_tests(&foo, true);
103
+
} else {
104
+
shadow_stack_call_tests(&foo_child, false);
105
+
}
106
+
107
+
if (pid) {
108
+
ksft_print_msg("Waiting on child to finish\n");
109
+
wait(&child_status);
110
+
} else {
111
+
/* exit child gracefully */
112
+
exit(0);
113
+
}
114
+
115
+
if (pid && WIFSIGNALED(child_status)) {
116
+
ksft_print_msg("Child faulted, fork test failed\n");
117
+
return false;
118
+
}
119
+
120
+
return true;
121
+
}
122
+
123
+
/* exercise 'map_shadow_stack', pivot to it and call some functions to ensure it works */
124
+
#define SHADOW_STACK_ALLOC_SIZE 4096
125
+
bool shadow_stack_map_test(unsigned long test_num, void *ctx)
126
+
{
127
+
unsigned long shdw_addr;
128
+
int ret = 0;
129
+
130
+
ksft_print_msg("Exercising shadow stack map test\n");
131
+
132
+
shdw_addr = my_syscall3(__NR_map_shadow_stack, NULL, SHADOW_STACK_ALLOC_SIZE, 0);
133
+
134
+
if (((long)shdw_addr) <= 0) {
135
+
ksft_print_msg("map_shadow_stack failed with error code %d\n",
136
+
(int)shdw_addr);
137
+
return false;
138
+
}
139
+
140
+
ret = munmap((void *)shdw_addr, SHADOW_STACK_ALLOC_SIZE);
141
+
142
+
if (ret) {
143
+
ksft_print_msg("munmap failed with error code %d\n", ret);
144
+
return false;
145
+
}
146
+
147
+
return true;
148
+
}
149
+
150
+
/*
151
+
* shadow stack protection tests. map a shadow stack and
152
+
* validate all memory protections work on it
153
+
*/
154
+
bool shadow_stack_protection_test(unsigned long test_num, void *ctx)
155
+
{
156
+
unsigned long shdw_addr;
157
+
unsigned long *write_addr = NULL;
158
+
int ret = 0, pid = 0, child_status = 0;
159
+
160
+
ksft_print_msg("Exercising shadow stack protection test (WPT)\n");
161
+
162
+
shdw_addr = my_syscall3(__NR_map_shadow_stack, NULL, SHADOW_STACK_ALLOC_SIZE, 0);
163
+
164
+
if (((long)shdw_addr) <= 0) {
165
+
ksft_print_msg("map_shadow_stack failed with error code %d\n",
166
+
(int)shdw_addr);
167
+
return false;
168
+
}
169
+
170
+
write_addr = (unsigned long *)shdw_addr;
171
+
pid = fork();
172
+
173
+
/* no child was created, return false */
174
+
if (pid == -1)
175
+
return false;
176
+
177
+
/*
178
+
* try to perform a store from child on shadow stack memory
179
+
* it should result in SIGSEGV
180
+
*/
181
+
if (!pid) {
182
+
/* below write must lead to SIGSEGV */
183
+
*write_addr = 0xdeadbeef;
184
+
} else {
185
+
wait(&child_status);
186
+
}
187
+
188
+
/* test fail, if 0xdeadbeef present on shadow stack address */
189
+
if (*write_addr == 0xdeadbeef) {
190
+
ksft_print_msg("Shadow stack WPT failed\n");
191
+
return false;
192
+
}
193
+
194
+
/* if child reached here, then fail */
195
+
if (!pid) {
196
+
ksft_print_msg("Shadow stack WPT failed: child reached unreachable state\n");
197
+
return false;
198
+
}
199
+
200
+
/* if child exited via signal handler but not for write on ss */
201
+
if (WIFEXITED(child_status) &&
202
+
WEXITSTATUS(child_status) != CHILD_EXIT_CODE_SSWRITE) {
203
+
ksft_print_msg("Shadow stack WPT failed: child wasn't signaled for write\n");
204
+
return false;
205
+
}
206
+
207
+
ret = munmap(write_addr, SHADOW_STACK_ALLOC_SIZE);
208
+
if (ret) {
209
+
ksft_print_msg("Shadow stack WPT failed: munmap failed, error code %d\n",
210
+
ret);
211
+
return false;
212
+
}
213
+
214
+
return true;
215
+
}
216
+
217
+
#define SS_MAGIC_WRITE_VAL 0xbeefdead
218
+
219
+
int gup_tests(int mem_fd, unsigned long *shdw_addr)
220
+
{
221
+
unsigned long val = 0;
222
+
223
+
lseek(mem_fd, (unsigned long)shdw_addr, SEEK_SET);
224
+
if (read(mem_fd, &val, sizeof(val)) < 0) {
225
+
ksft_print_msg("Reading shadow stack mem via gup failed\n");
226
+
return 1;
227
+
}
228
+
229
+
val = SS_MAGIC_WRITE_VAL;
230
+
lseek(mem_fd, (unsigned long)shdw_addr, SEEK_SET);
231
+
if (write(mem_fd, &val, sizeof(val)) < 0) {
232
+
ksft_print_msg("Writing shadow stack mem via gup failed\n");
233
+
return 1;
234
+
}
235
+
236
+
if (*shdw_addr != SS_MAGIC_WRITE_VAL) {
237
+
ksft_print_msg("GUP write to shadow stack memory failed\n");
238
+
return 1;
239
+
}
240
+
241
+
return 0;
242
+
}
243
+
244
+
bool shadow_stack_gup_tests(unsigned long test_num, void *ctx)
245
+
{
246
+
unsigned long shdw_addr = 0;
247
+
unsigned long *write_addr = NULL;
248
+
int fd = 0;
249
+
bool ret = false;
250
+
251
+
ksft_print_msg("Exercising shadow stack gup tests\n");
252
+
shdw_addr = my_syscall3(__NR_map_shadow_stack, NULL, SHADOW_STACK_ALLOC_SIZE, 0);
253
+
254
+
if (((long)shdw_addr) <= 0) {
255
+
ksft_print_msg("map_shadow_stack failed with error code %d\n", (int)shdw_addr);
256
+
return false;
257
+
}
258
+
259
+
write_addr = (unsigned long *)shdw_addr;
260
+
261
+
fd = open("/proc/self/mem", O_RDWR);
262
+
if (fd == -1)
263
+
return false;
264
+
265
+
if (gup_tests(fd, write_addr)) {
266
+
ksft_print_msg("gup tests failed\n");
267
+
goto out;
268
+
}
269
+
270
+
ret = true;
271
+
out:
272
+
if (shdw_addr && munmap(write_addr, SHADOW_STACK_ALLOC_SIZE)) {
273
+
ksft_print_msg("munmap failed with error code %d\n", ret);
274
+
ret = false;
275
+
}
276
+
277
+
return ret;
278
+
}
279
+
280
+
volatile bool break_loop;
281
+
282
+
void sigusr1_handler(int signo)
283
+
{
284
+
break_loop = true;
285
+
}
286
+
287
+
bool sigusr1_signal_test(void)
288
+
{
289
+
struct sigaction sa = {};
290
+
291
+
sa.sa_handler = sigusr1_handler;
292
+
sa.sa_flags = 0;
293
+
sigemptyset(&sa.sa_mask);
294
+
if (sigaction(SIGUSR1, &sa, NULL)) {
295
+
ksft_print_msg("Registering signal handler for SIGUSR1 failed\n");
296
+
return false;
297
+
}
298
+
299
+
return true;
300
+
}
301
+
302
+
/*
303
+
* shadow stack signal test. shadow stack must be enabled.
304
+
* register a signal, fork another thread which is waiting
305
+
* on signal. Send a signal from parent to child, verify
306
+
* that signal was received by child. If not test fails
307
+
*/
308
+
bool shadow_stack_signal_test(unsigned long test_num, void *ctx)
309
+
{
310
+
int pid = 0, child_status = 0, ret = 0;
311
+
unsigned long ss_status = 0;
312
+
313
+
ksft_print_msg("Exercising shadow stack signal test\n");
314
+
315
+
ret = my_syscall5(__NR_prctl, PR_GET_SHADOW_STACK_STATUS, &ss_status, 0, 0, 0);
316
+
if (ret) {
317
+
ksft_print_msg("Shadow stack get status prctl failed with errorcode %d\n", ret);
318
+
return false;
319
+
}
320
+
321
+
if (!(ss_status & PR_SHADOW_STACK_ENABLE))
322
+
ksft_print_msg("Shadow stack is not enabled, should be enabled via glibc\n");
323
+
324
+
/* this should be caught by signal handler and do an exit */
325
+
if (!sigusr1_signal_test()) {
326
+
ksft_print_msg("Registering sigusr1 handler failed\n");
327
+
exit(-1);
328
+
}
329
+
330
+
pid = fork();
331
+
332
+
if (pid == -1) {
333
+
ksft_print_msg("Signal test: fork failed\n");
334
+
goto out;
335
+
}
336
+
337
+
if (pid == 0) {
338
+
while (!break_loop)
339
+
sleep(1);
340
+
341
+
exit(11);
342
+
/* child shouldn't go beyond here */
343
+
}
344
+
345
+
/* send SIGUSR1 to child */
346
+
kill(pid, SIGUSR1);
347
+
wait(&child_status);
348
+
349
+
out:
350
+
351
+
return (WIFEXITED(child_status) &&
352
+
WEXITSTATUS(child_status) == 11);
353
+
}
354
+
355
+
int execute_shadow_stack_tests(void)
356
+
{
357
+
int ret = 0;
358
+
unsigned long test_count = 0;
359
+
unsigned long shstk_status = 0;
360
+
bool test_pass = false;
361
+
362
+
ksft_print_msg("Executing RISC-V shadow stack self tests\n");
363
+
ksft_set_plan(RISCV_SHADOW_STACK_TESTS);
364
+
365
+
ret = my_syscall5(__NR_prctl, PR_GET_SHADOW_STACK_STATUS, &shstk_status, 0, 0, 0);
366
+
367
+
if (ret != 0)
368
+
ksft_exit_fail_msg("Get shadow stack status failed with %d\n", ret);
369
+
370
+
/*
371
+
* If we are here that means get shadow stack status succeeded and
372
+
* thus shadow stack support is baked in the kernel.
373
+
*/
374
+
while (test_count < RISCV_SHADOW_STACK_TESTS) {
375
+
test_pass = (*shstk_tests[test_count].t_func)(test_count, NULL);
376
+
ksft_test_result(test_pass, shstk_tests[test_count].name);
377
+
test_count++;
378
+
}
379
+
380
+
ksft_finished();
381
+
382
+
return 0;
383
+
}
384
+
385
+
#pragma GCC pop_options
+27
tools/testing/selftests/riscv/cfi/shadowstack.h
+27
tools/testing/selftests/riscv/cfi/shadowstack.h
···
1
+
/* SPDX-License-Identifier: GPL-2.0-only */
2
+
3
+
#ifndef SELFTEST_SHADOWSTACK_TEST_H
4
+
#define SELFTEST_SHADOWSTACK_TEST_H
5
+
#include <stddef.h>
6
+
#include <linux/prctl.h>
7
+
8
+
/*
9
+
* A CFI test returns true for success or false for fail.
10
+
* Takes a test number to index into array, and a void pointer.
11
+
*/
12
+
typedef bool (*shstk_test_func)(unsigned long test_num, void *);
13
+
14
+
struct shadow_stack_tests {
15
+
char *name;
16
+
shstk_test_func t_func;
17
+
};
18
+
19
+
bool shadow_stack_fork_test(unsigned long test_num, void *ctx);
20
+
bool shadow_stack_map_test(unsigned long test_num, void *ctx);
21
+
bool shadow_stack_protection_test(unsigned long test_num, void *ctx);
22
+
bool shadow_stack_gup_tests(unsigned long test_num, void *ctx);
23
+
bool shadow_stack_signal_test(unsigned long test_num, void *ctx);
24
+
25
+
int execute_shadow_stack_tests(void);
26
+
27
+
#endif
+13
-5
tools/testing/selftests/riscv/hwprobe/which-cpus.c
+13
-5
tools/testing/selftests/riscv/hwprobe/which-cpus.c
···
83
83
84
84
int main(int argc, char **argv)
85
85
{
86
-
struct riscv_hwprobe pairs[2];
86
+
struct riscv_hwprobe pairs[3];
87
87
cpu_set_t cpus_aff, cpus;
88
-
__u64 ext0_all;
88
+
__u64 ext0_all, ext1_all;
89
89
long rc;
90
90
91
91
rc = sched_getaffinity(0, sizeof(cpu_set_t), &cpus_aff);
···
112
112
assert(rc == 0 && pairs[0].key == RISCV_HWPROBE_KEY_IMA_EXT_0);
113
113
ext0_all = pairs[0].value;
114
114
115
+
pairs[0] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_1, };
116
+
rc = riscv_hwprobe(pairs, 1, 0, NULL, 0);
117
+
assert(rc == 0 && pairs[0].key == RISCV_HWPROBE_KEY_IMA_EXT_1);
118
+
ext1_all = pairs[0].value;
119
+
115
120
pairs[0] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR, .value = RISCV_HWPROBE_BASE_BEHAVIOR_IMA, };
116
121
CPU_ZERO(&cpus);
117
122
rc = riscv_hwprobe(pairs, 1, 0, (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
···
139
134
140
135
pairs[0] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR, .value = RISCV_HWPROBE_BASE_BEHAVIOR_IMA, };
141
136
pairs[1] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_0, .value = ext0_all, };
137
+
pairs[2] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_1, .value = ext1_all, };
142
138
CPU_ZERO(&cpus);
143
-
rc = riscv_hwprobe(pairs, 2, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
139
+
rc = riscv_hwprobe(pairs, 3, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
144
140
ksft_test_result(rc == 0 && CPU_COUNT(&cpus) == sysconf(_SC_NPROCESSORS_ONLN), "set all cpus\n");
145
141
146
142
pairs[0] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR, .value = RISCV_HWPROBE_BASE_BEHAVIOR_IMA, };
147
143
pairs[1] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_0, .value = ext0_all, };
144
+
pairs[2] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_1, .value = ext1_all, };
148
145
memcpy(&cpus, &cpus_aff, sizeof(cpu_set_t));
149
-
rc = riscv_hwprobe(pairs, 2, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
146
+
rc = riscv_hwprobe(pairs, 3, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
150
147
ksft_test_result(rc == 0 && CPU_EQUAL(&cpus, &cpus_aff), "set all affinity cpus\n");
151
148
152
149
pairs[0] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR, .value = RISCV_HWPROBE_BASE_BEHAVIOR_IMA, };
153
150
pairs[1] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_0, .value = ~ext0_all, };
151
+
pairs[2] = (struct riscv_hwprobe){ .key = RISCV_HWPROBE_KEY_IMA_EXT_1, .value = ~ext1_all, };
154
152
memcpy(&cpus, &cpus_aff, sizeof(cpu_set_t));
155
-
rc = riscv_hwprobe(pairs, 2, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
153
+
rc = riscv_hwprobe(pairs, 3, sizeof(cpu_set_t), (unsigned long *)&cpus, RISCV_HWPROBE_WHICH_CPUS);
156
154
ksft_test_result(rc == 0 && CPU_COUNT(&cpus) == 0, "clear all cpus\n");
157
155
158
156
ksft_finished();
+2
tools/testing/selftests/riscv/vector/.gitignore
+2
tools/testing/selftests/riscv/vector/.gitignore
+9
-1
tools/testing/selftests/riscv/vector/Makefile
+9
-1
tools/testing/selftests/riscv/vector/Makefile
···
2
2
# Copyright (C) 2021 ARM Limited
3
3
# Originally tools/testing/arm64/abi/Makefile
4
4
5
-
TEST_GEN_PROGS := v_initval vstate_prctl vstate_ptrace
5
+
TEST_GEN_PROGS := v_initval vstate_prctl vstate_ptrace validate_v_ptrace
6
6
TEST_GEN_PROGS_EXTENDED := vstate_exec_nolibc v_exec_initval_nolibc
7
+
TEST_GEN_LIBS := v_helpers.c sys_hwprobe.c
7
8
8
9
include ../../lib.mk
10
+
11
+
TEST_GEN_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(TEST_GEN_LIBS))
9
12
10
13
$(OUTPUT)/sys_hwprobe.o: ../hwprobe/sys_hwprobe.S
11
14
$(CC) -static -c -o$@ $(CFLAGS) $^
···
32
29
33
30
$(OUTPUT)/vstate_ptrace: vstate_ptrace.c $(OUTPUT)/sys_hwprobe.o $(OUTPUT)/v_helpers.o
34
31
$(CC) -static -o$@ $(CFLAGS) $(LDFLAGS) $^
32
+
33
+
$(OUTPUT)/validate_v_ptrace: validate_v_ptrace.c $(OUTPUT)/sys_hwprobe.o $(OUTPUT)/v_helpers.o
34
+
$(CC) -static -o$@ $(CFLAGS) $(LDFLAGS) $^
35
+
36
+
EXTRA_CLEAN += $(TEST_GEN_OBJ)
+23
tools/testing/selftests/riscv/vector/v_helpers.c
+23
tools/testing/selftests/riscv/vector/v_helpers.c
···
26
26
return pair.value & RISCV_HWPROBE_EXT_ZVE32X;
27
27
}
28
28
29
+
unsigned long get_vr_len(void)
30
+
{
31
+
unsigned long vlenb;
32
+
33
+
if (is_vector_supported()) {
34
+
asm volatile("csrr %[vlenb], vlenb" : [vlenb] "=r"(vlenb));
35
+
return vlenb;
36
+
}
37
+
38
+
if (is_xtheadvector_supported()) {
39
+
asm volatile (
40
+
// 0 | zimm[10:0] | rs1 | 1 1 1 | rd | 1010111 | vsetvli
41
+
// vsetvli t4, x0, e8, m1, d1
42
+
".4byte 0b00000000000000000111111011010111\n\t"
43
+
"mv %[vlenb], t4\n\t"
44
+
: [vlenb] "=r"(vlenb) : : "memory", "t4");
45
+
return vlenb;
46
+
}
47
+
48
+
printf("WARNING: vector not supported\n");
49
+
return 0;
50
+
}
51
+
29
52
int launch_test(char *next_program, int test_inherit, int xtheadvector)
30
53
{
31
54
char *exec_argv[4], *exec_envp[1];
+2
tools/testing/selftests/riscv/vector/v_helpers.h
+2
tools/testing/selftests/riscv/vector/v_helpers.h
+915
tools/testing/selftests/riscv/vector/validate_v_ptrace.c
+915
tools/testing/selftests/riscv/vector/validate_v_ptrace.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-only
2
+
#include <sys/ptrace.h>
3
+
#include <sys/types.h>
4
+
#include <sys/wait.h>
5
+
#include <sys/uio.h>
6
+
#include <unistd.h>
7
+
#include <errno.h>
8
+
9
+
#include <linux/ptrace.h>
10
+
#include <linux/elf.h>
11
+
12
+
#include "kselftest_harness.h"
13
+
#include "v_helpers.h"
14
+
15
+
#define SR_FS_DIRTY 0x00006000UL
16
+
#define CSR_VXRM_SHIFT 1
17
+
18
+
volatile unsigned long chld_lock;
19
+
20
+
TEST(ptrace_v_not_enabled)
21
+
{
22
+
pid_t pid;
23
+
24
+
if (!(is_vector_supported() || is_xtheadvector_supported()))
25
+
SKIP(return, "Vector not supported");
26
+
27
+
chld_lock = 1;
28
+
pid = fork();
29
+
ASSERT_LE(0, pid)
30
+
TH_LOG("fork: %m");
31
+
32
+
if (pid == 0) {
33
+
while (chld_lock == 1)
34
+
asm volatile("" : : "g"(chld_lock) : "memory");
35
+
36
+
asm volatile ("ebreak" : : : );
37
+
} else {
38
+
struct __riscv_v_regset_state *regset_data;
39
+
unsigned long vlenb = get_vr_len();
40
+
size_t regset_size;
41
+
struct iovec iov;
42
+
int status;
43
+
int ret;
44
+
45
+
/* attach */
46
+
47
+
ASSERT_EQ(0, ptrace(PTRACE_ATTACH, pid, NULL, NULL));
48
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
49
+
ASSERT_TRUE(WIFSTOPPED(status));
50
+
51
+
/* unlock */
52
+
53
+
ASSERT_EQ(0, ptrace(PTRACE_POKEDATA, pid, &chld_lock, 0));
54
+
55
+
/* resume and wait for ebreak */
56
+
57
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
58
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
59
+
ASSERT_TRUE(WIFSTOPPED(status));
60
+
61
+
/* try to read vector registers from the tracee */
62
+
63
+
regset_size = sizeof(*regset_data) + vlenb * 32;
64
+
regset_data = calloc(1, regset_size);
65
+
66
+
iov.iov_base = regset_data;
67
+
iov.iov_len = regset_size;
68
+
69
+
/* V extension is available, but not yet enabled for the tracee */
70
+
71
+
errno = 0;
72
+
ret = ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov);
73
+
ASSERT_EQ(ENODATA, errno);
74
+
ASSERT_EQ(-1, ret);
75
+
76
+
/* cleanup */
77
+
78
+
ASSERT_EQ(0, kill(pid, SIGKILL));
79
+
}
80
+
}
81
+
82
+
TEST(ptrace_v_early_debug)
83
+
{
84
+
static volatile unsigned long vstart;
85
+
static volatile unsigned long vtype;
86
+
static volatile unsigned long vlenb;
87
+
static volatile unsigned long vcsr;
88
+
static volatile unsigned long vl;
89
+
bool xtheadvector;
90
+
pid_t pid;
91
+
92
+
if (!(is_vector_supported() || is_xtheadvector_supported()))
93
+
SKIP(return, "Vector not supported");
94
+
95
+
xtheadvector = is_xtheadvector_supported();
96
+
97
+
chld_lock = 1;
98
+
pid = fork();
99
+
ASSERT_LE(0, pid)
100
+
TH_LOG("fork: %m");
101
+
102
+
if (pid == 0) {
103
+
unsigned long vxsat, vxrm;
104
+
105
+
vlenb = get_vr_len();
106
+
107
+
while (chld_lock == 1)
108
+
asm volatile ("" : : "g"(chld_lock) : "memory");
109
+
110
+
asm volatile (
111
+
"csrr %[vstart], vstart\n"
112
+
"csrr %[vtype], vtype\n"
113
+
"csrr %[vl], vl\n"
114
+
: [vtype] "=r"(vtype), [vstart] "=r"(vstart), [vl] "=r"(vl)
115
+
:
116
+
: "memory");
117
+
118
+
/* no 'is_xtheadvector_supported()' here to avoid clobbering v-state by syscall */
119
+
if (xtheadvector) {
120
+
asm volatile (
121
+
"csrs sstatus, %[bit]\n"
122
+
"csrr %[vxsat], vxsat\n"
123
+
"csrr %[vxrm], vxrm\n"
124
+
: [vxsat] "=r"(vxsat), [vxrm] "=r"(vxrm)
125
+
: [bit] "r" (SR_FS_DIRTY)
126
+
: "memory");
127
+
vcsr = vxsat | vxrm << CSR_VXRM_SHIFT;
128
+
} else {
129
+
asm volatile (
130
+
"csrr %[vcsr], vcsr\n"
131
+
: [vcsr] "=r"(vcsr)
132
+
:
133
+
: "memory");
134
+
}
135
+
136
+
asm volatile (
137
+
".option push\n"
138
+
".option norvc\n"
139
+
"ebreak\n"
140
+
".option pop\n");
141
+
} else {
142
+
struct __riscv_v_regset_state *regset_data;
143
+
unsigned long vstart_csr;
144
+
unsigned long vlenb_csr;
145
+
unsigned long vtype_csr;
146
+
unsigned long vcsr_csr;
147
+
unsigned long vl_csr;
148
+
size_t regset_size;
149
+
struct iovec iov;
150
+
int status;
151
+
152
+
/* attach */
153
+
154
+
ASSERT_EQ(0, ptrace(PTRACE_ATTACH, pid, NULL, NULL));
155
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
156
+
ASSERT_TRUE(WIFSTOPPED(status));
157
+
158
+
/* unlock */
159
+
160
+
ASSERT_EQ(0, ptrace(PTRACE_POKEDATA, pid, &chld_lock, 0));
161
+
162
+
/* resume and wait for ebreak */
163
+
164
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
165
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
166
+
ASSERT_TRUE(WIFSTOPPED(status));
167
+
168
+
/* read tracee vector csr regs using ptrace PEEKDATA */
169
+
170
+
errno = 0;
171
+
vstart_csr = ptrace(PTRACE_PEEKDATA, pid, &vstart, NULL);
172
+
ASSERT_FALSE((errno != 0) && (vstart_csr == -1));
173
+
174
+
errno = 0;
175
+
vl_csr = ptrace(PTRACE_PEEKDATA, pid, &vl, NULL);
176
+
ASSERT_FALSE((errno != 0) && (vl_csr == -1));
177
+
178
+
errno = 0;
179
+
vtype_csr = ptrace(PTRACE_PEEKDATA, pid, &vtype, NULL);
180
+
ASSERT_FALSE((errno != 0) && (vtype_csr == -1));
181
+
182
+
errno = 0;
183
+
vcsr_csr = ptrace(PTRACE_PEEKDATA, pid, &vcsr, NULL);
184
+
ASSERT_FALSE((errno != 0) && (vcsr_csr == -1));
185
+
186
+
errno = 0;
187
+
vlenb_csr = ptrace(PTRACE_PEEKDATA, pid, &vlenb, NULL);
188
+
ASSERT_FALSE((errno != 0) && (vlenb_csr == -1));
189
+
190
+
/* read tracee csr regs using ptrace GETREGSET */
191
+
192
+
regset_size = sizeof(*regset_data) + vlenb_csr * 32;
193
+
regset_data = calloc(1, regset_size);
194
+
195
+
iov.iov_base = regset_data;
196
+
iov.iov_len = regset_size;
197
+
198
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
199
+
200
+
/* compare */
201
+
202
+
EXPECT_EQ(vstart_csr, regset_data->vstart);
203
+
EXPECT_EQ(vtype_csr, regset_data->vtype);
204
+
EXPECT_EQ(vlenb_csr, regset_data->vlenb);
205
+
EXPECT_EQ(vcsr_csr, regset_data->vcsr);
206
+
EXPECT_EQ(vl_csr, regset_data->vl);
207
+
208
+
/* cleanup */
209
+
210
+
ASSERT_EQ(0, kill(pid, SIGKILL));
211
+
}
212
+
}
213
+
214
+
TEST(ptrace_v_syscall_clobbering)
215
+
{
216
+
pid_t pid;
217
+
218
+
if (!is_vector_supported() && !is_xtheadvector_supported())
219
+
SKIP(return, "Vector not supported");
220
+
221
+
chld_lock = 1;
222
+
pid = fork();
223
+
ASSERT_LE(0, pid)
224
+
TH_LOG("fork: %m");
225
+
226
+
if (pid == 0) {
227
+
unsigned long vl;
228
+
229
+
while (chld_lock == 1)
230
+
asm volatile("" : : "g"(chld_lock) : "memory");
231
+
232
+
if (is_xtheadvector_supported()) {
233
+
asm volatile (
234
+
// 0 | zimm[10:0] | rs1 | 1 1 1 | rd |1010111| vsetvli
235
+
// vsetvli t4, x0, e16, m2, d1
236
+
".4byte 0b00000000010100000111111011010111\n"
237
+
"mv %[new_vl], t4\n"
238
+
: [new_vl] "=r" (vl) : : "t4");
239
+
} else {
240
+
asm volatile (
241
+
".option push\n"
242
+
".option arch, +zve32x\n"
243
+
"vsetvli %[new_vl], x0, e16, m2, tu, mu\n"
244
+
".option pop\n"
245
+
: [new_vl] "=r"(vl) : : );
246
+
}
247
+
248
+
while (1) {
249
+
asm volatile (
250
+
".option push\n"
251
+
".option norvc\n"
252
+
"ebreak\n"
253
+
".option pop\n");
254
+
255
+
sleep(0);
256
+
}
257
+
} else {
258
+
struct __riscv_v_regset_state *regset_data;
259
+
unsigned long vlenb = get_vr_len();
260
+
struct user_regs_struct regs;
261
+
size_t regset_size;
262
+
struct iovec iov;
263
+
int status;
264
+
265
+
/* attach */
266
+
267
+
ASSERT_EQ(0, ptrace(PTRACE_ATTACH, pid, NULL, NULL));
268
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
269
+
ASSERT_TRUE(WIFSTOPPED(status));
270
+
271
+
/* unlock */
272
+
273
+
ASSERT_EQ(0, ptrace(PTRACE_POKEDATA, pid, &chld_lock, 0));
274
+
275
+
/* resume and wait for the 1st ebreak */
276
+
277
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
278
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
279
+
ASSERT_TRUE(WIFSTOPPED(status));
280
+
281
+
/* read tracee vector csr regs using ptrace GETREGSET */
282
+
283
+
regset_size = sizeof(*regset_data) + vlenb * 32;
284
+
regset_data = calloc(1, regset_size);
285
+
286
+
iov.iov_base = regset_data;
287
+
iov.iov_len = regset_size;
288
+
289
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
290
+
291
+
/* verify initial vsetvli settings */
292
+
293
+
if (is_xtheadvector_supported())
294
+
EXPECT_EQ(5UL, regset_data->vtype);
295
+
else
296
+
EXPECT_EQ(9UL, regset_data->vtype);
297
+
298
+
EXPECT_EQ(regset_data->vlenb, regset_data->vl);
299
+
EXPECT_EQ(vlenb, regset_data->vlenb);
300
+
EXPECT_EQ(0UL, regset_data->vstart);
301
+
EXPECT_EQ(0UL, regset_data->vcsr);
302
+
303
+
/* skip 1st ebreak, then resume and wait for the 2nd ebreak */
304
+
305
+
iov.iov_base = ®s;
306
+
iov.iov_len = sizeof(regs);
307
+
308
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_PRSTATUS, &iov));
309
+
regs.pc += 4;
310
+
ASSERT_EQ(0, ptrace(PTRACE_SETREGSET, pid, NT_PRSTATUS, &iov));
311
+
312
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
313
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
314
+
ASSERT_TRUE(WIFSTOPPED(status));
315
+
316
+
/* read tracee vtype using ptrace GETREGSET */
317
+
318
+
iov.iov_base = regset_data;
319
+
iov.iov_len = regset_size;
320
+
321
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
322
+
323
+
/* verify that V state is illegal after syscall */
324
+
325
+
EXPECT_EQ((1UL << (__riscv_xlen - 1)), regset_data->vtype);
326
+
EXPECT_EQ(vlenb, regset_data->vlenb);
327
+
EXPECT_EQ(0UL, regset_data->vstart);
328
+
EXPECT_EQ(0UL, regset_data->vcsr);
329
+
EXPECT_EQ(0UL, regset_data->vl);
330
+
331
+
/* cleanup */
332
+
333
+
ASSERT_EQ(0, kill(pid, SIGKILL));
334
+
}
335
+
}
336
+
337
+
FIXTURE(v_csr_invalid)
338
+
{
339
+
};
340
+
341
+
FIXTURE_SETUP(v_csr_invalid)
342
+
{
343
+
}
344
+
345
+
FIXTURE_TEARDOWN(v_csr_invalid)
346
+
{
347
+
}
348
+
349
+
#define VECTOR_1_0 BIT(0)
350
+
#define XTHEAD_VECTOR_0_7 BIT(1)
351
+
352
+
#define vector_test(x) ((x) & VECTOR_1_0)
353
+
#define xthead_test(x) ((x) & XTHEAD_VECTOR_0_7)
354
+
355
+
/* modifications of the initial vsetvli settings */
356
+
FIXTURE_VARIANT(v_csr_invalid)
357
+
{
358
+
unsigned long vstart;
359
+
unsigned long vl;
360
+
unsigned long vtype;
361
+
unsigned long vcsr;
362
+
unsigned long vlenb_mul;
363
+
unsigned long vlenb_min;
364
+
unsigned long vlenb_max;
365
+
unsigned long spec;
366
+
};
367
+
368
+
/* unexpected vlenb value */
369
+
FIXTURE_VARIANT_ADD(v_csr_invalid, new_vlenb)
370
+
{
371
+
.vstart = 0x0,
372
+
.vl = 0x0,
373
+
.vtype = 0x3,
374
+
.vcsr = 0x0,
375
+
.vlenb_mul = 0x2,
376
+
.vlenb_min = 0x0,
377
+
.vlenb_max = 0x0,
378
+
.spec = VECTOR_1_0 | XTHEAD_VECTOR_0_7,
379
+
};
380
+
381
+
/* invalid reserved bits in vcsr */
382
+
FIXTURE_VARIANT_ADD(v_csr_invalid, vcsr_invalid_reserved_bits)
383
+
{
384
+
.vstart = 0x0,
385
+
.vl = 0x0,
386
+
.vtype = 0x3,
387
+
.vcsr = 0x1UL << 8,
388
+
.vlenb_mul = 0x1,
389
+
.vlenb_min = 0x0,
390
+
.vlenb_max = 0x0,
391
+
.spec = VECTOR_1_0 | XTHEAD_VECTOR_0_7,
392
+
};
393
+
394
+
/* invalid reserved bits in vtype */
395
+
FIXTURE_VARIANT_ADD(v_csr_invalid, vtype_invalid_reserved_bits)
396
+
{
397
+
.vstart = 0x0,
398
+
.vl = 0x0,
399
+
.vtype = (0x1UL << 8) | 0x3,
400
+
.vcsr = 0x0,
401
+
.vlenb_mul = 0x1,
402
+
.vlenb_min = 0x0,
403
+
.vlenb_max = 0x0,
404
+
.spec = VECTOR_1_0 | XTHEAD_VECTOR_0_7,
405
+
};
406
+
407
+
/* set vill bit */
408
+
FIXTURE_VARIANT_ADD(v_csr_invalid, invalid_vill_bit)
409
+
{
410
+
.vstart = 0x0,
411
+
.vl = 0x0,
412
+
.vtype = (0x1UL << (__riscv_xlen - 1)) | 0x3,
413
+
.vcsr = 0x0,
414
+
.vlenb_mul = 0x1,
415
+
.vlenb_min = 0x0,
416
+
.vlenb_max = 0x0,
417
+
.spec = VECTOR_1_0 | XTHEAD_VECTOR_0_7,
418
+
};
419
+
420
+
/* reserved vsew value: vsew > 3 */
421
+
FIXTURE_VARIANT_ADD(v_csr_invalid, reserved_vsew)
422
+
{
423
+
.vstart = 0x0,
424
+
.vl = 0x0,
425
+
.vtype = 0x4UL << 3,
426
+
.vcsr = 0x0,
427
+
.vlenb_mul = 0x1,
428
+
.vlenb_min = 0x0,
429
+
.vlenb_max = 0x0,
430
+
.spec = VECTOR_1_0,
431
+
};
432
+
433
+
/* XTheadVector: unsupported non-zero VEDIV value */
434
+
FIXTURE_VARIANT_ADD(v_csr_invalid, reserved_vediv)
435
+
{
436
+
.vstart = 0x0,
437
+
.vl = 0x0,
438
+
.vtype = 0x3UL << 5,
439
+
.vcsr = 0x0,
440
+
.vlenb_mul = 0x1,
441
+
.vlenb_min = 0x0,
442
+
.vlenb_max = 0x0,
443
+
.spec = XTHEAD_VECTOR_0_7,
444
+
};
445
+
446
+
/* reserved vlmul value: vlmul == 4 */
447
+
FIXTURE_VARIANT_ADD(v_csr_invalid, reserved_vlmul)
448
+
{
449
+
.vstart = 0x0,
450
+
.vl = 0x0,
451
+
.vtype = 0x4,
452
+
.vcsr = 0x0,
453
+
.vlenb_mul = 0x1,
454
+
.vlenb_min = 0x0,
455
+
.vlenb_max = 0x0,
456
+
.spec = VECTOR_1_0,
457
+
};
458
+
459
+
/* invalid fractional LMUL for VLEN <= 256: LMUL= 1/8, SEW = 64 */
460
+
FIXTURE_VARIANT_ADD(v_csr_invalid, frac_lmul1)
461
+
{
462
+
.vstart = 0x0,
463
+
.vl = 0x0,
464
+
.vtype = 0x1d,
465
+
.vcsr = 0x0,
466
+
.vlenb_mul = 0x1,
467
+
.vlenb_min = 0x0,
468
+
.vlenb_max = 0x20,
469
+
.spec = VECTOR_1_0,
470
+
};
471
+
472
+
/* invalid integral LMUL for VLEN <= 16: LMUL= 2, SEW = 64 */
473
+
FIXTURE_VARIANT_ADD(v_csr_invalid, int_lmul1)
474
+
{
475
+
.vstart = 0x0,
476
+
.vl = 0x0,
477
+
.vtype = 0x19,
478
+
.vcsr = 0x0,
479
+
.vlenb_mul = 0x1,
480
+
.vlenb_min = 0x0,
481
+
.vlenb_max = 0x2,
482
+
.spec = VECTOR_1_0,
483
+
};
484
+
485
+
/* XTheadVector: invalid integral LMUL for VLEN <= 16: LMUL= 2, SEW = 64 */
486
+
FIXTURE_VARIANT_ADD(v_csr_invalid, int_lmul2)
487
+
{
488
+
.vstart = 0x0,
489
+
.vl = 0x0,
490
+
.vtype = 0xd,
491
+
.vcsr = 0x0,
492
+
.vlenb_mul = 0x1,
493
+
.vlenb_min = 0x0,
494
+
.vlenb_max = 0x2,
495
+
.spec = XTHEAD_VECTOR_0_7,
496
+
};
497
+
498
+
/* invalid VL for VLEN <= 128: LMUL= 2, SEW = 64, VL = 8 */
499
+
FIXTURE_VARIANT_ADD(v_csr_invalid, vl1)
500
+
{
501
+
.vstart = 0x0,
502
+
.vl = 0x8,
503
+
.vtype = 0x19,
504
+
.vcsr = 0x0,
505
+
.vlenb_mul = 0x1,
506
+
.vlenb_min = 0x0,
507
+
.vlenb_max = 0x10,
508
+
.spec = VECTOR_1_0,
509
+
};
510
+
511
+
/* XTheadVector: invalid VL for VLEN <= 128: LMUL= 2, SEW = 64, VL = 8 */
512
+
FIXTURE_VARIANT_ADD(v_csr_invalid, vl2)
513
+
{
514
+
.vstart = 0x0,
515
+
.vl = 0x8,
516
+
.vtype = 0xd,
517
+
.vcsr = 0x0,
518
+
.vlenb_mul = 0x1,
519
+
.vlenb_min = 0x0,
520
+
.vlenb_max = 0x10,
521
+
.spec = XTHEAD_VECTOR_0_7,
522
+
};
523
+
524
+
TEST_F(v_csr_invalid, ptrace_v_invalid_values)
525
+
{
526
+
unsigned long vlenb;
527
+
pid_t pid;
528
+
529
+
if (!is_vector_supported() && !is_xtheadvector_supported())
530
+
SKIP(return, "Vectors not supported");
531
+
532
+
if (is_vector_supported() && !vector_test(variant->spec))
533
+
SKIP(return, "Test not supported for Vector");
534
+
535
+
if (is_xtheadvector_supported() && !xthead_test(variant->spec))
536
+
SKIP(return, "Test not supported for XTheadVector");
537
+
538
+
vlenb = get_vr_len();
539
+
540
+
if (variant->vlenb_min) {
541
+
if (vlenb < variant->vlenb_min)
542
+
SKIP(return, "This test does not support VLEN < %lu\n",
543
+
variant->vlenb_min * 8);
544
+
}
545
+
546
+
if (variant->vlenb_max) {
547
+
if (vlenb > variant->vlenb_max)
548
+
SKIP(return, "This test does not support VLEN > %lu\n",
549
+
variant->vlenb_max * 8);
550
+
}
551
+
552
+
chld_lock = 1;
553
+
pid = fork();
554
+
ASSERT_LE(0, pid)
555
+
TH_LOG("fork: %m");
556
+
557
+
if (pid == 0) {
558
+
unsigned long vl;
559
+
560
+
while (chld_lock == 1)
561
+
asm volatile("" : : "g"(chld_lock) : "memory");
562
+
563
+
if (is_xtheadvector_supported()) {
564
+
asm volatile (
565
+
// 0 | zimm[10:0] | rs1 | 1 1 1 | rd |1010111| vsetvli
566
+
// vsetvli t4, x0, e16, m2, d1
567
+
".4byte 0b00000000010100000111111011010111\n"
568
+
"mv %[new_vl], t4\n"
569
+
: [new_vl] "=r" (vl) : : "t4");
570
+
} else {
571
+
asm volatile (
572
+
".option push\n"
573
+
".option arch, +zve32x\n"
574
+
"vsetvli %[new_vl], x0, e16, m2, tu, mu\n"
575
+
".option pop\n"
576
+
: [new_vl] "=r"(vl) : : );
577
+
}
578
+
579
+
while (1) {
580
+
asm volatile (
581
+
".option push\n"
582
+
".option norvc\n"
583
+
"ebreak\n"
584
+
"nop\n"
585
+
".option pop\n");
586
+
}
587
+
} else {
588
+
struct __riscv_v_regset_state *regset_data;
589
+
size_t regset_size;
590
+
struct iovec iov;
591
+
int status;
592
+
int ret;
593
+
594
+
/* attach */
595
+
596
+
ASSERT_EQ(0, ptrace(PTRACE_ATTACH, pid, NULL, NULL));
597
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
598
+
ASSERT_TRUE(WIFSTOPPED(status));
599
+
600
+
/* unlock */
601
+
602
+
ASSERT_EQ(0, ptrace(PTRACE_POKEDATA, pid, &chld_lock, 0));
603
+
604
+
/* resume and wait for the 1st ebreak */
605
+
606
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
607
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
608
+
ASSERT_TRUE(WIFSTOPPED(status));
609
+
610
+
/* read tracee vector csr regs using ptrace GETREGSET */
611
+
612
+
regset_size = sizeof(*regset_data) + vlenb * 32;
613
+
regset_data = calloc(1, regset_size);
614
+
615
+
iov.iov_base = regset_data;
616
+
iov.iov_len = regset_size;
617
+
618
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
619
+
620
+
/* verify initial vsetvli settings */
621
+
622
+
if (is_xtheadvector_supported())
623
+
EXPECT_EQ(5UL, regset_data->vtype);
624
+
else
625
+
EXPECT_EQ(9UL, regset_data->vtype);
626
+
627
+
EXPECT_EQ(regset_data->vlenb, regset_data->vl);
628
+
EXPECT_EQ(vlenb, regset_data->vlenb);
629
+
EXPECT_EQ(0UL, regset_data->vstart);
630
+
EXPECT_EQ(0UL, regset_data->vcsr);
631
+
632
+
/* apply invalid settings from fixture variants */
633
+
634
+
regset_data->vlenb *= variant->vlenb_mul;
635
+
regset_data->vstart = variant->vstart;
636
+
regset_data->vtype = variant->vtype;
637
+
regset_data->vcsr = variant->vcsr;
638
+
regset_data->vl = variant->vl;
639
+
640
+
iov.iov_base = regset_data;
641
+
iov.iov_len = regset_size;
642
+
643
+
errno = 0;
644
+
ret = ptrace(PTRACE_SETREGSET, pid, NT_RISCV_VECTOR, &iov);
645
+
ASSERT_EQ(errno, EINVAL);
646
+
ASSERT_EQ(ret, -1);
647
+
648
+
/* cleanup */
649
+
650
+
ASSERT_EQ(0, kill(pid, SIGKILL));
651
+
}
652
+
}
653
+
654
+
FIXTURE(v_csr_valid)
655
+
{
656
+
};
657
+
658
+
FIXTURE_SETUP(v_csr_valid)
659
+
{
660
+
}
661
+
662
+
FIXTURE_TEARDOWN(v_csr_valid)
663
+
{
664
+
}
665
+
666
+
/* modifications of the initial vsetvli settings */
667
+
FIXTURE_VARIANT(v_csr_valid)
668
+
{
669
+
unsigned long vstart;
670
+
unsigned long vl;
671
+
unsigned long vtype;
672
+
unsigned long vcsr;
673
+
unsigned long vlenb_mul;
674
+
unsigned long vlenb_min;
675
+
unsigned long vlenb_max;
676
+
unsigned long spec;
677
+
};
678
+
679
+
/* valid for VLEN >= 128: LMUL= 1/4, SEW = 32 */
680
+
FIXTURE_VARIANT_ADD(v_csr_valid, frac_lmul1)
681
+
{
682
+
.vstart = 0x0,
683
+
.vl = 0x0,
684
+
.vtype = 0x16,
685
+
.vcsr = 0x0,
686
+
.vlenb_mul = 0x1,
687
+
.vlenb_min = 0x10,
688
+
.vlenb_max = 0x0,
689
+
.spec = VECTOR_1_0,
690
+
};
691
+
692
+
/* valid for VLEN >= 16: LMUL= 2, SEW = 32 */
693
+
FIXTURE_VARIANT_ADD(v_csr_valid, int_lmul1)
694
+
{
695
+
.vstart = 0x0,
696
+
.vl = 0x0,
697
+
.vtype = 0x11,
698
+
.vcsr = 0x0,
699
+
.vlenb_mul = 0x1,
700
+
.vlenb_min = 0x2,
701
+
.vlenb_max = 0x0,
702
+
.spec = VECTOR_1_0,
703
+
};
704
+
705
+
/* valid for XTheadVector VLEN >= 16: LMUL= 2, SEW = 32 */
706
+
FIXTURE_VARIANT_ADD(v_csr_valid, int_lmul2)
707
+
{
708
+
.vstart = 0x0,
709
+
.vl = 0x0,
710
+
.vtype = 0x9,
711
+
.vcsr = 0x0,
712
+
.vlenb_mul = 0x1,
713
+
.vlenb_min = 0x2,
714
+
.vlenb_max = 0x0,
715
+
.spec = XTHEAD_VECTOR_0_7,
716
+
};
717
+
718
+
/* valid for VLEN >= 32: LMUL= 2, SEW = 32, VL = 2 */
719
+
FIXTURE_VARIANT_ADD(v_csr_valid, int_lmul3)
720
+
{
721
+
.vstart = 0x0,
722
+
.vl = 0x2,
723
+
.vtype = 0x11,
724
+
.vcsr = 0x0,
725
+
.vlenb_mul = 0x1,
726
+
.vlenb_min = 0x4,
727
+
.vlenb_max = 0x0,
728
+
.spec = VECTOR_1_0,
729
+
};
730
+
731
+
TEST_F(v_csr_valid, ptrace_v_valid_values)
732
+
{
733
+
unsigned long vlenb;
734
+
pid_t pid;
735
+
736
+
if (!is_vector_supported() && !is_xtheadvector_supported())
737
+
SKIP(return, "Vectors not supported");
738
+
739
+
if (is_vector_supported() && !vector_test(variant->spec))
740
+
SKIP(return, "Test not supported for Vector");
741
+
742
+
if (is_xtheadvector_supported() && !xthead_test(variant->spec))
743
+
SKIP(return, "Test not supported for XTheadVector");
744
+
745
+
vlenb = get_vr_len();
746
+
747
+
if (variant->vlenb_min) {
748
+
if (vlenb < variant->vlenb_min)
749
+
SKIP(return, "This test does not support VLEN < %lu\n",
750
+
variant->vlenb_min * 8);
751
+
}
752
+
if (variant->vlenb_max) {
753
+
if (vlenb > variant->vlenb_max)
754
+
SKIP(return, "This test does not support VLEN > %lu\n",
755
+
variant->vlenb_max * 8);
756
+
}
757
+
758
+
chld_lock = 1;
759
+
pid = fork();
760
+
ASSERT_LE(0, pid)
761
+
TH_LOG("fork: %m");
762
+
763
+
if (pid == 0) {
764
+
unsigned long vl;
765
+
766
+
while (chld_lock == 1)
767
+
asm volatile("" : : "g"(chld_lock) : "memory");
768
+
769
+
if (is_xtheadvector_supported()) {
770
+
asm volatile (
771
+
// 0 | zimm[10:0] | rs1 | 1 1 1 | rd |1010111| vsetvli
772
+
// vsetvli t4, x0, e16, m2, d1
773
+
".4byte 0b00000000010100000111111011010111\n"
774
+
"mv %[new_vl], t4\n"
775
+
: [new_vl] "=r" (vl) : : "t4");
776
+
} else {
777
+
asm volatile (
778
+
".option push\n"
779
+
".option arch, +zve32x\n"
780
+
"vsetvli %[new_vl], x0, e16, m2, tu, mu\n"
781
+
".option pop\n"
782
+
: [new_vl] "=r"(vl) : : );
783
+
}
784
+
785
+
asm volatile (
786
+
".option push\n"
787
+
".option norvc\n"
788
+
".option arch, +zve32x\n"
789
+
"ebreak\n" /* breakpoint 1: apply new V state using ptrace */
790
+
"nop\n"
791
+
"ebreak\n" /* breakpoint 2: V state clean - context will not be saved */
792
+
"vmv.v.i v0, -1\n"
793
+
"ebreak\n" /* breakpoint 3: V state dirty - context will be saved */
794
+
".option pop\n");
795
+
} else {
796
+
struct __riscv_v_regset_state *regset_data;
797
+
struct user_regs_struct regs;
798
+
size_t regset_size;
799
+
struct iovec iov;
800
+
int status;
801
+
802
+
/* attach */
803
+
804
+
ASSERT_EQ(0, ptrace(PTRACE_ATTACH, pid, NULL, NULL));
805
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
806
+
ASSERT_TRUE(WIFSTOPPED(status));
807
+
808
+
/* unlock */
809
+
810
+
ASSERT_EQ(0, ptrace(PTRACE_POKEDATA, pid, &chld_lock, 0));
811
+
812
+
/* resume and wait for the 1st ebreak */
813
+
814
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
815
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
816
+
ASSERT_TRUE(WIFSTOPPED(status));
817
+
818
+
/* read tracee vector csr regs using ptrace GETREGSET */
819
+
820
+
regset_size = sizeof(*regset_data) + vlenb * 32;
821
+
regset_data = calloc(1, regset_size);
822
+
823
+
iov.iov_base = regset_data;
824
+
iov.iov_len = regset_size;
825
+
826
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
827
+
828
+
/* verify initial vsetvli settings */
829
+
830
+
if (is_xtheadvector_supported())
831
+
EXPECT_EQ(5UL, regset_data->vtype);
832
+
else
833
+
EXPECT_EQ(9UL, regset_data->vtype);
834
+
835
+
EXPECT_EQ(regset_data->vlenb, regset_data->vl);
836
+
EXPECT_EQ(vlenb, regset_data->vlenb);
837
+
EXPECT_EQ(0UL, regset_data->vstart);
838
+
EXPECT_EQ(0UL, regset_data->vcsr);
839
+
840
+
/* apply valid settings from fixture variants */
841
+
842
+
regset_data->vlenb *= variant->vlenb_mul;
843
+
regset_data->vstart = variant->vstart;
844
+
regset_data->vtype = variant->vtype;
845
+
regset_data->vcsr = variant->vcsr;
846
+
regset_data->vl = variant->vl;
847
+
848
+
iov.iov_base = regset_data;
849
+
iov.iov_len = regset_size;
850
+
851
+
ASSERT_EQ(0, ptrace(PTRACE_SETREGSET, pid, NT_RISCV_VECTOR, &iov));
852
+
853
+
/* skip 1st ebreak, then resume and wait for the 2nd ebreak */
854
+
855
+
iov.iov_base = ®s;
856
+
iov.iov_len = sizeof(regs);
857
+
858
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_PRSTATUS, &iov));
859
+
regs.pc += 4;
860
+
ASSERT_EQ(0, ptrace(PTRACE_SETREGSET, pid, NT_PRSTATUS, &iov));
861
+
862
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
863
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
864
+
ASSERT_TRUE(WIFSTOPPED(status));
865
+
866
+
/* read tracee vector csr regs using ptrace GETREGSET */
867
+
868
+
iov.iov_base = regset_data;
869
+
iov.iov_len = regset_size;
870
+
871
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
872
+
873
+
/* verify vector csr regs from tracee context */
874
+
875
+
EXPECT_EQ(regset_data->vstart, variant->vstart);
876
+
EXPECT_EQ(regset_data->vtype, variant->vtype);
877
+
EXPECT_EQ(regset_data->vcsr, variant->vcsr);
878
+
EXPECT_EQ(regset_data->vl, variant->vl);
879
+
EXPECT_EQ(regset_data->vlenb, vlenb);
880
+
881
+
/* skip 2nd ebreak, then resume and wait for the 3rd ebreak */
882
+
883
+
iov.iov_base = ®s;
884
+
iov.iov_len = sizeof(regs);
885
+
886
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_PRSTATUS, &iov));
887
+
regs.pc += 4;
888
+
ASSERT_EQ(0, ptrace(PTRACE_SETREGSET, pid, NT_PRSTATUS, &iov));
889
+
890
+
ASSERT_EQ(0, ptrace(PTRACE_CONT, pid, NULL, NULL));
891
+
ASSERT_EQ(pid, waitpid(pid, &status, 0));
892
+
ASSERT_TRUE(WIFSTOPPED(status));
893
+
894
+
/* read tracee vector csr regs using ptrace GETREGSET */
895
+
896
+
iov.iov_base = regset_data;
897
+
iov.iov_len = regset_size;
898
+
899
+
ASSERT_EQ(0, ptrace(PTRACE_GETREGSET, pid, NT_RISCV_VECTOR, &iov));
900
+
901
+
/* verify vector csr regs from tracee context */
902
+
903
+
EXPECT_EQ(regset_data->vstart, variant->vstart);
904
+
EXPECT_EQ(regset_data->vtype, variant->vtype);
905
+
EXPECT_EQ(regset_data->vcsr, variant->vcsr);
906
+
EXPECT_EQ(regset_data->vl, variant->vl);
907
+
EXPECT_EQ(regset_data->vlenb, vlenb);
908
+
909
+
/* cleanup */
910
+
911
+
ASSERT_EQ(0, kill(pid, SIGKILL));
912
+
}
913
+
}
914
+
915
+
TEST_HARNESS_MAIN
+4
-4
tools/testing/selftests/riscv/vector/vstate_exec_nolibc.c
+4
-4
tools/testing/selftests/riscv/vector/vstate_exec_nolibc.c
···
16
16
if (argc > 2 && strcmp(argv[2], "x"))
17
17
xtheadvector = 1;
18
18
19
-
ctrl = my_syscall1(__NR_prctl, PR_RISCV_V_GET_CONTROL);
20
-
if (ctrl < 0) {
19
+
ctrl = prctl(PR_RISCV_V_GET_CONTROL, 0, 0, 0, 0);
20
+
if (ctrl == -1) {
21
21
puts("PR_RISCV_V_GET_CONTROL is not supported\n");
22
-
return ctrl;
22
+
exit(-1);
23
23
}
24
24
25
25
if (test_inherit) {
···
51
51
}
52
52
53
53
if (!pid) {
54
-
rc = my_syscall1(__NR_prctl, PR_RISCV_V_GET_CONTROL);
54
+
rc = prctl(PR_RISCV_V_GET_CONTROL, 0, 0, 0, 0);
55
55
if (rc != ctrl) {
56
56
puts("child's vstate_ctrl not equal to parent's\n");
57
57
exit(-1);