Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'v6.13-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
Pull crypto updates from Herbert Xu:
"API:
- Add sig driver API
- Remove signing/verification from akcipher API
- Move crypto_simd_disabled_for_test to lib/crypto
- Add WARN_ON for return values from driver that indicates memory
corruption
Algorithms:
- Provide crc32-arch and crc32c-arch through Crypto API
- Optimise crc32c code size on x86
- Optimise crct10dif on arm/arm64
- Optimise p10-aes-gcm on powerpc
- Optimise aegis128 on x86
- Output full sample from test interface in jitter RNG
- Retry without padata when it fails in pcrypt
Drivers:
- Add support for Airoha EN7581 TRNG
- Add support for STM32MP25x platforms in stm32
- Enable iproc-r200 RNG driver on BCMBCA
- Add Broadcom BCM74110 RNG driver"
* tag 'v6.13-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (112 commits)
crypto: marvell/cesa - fix uninit value for struct mv_cesa_op_ctx
crypto: cavium - Fix an error handling path in cpt_ucode_load_fw()
crypto: aesni - Move back to module_init
crypto: lib/mpi - Export mpi_set_bit
crypto: aes-gcm-p10 - Use the correct bit to test for P10
hwrng: amd - remove reference to removed PPC_MAPLE config
crypto: arm/crct10dif - Implement plain NEON variant
crypto: arm/crct10dif - Macroify PMULL asm code
crypto: arm/crct10dif - Use existing mov_l macro instead of __adrl
crypto: arm64/crct10dif - Remove remaining 64x64 PMULL fallback code
crypto: arm64/crct10dif - Use faster 16x64 bit polynomial multiply
crypto: arm64/crct10dif - Remove obsolete chunking logic
crypto: bcm - add error check in the ahash_hmac_init function
crypto: caam - add error check to caam_rsa_set_priv_key_form
hwrng: bcm74110 - Add Broadcom BCM74110 RNG driver
dt-bindings: rng: add binding for BCM74110 RNG
padata: Clean up in padata_do_multithreaded()
crypto: inside-secure - Fix the return value of safexcel_xcbcmac_cra_init()
crypto: qat - Fix missing destroy_workqueue in adf_init_aer()
crypto: rsassa-pkcs1 - Reinstate support for legacy protocols
...
+6124
-4264
+7
Documentation/ABI/testing/debugfs-hisi-hpre
+7
Documentation/ABI/testing/debugfs-hisi-hpre
···
184
184
Contact: linux-crypto@vger.kernel.org
185
185
Description: Dump the total number of time out requests.
186
186
Available for both PF and VF, and take no other effect on HPRE.
187
+
188
+
What: /sys/kernel/debug/hisi_hpre/<bdf>/cap_regs
189
+
Date: Oct 2024
190
+
Contact: linux-crypto@vger.kernel.org
191
+
Description: Dump the values of the qm and hpre capability bit registers and
192
+
support the query of device specifications to facilitate fault locating.
193
+
Available for both PF and VF, and take no other effect on HPRE.
+7
Documentation/ABI/testing/debugfs-hisi-sec
+7
Documentation/ABI/testing/debugfs-hisi-sec
···
157
157
Description: Dump the total number of completed but marked error requests
158
158
to be received.
159
159
Available for both PF and VF, and take no other effect on SEC.
160
+
161
+
What: /sys/kernel/debug/hisi_sec2/<bdf>/cap_regs
162
+
Date: Oct 2024
163
+
Contact: linux-crypto@vger.kernel.org
164
+
Description: Dump the values of the qm and sec capability bit registers and
165
+
support the query of device specifications to facilitate fault locating.
166
+
Available for both PF and VF, and take no other effect on SEC.
+7
Documentation/ABI/testing/debugfs-hisi-zip
+7
Documentation/ABI/testing/debugfs-hisi-zip
···
158
158
Description: Dump the total number of BD type error requests
159
159
to be received.
160
160
Available for both PF and VF, and take no other effect on ZIP.
161
+
162
+
What: /sys/kernel/debug/hisi_zip/<bdf>/cap_regs
163
+
Date: Oct 2024
164
+
Contact: linux-crypto@vger.kernel.org
165
+
Description: Dump the values of the qm and zip capability bit registers and
166
+
support the query of device specifications to facilitate fault locating.
167
+
Available for both PF and VF, and take no other effect on ZIP.
+2
-2
Documentation/crypto/api-akcipher.rst
+2
-2
Documentation/crypto/api-akcipher.rst
···
8
8
---------------------
9
9
10
10
.. kernel-doc:: include/crypto/akcipher.h
11
-
:doc: Generic Public Key API
11
+
:doc: Generic Public Key Cipher API
12
12
13
13
.. kernel-doc:: include/crypto/akcipher.h
14
-
:functions: crypto_alloc_akcipher crypto_free_akcipher crypto_akcipher_set_pub_key crypto_akcipher_set_priv_key crypto_akcipher_maxsize crypto_akcipher_encrypt crypto_akcipher_decrypt crypto_akcipher_sign crypto_akcipher_verify
14
+
:functions: crypto_alloc_akcipher crypto_free_akcipher crypto_akcipher_set_pub_key crypto_akcipher_set_priv_key crypto_akcipher_maxsize crypto_akcipher_encrypt crypto_akcipher_decrypt
15
15
16
16
Asymmetric Cipher Request Handle
17
17
--------------------------------
+15
Documentation/crypto/api-sig.rst
+15
Documentation/crypto/api-sig.rst
···
1
+
Asymmetric Signature Algorithm Definitions
2
+
------------------------------------------
3
+
4
+
.. kernel-doc:: include/crypto/sig.h
5
+
:functions: sig_alg
6
+
7
+
Asymmetric Signature API
8
+
------------------------
9
+
10
+
.. kernel-doc:: include/crypto/sig.h
11
+
:doc: Generic Public Key Signature API
12
+
13
+
.. kernel-doc:: include/crypto/sig.h
14
+
:functions: crypto_alloc_sig crypto_free_sig crypto_sig_set_pubkey crypto_sig_set_privkey crypto_sig_keysize crypto_sig_maxsize crypto_sig_digestsize crypto_sig_sign crypto_sig_verify
15
+
+1
Documentation/crypto/api.rst
+1
Documentation/crypto/api.rst
+2
Documentation/crypto/architecture.rst
+2
Documentation/crypto/architecture.rst
···
214
214
215
215
- CRYPTO_ALG_TYPE_AKCIPHER Asymmetric cipher
216
216
217
+
- CRYPTO_ALG_TYPE_SIG Asymmetric signature
218
+
217
219
- CRYPTO_ALG_TYPE_PCOMPRESS Enhanced version of
218
220
CRYPTO_ALG_TYPE_COMPRESS allowing for segmented compression /
219
221
decompression instead of performing the operation on one segment
+1
Documentation/devicetree/bindings/crypto/qcom-qce.yaml
+1
Documentation/devicetree/bindings/crypto/qcom-qce.yaml
+38
Documentation/devicetree/bindings/rng/airoha,en7581-trng.yaml
+38
Documentation/devicetree/bindings/rng/airoha,en7581-trng.yaml
···
1
+
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2
+
%YAML 1.2
3
+
---
4
+
$id: http://devicetree.org/schemas/rng/airoha,en7581-trng.yaml#
5
+
$schema: http://devicetree.org/meta-schemas/core.yaml#
6
+
7
+
title: Airoha EN7851 True Random Number Generator
8
+
9
+
maintainers:
10
+
- Christian Marangi <ansuelsmth@gmail.com>
11
+
12
+
properties:
13
+
compatible:
14
+
const: airoha,en7581-trng
15
+
16
+
reg:
17
+
maxItems: 1
18
+
19
+
interrupts:
20
+
maxItems: 1
21
+
22
+
required:
23
+
- compatible
24
+
- reg
25
+
- interrupts
26
+
27
+
additionalProperties: false
28
+
29
+
examples:
30
+
- |
31
+
#include <dt-bindings/interrupt-controller/irq.h>
32
+
#include <dt-bindings/interrupt-controller/arm-gic.h>
33
+
34
+
rng@1faa1000 {
35
+
compatible = "airoha,en7581-trng";
36
+
reg = <0x1faa1000 0x1000>;
37
+
interrupts = <GIC_SPI 19 IRQ_TYPE_LEVEL_HIGH>;
38
+
};
+35
Documentation/devicetree/bindings/rng/brcm,bcm74110-rng.yaml
+35
Documentation/devicetree/bindings/rng/brcm,bcm74110-rng.yaml
···
1
+
# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
2
+
%YAML 1.2
3
+
---
4
+
$id: http://devicetree.org/schemas/rng/brcm,bcm74110-rng.yaml#
5
+
$schema: http://devicetree.org/meta-schemas/core.yaml#
6
+
7
+
title: BCM74110 Random number generator
8
+
9
+
description:
10
+
Random number generator used on the BCM74110.
11
+
12
+
maintainers:
13
+
- Markus Mayer <mmayer@broadcom.com>
14
+
- Florian Fainelli <florian.fainelli@broadcom.com>
15
+
16
+
properties:
17
+
compatible:
18
+
enum:
19
+
- brcm,bcm74110-rng
20
+
21
+
reg:
22
+
maxItems: 1
23
+
24
+
required:
25
+
- compatible
26
+
- reg
27
+
28
+
additionalProperties: false
29
+
30
+
examples:
31
+
- |
32
+
rng@83ba000 {
33
+
compatible = "brcm,bcm74110-rng";
34
+
reg = <0x83ba000 0x14>;
35
+
};
+1
-1
Documentation/devicetree/bindings/rng/imx-rng.yaml
+1
-1
Documentation/devicetree/bindings/rng/imx-rng.yaml
+11
-6
Documentation/devicetree/bindings/rng/omap_rng.yaml
Documentation/devicetree/bindings/rng/inside-secure,safexcel-eip76.yaml
+11
-6
Documentation/devicetree/bindings/rng/omap_rng.yaml
Documentation/devicetree/bindings/rng/inside-secure,safexcel-eip76.yaml
···
1
1
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2
2
%YAML 1.2
3
3
---
4
-
$id: http://devicetree.org/schemas/rng/omap_rng.yaml#
4
+
$id: http://devicetree.org/schemas/rng/inside-secure,safexcel-eip76.yaml#
5
5
$schema: http://devicetree.org/meta-schemas/core.yaml#
6
6
7
-
title: OMAP SoC and Inside-Secure HWRNG Module
7
+
title: Inside-Secure HWRNG Module
8
8
9
9
maintainers:
10
10
- Jayesh Choudhary <j-choudhary@ti.com>
11
11
12
12
properties:
13
13
compatible:
14
-
enum:
15
-
- ti,omap2-rng
16
-
- ti,omap4-rng
17
-
- inside-secure,safexcel-eip76
14
+
oneOf:
15
+
- enum:
16
+
- ti,omap2-rng
17
+
- ti,omap4-rng
18
+
- inside-secure,safexcel-eip76
19
+
- items:
20
+
- enum:
21
+
- marvell,armada-8k-rng
22
+
- const: inside-secure,safexcel-eip76
18
23
19
24
ti,hwmods:
20
25
const: rng
+27
-1
Documentation/devicetree/bindings/rng/st,stm32-rng.yaml
+27
-1
Documentation/devicetree/bindings/rng/st,stm32-rng.yaml
···
18
18
enum:
19
19
- st,stm32-rng
20
20
- st,stm32mp13-rng
21
+
- st,stm32mp25-rng
21
22
22
23
reg:
23
24
maxItems: 1
24
25
25
26
clocks:
26
-
maxItems: 1
27
+
minItems: 1
28
+
maxItems: 2
29
+
30
+
clock-names:
31
+
items:
32
+
- const: core
33
+
- const: bus
27
34
28
35
resets:
29
36
maxItems: 1
···
63
56
then:
64
57
properties:
65
58
st,rng-lock-conf: false
59
+
60
+
- if:
61
+
properties:
62
+
compatible:
63
+
contains:
64
+
enum:
65
+
- st,stm32-rng
66
+
- st,stm32mp13-rng
67
+
then:
68
+
properties:
69
+
clocks:
70
+
maxItems: 1
71
+
clock-names: false
72
+
else:
73
+
properties:
74
+
clocks:
75
+
minItems: 2
76
+
required:
77
+
- clock-names
66
78
67
79
additionalProperties: false
68
80
+1
-1
MAINTAINERS
+1
-1
MAINTAINERS
···
11447
11447
F: drivers/dma/ioat*
11448
11448
11449
11449
INTEL IAA CRYPTO DRIVER
11450
-
M: Tom Zanussi <tom.zanussi@linux.intel.com>
11450
+
M: Kristen Accardi <kristen.c.accardi@intel.com>
11451
11451
L: linux-crypto@vger.kernel.org
11452
11452
S: Supported
11453
11453
F: Documentation/driver-api/crypto/iaa/iaa-crypto.rst
+168
-81
arch/arm/crypto/crct10dif-ce-core.S
+168
-81
arch/arm/crypto/crct10dif-ce-core.S
···
112
112
FOLD_CONST_L .req q10l
113
113
FOLD_CONST_H .req q10h
114
114
115
+
/*
116
+
* Pairwise long polynomial multiplication of two 16-bit values
117
+
*
118
+
* { w0, w1 }, { y0, y1 }
119
+
*
120
+
* by two 64-bit values
121
+
*
122
+
* { x0, x1, x2, x3, x4, x5, x6, x7 }, { z0, z1, z2, z3, z4, z5, z6, z7 }
123
+
*
124
+
* where each vector element is a byte, ordered from least to most
125
+
* significant. The resulting 80-bit vectors are XOR'ed together.
126
+
*
127
+
* This can be implemented using 8x8 long polynomial multiplication, by
128
+
* reorganizing the input so that each pairwise 8x8 multiplication
129
+
* produces one of the terms from the decomposition below, and
130
+
* combining the results of each rank and shifting them into place.
131
+
*
132
+
* Rank
133
+
* 0 w0*x0 ^ | y0*z0 ^
134
+
* 1 (w0*x1 ^ w1*x0) << 8 ^ | (y0*z1 ^ y1*z0) << 8 ^
135
+
* 2 (w0*x2 ^ w1*x1) << 16 ^ | (y0*z2 ^ y1*z1) << 16 ^
136
+
* 3 (w0*x3 ^ w1*x2) << 24 ^ | (y0*z3 ^ y1*z2) << 24 ^
137
+
* 4 (w0*x4 ^ w1*x3) << 32 ^ | (y0*z4 ^ y1*z3) << 32 ^
138
+
* 5 (w0*x5 ^ w1*x4) << 40 ^ | (y0*z5 ^ y1*z4) << 40 ^
139
+
* 6 (w0*x6 ^ w1*x5) << 48 ^ | (y0*z6 ^ y1*z5) << 48 ^
140
+
* 7 (w0*x7 ^ w1*x6) << 56 ^ | (y0*z7 ^ y1*z6) << 56 ^
141
+
* 8 w1*x7 << 64 | y1*z7 << 64
142
+
*
143
+
* The inputs can be reorganized into
144
+
*
145
+
* { w0, w0, w0, w0, y0, y0, y0, y0 }, { w1, w1, w1, w1, y1, y1, y1, y1 }
146
+
* { x0, x2, x4, x6, z0, z2, z4, z6 }, { x1, x3, x5, x7, z1, z3, z5, z7 }
147
+
*
148
+
* and after performing 8x8->16 bit long polynomial multiplication of
149
+
* each of the halves of the first vector with those of the second one,
150
+
* we obtain the following four vectors of 16-bit elements:
151
+
*
152
+
* a := { w0*x0, w0*x2, w0*x4, w0*x6 }, { y0*z0, y0*z2, y0*z4, y0*z6 }
153
+
* b := { w0*x1, w0*x3, w0*x5, w0*x7 }, { y0*z1, y0*z3, y0*z5, y0*z7 }
154
+
* c := { w1*x0, w1*x2, w1*x4, w1*x6 }, { y1*z0, y1*z2, y1*z4, y1*z6 }
155
+
* d := { w1*x1, w1*x3, w1*x5, w1*x7 }, { y1*z1, y1*z3, y1*z5, y1*z7 }
156
+
*
157
+
* Results b and c can be XORed together, as the vector elements have
158
+
* matching ranks. Then, the final XOR can be pulled forward, and
159
+
* applied between the halves of each of the remaining three vectors,
160
+
* which are then shifted into place, and XORed together to produce the
161
+
* final 80-bit result.
162
+
*/
163
+
.macro pmull16x64_p8, v16, v64
164
+
vext.8 q11, \v64, \v64, #1
165
+
vld1.64 {q12}, [r4, :128]
166
+
vuzp.8 q11, \v64
167
+
vtbl.8 d24, {\v16\()_L-\v16\()_H}, d24
168
+
vtbl.8 d25, {\v16\()_L-\v16\()_H}, d25
169
+
bl __pmull16x64_p8
170
+
veor \v64, q12, q14
171
+
.endm
172
+
173
+
__pmull16x64_p8:
174
+
vmull.p8 q13, d23, d24
175
+
vmull.p8 q14, d23, d25
176
+
vmull.p8 q15, d22, d24
177
+
vmull.p8 q12, d22, d25
178
+
179
+
veor q14, q14, q15
180
+
veor d24, d24, d25
181
+
veor d26, d26, d27
182
+
veor d28, d28, d29
183
+
vmov.i32 d25, #0
184
+
vmov.i32 d29, #0
185
+
vext.8 q12, q12, q12, #14
186
+
vext.8 q14, q14, q14, #15
187
+
veor d24, d24, d26
188
+
bx lr
189
+
ENDPROC(__pmull16x64_p8)
190
+
191
+
.macro pmull16x64_p64, v16, v64
192
+
vmull.p64 q11, \v64\()l, \v16\()_L
193
+
vmull.p64 \v64, \v64\()h, \v16\()_H
194
+
veor \v64, \v64, q11
195
+
.endm
196
+
115
197
// Fold reg1, reg2 into the next 32 data bytes, storing the result back
116
198
// into reg1, reg2.
117
-
.macro fold_32_bytes, reg1, reg2
118
-
vld1.64 {q11-q12}, [buf]!
199
+
.macro fold_32_bytes, reg1, reg2, p
200
+
vld1.64 {q8-q9}, [buf]!
119
201
120
-
vmull.p64 q8, \reg1\()h, FOLD_CONST_H
121
-
vmull.p64 \reg1, \reg1\()l, FOLD_CONST_L
122
-
vmull.p64 q9, \reg2\()h, FOLD_CONST_H
123
-
vmull.p64 \reg2, \reg2\()l, FOLD_CONST_L
202
+
pmull16x64_\p FOLD_CONST, \reg1
203
+
pmull16x64_\p FOLD_CONST, \reg2
124
204
125
-
CPU_LE( vrev64.8 q11, q11 )
126
-
CPU_LE( vrev64.8 q12, q12 )
127
-
vswp q11l, q11h
128
-
vswp q12l, q12h
205
+
CPU_LE( vrev64.8 q8, q8 )
206
+
CPU_LE( vrev64.8 q9, q9 )
207
+
vswp q8l, q8h
208
+
vswp q9l, q9h
129
209
130
210
veor.8 \reg1, \reg1, q8
131
211
veor.8 \reg2, \reg2, q9
132
-
veor.8 \reg1, \reg1, q11
133
-
veor.8 \reg2, \reg2, q12
134
212
.endm
135
213
136
214
// Fold src_reg into dst_reg, optionally loading the next fold constants
137
-
.macro fold_16_bytes, src_reg, dst_reg, load_next_consts
138
-
vmull.p64 q8, \src_reg\()l, FOLD_CONST_L
139
-
vmull.p64 \src_reg, \src_reg\()h, FOLD_CONST_H
215
+
.macro fold_16_bytes, src_reg, dst_reg, p, load_next_consts
216
+
pmull16x64_\p FOLD_CONST, \src_reg
140
217
.ifnb \load_next_consts
141
218
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
142
219
.endif
143
-
veor.8 \dst_reg, \dst_reg, q8
144
220
veor.8 \dst_reg, \dst_reg, \src_reg
145
221
.endm
146
222
147
-
.macro __adrl, out, sym
148
-
movw \out, #:lower16:\sym
149
-
movt \out, #:upper16:\sym
150
-
.endm
151
-
152
-
//
153
-
// u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
154
-
//
155
-
// Assumes len >= 16.
156
-
//
157
-
ENTRY(crc_t10dif_pmull)
158
-
223
+
.macro crct10dif, p
159
224
// For sizes less than 256 bytes, we can't fold 128 bytes at a time.
160
225
cmp len, #256
161
-
blt .Lless_than_256_bytes
226
+
blt .Lless_than_256_bytes\@
162
227
163
-
__adrl fold_consts_ptr, .Lfold_across_128_bytes_consts
228
+
mov_l fold_consts_ptr, .Lfold_across_128_bytes_consts
164
229
165
230
// Load the first 128 data bytes. Byte swapping is necessary to make
166
231
// the bit order match the polynomial coefficient order.
···
264
199
265
200
// While >= 128 data bytes remain (not counting q0-q7), fold the 128
266
201
// bytes q0-q7 into them, storing the result back into q0-q7.
267
-
.Lfold_128_bytes_loop:
268
-
fold_32_bytes q0, q1
269
-
fold_32_bytes q2, q3
270
-
fold_32_bytes q4, q5
271
-
fold_32_bytes q6, q7
202
+
.Lfold_128_bytes_loop\@:
203
+
fold_32_bytes q0, q1, \p
204
+
fold_32_bytes q2, q3, \p
205
+
fold_32_bytes q4, q5, \p
206
+
fold_32_bytes q6, q7, \p
272
207
subs len, len, #128
273
-
bge .Lfold_128_bytes_loop
208
+
bge .Lfold_128_bytes_loop\@
274
209
275
210
// Now fold the 112 bytes in q0-q6 into the 16 bytes in q7.
276
211
277
212
// Fold across 64 bytes.
278
213
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
279
-
fold_16_bytes q0, q4
280
-
fold_16_bytes q1, q5
281
-
fold_16_bytes q2, q6
282
-
fold_16_bytes q3, q7, 1
214
+
fold_16_bytes q0, q4, \p
215
+
fold_16_bytes q1, q5, \p
216
+
fold_16_bytes q2, q6, \p
217
+
fold_16_bytes q3, q7, \p, 1
283
218
// Fold across 32 bytes.
284
-
fold_16_bytes q4, q6
285
-
fold_16_bytes q5, q7, 1
219
+
fold_16_bytes q4, q6, \p
220
+
fold_16_bytes q5, q7, \p, 1
286
221
// Fold across 16 bytes.
287
-
fold_16_bytes q6, q7
222
+
fold_16_bytes q6, q7, \p
288
223
289
224
// Add 128 to get the correct number of data bytes remaining in 0...127
290
225
// (not counting q7), following the previous extra subtraction by 128.
···
294
229
295
230
// While >= 16 data bytes remain (not counting q7), fold the 16 bytes q7
296
231
// into them, storing the result back into q7.
297
-
blt .Lfold_16_bytes_loop_done
298
-
.Lfold_16_bytes_loop:
299
-
vmull.p64 q8, q7l, FOLD_CONST_L
300
-
vmull.p64 q7, q7h, FOLD_CONST_H
301
-
veor.8 q7, q7, q8
232
+
blt .Lfold_16_bytes_loop_done\@
233
+
.Lfold_16_bytes_loop\@:
234
+
pmull16x64_\p FOLD_CONST, q7
302
235
vld1.64 {q0}, [buf]!
303
236
CPU_LE( vrev64.8 q0, q0 )
304
237
vswp q0l, q0h
305
238
veor.8 q7, q7, q0
306
239
subs len, len, #16
307
-
bge .Lfold_16_bytes_loop
240
+
bge .Lfold_16_bytes_loop\@
308
241
309
-
.Lfold_16_bytes_loop_done:
242
+
.Lfold_16_bytes_loop_done\@:
310
243
// Add 16 to get the correct number of data bytes remaining in 0...15
311
244
// (not counting q7), following the previous extra subtraction by 16.
312
245
adds len, len, #16
313
-
beq .Lreduce_final_16_bytes
246
+
beq .Lreduce_final_16_bytes\@
314
247
315
-
.Lhandle_partial_segment:
248
+
.Lhandle_partial_segment\@:
316
249
// Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first
317
250
// 16 bytes are in q7 and the rest are the remaining data in 'buf'. To
318
251
// do this without needing a fold constant for each possible 'len',
···
325
262
vswp q0l, q0h
326
263
327
264
// q1 = high order part of second chunk: q7 left-shifted by 'len' bytes.
328
-
__adrl r3, .Lbyteshift_table + 16
329
-
sub r3, r3, len
330
-
vld1.8 {q2}, [r3]
265
+
mov_l r1, .Lbyteshift_table + 16
266
+
sub r1, r1, len
267
+
vld1.8 {q2}, [r1]
331
268
vtbl.8 q1l, {q7l-q7h}, q2l
332
269
vtbl.8 q1h, {q7l-q7h}, q2h
333
270
···
345
282
vbsl.8 q2, q1, q0
346
283
347
284
// Fold the first chunk into the second chunk, storing the result in q7.
348
-
vmull.p64 q0, q3l, FOLD_CONST_L
349
-
vmull.p64 q7, q3h, FOLD_CONST_H
350
-
veor.8 q7, q7, q0
351
-
veor.8 q7, q7, q2
285
+
pmull16x64_\p FOLD_CONST, q3
286
+
veor.8 q7, q3, q2
287
+
b .Lreduce_final_16_bytes\@
352
288
353
-
.Lreduce_final_16_bytes:
289
+
.Lless_than_256_bytes\@:
290
+
// Checksumming a buffer of length 16...255 bytes
291
+
292
+
mov_l fold_consts_ptr, .Lfold_across_16_bytes_consts
293
+
294
+
// Load the first 16 data bytes.
295
+
vld1.64 {q7}, [buf]!
296
+
CPU_LE( vrev64.8 q7, q7 )
297
+
vswp q7l, q7h
298
+
299
+
// XOR the first 16 data *bits* with the initial CRC value.
300
+
vmov.i8 q0h, #0
301
+
vmov.u16 q0h[3], init_crc
302
+
veor.8 q7h, q7h, q0h
303
+
304
+
// Load the fold-across-16-bytes constants.
305
+
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
306
+
307
+
cmp len, #16
308
+
beq .Lreduce_final_16_bytes\@ // len == 16
309
+
subs len, len, #32
310
+
addlt len, len, #16
311
+
blt .Lhandle_partial_segment\@ // 17 <= len <= 31
312
+
b .Lfold_16_bytes_loop\@ // 32 <= len <= 255
313
+
314
+
.Lreduce_final_16_bytes\@:
315
+
.endm
316
+
317
+
//
318
+
// u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
319
+
//
320
+
// Assumes len >= 16.
321
+
//
322
+
ENTRY(crc_t10dif_pmull64)
323
+
crct10dif p64
324
+
354
325
// Reduce the 128-bit value M(x), stored in q7, to the final 16-bit CRC.
355
326
356
327
// Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
···
417
320
418
321
vmov.u16 r0, q0l[0]
419
322
bx lr
323
+
ENDPROC(crc_t10dif_pmull64)
420
324
421
-
.Lless_than_256_bytes:
422
-
// Checksumming a buffer of length 16...255 bytes
325
+
ENTRY(crc_t10dif_pmull8)
326
+
push {r4, lr}
327
+
mov_l r4, .L16x64perm
423
328
424
-
__adrl fold_consts_ptr, .Lfold_across_16_bytes_consts
329
+
crct10dif p8
425
330
426
-
// Load the first 16 data bytes.
427
-
vld1.64 {q7}, [buf]!
428
331
CPU_LE( vrev64.8 q7, q7 )
429
332
vswp q7l, q7h
430
-
431
-
// XOR the first 16 data *bits* with the initial CRC value.
432
-
vmov.i8 q0h, #0
433
-
vmov.u16 q0h[3], init_crc
434
-
veor.8 q7h, q7h, q0h
435
-
436
-
// Load the fold-across-16-bytes constants.
437
-
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
438
-
439
-
cmp len, #16
440
-
beq .Lreduce_final_16_bytes // len == 16
441
-
subs len, len, #32
442
-
addlt len, len, #16
443
-
blt .Lhandle_partial_segment // 17 <= len <= 31
444
-
b .Lfold_16_bytes_loop // 32 <= len <= 255
445
-
ENDPROC(crc_t10dif_pmull)
333
+
vst1.64 {q7}, [r3, :128]
334
+
pop {r4, pc}
335
+
ENDPROC(crc_t10dif_pmull8)
446
336
447
337
.section ".rodata", "a"
448
338
.align 4
···
463
379
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
464
380
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7
465
381
.byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe , 0x0
382
+
383
+
.L16x64perm:
384
+
.quad 0x808080800000000, 0x909090901010101
+45
-10
arch/arm/crypto/crct10dif-ce-glue.c
+45
-10
arch/arm/crypto/crct10dif-ce-glue.c
···
19
19
20
20
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
21
21
22
-
asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
22
+
asmlinkage u16 crc_t10dif_pmull64(u16 init_crc, const u8 *buf, size_t len);
23
+
asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u8 *buf, size_t len,
24
+
u8 out[16]);
23
25
24
26
static int crct10dif_init(struct shash_desc *desc)
25
27
{
···
31
29
return 0;
32
30
}
33
31
34
-
static int crct10dif_update(struct shash_desc *desc, const u8 *data,
35
-
unsigned int length)
32
+
static int crct10dif_update_ce(struct shash_desc *desc, const u8 *data,
33
+
unsigned int length)
36
34
{
37
35
u16 *crc = shash_desc_ctx(desc);
38
36
39
37
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
40
38
kernel_neon_begin();
41
-
*crc = crc_t10dif_pmull(*crc, data, length);
39
+
*crc = crc_t10dif_pmull64(*crc, data, length);
42
40
kernel_neon_end();
43
41
} else {
44
42
*crc = crc_t10dif_generic(*crc, data, length);
45
43
}
46
44
45
+
return 0;
46
+
}
47
+
48
+
static int crct10dif_update_neon(struct shash_desc *desc, const u8 *data,
49
+
unsigned int length)
50
+
{
51
+
u16 *crcp = shash_desc_ctx(desc);
52
+
u8 buf[16] __aligned(16);
53
+
u16 crc = *crcp;
54
+
55
+
if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
56
+
kernel_neon_begin();
57
+
crc_t10dif_pmull8(crc, data, length, buf);
58
+
kernel_neon_end();
59
+
60
+
crc = 0;
61
+
data = buf;
62
+
length = sizeof(buf);
63
+
}
64
+
65
+
*crcp = crc_t10dif_generic(crc, data, length);
47
66
return 0;
48
67
}
49
68
···
76
53
return 0;
77
54
}
78
55
79
-
static struct shash_alg crc_t10dif_alg = {
56
+
static struct shash_alg algs[] = {{
80
57
.digestsize = CRC_T10DIF_DIGEST_SIZE,
81
58
.init = crct10dif_init,
82
-
.update = crct10dif_update,
59
+
.update = crct10dif_update_neon,
60
+
.final = crct10dif_final,
61
+
.descsize = CRC_T10DIF_DIGEST_SIZE,
62
+
63
+
.base.cra_name = "crct10dif",
64
+
.base.cra_driver_name = "crct10dif-arm-neon",
65
+
.base.cra_priority = 150,
66
+
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
67
+
.base.cra_module = THIS_MODULE,
68
+
}, {
69
+
.digestsize = CRC_T10DIF_DIGEST_SIZE,
70
+
.init = crct10dif_init,
71
+
.update = crct10dif_update_ce,
83
72
.final = crct10dif_final,
84
73
.descsize = CRC_T10DIF_DIGEST_SIZE,
85
74
···
100
65
.base.cra_priority = 200,
101
66
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
102
67
.base.cra_module = THIS_MODULE,
103
-
};
68
+
}};
104
69
105
70
static int __init crc_t10dif_mod_init(void)
106
71
{
107
-
if (!(elf_hwcap2 & HWCAP2_PMULL))
72
+
if (!(elf_hwcap & HWCAP_NEON))
108
73
return -ENODEV;
109
74
110
-
return crypto_register_shash(&crc_t10dif_alg);
75
+
return crypto_register_shashes(algs, 1 + !!(elf_hwcap2 & HWCAP2_PMULL));
111
76
}
112
77
113
78
static void __exit crc_t10dif_mod_exit(void)
114
79
{
115
-
crypto_unregister_shash(&crc_t10dif_alg);
80
+
crypto_unregister_shashes(algs, 1 + !!(elf_hwcap2 & HWCAP2_PMULL));
116
81
}
117
82
118
83
module_init(crc_t10dif_mod_init);
+143
-188
arch/arm64/crypto/crct10dif-ce-core.S
+143
-188
arch/arm64/crypto/crct10dif-ce-core.S
···
1
1
//
2
2
// Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions
3
3
//
4
-
// Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
5
-
// Copyright (C) 2019 Google LLC <ebiggers@google.com>
4
+
// Copyright (C) 2016 Linaro Ltd
5
+
// Copyright (C) 2019-2024 Google LLC
6
+
//
7
+
// Authors: Ard Biesheuvel <ardb@google.com>
8
+
// Eric Biggers <ebiggers@google.com>
6
9
//
7
10
// This program is free software; you can redistribute it and/or modify
8
11
// it under the terms of the GNU General Public License version 2 as
···
74
71
init_crc .req w0
75
72
buf .req x1
76
73
len .req x2
77
-
fold_consts_ptr .req x3
74
+
fold_consts_ptr .req x5
78
75
79
76
fold_consts .req v10
80
-
81
-
ad .req v14
82
-
83
-
k00_16 .req v15
84
-
k32_48 .req v16
85
77
86
78
t3 .req v17
87
79
t4 .req v18
···
84
86
t6 .req v20
85
87
t7 .req v21
86
88
t8 .req v22
87
-
t9 .req v23
88
89
89
-
perm1 .req v24
90
-
perm2 .req v25
91
-
perm3 .req v26
92
-
perm4 .req v27
90
+
perm .req v27
93
91
94
-
bd1 .req v28
95
-
bd2 .req v29
96
-
bd3 .req v30
97
-
bd4 .req v31
98
-
99
-
.macro __pmull_init_p64
92
+
.macro pmull16x64_p64, a16, b64, c64
93
+
pmull2 \c64\().1q, \a16\().2d, \b64\().2d
94
+
pmull \b64\().1q, \a16\().1d, \b64\().1d
100
95
.endm
101
96
102
-
.macro __pmull_pre_p64, bd
97
+
/*
98
+
* Pairwise long polynomial multiplication of two 16-bit values
99
+
*
100
+
* { w0, w1 }, { y0, y1 }
101
+
*
102
+
* by two 64-bit values
103
+
*
104
+
* { x0, x1, x2, x3, x4, x5, x6, x7 }, { z0, z1, z2, z3, z4, z5, z6, z7 }
105
+
*
106
+
* where each vector element is a byte, ordered from least to most
107
+
* significant.
108
+
*
109
+
* This can be implemented using 8x8 long polynomial multiplication, by
110
+
* reorganizing the input so that each pairwise 8x8 multiplication
111
+
* produces one of the terms from the decomposition below, and
112
+
* combining the results of each rank and shifting them into place.
113
+
*
114
+
* Rank
115
+
* 0 w0*x0 ^ | y0*z0 ^
116
+
* 1 (w0*x1 ^ w1*x0) << 8 ^ | (y0*z1 ^ y1*z0) << 8 ^
117
+
* 2 (w0*x2 ^ w1*x1) << 16 ^ | (y0*z2 ^ y1*z1) << 16 ^
118
+
* 3 (w0*x3 ^ w1*x2) << 24 ^ | (y0*z3 ^ y1*z2) << 24 ^
119
+
* 4 (w0*x4 ^ w1*x3) << 32 ^ | (y0*z4 ^ y1*z3) << 32 ^
120
+
* 5 (w0*x5 ^ w1*x4) << 40 ^ | (y0*z5 ^ y1*z4) << 40 ^
121
+
* 6 (w0*x6 ^ w1*x5) << 48 ^ | (y0*z6 ^ y1*z5) << 48 ^
122
+
* 7 (w0*x7 ^ w1*x6) << 56 ^ | (y0*z7 ^ y1*z6) << 56 ^
123
+
* 8 w1*x7 << 64 | y1*z7 << 64
124
+
*
125
+
* The inputs can be reorganized into
126
+
*
127
+
* { w0, w0, w0, w0, y0, y0, y0, y0 }, { w1, w1, w1, w1, y1, y1, y1, y1 }
128
+
* { x0, x2, x4, x6, z0, z2, z4, z6 }, { x1, x3, x5, x7, z1, z3, z5, z7 }
129
+
*
130
+
* and after performing 8x8->16 bit long polynomial multiplication of
131
+
* each of the halves of the first vector with those of the second one,
132
+
* we obtain the following four vectors of 16-bit elements:
133
+
*
134
+
* a := { w0*x0, w0*x2, w0*x4, w0*x6 }, { y0*z0, y0*z2, y0*z4, y0*z6 }
135
+
* b := { w0*x1, w0*x3, w0*x5, w0*x7 }, { y0*z1, y0*z3, y0*z5, y0*z7 }
136
+
* c := { w1*x0, w1*x2, w1*x4, w1*x6 }, { y1*z0, y1*z2, y1*z4, y1*z6 }
137
+
* d := { w1*x1, w1*x3, w1*x5, w1*x7 }, { y1*z1, y1*z3, y1*z5, y1*z7 }
138
+
*
139
+
* Results b and c can be XORed together, as the vector elements have
140
+
* matching ranks. Then, the final XOR (*) can be pulled forward, and
141
+
* applied between the halves of each of the remaining three vectors,
142
+
* which are then shifted into place, and combined to produce two
143
+
* 80-bit results.
144
+
*
145
+
* (*) NOTE: the 16x64 bit polynomial multiply below is not equivalent
146
+
* to the 64x64 bit one above, but XOR'ing the outputs together will
147
+
* produce the expected result, and this is sufficient in the context of
148
+
* this algorithm.
149
+
*/
150
+
.macro pmull16x64_p8, a16, b64, c64
151
+
ext t7.16b, \b64\().16b, \b64\().16b, #1
152
+
tbl t5.16b, {\a16\().16b}, perm.16b
153
+
uzp1 t7.16b, \b64\().16b, t7.16b
154
+
bl __pmull_p8_16x64
155
+
ext \b64\().16b, t4.16b, t4.16b, #15
156
+
eor \c64\().16b, t8.16b, t5.16b
103
157
.endm
104
158
105
-
.macro __pmull_init_p8
106
-
// k00_16 := 0x0000000000000000_000000000000ffff
107
-
// k32_48 := 0x00000000ffffffff_0000ffffffffffff
108
-
movi k32_48.2d, #0xffffffff
109
-
mov k32_48.h[2], k32_48.h[0]
110
-
ushr k00_16.2d, k32_48.2d, #32
159
+
SYM_FUNC_START_LOCAL(__pmull_p8_16x64)
160
+
ext t6.16b, t5.16b, t5.16b, #8
111
161
112
-
// prepare the permutation vectors
113
-
mov_q x5, 0x080f0e0d0c0b0a09
114
-
movi perm4.8b, #8
115
-
dup perm1.2d, x5
116
-
eor perm1.16b, perm1.16b, perm4.16b
117
-
ushr perm2.2d, perm1.2d, #8
118
-
ushr perm3.2d, perm1.2d, #16
119
-
ushr perm4.2d, perm1.2d, #24
120
-
sli perm2.2d, perm1.2d, #56
121
-
sli perm3.2d, perm1.2d, #48
122
-
sli perm4.2d, perm1.2d, #40
123
-
.endm
162
+
pmull t3.8h, t7.8b, t5.8b
163
+
pmull t4.8h, t7.8b, t6.8b
164
+
pmull2 t5.8h, t7.16b, t5.16b
165
+
pmull2 t6.8h, t7.16b, t6.16b
124
166
125
-
.macro __pmull_pre_p8, bd
126
-
tbl bd1.16b, {\bd\().16b}, perm1.16b
127
-
tbl bd2.16b, {\bd\().16b}, perm2.16b
128
-
tbl bd3.16b, {\bd\().16b}, perm3.16b
129
-
tbl bd4.16b, {\bd\().16b}, perm4.16b
130
-
.endm
131
-
132
-
SYM_FUNC_START_LOCAL(__pmull_p8_core)
133
-
.L__pmull_p8_core:
134
-
ext t4.8b, ad.8b, ad.8b, #1 // A1
135
-
ext t5.8b, ad.8b, ad.8b, #2 // A2
136
-
ext t6.8b, ad.8b, ad.8b, #3 // A3
137
-
138
-
pmull t4.8h, t4.8b, fold_consts.8b // F = A1*B
139
-
pmull t8.8h, ad.8b, bd1.8b // E = A*B1
140
-
pmull t5.8h, t5.8b, fold_consts.8b // H = A2*B
141
-
pmull t7.8h, ad.8b, bd2.8b // G = A*B2
142
-
pmull t6.8h, t6.8b, fold_consts.8b // J = A3*B
143
-
pmull t9.8h, ad.8b, bd3.8b // I = A*B3
144
-
pmull t3.8h, ad.8b, bd4.8b // K = A*B4
145
-
b 0f
146
-
147
-
.L__pmull_p8_core2:
148
-
tbl t4.16b, {ad.16b}, perm1.16b // A1
149
-
tbl t5.16b, {ad.16b}, perm2.16b // A2
150
-
tbl t6.16b, {ad.16b}, perm3.16b // A3
151
-
152
-
pmull2 t4.8h, t4.16b, fold_consts.16b // F = A1*B
153
-
pmull2 t8.8h, ad.16b, bd1.16b // E = A*B1
154
-
pmull2 t5.8h, t5.16b, fold_consts.16b // H = A2*B
155
-
pmull2 t7.8h, ad.16b, bd2.16b // G = A*B2
156
-
pmull2 t6.8h, t6.16b, fold_consts.16b // J = A3*B
157
-
pmull2 t9.8h, ad.16b, bd3.16b // I = A*B3
158
-
pmull2 t3.8h, ad.16b, bd4.16b // K = A*B4
159
-
160
-
0: eor t4.16b, t4.16b, t8.16b // L = E + F
161
-
eor t5.16b, t5.16b, t7.16b // M = G + H
162
-
eor t6.16b, t6.16b, t9.16b // N = I + J
163
-
164
-
uzp1 t8.2d, t4.2d, t5.2d
165
-
uzp2 t4.2d, t4.2d, t5.2d
166
-
uzp1 t7.2d, t6.2d, t3.2d
167
-
uzp2 t6.2d, t6.2d, t3.2d
168
-
169
-
// t4 = (L) (P0 + P1) << 8
170
-
// t5 = (M) (P2 + P3) << 16
171
-
eor t8.16b, t8.16b, t4.16b
172
-
and t4.16b, t4.16b, k32_48.16b
173
-
174
-
// t6 = (N) (P4 + P5) << 24
175
-
// t7 = (K) (P6 + P7) << 32
176
-
eor t7.16b, t7.16b, t6.16b
177
-
and t6.16b, t6.16b, k00_16.16b
178
-
179
-
eor t8.16b, t8.16b, t4.16b
180
-
eor t7.16b, t7.16b, t6.16b
181
-
182
-
zip2 t5.2d, t8.2d, t4.2d
183
-
zip1 t4.2d, t8.2d, t4.2d
184
-
zip2 t3.2d, t7.2d, t6.2d
185
-
zip1 t6.2d, t7.2d, t6.2d
186
-
187
-
ext t4.16b, t4.16b, t4.16b, #15
167
+
ext t8.16b, t3.16b, t3.16b, #8
168
+
eor t4.16b, t4.16b, t6.16b
169
+
ext t7.16b, t5.16b, t5.16b, #8
170
+
ext t6.16b, t4.16b, t4.16b, #8
171
+
eor t8.8b, t8.8b, t3.8b
172
+
eor t5.8b, t5.8b, t7.8b
173
+
eor t4.8b, t4.8b, t6.8b
188
174
ext t5.16b, t5.16b, t5.16b, #14
189
-
ext t6.16b, t6.16b, t6.16b, #13
190
-
ext t3.16b, t3.16b, t3.16b, #12
191
-
192
-
eor t4.16b, t4.16b, t5.16b
193
-
eor t6.16b, t6.16b, t3.16b
194
175
ret
195
-
SYM_FUNC_END(__pmull_p8_core)
176
+
SYM_FUNC_END(__pmull_p8_16x64)
196
177
197
-
.macro __pmull_p8, rq, ad, bd, i
198
-
.ifnc \bd, fold_consts
199
-
.err
200
-
.endif
201
-
mov ad.16b, \ad\().16b
202
-
.ifb \i
203
-
pmull \rq\().8h, \ad\().8b, \bd\().8b // D = A*B
204
-
.else
205
-
pmull2 \rq\().8h, \ad\().16b, \bd\().16b // D = A*B
206
-
.endif
207
-
208
-
bl .L__pmull_p8_core\i
209
-
210
-
eor \rq\().16b, \rq\().16b, t4.16b
211
-
eor \rq\().16b, \rq\().16b, t6.16b
212
-
.endm
213
178
214
179
// Fold reg1, reg2 into the next 32 data bytes, storing the result back
215
180
// into reg1, reg2.
216
181
.macro fold_32_bytes, p, reg1, reg2
217
182
ldp q11, q12, [buf], #0x20
218
183
219
-
__pmull_\p v8, \reg1, fold_consts, 2
220
-
__pmull_\p \reg1, \reg1, fold_consts
184
+
pmull16x64_\p fold_consts, \reg1, v8
221
185
222
186
CPU_LE( rev64 v11.16b, v11.16b )
223
187
CPU_LE( rev64 v12.16b, v12.16b )
224
188
225
-
__pmull_\p v9, \reg2, fold_consts, 2
226
-
__pmull_\p \reg2, \reg2, fold_consts
189
+
pmull16x64_\p fold_consts, \reg2, v9
227
190
228
191
CPU_LE( ext v11.16b, v11.16b, v11.16b, #8 )
229
192
CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
···
197
238
198
239
// Fold src_reg into dst_reg, optionally loading the next fold constants
199
240
.macro fold_16_bytes, p, src_reg, dst_reg, load_next_consts
200
-
__pmull_\p v8, \src_reg, fold_consts
201
-
__pmull_\p \src_reg, \src_reg, fold_consts, 2
241
+
pmull16x64_\p fold_consts, \src_reg, v8
202
242
.ifnb \load_next_consts
203
243
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
204
-
__pmull_pre_\p fold_consts
205
244
.endif
206
245
eor \dst_reg\().16b, \dst_reg\().16b, v8.16b
207
246
eor \dst_reg\().16b, \dst_reg\().16b, \src_reg\().16b
208
247
.endm
209
248
210
-
.macro __pmull_p64, rd, rn, rm, n
211
-
.ifb \n
212
-
pmull \rd\().1q, \rn\().1d, \rm\().1d
213
-
.else
214
-
pmull2 \rd\().1q, \rn\().2d, \rm\().2d
215
-
.endif
216
-
.endm
217
-
218
249
.macro crc_t10dif_pmull, p
219
-
__pmull_init_\p
220
250
221
251
// For sizes less than 256 bytes, we can't fold 128 bytes at a time.
222
252
cmp len, #256
···
244
296
245
297
// Load the constants for folding across 128 bytes.
246
298
ld1 {fold_consts.2d}, [fold_consts_ptr]
247
-
__pmull_pre_\p fold_consts
248
299
249
300
// Subtract 128 for the 128 data bytes just consumed. Subtract another
250
301
// 128 to simplify the termination condition of the following loop.
···
265
318
// Fold across 64 bytes.
266
319
add fold_consts_ptr, fold_consts_ptr, #16
267
320
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
268
-
__pmull_pre_\p fold_consts
269
321
fold_16_bytes \p, v0, v4
270
322
fold_16_bytes \p, v1, v5
271
323
fold_16_bytes \p, v2, v6
···
285
339
// into them, storing the result back into v7.
286
340
b.lt .Lfold_16_bytes_loop_done_\@
287
341
.Lfold_16_bytes_loop_\@:
288
-
__pmull_\p v8, v7, fold_consts
289
-
__pmull_\p v7, v7, fold_consts, 2
342
+
pmull16x64_\p fold_consts, v7, v8
290
343
eor v7.16b, v7.16b, v8.16b
291
344
ldr q0, [buf], #16
292
345
CPU_LE( rev64 v0.16b, v0.16b )
···
332
387
bsl v2.16b, v1.16b, v0.16b
333
388
334
389
// Fold the first chunk into the second chunk, storing the result in v7.
335
-
__pmull_\p v0, v3, fold_consts
336
-
__pmull_\p v7, v3, fold_consts, 2
337
-
eor v7.16b, v7.16b, v0.16b
390
+
pmull16x64_\p fold_consts, v3, v0
391
+
eor v7.16b, v3.16b, v0.16b
338
392
eor v7.16b, v7.16b, v2.16b
339
-
340
-
.Lreduce_final_16_bytes_\@:
341
-
// Reduce the 128-bit value M(x), stored in v7, to the final 16-bit CRC.
342
-
343
-
movi v2.16b, #0 // init zero register
344
-
345
-
// Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
346
-
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
347
-
__pmull_pre_\p fold_consts
348
-
349
-
// Fold the high 64 bits into the low 64 bits, while also multiplying by
350
-
// x^64. This produces a 128-bit value congruent to x^64 * M(x) and
351
-
// whose low 48 bits are 0.
352
-
ext v0.16b, v2.16b, v7.16b, #8
353
-
__pmull_\p v7, v7, fold_consts, 2 // high bits * x^48 * (x^80 mod G(x))
354
-
eor v0.16b, v0.16b, v7.16b // + low bits * x^64
355
-
356
-
// Fold the high 32 bits into the low 96 bits. This produces a 96-bit
357
-
// value congruent to x^64 * M(x) and whose low 48 bits are 0.
358
-
ext v1.16b, v0.16b, v2.16b, #12 // extract high 32 bits
359
-
mov v0.s[3], v2.s[0] // zero high 32 bits
360
-
__pmull_\p v1, v1, fold_consts // high 32 bits * x^48 * (x^48 mod G(x))
361
-
eor v0.16b, v0.16b, v1.16b // + low bits
362
-
363
-
// Load G(x) and floor(x^48 / G(x)).
364
-
ld1 {fold_consts.2d}, [fold_consts_ptr]
365
-
__pmull_pre_\p fold_consts
366
-
367
-
// Use Barrett reduction to compute the final CRC value.
368
-
__pmull_\p v1, v0, fold_consts, 2 // high 32 bits * floor(x^48 / G(x))
369
-
ushr v1.2d, v1.2d, #32 // /= x^32
370
-
__pmull_\p v1, v1, fold_consts // *= G(x)
371
-
ushr v0.2d, v0.2d, #48
372
-
eor v0.16b, v0.16b, v1.16b // + low 16 nonzero bits
373
-
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of v0.
374
-
375
-
umov w0, v0.h[0]
376
-
.ifc \p, p8
377
-
frame_pop
378
-
.endif
379
-
ret
393
+
b .Lreduce_final_16_bytes_\@
380
394
381
395
.Lless_than_256_bytes_\@:
382
396
// Checksumming a buffer of length 16...255 bytes
···
354
450
355
451
// Load the fold-across-16-bytes constants.
356
452
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
357
-
__pmull_pre_\p fold_consts
358
453
359
454
cmp len, #16
360
455
b.eq .Lreduce_final_16_bytes_\@ // len == 16
···
361
458
b.ge .Lfold_16_bytes_loop_\@ // 32 <= len <= 255
362
459
add len, len, #16
363
460
b .Lhandle_partial_segment_\@ // 17 <= len <= 31
461
+
462
+
.Lreduce_final_16_bytes_\@:
364
463
.endm
365
464
366
465
//
···
372
467
//
373
468
SYM_FUNC_START(crc_t10dif_pmull_p8)
374
469
frame_push 1
470
+
471
+
// Compose { 0,0,0,0, 8,8,8,8, 1,1,1,1, 9,9,9,9 }
472
+
movi perm.4h, #8, lsl #8
473
+
orr perm.2s, #1, lsl #16
474
+
orr perm.2s, #1, lsl #24
475
+
zip1 perm.16b, perm.16b, perm.16b
476
+
zip1 perm.16b, perm.16b, perm.16b
477
+
375
478
crc_t10dif_pmull p8
479
+
480
+
CPU_LE( rev64 v7.16b, v7.16b )
481
+
CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
482
+
str q7, [x3]
483
+
484
+
frame_pop
485
+
ret
376
486
SYM_FUNC_END(crc_t10dif_pmull_p8)
377
487
378
488
.align 5
···
398
478
//
399
479
SYM_FUNC_START(crc_t10dif_pmull_p64)
400
480
crc_t10dif_pmull p64
481
+
482
+
// Reduce the 128-bit value M(x), stored in v7, to the final 16-bit CRC.
483
+
484
+
movi v2.16b, #0 // init zero register
485
+
486
+
// Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
487
+
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
488
+
489
+
// Fold the high 64 bits into the low 64 bits, while also multiplying by
490
+
// x^64. This produces a 128-bit value congruent to x^64 * M(x) and
491
+
// whose low 48 bits are 0.
492
+
ext v0.16b, v2.16b, v7.16b, #8
493
+
pmull2 v7.1q, v7.2d, fold_consts.2d // high bits * x^48 * (x^80 mod G(x))
494
+
eor v0.16b, v0.16b, v7.16b // + low bits * x^64
495
+
496
+
// Fold the high 32 bits into the low 96 bits. This produces a 96-bit
497
+
// value congruent to x^64 * M(x) and whose low 48 bits are 0.
498
+
ext v1.16b, v0.16b, v2.16b, #12 // extract high 32 bits
499
+
mov v0.s[3], v2.s[0] // zero high 32 bits
500
+
pmull v1.1q, v1.1d, fold_consts.1d // high 32 bits * x^48 * (x^48 mod G(x))
501
+
eor v0.16b, v0.16b, v1.16b // + low bits
502
+
503
+
// Load G(x) and floor(x^48 / G(x)).
504
+
ld1 {fold_consts.2d}, [fold_consts_ptr]
505
+
506
+
// Use Barrett reduction to compute the final CRC value.
507
+
pmull2 v1.1q, v0.2d, fold_consts.2d // high 32 bits * floor(x^48 / G(x))
508
+
ushr v1.2d, v1.2d, #32 // /= x^32
509
+
pmull v1.1q, v1.1d, fold_consts.1d // *= G(x)
510
+
ushr v0.2d, v0.2d, #48
511
+
eor v0.16b, v0.16b, v1.16b // + low 16 nonzero bits
512
+
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of v0.
513
+
514
+
umov w0, v0.h[0]
515
+
ret
401
516
SYM_FUNC_END(crc_t10dif_pmull_p64)
402
517
403
518
.section ".rodata", "a"
+16
-28
arch/arm64/crypto/crct10dif-ce-glue.c
+16
-28
arch/arm64/crypto/crct10dif-ce-glue.c
···
20
20
21
21
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
22
22
23
-
asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len);
23
+
asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len,
24
+
u8 out[16]);
24
25
asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
25
26
26
27
static int crct10dif_init(struct shash_desc *desc)
···
35
34
static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data,
36
35
unsigned int length)
37
36
{
38
-
u16 *crc = shash_desc_ctx(desc);
37
+
u16 *crcp = shash_desc_ctx(desc);
38
+
u16 crc = *crcp;
39
+
u8 buf[16];
39
40
40
-
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
41
-
do {
42
-
unsigned int chunk = length;
41
+
if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
42
+
kernel_neon_begin();
43
+
crc_t10dif_pmull_p8(crc, data, length, buf);
44
+
kernel_neon_end();
43
45
44
-
if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE)
45
-
chunk = SZ_4K;
46
-
47
-
kernel_neon_begin();
48
-
*crc = crc_t10dif_pmull_p8(*crc, data, chunk);
49
-
kernel_neon_end();
50
-
data += chunk;
51
-
length -= chunk;
52
-
} while (length);
53
-
} else {
54
-
*crc = crc_t10dif_generic(*crc, data, length);
46
+
crc = 0;
47
+
data = buf;
48
+
length = sizeof(buf);
55
49
}
56
50
51
+
*crcp = crc_t10dif_generic(crc, data, length);
57
52
return 0;
58
53
}
59
54
···
59
62
u16 *crc = shash_desc_ctx(desc);
60
63
61
64
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
62
-
do {
63
-
unsigned int chunk = length;
64
-
65
-
if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE)
66
-
chunk = SZ_4K;
67
-
68
-
kernel_neon_begin();
69
-
*crc = crc_t10dif_pmull_p64(*crc, data, chunk);
70
-
kernel_neon_end();
71
-
data += chunk;
72
-
length -= chunk;
73
-
} while (length);
65
+
kernel_neon_begin();
66
+
*crc = crc_t10dif_pmull_p64(*crc, data, length);
67
+
kernel_neon_end();
74
68
} else {
75
69
*crc = crc_t10dif_generic(*crc, data, length);
76
70
}
+1
-1
arch/powerpc/crypto/Kconfig
+1
-1
arch/powerpc/crypto/Kconfig
···
107
107
108
108
config CRYPTO_AES_GCM_P10
109
109
tristate "Stitched AES/GCM acceleration support on P10 or later CPU (PPC)"
110
-
depends on BROKEN
111
110
depends on PPC64 && CPU_LITTLE_ENDIAN && VSX
112
111
select CRYPTO_LIB_AES
113
112
select CRYPTO_ALGAPI
114
113
select CRYPTO_AEAD
115
114
select CRYPTO_SKCIPHER
115
+
select CRYPTO_SIMD
116
116
help
117
117
AEAD cipher: AES cipher algorithms (FIPS-197)
118
118
GCM (Galois/Counter Mode) authenticated encryption mode (NIST SP800-38D)
+118
-23
arch/powerpc/crypto/aes-gcm-p10-glue.c
+118
-23
arch/powerpc/crypto/aes-gcm-p10-glue.c
···
8
8
#include <linux/unaligned.h>
9
9
#include <asm/simd.h>
10
10
#include <asm/switch_to.h>
11
+
#include <crypto/gcm.h>
11
12
#include <crypto/aes.h>
12
13
#include <crypto/algapi.h>
13
14
#include <crypto/b128ops.h>
···
25
24
26
25
#define PPC_ALIGN 16
27
26
#define GCM_IV_SIZE 12
27
+
#define RFC4106_NONCE_SIZE 4
28
28
29
29
MODULE_DESCRIPTION("PPC64le AES-GCM with Stitched implementation");
30
30
MODULE_AUTHOR("Danny Tsen <dtsen@linux.ibm.com");
···
33
31
MODULE_ALIAS_CRYPTO("aes");
34
32
35
33
asmlinkage int aes_p10_set_encrypt_key(const u8 *userKey, const int bits,
36
-
void *key);
34
+
void *key);
37
35
asmlinkage void aes_p10_encrypt(const u8 *in, u8 *out, const void *key);
38
36
asmlinkage void aes_p10_gcm_encrypt(u8 *in, u8 *out, size_t len,
39
37
void *rkey, u8 *iv, void *Xi);
···
41
39
void *rkey, u8 *iv, void *Xi);
42
40
asmlinkage void gcm_init_htable(unsigned char htable[], unsigned char Xi[]);
43
41
asmlinkage void gcm_ghash_p10(unsigned char *Xi, unsigned char *Htable,
44
-
unsigned char *aad, unsigned int alen);
42
+
unsigned char *aad, unsigned int alen);
43
+
asmlinkage void gcm_update(u8 *iv, void *Xi);
45
44
46
45
struct aes_key {
47
46
u8 key[AES_MAX_KEYLENGTH];
···
55
52
u8 aad_hash[16];
56
53
u64 aadLen;
57
54
u64 Plen; /* offset 56 - used in aes_p10_gcm_{en/de}crypt */
55
+
u8 pblock[16];
58
56
};
59
57
struct Hash_ctx {
60
58
u8 H[16]; /* subkey */
···
64
60
65
61
struct p10_aes_gcm_ctx {
66
62
struct aes_key enc_key;
63
+
u8 nonce[RFC4106_NONCE_SIZE];
67
64
};
68
65
69
66
static void vsx_begin(void)
70
67
{
71
68
preempt_disable();
69
+
pagefault_disable();
72
70
enable_kernel_vsx();
73
71
}
74
72
75
73
static void vsx_end(void)
76
74
{
77
75
disable_kernel_vsx();
76
+
pagefault_enable();
78
77
preempt_enable();
79
78
}
80
79
···
192
185
}
193
186
194
187
static int p10_aes_gcm_setkey(struct crypto_aead *aead, const u8 *key,
195
-
unsigned int keylen)
188
+
unsigned int keylen)
196
189
{
197
190
struct crypto_tfm *tfm = crypto_aead_tfm(aead);
198
191
struct p10_aes_gcm_ctx *ctx = crypto_tfm_ctx(tfm);
···
205
198
return ret ? -EINVAL : 0;
206
199
}
207
200
208
-
static int p10_aes_gcm_crypt(struct aead_request *req, int enc)
201
+
static int p10_aes_gcm_crypt(struct aead_request *req, u8 *riv,
202
+
int assoclen, int enc)
209
203
{
210
204
struct crypto_tfm *tfm = req->base.tfm;
211
205
struct p10_aes_gcm_ctx *ctx = crypto_tfm_ctx(tfm);
···
218
210
struct skcipher_walk walk;
219
211
u8 *assocmem = NULL;
220
212
u8 *assoc;
221
-
unsigned int assoclen = req->assoclen;
222
213
unsigned int cryptlen = req->cryptlen;
223
214
unsigned char ivbuf[AES_BLOCK_SIZE+PPC_ALIGN];
224
215
unsigned char *iv = PTR_ALIGN((void *)ivbuf, PPC_ALIGN);
···
225
218
unsigned long auth_tag_len = crypto_aead_authsize(__crypto_aead_cast(tfm));
226
219
u8 otag[16];
227
220
int total_processed = 0;
221
+
int nbytes;
228
222
229
223
memset(databuf, 0, sizeof(databuf));
230
224
memset(hashbuf, 0, sizeof(hashbuf));
231
225
memset(ivbuf, 0, sizeof(ivbuf));
232
-
memcpy(iv, req->iv, GCM_IV_SIZE);
226
+
memcpy(iv, riv, GCM_IV_SIZE);
233
227
234
228
/* Linearize assoc, if not already linear */
235
229
if (req->src->length >= assoclen && req->src->length) {
···
265
257
if (ret)
266
258
return ret;
267
259
268
-
while (walk.nbytes > 0 && ret == 0) {
260
+
while ((nbytes = walk.nbytes) > 0 && ret == 0) {
261
+
u8 *src = walk.src.virt.addr;
262
+
u8 *dst = walk.dst.virt.addr;
263
+
u8 buf[AES_BLOCK_SIZE];
264
+
265
+
if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE))
266
+
src = dst = memcpy(buf, src, nbytes);
269
267
270
268
vsx_begin();
271
269
if (enc)
272
-
aes_p10_gcm_encrypt(walk.src.virt.addr,
273
-
walk.dst.virt.addr,
274
-
walk.nbytes,
270
+
aes_p10_gcm_encrypt(src, dst, nbytes,
275
271
&ctx->enc_key, gctx->iv, hash->Htable);
276
272
else
277
-
aes_p10_gcm_decrypt(walk.src.virt.addr,
278
-
walk.dst.virt.addr,
279
-
walk.nbytes,
273
+
aes_p10_gcm_decrypt(src, dst, nbytes,
280
274
&ctx->enc_key, gctx->iv, hash->Htable);
275
+
276
+
if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE))
277
+
memcpy(walk.dst.virt.addr, buf, nbytes);
278
+
281
279
vsx_end();
282
280
283
281
total_processed += walk.nbytes;
···
295
281
296
282
/* Finalize hash */
297
283
vsx_begin();
284
+
gcm_update(gctx->iv, hash->Htable);
298
285
finish_tag(gctx, hash, total_processed);
299
286
vsx_end();
300
287
···
317
302
return 0;
318
303
}
319
304
305
+
static int rfc4106_setkey(struct crypto_aead *tfm, const u8 *inkey,
306
+
unsigned int keylen)
307
+
{
308
+
struct p10_aes_gcm_ctx *ctx = crypto_aead_ctx(tfm);
309
+
int err;
310
+
311
+
keylen -= RFC4106_NONCE_SIZE;
312
+
err = p10_aes_gcm_setkey(tfm, inkey, keylen);
313
+
if (err)
314
+
return err;
315
+
316
+
memcpy(ctx->nonce, inkey + keylen, RFC4106_NONCE_SIZE);
317
+
return 0;
318
+
}
319
+
320
+
static int rfc4106_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
321
+
{
322
+
return crypto_rfc4106_check_authsize(authsize);
323
+
}
324
+
325
+
static int rfc4106_encrypt(struct aead_request *req)
326
+
{
327
+
struct crypto_aead *aead = crypto_aead_reqtfm(req);
328
+
struct p10_aes_gcm_ctx *ctx = crypto_aead_ctx(aead);
329
+
u8 iv[AES_BLOCK_SIZE];
330
+
331
+
memcpy(iv, ctx->nonce, RFC4106_NONCE_SIZE);
332
+
memcpy(iv + RFC4106_NONCE_SIZE, req->iv, GCM_RFC4106_IV_SIZE);
333
+
334
+
return crypto_ipsec_check_assoclen(req->assoclen) ?:
335
+
p10_aes_gcm_crypt(req, iv, req->assoclen - GCM_RFC4106_IV_SIZE, 1);
336
+
}
337
+
338
+
static int rfc4106_decrypt(struct aead_request *req)
339
+
{
340
+
struct crypto_aead *aead = crypto_aead_reqtfm(req);
341
+
struct p10_aes_gcm_ctx *ctx = crypto_aead_ctx(aead);
342
+
u8 iv[AES_BLOCK_SIZE];
343
+
344
+
memcpy(iv, ctx->nonce, RFC4106_NONCE_SIZE);
345
+
memcpy(iv + RFC4106_NONCE_SIZE, req->iv, GCM_RFC4106_IV_SIZE);
346
+
347
+
return crypto_ipsec_check_assoclen(req->assoclen) ?:
348
+
p10_aes_gcm_crypt(req, iv, req->assoclen - GCM_RFC4106_IV_SIZE, 0);
349
+
}
350
+
320
351
static int p10_aes_gcm_encrypt(struct aead_request *req)
321
352
{
322
-
return p10_aes_gcm_crypt(req, 1);
353
+
return p10_aes_gcm_crypt(req, req->iv, req->assoclen, 1);
323
354
}
324
355
325
356
static int p10_aes_gcm_decrypt(struct aead_request *req)
326
357
{
327
-
return p10_aes_gcm_crypt(req, 0);
358
+
return p10_aes_gcm_crypt(req, req->iv, req->assoclen, 0);
328
359
}
329
360
330
-
static struct aead_alg gcm_aes_alg = {
361
+
static struct aead_alg gcm_aes_algs[] = {{
331
362
.ivsize = GCM_IV_SIZE,
332
363
.maxauthsize = 16,
333
364
···
382
321
.encrypt = p10_aes_gcm_encrypt,
383
322
.decrypt = p10_aes_gcm_decrypt,
384
323
385
-
.base.cra_name = "gcm(aes)",
386
-
.base.cra_driver_name = "aes_gcm_p10",
324
+
.base.cra_name = "__gcm(aes)",
325
+
.base.cra_driver_name = "__aes_gcm_p10",
387
326
.base.cra_priority = 2100,
388
327
.base.cra_blocksize = 1,
389
-
.base.cra_ctxsize = sizeof(struct p10_aes_gcm_ctx),
328
+
.base.cra_ctxsize = sizeof(struct p10_aes_gcm_ctx)+
329
+
4 * sizeof(u64[2]),
390
330
.base.cra_module = THIS_MODULE,
391
-
};
331
+
.base.cra_flags = CRYPTO_ALG_INTERNAL,
332
+
}, {
333
+
.ivsize = GCM_RFC4106_IV_SIZE,
334
+
.maxauthsize = 16,
335
+
.setkey = rfc4106_setkey,
336
+
.setauthsize = rfc4106_setauthsize,
337
+
.encrypt = rfc4106_encrypt,
338
+
.decrypt = rfc4106_decrypt,
339
+
340
+
.base.cra_name = "__rfc4106(gcm(aes))",
341
+
.base.cra_driver_name = "__rfc4106_aes_gcm_p10",
342
+
.base.cra_priority = 2100,
343
+
.base.cra_blocksize = 1,
344
+
.base.cra_ctxsize = sizeof(struct p10_aes_gcm_ctx) +
345
+
4 * sizeof(u64[2]),
346
+
.base.cra_module = THIS_MODULE,
347
+
.base.cra_flags = CRYPTO_ALG_INTERNAL,
348
+
}};
349
+
350
+
static struct simd_aead_alg *p10_simd_aeads[ARRAY_SIZE(gcm_aes_algs)];
392
351
393
352
static int __init p10_init(void)
394
353
{
395
-
return crypto_register_aead(&gcm_aes_alg);
354
+
int ret;
355
+
356
+
if (!cpu_has_feature(CPU_FTR_ARCH_31))
357
+
return 0;
358
+
359
+
ret = simd_register_aeads_compat(gcm_aes_algs,
360
+
ARRAY_SIZE(gcm_aes_algs),
361
+
p10_simd_aeads);
362
+
if (ret) {
363
+
simd_unregister_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs),
364
+
p10_simd_aeads);
365
+
return ret;
366
+
}
367
+
return 0;
396
368
}
397
369
398
370
static void __exit p10_exit(void)
399
371
{
400
-
crypto_unregister_aead(&gcm_aes_alg);
372
+
simd_unregister_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs),
373
+
p10_simd_aeads);
401
374
}
402
375
403
-
module_cpu_feature_match(PPC_MODULE_FEATURE_P10, p10_init);
376
+
module_init(p10_init);
404
377
module_exit(p10_exit);
+1098
-1383
arch/powerpc/crypto/aes-gcm-p10.S
+1098
-1383
arch/powerpc/crypto/aes-gcm-p10.S
···
1
1
/* SPDX-License-Identifier: GPL-2.0-or-later */
2
-
#
3
-
# Accelerated AES-GCM stitched implementation for ppc64le.
4
-
#
5
-
# Copyright 2022- IBM Inc. All rights reserved
6
-
#
7
-
#===================================================================================
8
-
# Written by Danny Tsen <dtsen@linux.ibm.com>
9
-
#
10
-
# GHASH is based on the Karatsuba multiplication method.
11
-
#
12
-
# Xi xor X1
13
-
#
14
-
# X1 * H^4 + X2 * H^3 + x3 * H^2 + X4 * H =
15
-
# (X1.h * H4.h + xX.l * H4.l + X1 * H4) +
16
-
# (X2.h * H3.h + X2.l * H3.l + X2 * H3) +
17
-
# (X3.h * H2.h + X3.l * H2.l + X3 * H2) +
18
-
# (X4.h * H.h + X4.l * H.l + X4 * H)
19
-
#
20
-
# Xi = v0
21
-
# H Poly = v2
22
-
# Hash keys = v3 - v14
23
-
# ( H.l, H, H.h)
24
-
# ( H^2.l, H^2, H^2.h)
25
-
# ( H^3.l, H^3, H^3.h)
26
-
# ( H^4.l, H^4, H^4.h)
27
-
#
28
-
# v30 is IV
29
-
# v31 - counter 1
30
-
#
31
-
# AES used,
32
-
# vs0 - vs14 for round keys
33
-
# v15, v16, v17, v18, v19, v20, v21, v22 for 8 blocks (encrypted)
34
-
#
35
-
# This implementation uses stitched AES-GCM approach to improve overall performance.
36
-
# AES is implemented with 8x blocks and GHASH is using 2 4x blocks.
37
-
#
38
-
# ===================================================================================
39
-
#
2
+
#
3
+
# Accelerated AES-GCM stitched implementation for ppc64le.
4
+
#
5
+
# Copyright 2024- IBM Inc.
6
+
#
7
+
#===================================================================================
8
+
# Written by Danny Tsen <dtsen@us.ibm.com>
9
+
#
10
+
# GHASH is based on the Karatsuba multiplication method.
11
+
#
12
+
# Xi xor X1
13
+
#
14
+
# X1 * H^4 + X2 * H^3 + x3 * H^2 + X4 * H =
15
+
# (X1.h * H4.h + xX.l * H4.l + X1 * H4) +
16
+
# (X2.h * H3.h + X2.l * H3.l + X2 * H3) +
17
+
# (X3.h * H2.h + X3.l * H2.l + X3 * H2) +
18
+
# (X4.h * H.h + X4.l * H.l + X4 * H)
19
+
#
20
+
# Xi = v0
21
+
# H Poly = v2
22
+
# Hash keys = v3 - v14
23
+
# ( H.l, H, H.h)
24
+
# ( H^2.l, H^2, H^2.h)
25
+
# ( H^3.l, H^3, H^3.h)
26
+
# ( H^4.l, H^4, H^4.h)
27
+
#
28
+
# v30 is IV
29
+
# v31 - counter 1
30
+
#
31
+
# AES used,
32
+
# vs0 - round key 0
33
+
# v15, v16, v17, v18, v19, v20, v21, v22 for 8 blocks (encrypted)
34
+
#
35
+
# This implementation uses stitched AES-GCM approach to improve overall performance.
36
+
# AES is implemented with 8x blocks and GHASH is using 2 4x blocks.
37
+
#
38
+
# ===================================================================================
39
+
#
40
40
41
41
#include <asm/ppc_asm.h>
42
42
#include <linux/linkage.h>
···
44
44
.machine "any"
45
45
.text
46
46
47
-
# 4x loops
48
-
# v15 - v18 - input states
49
-
# vs1 - vs9 - round keys
50
-
#
51
-
.macro Loop_aes_middle4x
52
-
xxlor 19+32, 1, 1
53
-
xxlor 20+32, 2, 2
54
-
xxlor 21+32, 3, 3
55
-
xxlor 22+32, 4, 4
56
-
57
-
vcipher 15, 15, 19
58
-
vcipher 16, 16, 19
59
-
vcipher 17, 17, 19
60
-
vcipher 18, 18, 19
61
-
62
-
vcipher 15, 15, 20
63
-
vcipher 16, 16, 20
64
-
vcipher 17, 17, 20
65
-
vcipher 18, 18, 20
66
-
67
-
vcipher 15, 15, 21
68
-
vcipher 16, 16, 21
69
-
vcipher 17, 17, 21
70
-
vcipher 18, 18, 21
71
-
72
-
vcipher 15, 15, 22
73
-
vcipher 16, 16, 22
74
-
vcipher 17, 17, 22
75
-
vcipher 18, 18, 22
76
-
77
-
xxlor 19+32, 5, 5
78
-
xxlor 20+32, 6, 6
79
-
xxlor 21+32, 7, 7
80
-
xxlor 22+32, 8, 8
81
-
82
-
vcipher 15, 15, 19
83
-
vcipher 16, 16, 19
84
-
vcipher 17, 17, 19
85
-
vcipher 18, 18, 19
86
-
87
-
vcipher 15, 15, 20
88
-
vcipher 16, 16, 20
89
-
vcipher 17, 17, 20
90
-
vcipher 18, 18, 20
91
-
92
-
vcipher 15, 15, 21
93
-
vcipher 16, 16, 21
94
-
vcipher 17, 17, 21
95
-
vcipher 18, 18, 21
96
-
97
-
vcipher 15, 15, 22
98
-
vcipher 16, 16, 22
99
-
vcipher 17, 17, 22
100
-
vcipher 18, 18, 22
101
-
102
-
xxlor 23+32, 9, 9
103
-
vcipher 15, 15, 23
104
-
vcipher 16, 16, 23
105
-
vcipher 17, 17, 23
106
-
vcipher 18, 18, 23
47
+
.macro SAVE_GPR GPR OFFSET FRAME
48
+
std \GPR,\OFFSET(\FRAME)
107
49
.endm
108
50
109
-
# 8x loops
110
-
# v15 - v22 - input states
111
-
# vs1 - vs9 - round keys
112
-
#
113
-
.macro Loop_aes_middle8x
114
-
xxlor 23+32, 1, 1
115
-
xxlor 24+32, 2, 2
116
-
xxlor 25+32, 3, 3
117
-
xxlor 26+32, 4, 4
118
-
119
-
vcipher 15, 15, 23
120
-
vcipher 16, 16, 23
121
-
vcipher 17, 17, 23
122
-
vcipher 18, 18, 23
123
-
vcipher 19, 19, 23
124
-
vcipher 20, 20, 23
125
-
vcipher 21, 21, 23
126
-
vcipher 22, 22, 23
127
-
128
-
vcipher 15, 15, 24
129
-
vcipher 16, 16, 24
130
-
vcipher 17, 17, 24
131
-
vcipher 18, 18, 24
132
-
vcipher 19, 19, 24
133
-
vcipher 20, 20, 24
134
-
vcipher 21, 21, 24
135
-
vcipher 22, 22, 24
136
-
137
-
vcipher 15, 15, 25
138
-
vcipher 16, 16, 25
139
-
vcipher 17, 17, 25
140
-
vcipher 18, 18, 25
141
-
vcipher 19, 19, 25
142
-
vcipher 20, 20, 25
143
-
vcipher 21, 21, 25
144
-
vcipher 22, 22, 25
145
-
146
-
vcipher 15, 15, 26
147
-
vcipher 16, 16, 26
148
-
vcipher 17, 17, 26
149
-
vcipher 18, 18, 26
150
-
vcipher 19, 19, 26
151
-
vcipher 20, 20, 26
152
-
vcipher 21, 21, 26
153
-
vcipher 22, 22, 26
154
-
155
-
xxlor 23+32, 5, 5
156
-
xxlor 24+32, 6, 6
157
-
xxlor 25+32, 7, 7
158
-
xxlor 26+32, 8, 8
159
-
160
-
vcipher 15, 15, 23
161
-
vcipher 16, 16, 23
162
-
vcipher 17, 17, 23
163
-
vcipher 18, 18, 23
164
-
vcipher 19, 19, 23
165
-
vcipher 20, 20, 23
166
-
vcipher 21, 21, 23
167
-
vcipher 22, 22, 23
168
-
169
-
vcipher 15, 15, 24
170
-
vcipher 16, 16, 24
171
-
vcipher 17, 17, 24
172
-
vcipher 18, 18, 24
173
-
vcipher 19, 19, 24
174
-
vcipher 20, 20, 24
175
-
vcipher 21, 21, 24
176
-
vcipher 22, 22, 24
177
-
178
-
vcipher 15, 15, 25
179
-
vcipher 16, 16, 25
180
-
vcipher 17, 17, 25
181
-
vcipher 18, 18, 25
182
-
vcipher 19, 19, 25
183
-
vcipher 20, 20, 25
184
-
vcipher 21, 21, 25
185
-
vcipher 22, 22, 25
186
-
187
-
vcipher 15, 15, 26
188
-
vcipher 16, 16, 26
189
-
vcipher 17, 17, 26
190
-
vcipher 18, 18, 26
191
-
vcipher 19, 19, 26
192
-
vcipher 20, 20, 26
193
-
vcipher 21, 21, 26
194
-
vcipher 22, 22, 26
195
-
196
-
xxlor 23+32, 9, 9
197
-
vcipher 15, 15, 23
198
-
vcipher 16, 16, 23
199
-
vcipher 17, 17, 23
200
-
vcipher 18, 18, 23
201
-
vcipher 19, 19, 23
202
-
vcipher 20, 20, 23
203
-
vcipher 21, 21, 23
204
-
vcipher 22, 22, 23
51
+
.macro SAVE_VRS VRS OFFSET FRAME
52
+
stxv \VRS+32, \OFFSET(\FRAME)
205
53
.endm
206
54
207
-
.macro Loop_aes_middle_1x
208
-
xxlor 19+32, 1, 1
209
-
xxlor 20+32, 2, 2
210
-
xxlor 21+32, 3, 3
211
-
xxlor 22+32, 4, 4
212
-
213
-
vcipher 15, 15, 19
214
-
vcipher 15, 15, 20
215
-
vcipher 15, 15, 21
216
-
vcipher 15, 15, 22
217
-
218
-
xxlor 19+32, 5, 5
219
-
xxlor 20+32, 6, 6
220
-
xxlor 21+32, 7, 7
221
-
xxlor 22+32, 8, 8
222
-
223
-
vcipher 15, 15, 19
224
-
vcipher 15, 15, 20
225
-
vcipher 15, 15, 21
226
-
vcipher 15, 15, 22
227
-
228
-
xxlor 19+32, 9, 9
229
-
vcipher 15, 15, 19
55
+
.macro RESTORE_GPR GPR OFFSET FRAME
56
+
ld \GPR,\OFFSET(\FRAME)
230
57
.endm
231
58
232
-
#
233
-
# Compute 4x hash values based on Karatsuba method.
234
-
#
235
-
.macro ppc_aes_gcm_ghash
236
-
vxor 15, 15, 0
237
-
238
-
vpmsumd 23, 12, 15 # H4.L * X.L
239
-
vpmsumd 24, 9, 16
240
-
vpmsumd 25, 6, 17
241
-
vpmsumd 26, 3, 18
242
-
243
-
vxor 23, 23, 24
244
-
vxor 23, 23, 25
245
-
vxor 23, 23, 26 # L
246
-
247
-
vpmsumd 24, 13, 15 # H4.L * X.H + H4.H * X.L
248
-
vpmsumd 25, 10, 16 # H3.L * X1.H + H3.H * X1.L
249
-
vpmsumd 26, 7, 17
250
-
vpmsumd 27, 4, 18
251
-
252
-
vxor 24, 24, 25
253
-
vxor 24, 24, 26
254
-
vxor 24, 24, 27 # M
255
-
256
-
# sum hash and reduction with H Poly
257
-
vpmsumd 28, 23, 2 # reduction
258
-
259
-
vxor 29, 29, 29
260
-
vsldoi 26, 24, 29, 8 # mL
261
-
vsldoi 29, 29, 24, 8 # mH
262
-
vxor 23, 23, 26 # mL + L
263
-
264
-
vsldoi 23, 23, 23, 8 # swap
265
-
vxor 23, 23, 28
266
-
267
-
vpmsumd 24, 14, 15 # H4.H * X.H
268
-
vpmsumd 25, 11, 16
269
-
vpmsumd 26, 8, 17
270
-
vpmsumd 27, 5, 18
271
-
272
-
vxor 24, 24, 25
273
-
vxor 24, 24, 26
274
-
vxor 24, 24, 27
275
-
276
-
vxor 24, 24, 29
277
-
278
-
# sum hash and reduction with H Poly
279
-
vsldoi 27, 23, 23, 8 # swap
280
-
vpmsumd 23, 23, 2
281
-
vxor 27, 27, 24
282
-
vxor 23, 23, 27
283
-
284
-
xxlor 32, 23+32, 23+32 # update hash
285
-
286
-
.endm
287
-
288
-
#
289
-
# Combine two 4x ghash
290
-
# v15 - v22 - input blocks
291
-
#
292
-
.macro ppc_aes_gcm_ghash2_4x
293
-
# first 4x hash
294
-
vxor 15, 15, 0 # Xi + X
295
-
296
-
vpmsumd 23, 12, 15 # H4.L * X.L
297
-
vpmsumd 24, 9, 16
298
-
vpmsumd 25, 6, 17
299
-
vpmsumd 26, 3, 18
300
-
301
-
vxor 23, 23, 24
302
-
vxor 23, 23, 25
303
-
vxor 23, 23, 26 # L
304
-
305
-
vpmsumd 24, 13, 15 # H4.L * X.H + H4.H * X.L
306
-
vpmsumd 25, 10, 16 # H3.L * X1.H + H3.H * X1.L
307
-
vpmsumd 26, 7, 17
308
-
vpmsumd 27, 4, 18
309
-
310
-
vxor 24, 24, 25
311
-
vxor 24, 24, 26
312
-
313
-
# sum hash and reduction with H Poly
314
-
vpmsumd 28, 23, 2 # reduction
315
-
316
-
vxor 29, 29, 29
317
-
318
-
vxor 24, 24, 27 # M
319
-
vsldoi 26, 24, 29, 8 # mL
320
-
vsldoi 29, 29, 24, 8 # mH
321
-
vxor 23, 23, 26 # mL + L
322
-
323
-
vsldoi 23, 23, 23, 8 # swap
324
-
vxor 23, 23, 28
325
-
326
-
vpmsumd 24, 14, 15 # H4.H * X.H
327
-
vpmsumd 25, 11, 16
328
-
vpmsumd 26, 8, 17
329
-
vpmsumd 27, 5, 18
330
-
331
-
vxor 24, 24, 25
332
-
vxor 24, 24, 26
333
-
vxor 24, 24, 27 # H
334
-
335
-
vxor 24, 24, 29 # H + mH
336
-
337
-
# sum hash and reduction with H Poly
338
-
vsldoi 27, 23, 23, 8 # swap
339
-
vpmsumd 23, 23, 2
340
-
vxor 27, 27, 24
341
-
vxor 27, 23, 27 # 1st Xi
342
-
343
-
# 2nd 4x hash
344
-
vpmsumd 24, 9, 20
345
-
vpmsumd 25, 6, 21
346
-
vpmsumd 26, 3, 22
347
-
vxor 19, 19, 27 # Xi + X
348
-
vpmsumd 23, 12, 19 # H4.L * X.L
349
-
350
-
vxor 23, 23, 24
351
-
vxor 23, 23, 25
352
-
vxor 23, 23, 26 # L
353
-
354
-
vpmsumd 24, 13, 19 # H4.L * X.H + H4.H * X.L
355
-
vpmsumd 25, 10, 20 # H3.L * X1.H + H3.H * X1.L
356
-
vpmsumd 26, 7, 21
357
-
vpmsumd 27, 4, 22
358
-
359
-
vxor 24, 24, 25
360
-
vxor 24, 24, 26
361
-
362
-
# sum hash and reduction with H Poly
363
-
vpmsumd 28, 23, 2 # reduction
364
-
365
-
vxor 29, 29, 29
366
-
367
-
vxor 24, 24, 27 # M
368
-
vsldoi 26, 24, 29, 8 # mL
369
-
vsldoi 29, 29, 24, 8 # mH
370
-
vxor 23, 23, 26 # mL + L
371
-
372
-
vsldoi 23, 23, 23, 8 # swap
373
-
vxor 23, 23, 28
374
-
375
-
vpmsumd 24, 14, 19 # H4.H * X.H
376
-
vpmsumd 25, 11, 20
377
-
vpmsumd 26, 8, 21
378
-
vpmsumd 27, 5, 22
379
-
380
-
vxor 24, 24, 25
381
-
vxor 24, 24, 26
382
-
vxor 24, 24, 27 # H
383
-
384
-
vxor 24, 24, 29 # H + mH
385
-
386
-
# sum hash and reduction with H Poly
387
-
vsldoi 27, 23, 23, 8 # swap
388
-
vpmsumd 23, 23, 2
389
-
vxor 27, 27, 24
390
-
vxor 23, 23, 27
391
-
392
-
xxlor 32, 23+32, 23+32 # update hash
393
-
394
-
.endm
395
-
396
-
#
397
-
# Compute update single hash
398
-
#
399
-
.macro ppc_update_hash_1x
400
-
vxor 28, 28, 0
401
-
402
-
vxor 19, 19, 19
403
-
404
-
vpmsumd 22, 3, 28 # L
405
-
vpmsumd 23, 4, 28 # M
406
-
vpmsumd 24, 5, 28 # H
407
-
408
-
vpmsumd 27, 22, 2 # reduction
409
-
410
-
vsldoi 25, 23, 19, 8 # mL
411
-
vsldoi 26, 19, 23, 8 # mH
412
-
vxor 22, 22, 25 # LL + LL
413
-
vxor 24, 24, 26 # HH + HH
414
-
415
-
vsldoi 22, 22, 22, 8 # swap
416
-
vxor 22, 22, 27
417
-
418
-
vsldoi 20, 22, 22, 8 # swap
419
-
vpmsumd 22, 22, 2 # reduction
420
-
vxor 20, 20, 24
421
-
vxor 22, 22, 20
422
-
423
-
vmr 0, 22 # update hash
424
-
59
+
.macro RESTORE_VRS VRS OFFSET FRAME
60
+
lxv \VRS+32, \OFFSET(\FRAME)
425
61
.endm
426
62
427
63
.macro SAVE_REGS
428
-
stdu 1,-640(1)
429
64
mflr 0
65
+
std 0, 16(1)
66
+
stdu 1,-512(1)
430
67
431
-
std 14,112(1)
432
-
std 15,120(1)
433
-
std 16,128(1)
434
-
std 17,136(1)
435
-
std 18,144(1)
436
-
std 19,152(1)
437
-
std 20,160(1)
438
-
std 21,168(1)
439
-
li 9, 256
440
-
stvx 20, 9, 1
441
-
addi 9, 9, 16
442
-
stvx 21, 9, 1
443
-
addi 9, 9, 16
444
-
stvx 22, 9, 1
445
-
addi 9, 9, 16
446
-
stvx 23, 9, 1
447
-
addi 9, 9, 16
448
-
stvx 24, 9, 1
449
-
addi 9, 9, 16
450
-
stvx 25, 9, 1
451
-
addi 9, 9, 16
452
-
stvx 26, 9, 1
453
-
addi 9, 9, 16
454
-
stvx 27, 9, 1
455
-
addi 9, 9, 16
456
-
stvx 28, 9, 1
457
-
addi 9, 9, 16
458
-
stvx 29, 9, 1
459
-
addi 9, 9, 16
460
-
stvx 30, 9, 1
461
-
addi 9, 9, 16
462
-
stvx 31, 9, 1
463
-
stxv 14, 464(1)
464
-
stxv 15, 480(1)
465
-
stxv 16, 496(1)
466
-
stxv 17, 512(1)
467
-
stxv 18, 528(1)
468
-
stxv 19, 544(1)
469
-
stxv 20, 560(1)
470
-
stxv 21, 576(1)
471
-
stxv 22, 592(1)
472
-
std 0, 656(1)
473
-
.endm
68
+
SAVE_GPR 14, 112, 1
69
+
SAVE_GPR 15, 120, 1
70
+
SAVE_GPR 16, 128, 1
71
+
SAVE_GPR 17, 136, 1
72
+
SAVE_GPR 18, 144, 1
73
+
SAVE_GPR 19, 152, 1
74
+
SAVE_GPR 20, 160, 1
75
+
SAVE_GPR 21, 168, 1
76
+
SAVE_GPR 22, 176, 1
77
+
SAVE_GPR 23, 184, 1
78
+
SAVE_GPR 24, 192, 1
79
+
80
+
addi 9, 1, 256
81
+
SAVE_VRS 20, 0, 9
82
+
SAVE_VRS 21, 16, 9
83
+
SAVE_VRS 22, 32, 9
84
+
SAVE_VRS 23, 48, 9
85
+
SAVE_VRS 24, 64, 9
86
+
SAVE_VRS 25, 80, 9
87
+
SAVE_VRS 26, 96, 9
88
+
SAVE_VRS 27, 112, 9
89
+
SAVE_VRS 28, 128, 9
90
+
SAVE_VRS 29, 144, 9
91
+
SAVE_VRS 30, 160, 9
92
+
SAVE_VRS 31, 176, 9
93
+
.endm # SAVE_REGS
474
94
475
95
.macro RESTORE_REGS
476
-
lxv 14, 464(1)
477
-
lxv 15, 480(1)
478
-
lxv 16, 496(1)
479
-
lxv 17, 512(1)
480
-
lxv 18, 528(1)
481
-
lxv 19, 544(1)
482
-
lxv 20, 560(1)
483
-
lxv 21, 576(1)
484
-
lxv 22, 592(1)
485
-
li 9, 256
486
-
lvx 20, 9, 1
487
-
addi 9, 9, 16
488
-
lvx 21, 9, 1
489
-
addi 9, 9, 16
490
-
lvx 22, 9, 1
491
-
addi 9, 9, 16
492
-
lvx 23, 9, 1
493
-
addi 9, 9, 16
494
-
lvx 24, 9, 1
495
-
addi 9, 9, 16
496
-
lvx 25, 9, 1
497
-
addi 9, 9, 16
498
-
lvx 26, 9, 1
499
-
addi 9, 9, 16
500
-
lvx 27, 9, 1
501
-
addi 9, 9, 16
502
-
lvx 28, 9, 1
503
-
addi 9, 9, 16
504
-
lvx 29, 9, 1
505
-
addi 9, 9, 16
506
-
lvx 30, 9, 1
507
-
addi 9, 9, 16
508
-
lvx 31, 9, 1
96
+
addi 9, 1, 256
97
+
RESTORE_VRS 20, 0, 9
98
+
RESTORE_VRS 21, 16, 9
99
+
RESTORE_VRS 22, 32, 9
100
+
RESTORE_VRS 23, 48, 9
101
+
RESTORE_VRS 24, 64, 9
102
+
RESTORE_VRS 25, 80, 9
103
+
RESTORE_VRS 26, 96, 9
104
+
RESTORE_VRS 27, 112, 9
105
+
RESTORE_VRS 28, 128, 9
106
+
RESTORE_VRS 29, 144, 9
107
+
RESTORE_VRS 30, 160, 9
108
+
RESTORE_VRS 31, 176, 9
509
109
510
-
ld 0, 656(1)
511
-
ld 14,112(1)
512
-
ld 15,120(1)
513
-
ld 16,128(1)
514
-
ld 17,136(1)
515
-
ld 18,144(1)
516
-
ld 19,152(1)
517
-
ld 20,160(1)
518
-
ld 21,168(1)
110
+
RESTORE_GPR 14, 112, 1
111
+
RESTORE_GPR 15, 120, 1
112
+
RESTORE_GPR 16, 128, 1
113
+
RESTORE_GPR 17, 136, 1
114
+
RESTORE_GPR 18, 144, 1
115
+
RESTORE_GPR 19, 152, 1
116
+
RESTORE_GPR 20, 160, 1
117
+
RESTORE_GPR 21, 168, 1
118
+
RESTORE_GPR 22, 176, 1
119
+
RESTORE_GPR 23, 184, 1
120
+
RESTORE_GPR 24, 192, 1
519
121
520
-
mtlr 0
521
-
addi 1, 1, 640
122
+
addi 1, 1, 512
123
+
ld 0, 16(1)
124
+
mtlr 0
125
+
.endm # RESTORE_REGS
126
+
127
+
# 4x loops
128
+
.macro AES_CIPHER_4x _VCIPHER ST r
129
+
\_VCIPHER \ST, \ST, \r
130
+
\_VCIPHER \ST+1, \ST+1, \r
131
+
\_VCIPHER \ST+2, \ST+2, \r
132
+
\_VCIPHER \ST+3, \ST+3, \r
522
133
.endm
523
134
135
+
# 8x loops
136
+
.macro AES_CIPHER_8x _VCIPHER ST r
137
+
\_VCIPHER \ST, \ST, \r
138
+
\_VCIPHER \ST+1, \ST+1, \r
139
+
\_VCIPHER \ST+2, \ST+2, \r
140
+
\_VCIPHER \ST+3, \ST+3, \r
141
+
\_VCIPHER \ST+4, \ST+4, \r
142
+
\_VCIPHER \ST+5, \ST+5, \r
143
+
\_VCIPHER \ST+6, \ST+6, \r
144
+
\_VCIPHER \ST+7, \ST+7, \r
145
+
.endm
146
+
147
+
.macro LOOP_8AES_STATE
148
+
xxlor 32+23, 1, 1
149
+
xxlor 32+24, 2, 2
150
+
xxlor 32+25, 3, 3
151
+
xxlor 32+26, 4, 4
152
+
AES_CIPHER_8x vcipher, 15, 23
153
+
AES_CIPHER_8x vcipher, 15, 24
154
+
AES_CIPHER_8x vcipher, 15, 25
155
+
AES_CIPHER_8x vcipher, 15, 26
156
+
xxlor 32+23, 5, 5
157
+
xxlor 32+24, 6, 6
158
+
xxlor 32+25, 7, 7
159
+
xxlor 32+26, 8, 8
160
+
AES_CIPHER_8x vcipher, 15, 23
161
+
AES_CIPHER_8x vcipher, 15, 24
162
+
AES_CIPHER_8x vcipher, 15, 25
163
+
AES_CIPHER_8x vcipher, 15, 26
164
+
.endm
165
+
166
+
#
167
+
# PPC_GHASH4x(H, S1, S2, S3, S4): Compute 4x hash values based on Karatsuba method.
168
+
# H: returning digest
169
+
# S#: states
170
+
#
171
+
# S1 should xor with the previous digest
172
+
#
173
+
# Xi = v0
174
+
# H Poly = v2
175
+
# Hash keys = v3 - v14
176
+
# Scratch: v23 - v29
177
+
#
178
+
.macro PPC_GHASH4x H S1 S2 S3 S4
179
+
180
+
vpmsumd 23, 12, \S1 # H4.L * X.L
181
+
vpmsumd 24, 9, \S2
182
+
vpmsumd 25, 6, \S3
183
+
vpmsumd 26, 3, \S4
184
+
185
+
vpmsumd 27, 13, \S1 # H4.L * X.H + H4.H * X.L
186
+
vpmsumd 28, 10, \S2 # H3.L * X1.H + H3.H * X1.L
187
+
188
+
vxor 23, 23, 24
189
+
vxor 23, 23, 25
190
+
vxor 23, 23, 26 # L
191
+
192
+
vxor 24, 27, 28
193
+
vpmsumd 25, 7, \S3
194
+
vpmsumd 26, 4, \S4
195
+
196
+
vxor 24, 24, 25
197
+
vxor 24, 24, 26 # M
198
+
199
+
# sum hash and reduction with H Poly
200
+
vpmsumd 28, 23, 2 # reduction
201
+
202
+
vxor 1, 1, 1
203
+
vsldoi 25, 24, 1, 8 # mL
204
+
vsldoi 1, 1, 24, 8 # mH
205
+
vxor 23, 23, 25 # mL + L
206
+
207
+
# This performs swap and xor like,
208
+
# vsldoi 23, 23, 23, 8 # swap
209
+
# vxor 23, 23, 28
210
+
xxlor 32+25, 10, 10
211
+
vpermxor 23, 23, 28, 25
212
+
213
+
vpmsumd 26, 14, \S1 # H4.H * X.H
214
+
vpmsumd 27, 11, \S2
215
+
vpmsumd 28, 8, \S3
216
+
vpmsumd 29, 5, \S4
217
+
218
+
vxor 24, 26, 27
219
+
vxor 24, 24, 28
220
+
vxor 24, 24, 29
221
+
222
+
vxor 24, 24, 1
223
+
224
+
# sum hash and reduction with H Poly
225
+
vsldoi 25, 23, 23, 8 # swap
226
+
vpmsumd 23, 23, 2
227
+
vxor 27, 25, 24
228
+
vxor \H, 23, 27
229
+
.endm
230
+
231
+
#
232
+
# Compute update single ghash
233
+
# scratch: v1, v22..v27
234
+
#
235
+
.macro PPC_GHASH1x H S1
236
+
237
+
vxor 1, 1, 1
238
+
239
+
vpmsumd 22, 3, \S1 # L
240
+
vpmsumd 23, 4, \S1 # M
241
+
vpmsumd 24, 5, \S1 # H
242
+
243
+
vpmsumd 27, 22, 2 # reduction
244
+
245
+
vsldoi 25, 23, 1, 8 # mL
246
+
vsldoi 26, 1, 23, 8 # mH
247
+
vxor 22, 22, 25 # LL + LL
248
+
vxor 24, 24, 26 # HH + HH
249
+
250
+
xxlor 32+25, 10, 10
251
+
vpermxor 22, 22, 27, 25
252
+
253
+
vsldoi 23, 22, 22, 8 # swap
254
+
vpmsumd 22, 22, 2 # reduction
255
+
vxor 23, 23, 24
256
+
vxor \H, 22, 23
257
+
.endm
258
+
259
+
#
260
+
# LOAD_HASH_TABLE
261
+
# Xi = v0
262
+
# H Poly = v2
263
+
# Hash keys = v3 - v14
264
+
#
524
265
.macro LOAD_HASH_TABLE
525
266
# Load Xi
526
267
lxvb16x 32, 0, 8 # load Xi
···
298
557
lxvd2x 14+32, 10, 8 # H^4h
299
558
.endm
300
559
301
-
#
302
-
# aes_p10_gcm_encrypt (const void *inp, void *out, size_t len,
303
-
# const char *rk, unsigned char iv[16], void *Xip);
304
-
#
305
-
# r3 - inp
306
-
# r4 - out
307
-
# r5 - len
308
-
# r6 - AES round keys
309
-
# r7 - iv and other data
310
-
# r8 - Xi, HPoli, hash keys
311
-
#
312
-
# rounds is at offset 240 in rk
313
-
# Xi is at 0 in gcm_table (Xip).
314
-
#
315
-
_GLOBAL(aes_p10_gcm_encrypt)
316
-
.align 5
560
+
################################################################################
561
+
# Compute AES and ghash one block at a time.
562
+
# r23: AES rounds
563
+
# v30: current IV
564
+
# vs0: roundkey 0
565
+
#
566
+
################################################################################
567
+
SYM_FUNC_START_LOCAL(aes_gcm_crypt_1x)
568
+
569
+
cmpdi 5, 16
570
+
bge __More_1x
571
+
blr
572
+
__More_1x:
573
+
li 10, 16
574
+
divdu 12, 5, 10
575
+
576
+
xxlxor 32+15, 32+30, 0
577
+
578
+
# Pre-load 8 AES rounds to scratch vectors.
579
+
xxlor 32+16, 1, 1
580
+
xxlor 32+17, 2, 2
581
+
xxlor 32+18, 3, 3
582
+
xxlor 32+19, 4, 4
583
+
xxlor 32+20, 5, 5
584
+
xxlor 32+21, 6, 6
585
+
xxlor 32+28, 7, 7
586
+
xxlor 32+29, 8, 8
587
+
lwz 23, 240(6) # n rounds
588
+
addi 22, 23, -9 # remaing AES rounds
589
+
590
+
cmpdi 12, 0
591
+
bgt __Loop_1x
592
+
blr
593
+
594
+
__Loop_1x:
595
+
mtctr 22
596
+
addi 10, 6, 144
597
+
vcipher 15, 15, 16
598
+
vcipher 15, 15, 17
599
+
vcipher 15, 15, 18
600
+
vcipher 15, 15, 19
601
+
vcipher 15, 15, 20
602
+
vcipher 15, 15, 21
603
+
vcipher 15, 15, 28
604
+
vcipher 15, 15, 29
605
+
606
+
__Loop_aes_1state:
607
+
lxv 32+1, 0(10)
608
+
vcipher 15, 15, 1
609
+
addi 10, 10, 16
610
+
bdnz __Loop_aes_1state
611
+
lxv 32+1, 0(10) # last round key
612
+
lxvb16x 11, 0, 14 # load input block
613
+
vcipherlast 15, 15, 1
614
+
615
+
xxlxor 32+15, 32+15, 11
616
+
stxvb16x 32+15, 0, 9 # store output
617
+
addi 14, 14, 16
618
+
addi 9, 9, 16
619
+
620
+
cmpdi 24, 0 # decrypt?
621
+
bne __Encrypt_1x
622
+
xxlor 15+32, 11, 11
623
+
__Encrypt_1x:
624
+
vxor 15, 15, 0
625
+
PPC_GHASH1x 0, 15
626
+
627
+
addi 5, 5, -16
628
+
addi 11, 11, 16
629
+
630
+
vadduwm 30, 30, 31 # IV + counter
631
+
xxlxor 32+15, 32+30, 0
632
+
addi 12, 12, -1
633
+
cmpdi 12, 0
634
+
bgt __Loop_1x
635
+
636
+
stxvb16x 32+30, 0, 7 # update IV
637
+
stxvb16x 32+0, 0, 8 # update Xi
638
+
blr
639
+
SYM_FUNC_END(aes_gcm_crypt_1x)
640
+
641
+
################################################################################
642
+
# Process a normal partial block when we come here.
643
+
# Compute partial mask, Load and store partial block to stack.
644
+
# Update partial_len and pblock.
645
+
# pblock is (encrypted ^ AES state) for encrypt
646
+
# and (input ^ AES state) for decrypt.
647
+
#
648
+
################################################################################
649
+
SYM_FUNC_START_LOCAL(__Process_partial)
650
+
651
+
# create partial mask
652
+
vspltisb 16, -1
653
+
li 12, 16
654
+
sub 12, 12, 5
655
+
sldi 12, 12, 3
656
+
mtvsrdd 32+17, 0, 12
657
+
vslo 16, 16, 17 # partial block mask
658
+
659
+
lxvb16x 11, 0, 14 # load partial block
660
+
xxland 11, 11, 32+16
661
+
662
+
# AES crypt partial
663
+
xxlxor 32+15, 32+30, 0
664
+
lwz 23, 240(6) # n rounds
665
+
addi 22, 23, -1 # loop - 1
666
+
mtctr 22
667
+
addi 10, 6, 16
668
+
669
+
__Loop_aes_pstate:
670
+
lxv 32+1, 0(10)
671
+
vcipher 15, 15, 1
672
+
addi 10, 10, 16
673
+
bdnz __Loop_aes_pstate
674
+
lxv 32+1, 0(10) # last round key
675
+
vcipherlast 15, 15, 1
676
+
677
+
xxlxor 32+15, 32+15, 11
678
+
vand 15, 15, 16
679
+
680
+
# AES crypt output v15
681
+
# Write partial
682
+
li 10, 224
683
+
stxvb16x 15+32, 10, 1 # write v15 to stack
684
+
addi 10, 1, 223
685
+
addi 12, 9, -1
686
+
mtctr 5 # partial block len
687
+
__Write_partial:
688
+
lbzu 22, 1(10)
689
+
stbu 22, 1(12)
690
+
bdnz __Write_partial
691
+
692
+
cmpdi 24, 0 # decrypt?
693
+
bne __Encrypt_partial
694
+
xxlor 32+15, 11, 11 # decrypt using the input block
695
+
__Encrypt_partial:
696
+
#vxor 15, 15, 0 # ^ previous hash
697
+
#PPC_GHASH1x 0, 15
698
+
699
+
add 14, 14, 5
700
+
add 9, 9, 5
701
+
std 5, 56(7) # update partial
702
+
sub 11, 11, 5
703
+
li 5, 0 # done last byte
704
+
705
+
#
706
+
# Don't increase IV since this is the last partial.
707
+
# It should get updated in gcm_update if no more data blocks.
708
+
#vadduwm 30, 30, 31 # increase IV
709
+
stxvb16x 32+30, 0, 7 # update IV
710
+
li 10, 64
711
+
stxvb16x 32+0, 0, 8 # Update X1
712
+
stxvb16x 32+15, 10, 7 # Update pblock
713
+
blr
714
+
SYM_FUNC_END(__Process_partial)
715
+
716
+
################################################################################
717
+
# Combine partial blocks and ghash when we come here.
718
+
#
719
+
# The partial block has to be shifted to the right location to encrypt/decrypt
720
+
# and compute ghash if combing the previous partial block is needed.
721
+
# - Compute ghash for a full block. Clear Partial_len and pblock. Update IV.
722
+
# Write Xi.
723
+
# - Don't compute ghash if not full block. gcm_update will take care of it
724
+
# is the last block. Update Partial_len and pblock.
725
+
#
726
+
################################################################################
727
+
SYM_FUNC_START_LOCAL(__Combine_partial)
728
+
729
+
ld 12, 56(7)
730
+
mr 21, 5 # these bytes to be processed
731
+
732
+
li 17, 0
733
+
li 16, 16
734
+
sub 22, 16, 12 # bytes to complete a block
735
+
sub 17, 22, 5 # remaining bytes in a block
736
+
cmpdi 5, 16
737
+
ble __Inp_msg_less16
738
+
li 17, 0
739
+
mr 21, 22
740
+
b __Combine_continue
741
+
__Inp_msg_less16:
742
+
cmpd 22, 5
743
+
bgt __Combine_continue
744
+
li 17, 0
745
+
mr 21, 22 # these bytes to be processed
746
+
747
+
__Combine_continue:
748
+
# load msg and shift to the proper location and mask
749
+
vspltisb 16, -1
750
+
sldi 15, 12, 3
751
+
mtvsrdd 32+17, 0, 15
752
+
vslo 16, 16, 17
753
+
vsro 16, 16, 17
754
+
sldi 15, 17, 3
755
+
mtvsrdd 32+17, 0, 15
756
+
vsro 16, 16, 17
757
+
vslo 16, 16, 17 # mask
758
+
759
+
lxvb16x 32+19, 0, 14 # load partial block
760
+
sldi 15, 12, 3
761
+
mtvsrdd 32+17, 0, 15
762
+
vsro 19, 19, 17 # 0x00..xxxx??..??
763
+
sldi 15, 17, 3
764
+
mtvsrdd 32+17, 0, 15
765
+
vsro 19, 19, 17 # 0x00..xxxx
766
+
vslo 19, 19, 17 # shift back to form 0x00..xxxx00..00
767
+
768
+
# AES crypt partial
769
+
xxlxor 32+15, 32+30, 0
770
+
lwz 23, 240(6) # n rounds
771
+
addi 22, 23, -1 # loop - 1
772
+
mtctr 22
773
+
addi 10, 6, 16
774
+
775
+
__Loop_aes_cpstate:
776
+
lxv 32+1, 0(10)
777
+
vcipher 15, 15, 1
778
+
addi 10, 10, 16
779
+
bdnz __Loop_aes_cpstate
780
+
lxv 32+1, 0(10) # last round key
781
+
vcipherlast 15, 15, 1
782
+
783
+
vxor 15, 15, 19
784
+
vand 15, 15, 16
785
+
786
+
# AES crypt output v15
787
+
# Write partial
788
+
li 10, 224
789
+
stxvb16x 15+32, 10, 1 # write v15 to stack
790
+
addi 10, 1, 223
791
+
add 10, 10, 12 # add offset
792
+
addi 15, 9, -1
793
+
mtctr 21 # partial block len
794
+
__Write_combine_partial:
795
+
lbzu 22, 1(10)
796
+
stbu 22, 1(15)
797
+
bdnz __Write_combine_partial
798
+
799
+
add 14, 14, 21
800
+
add 11, 11, 21
801
+
add 9, 9, 21
802
+
sub 5, 5, 21
803
+
804
+
# Encrypt/Decrypt?
805
+
cmpdi 24, 0 # decrypt?
806
+
bne __Encrypt_combine_partial
807
+
vmr 15, 19 # decrypt using the input block
808
+
809
+
__Encrypt_combine_partial:
810
+
#
811
+
# Update partial flag and combine ghash.
812
+
__Update_partial_ghash:
813
+
li 10, 64
814
+
lxvb16x 32+17, 10, 7 # load previous pblock
815
+
add 12, 12, 21 # combined pprocessed
816
+
vxor 15, 15, 17 # combined pblock
817
+
818
+
cmpdi 12, 16
819
+
beq __Clear_partial_flag
820
+
std 12, 56(7) # update partial len
821
+
stxvb16x 32+15, 10, 7 # Update current pblock
822
+
blr
823
+
824
+
__Clear_partial_flag:
825
+
li 12, 0
826
+
std 12, 56(7)
827
+
# Update IV and ghash here
828
+
vadduwm 30, 30, 31 # increase IV
829
+
stxvb16x 32+30, 0, 7 # update IV
830
+
831
+
# v15 either is either (input blockor encrypted)^(AES state)
832
+
vxor 15, 15, 0
833
+
PPC_GHASH1x 0, 15
834
+
stxvb16x 32+0, 10, 7 # update pblock for debug?
835
+
stxvb16x 32+0, 0, 8 # update Xi
836
+
blr
837
+
SYM_FUNC_END(__Combine_partial)
838
+
839
+
################################################################################
840
+
# gcm_update(iv, Xi) - compute last hash
841
+
#
842
+
################################################################################
843
+
SYM_FUNC_START(gcm_update)
844
+
845
+
ld 10, 56(3)
846
+
cmpdi 10, 0
847
+
beq __no_update
848
+
849
+
lxvb16x 32, 0, 4 # load Xi
850
+
# load Hash - h^4, h^3, h^2, h
851
+
li 10, 32
852
+
lxvd2x 2+32, 10, 4 # H Poli
853
+
li 10, 48
854
+
lxvd2x 3+32, 10, 4 # Hl
855
+
li 10, 64
856
+
lxvd2x 4+32, 10, 4 # H
857
+
li 10, 80
858
+
lxvd2x 5+32, 10, 4 # Hh
859
+
860
+
addis 11, 2, permx@toc@ha
861
+
addi 11, 11, permx@toc@l
862
+
lxv 10, 0(11) # vs10: vpermxor vector
863
+
864
+
li 9, 64
865
+
lxvb16x 32+6, 9, 3 # load pblock
866
+
vxor 6, 6, 0
867
+
868
+
vxor 1, 1, 1
869
+
vpmsumd 12, 3, 6 # L
870
+
vpmsumd 13, 4, 6 # M
871
+
vpmsumd 14, 5, 6 # H
872
+
vpmsumd 17, 12, 2 # reduction
873
+
vsldoi 15, 13, 1, 8 # mL
874
+
vsldoi 16, 1, 13, 8 # mH
875
+
vxor 12, 12, 15 # LL + LL
876
+
vxor 14, 14, 16 # HH + HH
877
+
xxlor 32+15, 10, 10
878
+
vpermxor 12, 12, 17, 15
879
+
vsldoi 13, 12, 12, 8 # swap
880
+
vpmsumd 12, 12, 2 # reduction
881
+
vxor 13, 13, 14
882
+
vxor 7, 12, 13
883
+
884
+
#vxor 0, 0, 0
885
+
#stxvb16x 32+0, 9, 3
886
+
li 10, 0
887
+
std 10, 56(3)
888
+
stxvb16x 32+7, 0, 4
889
+
890
+
__no_update:
891
+
blr
892
+
SYM_FUNC_END(gcm_update)
893
+
894
+
################################################################################
895
+
# aes_p10_gcm_encrypt (const void *inp, void *out, size_t len,
896
+
# const char *rk, unsigned char iv[16], void *Xip);
897
+
#
898
+
# r3 - inp
899
+
# r4 - out
900
+
# r5 - len
901
+
# r6 - AES round keys
902
+
# r7 - iv and other data
903
+
# r8 - Xi, HPoli, hash keys
904
+
#
905
+
# rounds is at offset 240 in rk
906
+
# Xi is at 0 in gcm_table (Xip).
907
+
#
908
+
################################################################################
909
+
SYM_FUNC_START(aes_p10_gcm_encrypt)
910
+
911
+
cmpdi 5, 0
912
+
ble __Invalid_msg_len
317
913
318
914
SAVE_REGS
319
-
320
915
LOAD_HASH_TABLE
321
916
322
917
# initialize ICB: GHASH( IV ), IV - r7
323
918
lxvb16x 30+32, 0, 7 # load IV - v30
324
919
325
-
mr 12, 5 # length
326
-
li 11, 0 # block index
920
+
mr 14, 3
921
+
mr 9, 4
327
922
328
923
# counter 1
329
924
vxor 31, 31, 31
330
925
vspltisb 22, 1
331
926
vsldoi 31, 31, 22,1 # counter 1
332
927
333
-
# load round key to VSR
334
-
lxv 0, 0(6)
335
-
lxv 1, 0x10(6)
336
-
lxv 2, 0x20(6)
337
-
lxv 3, 0x30(6)
338
-
lxv 4, 0x40(6)
339
-
lxv 5, 0x50(6)
340
-
lxv 6, 0x60(6)
341
-
lxv 7, 0x70(6)
342
-
lxv 8, 0x80(6)
343
-
lxv 9, 0x90(6)
344
-
lxv 10, 0xa0(6)
928
+
addis 11, 2, permx@toc@ha
929
+
addi 11, 11, permx@toc@l
930
+
lxv 10, 0(11) # vs10: vpermxor vector
931
+
li 11, 0
932
+
933
+
# load 9 round keys to VSR
934
+
lxv 0, 0(6) # round key 0
935
+
lxv 1, 16(6) # round key 1
936
+
lxv 2, 32(6) # round key 2
937
+
lxv 3, 48(6) # round key 3
938
+
lxv 4, 64(6) # round key 4
939
+
lxv 5, 80(6) # round key 5
940
+
lxv 6, 96(6) # round key 6
941
+
lxv 7, 112(6) # round key 7
942
+
lxv 8, 128(6) # round key 8
345
943
346
944
# load rounds - 10 (128), 12 (192), 14 (256)
347
-
lwz 9,240(6)
945
+
lwz 23, 240(6) # n rounds
946
+
li 24, 1 # encrypt
348
947
948
+
__Process_encrypt:
349
949
#
350
-
# vxor state, state, w # addroundkey
351
-
xxlor 32+29, 0, 0
352
-
vxor 15, 30, 29 # IV + round key - add round key 0
353
-
354
-
cmpdi 9, 10
355
-
beq Loop_aes_gcm_8x
356
-
357
-
# load 2 more round keys (v11, v12)
358
-
lxv 11, 0xb0(6)
359
-
lxv 12, 0xc0(6)
360
-
361
-
cmpdi 9, 12
362
-
beq Loop_aes_gcm_8x
363
-
364
-
# load 2 more round keys (v11, v12, v13, v14)
365
-
lxv 13, 0xd0(6)
366
-
lxv 14, 0xe0(6)
367
-
cmpdi 9, 14
368
-
beq Loop_aes_gcm_8x
369
-
370
-
b aes_gcm_out
371
-
372
-
.align 5
373
-
Loop_aes_gcm_8x:
374
-
mr 14, 3
375
-
mr 9, 4
376
-
950
+
# Process different blocks
377
951
#
378
-
# check partial block
379
-
#
380
-
Continue_partial_check:
381
-
ld 15, 56(7)
382
-
cmpdi 15, 0
383
-
beq Continue
384
-
bgt Final_block
385
-
cmpdi 15, 16
386
-
blt Final_block
952
+
ld 12, 56(7)
953
+
cmpdi 12, 0
954
+
bgt __Do_combine_enc
955
+
cmpdi 5, 128
956
+
blt __Process_more_enc
387
957
388
-
Continue:
389
-
# n blcoks
958
+
#
959
+
# Process 8x AES/GCM blocks
960
+
#
961
+
__Process_8x_enc:
962
+
# 8x blcoks
390
963
li 10, 128
391
-
divdu 10, 12, 10 # n 128 bytes-blocks
392
-
cmpdi 10, 0
393
-
beq Loop_last_block
964
+
divdu 12, 5, 10 # n 128 bytes-blocks
394
965
395
-
vaddudm 30, 30, 31 # IV + counter
396
-
vxor 16, 30, 29
397
-
vaddudm 30, 30, 31
398
-
vxor 17, 30, 29
399
-
vaddudm 30, 30, 31
400
-
vxor 18, 30, 29
401
-
vaddudm 30, 30, 31
402
-
vxor 19, 30, 29
403
-
vaddudm 30, 30, 31
404
-
vxor 20, 30, 29
405
-
vaddudm 30, 30, 31
406
-
vxor 21, 30, 29
407
-
vaddudm 30, 30, 31
408
-
vxor 22, 30, 29
966
+
addi 12, 12, -1 # loop - 1
409
967
410
-
mtctr 10
968
+
vmr 15, 30 # first state: IV
969
+
vadduwm 16, 15, 31 # state + counter
970
+
vadduwm 17, 16, 31
971
+
vadduwm 18, 17, 31
972
+
vadduwm 19, 18, 31
973
+
vadduwm 20, 19, 31
974
+
vadduwm 21, 20, 31
975
+
vadduwm 22, 21, 31
976
+
xxlor 9, 32+22, 32+22 # save last state
977
+
978
+
# vxor state, state, w # addroundkey
979
+
xxlor 32+29, 0, 0
980
+
vxor 15, 15, 29 # IV + round key - add round key 0
981
+
vxor 16, 16, 29
982
+
vxor 17, 17, 29
983
+
vxor 18, 18, 29
984
+
vxor 19, 19, 29
985
+
vxor 20, 20, 29
986
+
vxor 21, 21, 29
987
+
vxor 22, 22, 29
411
988
412
989
li 15, 16
413
990
li 16, 32
···
735
676
li 20, 96
736
677
li 21, 112
737
678
738
-
lwz 10, 240(6)
739
-
740
-
Loop_8x_block:
741
-
742
-
lxvb16x 15, 0, 14 # load block
743
-
lxvb16x 16, 15, 14 # load block
744
-
lxvb16x 17, 16, 14 # load block
745
-
lxvb16x 18, 17, 14 # load block
746
-
lxvb16x 19, 18, 14 # load block
747
-
lxvb16x 20, 19, 14 # load block
748
-
lxvb16x 21, 20, 14 # load block
749
-
lxvb16x 22, 21, 14 # load block
750
-
addi 14, 14, 128
751
-
752
-
Loop_aes_middle8x
753
-
754
-
xxlor 23+32, 10, 10
755
-
756
-
cmpdi 10, 10
757
-
beq Do_next_ghash
758
-
759
-
# 192 bits
760
-
xxlor 24+32, 11, 11
761
-
762
-
vcipher 15, 15, 23
763
-
vcipher 16, 16, 23
764
-
vcipher 17, 17, 23
765
-
vcipher 18, 18, 23
766
-
vcipher 19, 19, 23
767
-
vcipher 20, 20, 23
768
-
vcipher 21, 21, 23
769
-
vcipher 22, 22, 23
770
-
771
-
vcipher 15, 15, 24
772
-
vcipher 16, 16, 24
773
-
vcipher 17, 17, 24
774
-
vcipher 18, 18, 24
775
-
vcipher 19, 19, 24
776
-
vcipher 20, 20, 24
777
-
vcipher 21, 21, 24
778
-
vcipher 22, 22, 24
779
-
780
-
xxlor 23+32, 12, 12
781
-
782
-
cmpdi 10, 12
783
-
beq Do_next_ghash
784
-
785
-
# 256 bits
786
-
xxlor 24+32, 13, 13
787
-
788
-
vcipher 15, 15, 23
789
-
vcipher 16, 16, 23
790
-
vcipher 17, 17, 23
791
-
vcipher 18, 18, 23
792
-
vcipher 19, 19, 23
793
-
vcipher 20, 20, 23
794
-
vcipher 21, 21, 23
795
-
vcipher 22, 22, 23
796
-
797
-
vcipher 15, 15, 24
798
-
vcipher 16, 16, 24
799
-
vcipher 17, 17, 24
800
-
vcipher 18, 18, 24
801
-
vcipher 19, 19, 24
802
-
vcipher 20, 20, 24
803
-
vcipher 21, 21, 24
804
-
vcipher 22, 22, 24
805
-
806
-
xxlor 23+32, 14, 14
807
-
808
-
cmpdi 10, 14
809
-
beq Do_next_ghash
810
-
b aes_gcm_out
811
-
812
-
Do_next_ghash:
813
-
814
679
#
815
-
# last round
816
-
vcipherlast 15, 15, 23
817
-
vcipherlast 16, 16, 23
680
+
# Pre-compute first 8 AES state and leave 1/3/5 more rounds
681
+
# for the loop.
682
+
#
683
+
addi 22, 23, -9 # process 8 keys
684
+
mtctr 22 # AES key loop
685
+
addi 10, 6, 144
818
686
819
-
xxlxor 47, 47, 15
820
-
stxvb16x 47, 0, 9 # store output
821
-
xxlxor 48, 48, 16
822
-
stxvb16x 48, 15, 9 # store output
687
+
LOOP_8AES_STATE # process 8 AES keys
823
688
824
-
vcipherlast 17, 17, 23
825
-
vcipherlast 18, 18, 23
689
+
__PreLoop_aes_state:
690
+
lxv 32+1, 0(10) # round key
691
+
AES_CIPHER_8x vcipher 15 1
692
+
addi 10, 10, 16
693
+
bdnz __PreLoop_aes_state
694
+
lxv 32+1, 0(10) # last round key (v1)
826
695
827
-
xxlxor 49, 49, 17
828
-
stxvb16x 49, 16, 9 # store output
829
-
xxlxor 50, 50, 18
830
-
stxvb16x 50, 17, 9 # store output
696
+
cmpdi 12, 0 # Only one loop (8 block)
697
+
beq __Finish_ghash
831
698
832
-
vcipherlast 19, 19, 23
833
-
vcipherlast 20, 20, 23
699
+
#
700
+
# Loop 8x blocks and compute ghash
701
+
#
702
+
__Loop_8x_block_enc:
703
+
vcipherlast 15, 15, 1
704
+
vcipherlast 16, 16, 1
705
+
vcipherlast 17, 17, 1
706
+
vcipherlast 18, 18, 1
707
+
vcipherlast 19, 19, 1
708
+
vcipherlast 20, 20, 1
709
+
vcipherlast 21, 21, 1
710
+
vcipherlast 22, 22, 1
834
711
835
-
xxlxor 51, 51, 19
836
-
stxvb16x 51, 18, 9 # store output
837
-
xxlxor 52, 52, 20
838
-
stxvb16x 52, 19, 9 # store output
712
+
lxvb16x 32+23, 0, 14 # load block
713
+
lxvb16x 32+24, 15, 14 # load block
714
+
lxvb16x 32+25, 16, 14 # load block
715
+
lxvb16x 32+26, 17, 14 # load block
716
+
lxvb16x 32+27, 18, 14 # load block
717
+
lxvb16x 32+28, 19, 14 # load block
718
+
lxvb16x 32+29, 20, 14 # load block
719
+
lxvb16x 32+30, 21, 14 # load block
720
+
addi 14, 14, 128
839
721
840
-
vcipherlast 21, 21, 23
841
-
vcipherlast 22, 22, 23
722
+
vxor 15, 15, 23
723
+
vxor 16, 16, 24
724
+
vxor 17, 17, 25
725
+
vxor 18, 18, 26
726
+
vxor 19, 19, 27
727
+
vxor 20, 20, 28
728
+
vxor 21, 21, 29
729
+
vxor 22, 22, 30
842
730
843
-
xxlxor 53, 53, 21
844
-
stxvb16x 53, 20, 9 # store output
845
-
xxlxor 54, 54, 22
846
-
stxvb16x 54, 21, 9 # store output
847
-
848
-
addi 9, 9, 128
731
+
stxvb16x 47, 0, 9 # store output
732
+
stxvb16x 48, 15, 9 # store output
733
+
stxvb16x 49, 16, 9 # store output
734
+
stxvb16x 50, 17, 9 # store output
735
+
stxvb16x 51, 18, 9 # store output
736
+
stxvb16x 52, 19, 9 # store output
737
+
stxvb16x 53, 20, 9 # store output
738
+
stxvb16x 54, 21, 9 # store output
739
+
addi 9, 9, 128
849
740
850
741
# ghash here
851
-
ppc_aes_gcm_ghash2_4x
742
+
vxor 15, 15, 0
743
+
PPC_GHASH4x 0, 15, 16, 17, 18
852
744
853
-
xxlor 27+32, 0, 0
854
-
vaddudm 30, 30, 31 # IV + counter
855
-
vmr 29, 30
856
-
vxor 15, 30, 27 # add round key
857
-
vaddudm 30, 30, 31
858
-
vxor 16, 30, 27
859
-
vaddudm 30, 30, 31
860
-
vxor 17, 30, 27
861
-
vaddudm 30, 30, 31
862
-
vxor 18, 30, 27
863
-
vaddudm 30, 30, 31
864
-
vxor 19, 30, 27
865
-
vaddudm 30, 30, 31
866
-
vxor 20, 30, 27
867
-
vaddudm 30, 30, 31
868
-
vxor 21, 30, 27
869
-
vaddudm 30, 30, 31
870
-
vxor 22, 30, 27
745
+
vxor 19, 19, 0
746
+
PPC_GHASH4x 0, 19, 20, 21, 22
871
747
872
-
addi 12, 12, -128
748
+
xxlor 32+15, 9, 9 # last state
749
+
vadduwm 15, 15, 31 # state + counter
750
+
vadduwm 16, 15, 31
751
+
vadduwm 17, 16, 31
752
+
vadduwm 18, 17, 31
753
+
vadduwm 19, 18, 31
754
+
vadduwm 20, 19, 31
755
+
vadduwm 21, 20, 31
756
+
vadduwm 22, 21, 31
757
+
xxlor 9, 32+22, 32+22 # save last state
758
+
759
+
xxlor 32+27, 0, 0 # restore roundkey 0
760
+
vxor 15, 15, 27 # IV + round key - add round key 0
761
+
vxor 16, 16, 27
762
+
vxor 17, 17, 27
763
+
vxor 18, 18, 27
764
+
vxor 19, 19, 27
765
+
vxor 20, 20, 27
766
+
vxor 21, 21, 27
767
+
vxor 22, 22, 27
768
+
769
+
addi 5, 5, -128
873
770
addi 11, 11, 128
874
771
875
-
bdnz Loop_8x_block
772
+
LOOP_8AES_STATE # process 8 AES keys
773
+
mtctr 22 # AES key loop
774
+
addi 10, 6, 144
775
+
__LastLoop_aes_state:
776
+
lxv 32+1, 0(10) # round key
777
+
AES_CIPHER_8x vcipher 15 1
778
+
addi 10, 10, 16
779
+
bdnz __LastLoop_aes_state
780
+
lxv 32+1, 0(10) # last round key (v1)
876
781
877
-
vmr 30, 29
878
-
stxvb16x 30+32, 0, 7 # update IV
782
+
addi 12, 12, -1
783
+
cmpdi 12, 0
784
+
bne __Loop_8x_block_enc
879
785
880
-
Loop_last_block:
881
-
cmpdi 12, 0
786
+
__Finish_ghash:
787
+
vcipherlast 15, 15, 1
788
+
vcipherlast 16, 16, 1
789
+
vcipherlast 17, 17, 1
790
+
vcipherlast 18, 18, 1
791
+
vcipherlast 19, 19, 1
792
+
vcipherlast 20, 20, 1
793
+
vcipherlast 21, 21, 1
794
+
vcipherlast 22, 22, 1
795
+
796
+
lxvb16x 32+23, 0, 14 # load block
797
+
lxvb16x 32+24, 15, 14 # load block
798
+
lxvb16x 32+25, 16, 14 # load block
799
+
lxvb16x 32+26, 17, 14 # load block
800
+
lxvb16x 32+27, 18, 14 # load block
801
+
lxvb16x 32+28, 19, 14 # load block
802
+
lxvb16x 32+29, 20, 14 # load block
803
+
lxvb16x 32+30, 21, 14 # load block
804
+
addi 14, 14, 128
805
+
806
+
vxor 15, 15, 23
807
+
vxor 16, 16, 24
808
+
vxor 17, 17, 25
809
+
vxor 18, 18, 26
810
+
vxor 19, 19, 27
811
+
vxor 20, 20, 28
812
+
vxor 21, 21, 29
813
+
vxor 22, 22, 30
814
+
815
+
stxvb16x 47, 0, 9 # store output
816
+
stxvb16x 48, 15, 9 # store output
817
+
stxvb16x 49, 16, 9 # store output
818
+
stxvb16x 50, 17, 9 # store output
819
+
stxvb16x 51, 18, 9 # store output
820
+
stxvb16x 52, 19, 9 # store output
821
+
stxvb16x 53, 20, 9 # store output
822
+
stxvb16x 54, 21, 9 # store output
823
+
addi 9, 9, 128
824
+
825
+
vxor 15, 15, 0
826
+
PPC_GHASH4x 0, 15, 16, 17, 18
827
+
828
+
vxor 19, 19, 0
829
+
PPC_GHASH4x 0, 19, 20, 21, 22
830
+
831
+
xxlor 30+32, 9, 9 # last ctr
832
+
vadduwm 30, 30, 31 # increase ctr
833
+
stxvb16x 32+30, 0, 7 # update IV
834
+
stxvb16x 32+0, 0, 8 # update Xi
835
+
836
+
addi 5, 5, -128
837
+
addi 11, 11, 128
838
+
839
+
#
840
+
# Done 8x blocks
841
+
#
842
+
843
+
cmpdi 5, 0
882
844
beq aes_gcm_out
883
845
884
-
# loop last few blocks
885
-
li 10, 16
886
-
divdu 10, 12, 10
846
+
__Process_more_enc:
847
+
li 24, 1 # encrypt
848
+
bl aes_gcm_crypt_1x
849
+
cmpdi 5, 0
850
+
beq aes_gcm_out
887
851
888
-
mtctr 10
852
+
bl __Process_partial
853
+
cmpdi 5, 0
854
+
beq aes_gcm_out
855
+
__Do_combine_enc:
856
+
bl __Combine_partial
857
+
cmpdi 5, 0
858
+
bgt __Process_encrypt
859
+
b aes_gcm_out
889
860
890
-
lwz 10, 240(6)
861
+
SYM_FUNC_END(aes_p10_gcm_encrypt)
891
862
892
-
cmpdi 12, 16
893
-
blt Final_block
863
+
################################################################################
864
+
# aes_p10_gcm_decrypt (const void *inp, void *out, size_t len,
865
+
# const char *rk, unsigned char iv[16], void *Xip);
866
+
# 8x Decrypt
867
+
#
868
+
################################################################################
869
+
SYM_FUNC_START(aes_p10_gcm_decrypt)
894
870
895
-
Next_rem_block:
896
-
lxvb16x 15, 0, 14 # load block
871
+
cmpdi 5, 0
872
+
ble __Invalid_msg_len
897
873
898
-
Loop_aes_middle_1x
874
+
SAVE_REGS
875
+
LOAD_HASH_TABLE
899
876
900
-
xxlor 23+32, 10, 10
877
+
# initialize ICB: GHASH( IV ), IV - r7
878
+
lxvb16x 30+32, 0, 7 # load IV - v30
901
879
902
-
cmpdi 10, 10
903
-
beq Do_next_1x
880
+
mr 14, 3
881
+
mr 9, 4
904
882
905
-
# 192 bits
906
-
xxlor 24+32, 11, 11
883
+
# counter 1
884
+
vxor 31, 31, 31
885
+
vspltisb 22, 1
886
+
vsldoi 31, 31, 22,1 # counter 1
907
887
908
-
vcipher 15, 15, 23
909
-
vcipher 15, 15, 24
888
+
addis 11, 2, permx@toc@ha
889
+
addi 11, 11, permx@toc@l
890
+
lxv 10, 0(11) # vs10: vpermxor vector
891
+
li 11, 0
910
892
911
-
xxlor 23+32, 12, 12
893
+
# load 9 round keys to VSR
894
+
lxv 0, 0(6) # round key 0
895
+
lxv 1, 16(6) # round key 1
896
+
lxv 2, 32(6) # round key 2
897
+
lxv 3, 48(6) # round key 3
898
+
lxv 4, 64(6) # round key 4
899
+
lxv 5, 80(6) # round key 5
900
+
lxv 6, 96(6) # round key 6
901
+
lxv 7, 112(6) # round key 7
902
+
lxv 8, 128(6) # round key 8
912
903
913
-
cmpdi 10, 12
914
-
beq Do_next_1x
904
+
# load rounds - 10 (128), 12 (192), 14 (256)
905
+
lwz 23, 240(6) # n rounds
906
+
li 24, 0 # decrypt
915
907
916
-
# 256 bits
917
-
xxlor 24+32, 13, 13
918
-
919
-
vcipher 15, 15, 23
920
-
vcipher 15, 15, 24
921
-
922
-
xxlor 23+32, 14, 14
923
-
924
-
cmpdi 10, 14
925
-
beq Do_next_1x
926
-
927
-
Do_next_1x:
928
-
vcipherlast 15, 15, 23
929
-
930
-
xxlxor 47, 47, 15
931
-
stxvb16x 47, 0, 9 # store output
932
-
addi 14, 14, 16
933
-
addi 9, 9, 16
934
-
935
-
vmr 28, 15
936
-
ppc_update_hash_1x
937
-
938
-
addi 12, 12, -16
939
-
addi 11, 11, 16
940
-
xxlor 19+32, 0, 0
941
-
vaddudm 30, 30, 31 # IV + counter
942
-
vxor 15, 30, 19 # add round key
943
-
944
-
bdnz Next_rem_block
945
-
946
-
li 15, 0
947
-
std 15, 56(7) # clear partial?
948
-
stxvb16x 30+32, 0, 7 # update IV
908
+
__Process_decrypt:
909
+
#
910
+
# Process different blocks
911
+
#
912
+
ld 12, 56(7)
949
913
cmpdi 12, 0
950
-
beq aes_gcm_out
914
+
bgt __Do_combine_dec
915
+
cmpdi 5, 128
916
+
blt __Process_more_dec
951
917
952
-
Final_block:
953
-
lwz 10, 240(6)
954
-
Loop_aes_middle_1x
918
+
#
919
+
# Process 8x AES/GCM blocks
920
+
#
921
+
__Process_8x_dec:
922
+
# 8x blcoks
923
+
li 10, 128
924
+
divdu 12, 5, 10 # n 128 bytes-blocks
955
925
956
-
xxlor 23+32, 10, 10
926
+
addi 12, 12, -1 # loop - 1
957
927
958
-
cmpdi 10, 10
959
-
beq Do_final_1x
928
+
vmr 15, 30 # first state: IV
929
+
vadduwm 16, 15, 31 # state + counter
930
+
vadduwm 17, 16, 31
931
+
vadduwm 18, 17, 31
932
+
vadduwm 19, 18, 31
933
+
vadduwm 20, 19, 31
934
+
vadduwm 21, 20, 31
935
+
vadduwm 22, 21, 31
936
+
xxlor 9, 32+22, 32+22 # save last state
960
937
961
-
# 192 bits
962
-
xxlor 24+32, 11, 11
963
-
964
-
vcipher 15, 15, 23
965
-
vcipher 15, 15, 24
966
-
967
-
xxlor 23+32, 12, 12
968
-
969
-
cmpdi 10, 12
970
-
beq Do_final_1x
971
-
972
-
# 256 bits
973
-
xxlor 24+32, 13, 13
974
-
975
-
vcipher 15, 15, 23
976
-
vcipher 15, 15, 24
977
-
978
-
xxlor 23+32, 14, 14
979
-
980
-
cmpdi 10, 14
981
-
beq Do_final_1x
982
-
983
-
Do_final_1x:
984
-
vcipherlast 15, 15, 23
985
-
986
-
# check partial block
987
-
li 21, 0 # encrypt
988
-
ld 15, 56(7) # partial?
989
-
cmpdi 15, 0
990
-
beq Normal_block
991
-
bl Do_partial_block
992
-
993
-
cmpdi 12, 0
994
-
ble aes_gcm_out
995
-
996
-
b Continue_partial_check
997
-
998
-
Normal_block:
999
-
lxvb16x 15, 0, 14 # load last block
1000
-
xxlxor 47, 47, 15
1001
-
1002
-
# create partial block mask
1003
-
li 15, 16
1004
-
sub 15, 15, 12 # index to the mask
1005
-
1006
-
vspltisb 16, -1 # first 16 bytes - 0xffff...ff
1007
-
vspltisb 17, 0 # second 16 bytes - 0x0000...00
1008
-
li 10, 192
1009
-
stvx 16, 10, 1
1010
-
addi 10, 10, 16
1011
-
stvx 17, 10, 1
1012
-
1013
-
addi 10, 1, 192
1014
-
lxvb16x 16, 15, 10 # load partial block mask
1015
-
xxland 47, 47, 16
1016
-
1017
-
vmr 28, 15
1018
-
ppc_update_hash_1x
1019
-
1020
-
# * should store only the remaining bytes.
1021
-
bl Write_partial_block
1022
-
1023
-
stxvb16x 30+32, 0, 7 # update IV
1024
-
std 12, 56(7) # update partial?
1025
-
li 16, 16
1026
-
1027
-
stxvb16x 32, 0, 8 # write out Xi
1028
-
stxvb16x 32, 16, 8 # write out Xi
1029
-
b aes_gcm_out
1030
-
1031
-
#
1032
-
# Compute data mask
1033
-
#
1034
-
.macro GEN_MASK _mask _start _end
1035
-
vspltisb 16, -1 # first 16 bytes - 0xffff...ff
1036
-
vspltisb 17, 0 # second 16 bytes - 0x0000...00
1037
-
li 10, 192
1038
-
stxvb16x 17+32, 10, 1
1039
-
add 10, 10, \_start
1040
-
stxvb16x 16+32, 10, 1
1041
-
add 10, 10, \_end
1042
-
stxvb16x 17+32, 10, 1
1043
-
1044
-
addi 10, 1, 192
1045
-
lxvb16x \_mask, 0, 10 # load partial block mask
1046
-
.endm
1047
-
1048
-
#
1049
-
# Handle multiple partial blocks for encrypt and decrypt
1050
-
# operations.
1051
-
#
1052
-
SYM_FUNC_START_LOCAL(Do_partial_block)
1053
-
add 17, 15, 5
1054
-
cmpdi 17, 16
1055
-
bgt Big_block
1056
-
GEN_MASK 18, 15, 5
1057
-
b _Partial
1058
-
SYM_FUNC_END(Do_partial_block)
1059
-
Big_block:
1060
-
li 16, 16
1061
-
GEN_MASK 18, 15, 16
1062
-
1063
-
_Partial:
1064
-
lxvb16x 17+32, 0, 14 # load last block
1065
-
sldi 16, 15, 3
1066
-
mtvsrdd 32+16, 0, 16
1067
-
vsro 17, 17, 16
1068
-
xxlxor 47, 47, 17+32
1069
-
xxland 47, 47, 18
1070
-
1071
-
vxor 0, 0, 0 # clear Xi
1072
-
vmr 28, 15
1073
-
1074
-
cmpdi 21, 0 # encrypt/decrypt ops?
1075
-
beq Skip_decrypt
1076
-
xxland 32+28, 32+17, 18
1077
-
1078
-
Skip_decrypt:
1079
-
1080
-
ppc_update_hash_1x
1081
-
1082
-
li 16, 16
1083
-
lxvb16x 32+29, 16, 8
1084
-
vxor 0, 0, 29
1085
-
stxvb16x 32, 0, 8 # save Xi
1086
-
stxvb16x 32, 16, 8 # save Xi
1087
-
1088
-
# store partial block
1089
-
# loop the rest of the stream if any
1090
-
sldi 16, 15, 3
1091
-
mtvsrdd 32+16, 0, 16
1092
-
vslo 15, 15, 16
1093
-
#stxvb16x 15+32, 0, 9 # last block
1094
-
1095
-
li 16, 16
1096
-
sub 17, 16, 15 # 16 - partial
1097
-
1098
-
add 16, 15, 5
1099
-
cmpdi 16, 16
1100
-
bgt Larger_16
1101
-
mr 17, 5
1102
-
Larger_16:
1103
-
1104
-
# write partial
1105
-
li 10, 192
1106
-
stxvb16x 15+32, 10, 1 # save current block
1107
-
1108
-
addi 10, 9, -1
1109
-
addi 16, 1, 191
1110
-
mtctr 17 # move partial byte count
1111
-
1112
-
Write_last_partial:
1113
-
lbzu 18, 1(16)
1114
-
stbu 18, 1(10)
1115
-
bdnz Write_last_partial
1116
-
# Complete loop partial
1117
-
1118
-
add 14, 14, 17
1119
-
add 9, 9, 17
1120
-
sub 12, 12, 17
1121
-
add 11, 11, 17
1122
-
1123
-
add 15, 15, 5
1124
-
cmpdi 15, 16
1125
-
blt Save_partial
1126
-
1127
-
vaddudm 30, 30, 31
1128
-
stxvb16x 30+32, 0, 7 # update IV
938
+
# vxor state, state, w # addroundkey
1129
939
xxlor 32+29, 0, 0
1130
-
vxor 15, 30, 29 # IV + round key - add round key 0
1131
-
li 15, 0
1132
-
std 15, 56(7) # partial done - clear
1133
-
b Partial_done
1134
-
Save_partial:
1135
-
std 15, 56(7) # partial
940
+
vxor 15, 15, 29 # IV + round key - add round key 0
941
+
vxor 16, 16, 29
942
+
vxor 17, 17, 29
943
+
vxor 18, 18, 29
944
+
vxor 19, 19, 29
945
+
vxor 20, 20, 29
946
+
vxor 21, 21, 29
947
+
vxor 22, 22, 29
1136
948
1137
-
Partial_done:
1138
-
blr
949
+
li 15, 16
950
+
li 16, 32
951
+
li 17, 48
952
+
li 18, 64
953
+
li 19, 80
954
+
li 20, 96
955
+
li 21, 112
1139
956
1140
-
#
1141
-
# Write partial block
1142
-
# r9 - output
1143
-
# r12 - remaining bytes
1144
-
# v15 - partial input data
1145
-
#
1146
-
SYM_FUNC_START_LOCAL(Write_partial_block)
1147
-
li 10, 192
1148
-
stxvb16x 15+32, 10, 1 # last block
957
+
#
958
+
# Pre-compute first 8 AES state and leave 1/3/5 more rounds
959
+
# for the loop.
960
+
#
961
+
addi 22, 23, -9 # process 8 keys
962
+
mtctr 22 # AES key loop
963
+
addi 10, 6, 144
1149
964
1150
-
addi 10, 9, -1
1151
-
addi 16, 1, 191
965
+
LOOP_8AES_STATE # process 8 AES keys
1152
966
1153
-
mtctr 12 # remaining bytes
1154
-
li 15, 0
967
+
__PreLoop_aes_state_dec:
968
+
lxv 32+1, 0(10) # round key
969
+
AES_CIPHER_8x vcipher 15 1
970
+
addi 10, 10, 16
971
+
bdnz __PreLoop_aes_state_dec
972
+
lxv 32+1, 0(10) # last round key (v1)
1155
973
1156
-
Write_last_byte:
1157
-
lbzu 14, 1(16)
1158
-
stbu 14, 1(10)
1159
-
bdnz Write_last_byte
1160
-
blr
1161
-
SYM_FUNC_END(Write_partial_block)
974
+
cmpdi 12, 0 # Only one loop (8 block)
975
+
beq __Finish_ghash_dec
1162
976
1163
-
aes_gcm_out:
1164
-
# out = state
1165
-
stxvb16x 32, 0, 8 # write out Xi
1166
-
add 3, 11, 12 # return count
977
+
#
978
+
# Loop 8x blocks and compute ghash
979
+
#
980
+
__Loop_8x_block_dec:
981
+
vcipherlast 15, 15, 1
982
+
vcipherlast 16, 16, 1
983
+
vcipherlast 17, 17, 1
984
+
vcipherlast 18, 18, 1
985
+
vcipherlast 19, 19, 1
986
+
vcipherlast 20, 20, 1
987
+
vcipherlast 21, 21, 1
988
+
vcipherlast 22, 22, 1
989
+
990
+
lxvb16x 32+23, 0, 14 # load block
991
+
lxvb16x 32+24, 15, 14 # load block
992
+
lxvb16x 32+25, 16, 14 # load block
993
+
lxvb16x 32+26, 17, 14 # load block
994
+
lxvb16x 32+27, 18, 14 # load block
995
+
lxvb16x 32+28, 19, 14 # load block
996
+
lxvb16x 32+29, 20, 14 # load block
997
+
lxvb16x 32+30, 21, 14 # load block
998
+
addi 14, 14, 128
999
+
1000
+
vxor 15, 15, 23
1001
+
vxor 16, 16, 24
1002
+
vxor 17, 17, 25
1003
+
vxor 18, 18, 26
1004
+
vxor 19, 19, 27
1005
+
vxor 20, 20, 28
1006
+
vxor 21, 21, 29
1007
+
vxor 22, 22, 30
1008
+
1009
+
stxvb16x 47, 0, 9 # store output
1010
+
stxvb16x 48, 15, 9 # store output
1011
+
stxvb16x 49, 16, 9 # store output
1012
+
stxvb16x 50, 17, 9 # store output
1013
+
stxvb16x 51, 18, 9 # store output
1014
+
stxvb16x 52, 19, 9 # store output
1015
+
stxvb16x 53, 20, 9 # store output
1016
+
stxvb16x 54, 21, 9 # store output
1017
+
1018
+
addi 9, 9, 128
1019
+
1020
+
vmr 15, 23
1021
+
vmr 16, 24
1022
+
vmr 17, 25
1023
+
vmr 18, 26
1024
+
vmr 19, 27
1025
+
vmr 20, 28
1026
+
vmr 21, 29
1027
+
vmr 22, 30
1028
+
1029
+
# ghash here
1030
+
vxor 15, 15, 0
1031
+
PPC_GHASH4x 0, 15, 16, 17, 18
1032
+
1033
+
vxor 19, 19, 0
1034
+
PPC_GHASH4x 0, 19, 20, 21, 22
1035
+
1036
+
xxlor 32+15, 9, 9 # last state
1037
+
vadduwm 15, 15, 31 # state + counter
1038
+
vadduwm 16, 15, 31
1039
+
vadduwm 17, 16, 31
1040
+
vadduwm 18, 17, 31
1041
+
vadduwm 19, 18, 31
1042
+
vadduwm 20, 19, 31
1043
+
vadduwm 21, 20, 31
1044
+
vadduwm 22, 21, 31
1045
+
xxlor 9, 32+22, 32+22 # save last state
1046
+
1047
+
xxlor 32+27, 0, 0 # restore roundkey 0
1048
+
vxor 15, 15, 27 # IV + round key - add round key 0
1049
+
vxor 16, 16, 27
1050
+
vxor 17, 17, 27
1051
+
vxor 18, 18, 27
1052
+
vxor 19, 19, 27
1053
+
vxor 20, 20, 27
1054
+
vxor 21, 21, 27
1055
+
vxor 22, 22, 27
1056
+
1057
+
addi 5, 5, -128
1058
+
addi 11, 11, 128
1059
+
1060
+
LOOP_8AES_STATE # process 8 AES keys
1061
+
mtctr 22 # AES key loop
1062
+
addi 10, 6, 144
1063
+
__LastLoop_aes_state_dec:
1064
+
lxv 32+1, 0(10) # round key
1065
+
AES_CIPHER_8x vcipher 15 1
1066
+
addi 10, 10, 16
1067
+
bdnz __LastLoop_aes_state_dec
1068
+
lxv 32+1, 0(10) # last round key (v1)
1069
+
1070
+
addi 12, 12, -1
1071
+
cmpdi 12, 0
1072
+
bne __Loop_8x_block_dec
1073
+
1074
+
__Finish_ghash_dec:
1075
+
vcipherlast 15, 15, 1
1076
+
vcipherlast 16, 16, 1
1077
+
vcipherlast 17, 17, 1
1078
+
vcipherlast 18, 18, 1
1079
+
vcipherlast 19, 19, 1
1080
+
vcipherlast 20, 20, 1
1081
+
vcipherlast 21, 21, 1
1082
+
vcipherlast 22, 22, 1
1083
+
1084
+
lxvb16x 32+23, 0, 14 # load block
1085
+
lxvb16x 32+24, 15, 14 # load block
1086
+
lxvb16x 32+25, 16, 14 # load block
1087
+
lxvb16x 32+26, 17, 14 # load block
1088
+
lxvb16x 32+27, 18, 14 # load block
1089
+
lxvb16x 32+28, 19, 14 # load block
1090
+
lxvb16x 32+29, 20, 14 # load block
1091
+
lxvb16x 32+30, 21, 14 # load block
1092
+
addi 14, 14, 128
1093
+
1094
+
vxor 15, 15, 23
1095
+
vxor 16, 16, 24
1096
+
vxor 17, 17, 25
1097
+
vxor 18, 18, 26
1098
+
vxor 19, 19, 27
1099
+
vxor 20, 20, 28
1100
+
vxor 21, 21, 29
1101
+
vxor 22, 22, 30
1102
+
1103
+
stxvb16x 47, 0, 9 # store output
1104
+
stxvb16x 48, 15, 9 # store output
1105
+
stxvb16x 49, 16, 9 # store output
1106
+
stxvb16x 50, 17, 9 # store output
1107
+
stxvb16x 51, 18, 9 # store output
1108
+
stxvb16x 52, 19, 9 # store output
1109
+
stxvb16x 53, 20, 9 # store output
1110
+
stxvb16x 54, 21, 9 # store output
1111
+
addi 9, 9, 128
1112
+
1113
+
#vmr 15, 23
1114
+
vxor 15, 23, 0
1115
+
vmr 16, 24
1116
+
vmr 17, 25
1117
+
vmr 18, 26
1118
+
vmr 19, 27
1119
+
vmr 20, 28
1120
+
vmr 21, 29
1121
+
vmr 22, 30
1122
+
1123
+
#vxor 15, 15, 0
1124
+
PPC_GHASH4x 0, 15, 16, 17, 18
1125
+
1126
+
vxor 19, 19, 0
1127
+
PPC_GHASH4x 0, 19, 20, 21, 22
1128
+
1129
+
xxlor 30+32, 9, 9 # last ctr
1130
+
vadduwm 30, 30, 31 # increase ctr
1131
+
stxvb16x 32+30, 0, 7 # update IV
1132
+
stxvb16x 32+0, 0, 8 # update Xi
1133
+
1134
+
addi 5, 5, -128
1135
+
addi 11, 11, 128
1136
+
1137
+
#
1138
+
# Done 8x blocks
1139
+
#
1140
+
1141
+
cmpdi 5, 0
1142
+
beq aes_gcm_out
1143
+
1144
+
__Process_more_dec:
1145
+
li 24, 0 # decrypt
1146
+
bl aes_gcm_crypt_1x
1147
+
cmpdi 5, 0
1148
+
beq aes_gcm_out
1149
+
1150
+
bl __Process_partial
1151
+
cmpdi 5, 0
1152
+
beq aes_gcm_out
1153
+
__Do_combine_dec:
1154
+
bl __Combine_partial
1155
+
cmpdi 5, 0
1156
+
bgt __Process_decrypt
1157
+
b aes_gcm_out
1158
+
SYM_FUNC_END(aes_p10_gcm_decrypt)
1159
+
1160
+
SYM_FUNC_START_LOCAL(aes_gcm_out)
1161
+
1162
+
mr 3, 11 # return count
1167
1163
1168
1164
RESTORE_REGS
1169
1165
blr
1170
1166
1171
-
#
1172
-
# 8x Decrypt
1173
-
#
1174
-
_GLOBAL(aes_p10_gcm_decrypt)
1175
-
.align 5
1176
-
1177
-
SAVE_REGS
1178
-
1179
-
LOAD_HASH_TABLE
1180
-
1181
-
# initialize ICB: GHASH( IV ), IV - r7
1182
-
lxvb16x 30+32, 0, 7 # load IV - v30
1183
-
1184
-
mr 12, 5 # length
1185
-
li 11, 0 # block index
1186
-
1187
-
# counter 1
1188
-
vxor 31, 31, 31
1189
-
vspltisb 22, 1
1190
-
vsldoi 31, 31, 22,1 # counter 1
1191
-
1192
-
# load round key to VSR
1193
-
lxv 0, 0(6)
1194
-
lxv 1, 0x10(6)
1195
-
lxv 2, 0x20(6)
1196
-
lxv 3, 0x30(6)
1197
-
lxv 4, 0x40(6)
1198
-
lxv 5, 0x50(6)
1199
-
lxv 6, 0x60(6)
1200
-
lxv 7, 0x70(6)
1201
-
lxv 8, 0x80(6)
1202
-
lxv 9, 0x90(6)
1203
-
lxv 10, 0xa0(6)
1204
-
1205
-
# load rounds - 10 (128), 12 (192), 14 (256)
1206
-
lwz 9,240(6)
1207
-
1208
-
#
1209
-
# vxor state, state, w # addroundkey
1210
-
xxlor 32+29, 0, 0
1211
-
vxor 15, 30, 29 # IV + round key - add round key 0
1212
-
1213
-
cmpdi 9, 10
1214
-
beq Loop_aes_gcm_8x_dec
1215
-
1216
-
# load 2 more round keys (v11, v12)
1217
-
lxv 11, 0xb0(6)
1218
-
lxv 12, 0xc0(6)
1219
-
1220
-
cmpdi 9, 12
1221
-
beq Loop_aes_gcm_8x_dec
1222
-
1223
-
# load 2 more round keys (v11, v12, v13, v14)
1224
-
lxv 13, 0xd0(6)
1225
-
lxv 14, 0xe0(6)
1226
-
cmpdi 9, 14
1227
-
beq Loop_aes_gcm_8x_dec
1228
-
1229
-
b aes_gcm_out
1230
-
1231
-
.align 5
1232
-
Loop_aes_gcm_8x_dec:
1233
-
mr 14, 3
1234
-
mr 9, 4
1235
-
1236
-
#
1237
-
# check partial block
1238
-
#
1239
-
Continue_partial_check_dec:
1240
-
ld 15, 56(7)
1241
-
cmpdi 15, 0
1242
-
beq Continue_dec
1243
-
bgt Final_block_dec
1244
-
cmpdi 15, 16
1245
-
blt Final_block_dec
1246
-
1247
-
Continue_dec:
1248
-
# n blcoks
1249
-
li 10, 128
1250
-
divdu 10, 12, 10 # n 128 bytes-blocks
1251
-
cmpdi 10, 0
1252
-
beq Loop_last_block_dec
1253
-
1254
-
vaddudm 30, 30, 31 # IV + counter
1255
-
vxor 16, 30, 29
1256
-
vaddudm 30, 30, 31
1257
-
vxor 17, 30, 29
1258
-
vaddudm 30, 30, 31
1259
-
vxor 18, 30, 29
1260
-
vaddudm 30, 30, 31
1261
-
vxor 19, 30, 29
1262
-
vaddudm 30, 30, 31
1263
-
vxor 20, 30, 29
1264
-
vaddudm 30, 30, 31
1265
-
vxor 21, 30, 29
1266
-
vaddudm 30, 30, 31
1267
-
vxor 22, 30, 29
1268
-
1269
-
mtctr 10
1270
-
1271
-
li 15, 16
1272
-
li 16, 32
1273
-
li 17, 48
1274
-
li 18, 64
1275
-
li 19, 80
1276
-
li 20, 96
1277
-
li 21, 112
1278
-
1279
-
lwz 10, 240(6)
1280
-
1281
-
Loop_8x_block_dec:
1282
-
1283
-
lxvb16x 15, 0, 14 # load block
1284
-
lxvb16x 16, 15, 14 # load block
1285
-
lxvb16x 17, 16, 14 # load block
1286
-
lxvb16x 18, 17, 14 # load block
1287
-
lxvb16x 19, 18, 14 # load block
1288
-
lxvb16x 20, 19, 14 # load block
1289
-
lxvb16x 21, 20, 14 # load block
1290
-
lxvb16x 22, 21, 14 # load block
1291
-
addi 14, 14, 128
1292
-
1293
-
Loop_aes_middle8x
1294
-
1295
-
xxlor 23+32, 10, 10
1296
-
1297
-
cmpdi 10, 10
1298
-
beq Do_next_ghash_dec
1299
-
1300
-
# 192 bits
1301
-
xxlor 24+32, 11, 11
1302
-
1303
-
vcipher 15, 15, 23
1304
-
vcipher 16, 16, 23
1305
-
vcipher 17, 17, 23
1306
-
vcipher 18, 18, 23
1307
-
vcipher 19, 19, 23
1308
-
vcipher 20, 20, 23
1309
-
vcipher 21, 21, 23
1310
-
vcipher 22, 22, 23
1311
-
1312
-
vcipher 15, 15, 24
1313
-
vcipher 16, 16, 24
1314
-
vcipher 17, 17, 24
1315
-
vcipher 18, 18, 24
1316
-
vcipher 19, 19, 24
1317
-
vcipher 20, 20, 24
1318
-
vcipher 21, 21, 24
1319
-
vcipher 22, 22, 24
1320
-
1321
-
xxlor 23+32, 12, 12
1322
-
1323
-
cmpdi 10, 12
1324
-
beq Do_next_ghash_dec
1325
-
1326
-
# 256 bits
1327
-
xxlor 24+32, 13, 13
1328
-
1329
-
vcipher 15, 15, 23
1330
-
vcipher 16, 16, 23
1331
-
vcipher 17, 17, 23
1332
-
vcipher 18, 18, 23
1333
-
vcipher 19, 19, 23
1334
-
vcipher 20, 20, 23
1335
-
vcipher 21, 21, 23
1336
-
vcipher 22, 22, 23
1337
-
1338
-
vcipher 15, 15, 24
1339
-
vcipher 16, 16, 24
1340
-
vcipher 17, 17, 24
1341
-
vcipher 18, 18, 24
1342
-
vcipher 19, 19, 24
1343
-
vcipher 20, 20, 24
1344
-
vcipher 21, 21, 24
1345
-
vcipher 22, 22, 24
1346
-
1347
-
xxlor 23+32, 14, 14
1348
-
1349
-
cmpdi 10, 14
1350
-
beq Do_next_ghash_dec
1351
-
b aes_gcm_out
1352
-
1353
-
Do_next_ghash_dec:
1354
-
1355
-
#
1356
-
# last round
1357
-
vcipherlast 15, 15, 23
1358
-
vcipherlast 16, 16, 23
1359
-
1360
-
xxlxor 47, 47, 15
1361
-
stxvb16x 47, 0, 9 # store output
1362
-
xxlxor 48, 48, 16
1363
-
stxvb16x 48, 15, 9 # store output
1364
-
1365
-
vcipherlast 17, 17, 23
1366
-
vcipherlast 18, 18, 23
1367
-
1368
-
xxlxor 49, 49, 17
1369
-
stxvb16x 49, 16, 9 # store output
1370
-
xxlxor 50, 50, 18
1371
-
stxvb16x 50, 17, 9 # store output
1372
-
1373
-
vcipherlast 19, 19, 23
1374
-
vcipherlast 20, 20, 23
1375
-
1376
-
xxlxor 51, 51, 19
1377
-
stxvb16x 51, 18, 9 # store output
1378
-
xxlxor 52, 52, 20
1379
-
stxvb16x 52, 19, 9 # store output
1380
-
1381
-
vcipherlast 21, 21, 23
1382
-
vcipherlast 22, 22, 23
1383
-
1384
-
xxlxor 53, 53, 21
1385
-
stxvb16x 53, 20, 9 # store output
1386
-
xxlxor 54, 54, 22
1387
-
stxvb16x 54, 21, 9 # store output
1388
-
1389
-
addi 9, 9, 128
1390
-
1391
-
xxlor 15+32, 15, 15
1392
-
xxlor 16+32, 16, 16
1393
-
xxlor 17+32, 17, 17
1394
-
xxlor 18+32, 18, 18
1395
-
xxlor 19+32, 19, 19
1396
-
xxlor 20+32, 20, 20
1397
-
xxlor 21+32, 21, 21
1398
-
xxlor 22+32, 22, 22
1399
-
1400
-
# ghash here
1401
-
ppc_aes_gcm_ghash2_4x
1402
-
1403
-
xxlor 27+32, 0, 0
1404
-
vaddudm 30, 30, 31 # IV + counter
1405
-
vmr 29, 30
1406
-
vxor 15, 30, 27 # add round key
1407
-
vaddudm 30, 30, 31
1408
-
vxor 16, 30, 27
1409
-
vaddudm 30, 30, 31
1410
-
vxor 17, 30, 27
1411
-
vaddudm 30, 30, 31
1412
-
vxor 18, 30, 27
1413
-
vaddudm 30, 30, 31
1414
-
vxor 19, 30, 27
1415
-
vaddudm 30, 30, 31
1416
-
vxor 20, 30, 27
1417
-
vaddudm 30, 30, 31
1418
-
vxor 21, 30, 27
1419
-
vaddudm 30, 30, 31
1420
-
vxor 22, 30, 27
1421
-
1422
-
addi 12, 12, -128
1423
-
addi 11, 11, 128
1424
-
1425
-
bdnz Loop_8x_block_dec
1426
-
1427
-
vmr 30, 29
1428
-
stxvb16x 30+32, 0, 7 # update IV
1429
-
1430
-
Loop_last_block_dec:
1431
-
cmpdi 12, 0
1432
-
beq aes_gcm_out
1433
-
1434
-
# loop last few blocks
1435
-
li 10, 16
1436
-
divdu 10, 12, 10
1437
-
1438
-
mtctr 10
1439
-
1440
-
lwz 10, 240(6)
1441
-
1442
-
cmpdi 12, 16
1443
-
blt Final_block_dec
1444
-
1445
-
Next_rem_block_dec:
1446
-
lxvb16x 15, 0, 14 # load block
1447
-
1448
-
Loop_aes_middle_1x
1449
-
1450
-
xxlor 23+32, 10, 10
1451
-
1452
-
cmpdi 10, 10
1453
-
beq Do_next_1x_dec
1454
-
1455
-
# 192 bits
1456
-
xxlor 24+32, 11, 11
1457
-
1458
-
vcipher 15, 15, 23
1459
-
vcipher 15, 15, 24
1460
-
1461
-
xxlor 23+32, 12, 12
1462
-
1463
-
cmpdi 10, 12
1464
-
beq Do_next_1x_dec
1465
-
1466
-
# 256 bits
1467
-
xxlor 24+32, 13, 13
1468
-
1469
-
vcipher 15, 15, 23
1470
-
vcipher 15, 15, 24
1471
-
1472
-
xxlor 23+32, 14, 14
1473
-
1474
-
cmpdi 10, 14
1475
-
beq Do_next_1x_dec
1476
-
1477
-
Do_next_1x_dec:
1478
-
vcipherlast 15, 15, 23
1479
-
1480
-
xxlxor 47, 47, 15
1481
-
stxvb16x 47, 0, 9 # store output
1482
-
addi 14, 14, 16
1483
-
addi 9, 9, 16
1484
-
1485
-
xxlor 28+32, 15, 15
1486
-
#vmr 28, 15
1487
-
ppc_update_hash_1x
1488
-
1489
-
addi 12, 12, -16
1490
-
addi 11, 11, 16
1491
-
xxlor 19+32, 0, 0
1492
-
vaddudm 30, 30, 31 # IV + counter
1493
-
vxor 15, 30, 19 # add round key
1494
-
1495
-
bdnz Next_rem_block_dec
1496
-
1497
-
li 15, 0
1498
-
std 15, 56(7) # clear partial?
1499
-
stxvb16x 30+32, 0, 7 # update IV
1500
-
cmpdi 12, 0
1501
-
beq aes_gcm_out
1502
-
1503
-
Final_block_dec:
1504
-
lwz 10, 240(6)
1505
-
Loop_aes_middle_1x
1506
-
1507
-
xxlor 23+32, 10, 10
1508
-
1509
-
cmpdi 10, 10
1510
-
beq Do_final_1x_dec
1511
-
1512
-
# 192 bits
1513
-
xxlor 24+32, 11, 11
1514
-
1515
-
vcipher 15, 15, 23
1516
-
vcipher 15, 15, 24
1517
-
1518
-
xxlor 23+32, 12, 12
1519
-
1520
-
cmpdi 10, 12
1521
-
beq Do_final_1x_dec
1522
-
1523
-
# 256 bits
1524
-
xxlor 24+32, 13, 13
1525
-
1526
-
vcipher 15, 15, 23
1527
-
vcipher 15, 15, 24
1528
-
1529
-
xxlor 23+32, 14, 14
1530
-
1531
-
cmpdi 10, 14
1532
-
beq Do_final_1x_dec
1533
-
1534
-
Do_final_1x_dec:
1535
-
vcipherlast 15, 15, 23
1536
-
1537
-
# check partial block
1538
-
li 21, 1 # decrypt
1539
-
ld 15, 56(7) # partial?
1540
-
cmpdi 15, 0
1541
-
beq Normal_block_dec
1542
-
bl Do_partial_block
1543
-
cmpdi 12, 0
1544
-
ble aes_gcm_out
1545
-
1546
-
b Continue_partial_check_dec
1547
-
1548
-
Normal_block_dec:
1549
-
lxvb16x 15, 0, 14 # load last block
1550
-
xxlxor 47, 47, 15
1551
-
1552
-
# create partial block mask
1553
-
li 15, 16
1554
-
sub 15, 15, 12 # index to the mask
1555
-
1556
-
vspltisb 16, -1 # first 16 bytes - 0xffff...ff
1557
-
vspltisb 17, 0 # second 16 bytes - 0x0000...00
1558
-
li 10, 192
1559
-
stvx 16, 10, 1
1560
-
addi 10, 10, 16
1561
-
stvx 17, 10, 1
1562
-
1563
-
addi 10, 1, 192
1564
-
lxvb16x 16, 15, 10 # load partial block mask
1565
-
xxland 47, 47, 16
1566
-
1567
-
xxland 32+28, 15, 16
1568
-
#vmr 28, 15
1569
-
ppc_update_hash_1x
1570
-
1571
-
# * should store only the remaining bytes.
1572
-
bl Write_partial_block
1573
-
1574
-
stxvb16x 30+32, 0, 7 # update IV
1575
-
std 12, 56(7) # update partial?
1576
-
li 16, 16
1577
-
1578
-
stxvb16x 32, 0, 8 # write out Xi
1579
-
stxvb16x 32, 16, 8 # write out Xi
1580
-
b aes_gcm_out
1167
+
__Invalid_msg_len:
1168
+
li 3, 0
1169
+
blr
1170
+
SYM_FUNC_END(aes_gcm_out)
1171
+
1172
+
SYM_DATA_START_LOCAL(PERMX)
1173
+
.align 4
1174
+
# for vector permute and xor
1175
+
permx:
1176
+
.long 0x4c5d6e7f, 0x08192a3b, 0xc4d5e6f7, 0x8091a2b3
1177
+
SYM_DATA_END(permx)
+2
-2
arch/x86/crypto/Kconfig
+2
-2
arch/x86/crypto/Kconfig
···
363
363
- AVX-512VL (Advanced Vector Extensions-512VL)
364
364
365
365
config CRYPTO_AEGIS128_AESNI_SSE2
366
-
tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE2)"
366
+
tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE4.1)"
367
367
depends on X86 && 64BIT
368
368
select CRYPTO_AEAD
369
369
select CRYPTO_SIMD
···
372
372
373
373
Architecture: x86_64 using:
374
374
- AES-NI (AES New Instructions)
375
-
- SSE2 (Streaming SIMD Extensions 2)
375
+
- SSE4.1 (Streaming SIMD Extensions 4.1)
376
376
377
377
config CRYPTO_NHPOLY1305_SSE2
378
378
tristate "Hash functions: NHPoly1305 (SSE2)"
+185
-331
arch/x86/crypto/aegis128-aesni-asm.S
+185
-331
arch/x86/crypto/aegis128-aesni-asm.S
···
1
1
/* SPDX-License-Identifier: GPL-2.0-only */
2
2
/*
3
-
* AES-NI + SSE2 implementation of AEGIS-128
3
+
* AES-NI + SSE4.1 implementation of AEGIS-128
4
4
*
5
5
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
6
6
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
7
+
* Copyright 2024 Google LLC
7
8
*/
8
9
9
10
#include <linux/linkage.h>
10
-
#include <linux/cfi_types.h>
11
-
#include <asm/frame.h>
12
11
13
12
#define STATE0 %xmm0
14
13
#define STATE1 %xmm1
···
19
20
#define T0 %xmm6
20
21
#define T1 %xmm7
21
22
22
-
#define STATEP %rdi
23
-
#define LEN %rsi
24
-
#define SRC %rdx
25
-
#define DST %rcx
26
-
27
23
.section .rodata.cst16.aegis128_const, "aM", @progbits, 32
28
24
.align 16
29
25
.Laegis128_const_0:
···
28
34
.byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1
29
35
.byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd
30
36
31
-
.section .rodata.cst16.aegis128_counter, "aM", @progbits, 16
32
-
.align 16
33
-
.Laegis128_counter:
34
-
.byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
35
-
.byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
37
+
.section .rodata.cst32.zeropad_mask, "aM", @progbits, 32
38
+
.align 32
39
+
.Lzeropad_mask:
40
+
.octa 0xffffffffffffffffffffffffffffffff
41
+
.octa 0
36
42
37
43
.text
38
44
···
55
61
.endm
56
62
57
63
/*
58
-
* __load_partial: internal ABI
59
-
* input:
60
-
* LEN - bytes
61
-
* SRC - src
62
-
* output:
63
-
* MSG - message block
64
-
* changed:
65
-
* T0
66
-
* %r8
67
-
* %r9
64
+
* Load 1 <= LEN (%ecx) <= 15 bytes from the pointer SRC into the xmm register
65
+
* MSG and zeroize any remaining bytes. Clobbers %rax, %rcx, and %r8.
68
66
*/
69
-
SYM_FUNC_START_LOCAL(__load_partial)
70
-
xor %r9d, %r9d
71
-
pxor MSG, MSG
67
+
.macro load_partial
68
+
sub $8, %ecx /* LEN - 8 */
69
+
jle .Lle8\@
72
70
73
-
mov LEN, %r8
74
-
and $0x1, %r8
75
-
jz .Lld_partial_1
71
+
/* Load 9 <= LEN <= 15 bytes: */
72
+
movq (SRC), MSG /* Load first 8 bytes */
73
+
mov (SRC, %rcx), %rax /* Load last 8 bytes */
74
+
neg %ecx
75
+
shl $3, %ecx
76
+
shr %cl, %rax /* Discard overlapping bytes */
77
+
pinsrq $1, %rax, MSG
78
+
jmp .Ldone\@
76
79
77
-
mov LEN, %r8
78
-
and $0x1E, %r8
79
-
add SRC, %r8
80
-
mov (%r8), %r9b
80
+
.Lle8\@:
81
+
add $4, %ecx /* LEN - 4 */
82
+
jl .Llt4\@
81
83
82
-
.Lld_partial_1:
83
-
mov LEN, %r8
84
-
and $0x2, %r8
85
-
jz .Lld_partial_2
84
+
/* Load 4 <= LEN <= 8 bytes: */
85
+
mov (SRC), %eax /* Load first 4 bytes */
86
+
mov (SRC, %rcx), %r8d /* Load last 4 bytes */
87
+
jmp .Lcombine\@
86
88
87
-
mov LEN, %r8
88
-
and $0x1C, %r8
89
-
add SRC, %r8
90
-
shl $0x10, %r9
91
-
mov (%r8), %r9w
92
-
93
-
.Lld_partial_2:
94
-
mov LEN, %r8
95
-
and $0x4, %r8
96
-
jz .Lld_partial_4
97
-
98
-
mov LEN, %r8
99
-
and $0x18, %r8
100
-
add SRC, %r8
101
-
shl $32, %r9
102
-
mov (%r8), %r8d
103
-
xor %r8, %r9
104
-
105
-
.Lld_partial_4:
106
-
movq %r9, MSG
107
-
108
-
mov LEN, %r8
109
-
and $0x8, %r8
110
-
jz .Lld_partial_8
111
-
112
-
mov LEN, %r8
113
-
and $0x10, %r8
114
-
add SRC, %r8
115
-
pslldq $8, MSG
116
-
movq (%r8), T0
117
-
pxor T0, MSG
118
-
119
-
.Lld_partial_8:
120
-
RET
121
-
SYM_FUNC_END(__load_partial)
89
+
.Llt4\@:
90
+
/* Load 1 <= LEN <= 3 bytes: */
91
+
add $2, %ecx /* LEN - 2 */
92
+
movzbl (SRC), %eax /* Load first byte */
93
+
jl .Lmovq\@
94
+
movzwl (SRC, %rcx), %r8d /* Load last 2 bytes */
95
+
.Lcombine\@:
96
+
shl $3, %ecx
97
+
shl %cl, %r8
98
+
or %r8, %rax /* Combine the two parts */
99
+
.Lmovq\@:
100
+
movq %rax, MSG
101
+
.Ldone\@:
102
+
.endm
122
103
123
104
/*
124
-
* __store_partial: internal ABI
125
-
* input:
126
-
* LEN - bytes
127
-
* DST - dst
128
-
* output:
129
-
* T0 - message block
130
-
* changed:
131
-
* %r8
132
-
* %r9
133
-
* %r10
105
+
* Store 1 <= LEN (%ecx) <= 15 bytes from the xmm register \msg to the pointer
106
+
* DST. Clobbers %rax, %rcx, and %r8.
134
107
*/
135
-
SYM_FUNC_START_LOCAL(__store_partial)
136
-
mov LEN, %r8
137
-
mov DST, %r9
108
+
.macro store_partial msg
109
+
sub $8, %ecx /* LEN - 8 */
110
+
jl .Llt8\@
138
111
139
-
movq T0, %r10
112
+
/* Store 8 <= LEN <= 15 bytes: */
113
+
pextrq $1, \msg, %rax
114
+
mov %ecx, %r8d
115
+
shl $3, %ecx
116
+
ror %cl, %rax
117
+
mov %rax, (DST, %r8) /* Store last LEN - 8 bytes */
118
+
movq \msg, (DST) /* Store first 8 bytes */
119
+
jmp .Ldone\@
140
120
141
-
cmp $8, %r8
142
-
jl .Lst_partial_8
121
+
.Llt8\@:
122
+
add $4, %ecx /* LEN - 4 */
123
+
jl .Llt4\@
143
124
144
-
mov %r10, (%r9)
145
-
psrldq $8, T0
146
-
movq T0, %r10
125
+
/* Store 4 <= LEN <= 7 bytes: */
126
+
pextrd $1, \msg, %eax
127
+
mov %ecx, %r8d
128
+
shl $3, %ecx
129
+
ror %cl, %eax
130
+
mov %eax, (DST, %r8) /* Store last LEN - 4 bytes */
131
+
movd \msg, (DST) /* Store first 4 bytes */
132
+
jmp .Ldone\@
147
133
148
-
sub $8, %r8
149
-
add $8, %r9
150
-
151
-
.Lst_partial_8:
152
-
cmp $4, %r8
153
-
jl .Lst_partial_4
154
-
155
-
mov %r10d, (%r9)
156
-
shr $32, %r10
157
-
158
-
sub $4, %r8
159
-
add $4, %r9
160
-
161
-
.Lst_partial_4:
162
-
cmp $2, %r8
163
-
jl .Lst_partial_2
164
-
165
-
mov %r10w, (%r9)
166
-
shr $0x10, %r10
167
-
168
-
sub $2, %r8
169
-
add $2, %r9
170
-
171
-
.Lst_partial_2:
172
-
cmp $1, %r8
173
-
jl .Lst_partial_1
174
-
175
-
mov %r10b, (%r9)
176
-
177
-
.Lst_partial_1:
178
-
RET
179
-
SYM_FUNC_END(__store_partial)
134
+
.Llt4\@:
135
+
/* Store 1 <= LEN <= 3 bytes: */
136
+
pextrb $0, \msg, 0(DST)
137
+
cmp $-2, %ecx /* LEN - 4 == -2, i.e. LEN == 2? */
138
+
jl .Ldone\@
139
+
pextrb $1, \msg, 1(DST)
140
+
je .Ldone\@
141
+
pextrb $2, \msg, 2(DST)
142
+
.Ldone\@:
143
+
.endm
180
144
181
145
/*
182
-
* void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv);
146
+
* void aegis128_aesni_init(struct aegis_state *state,
147
+
* const struct aegis_block *key,
148
+
* const u8 iv[AEGIS128_NONCE_SIZE]);
183
149
*/
184
-
SYM_FUNC_START(crypto_aegis128_aesni_init)
185
-
FRAME_BEGIN
150
+
SYM_FUNC_START(aegis128_aesni_init)
151
+
.set STATEP, %rdi
152
+
.set KEYP, %rsi
153
+
.set IVP, %rdx
186
154
187
155
/* load IV: */
188
-
movdqu (%rdx), T1
156
+
movdqu (IVP), T1
189
157
190
158
/* load key: */
191
-
movdqa (%rsi), KEY
159
+
movdqa (KEYP), KEY
192
160
pxor KEY, T1
193
161
movdqa T1, STATE0
194
162
movdqa KEY, STATE3
···
180
224
movdqu STATE2, 0x20(STATEP)
181
225
movdqu STATE3, 0x30(STATEP)
182
226
movdqu STATE4, 0x40(STATEP)
183
-
184
-
FRAME_END
185
227
RET
186
-
SYM_FUNC_END(crypto_aegis128_aesni_init)
228
+
SYM_FUNC_END(aegis128_aesni_init)
187
229
188
230
/*
189
-
* void crypto_aegis128_aesni_ad(void *state, unsigned int length,
190
-
* const void *data);
231
+
* void aegis128_aesni_ad(struct aegis_state *state, const u8 *data,
232
+
* unsigned int len);
233
+
*
234
+
* len must be a multiple of 16.
191
235
*/
192
-
SYM_FUNC_START(crypto_aegis128_aesni_ad)
193
-
FRAME_BEGIN
236
+
SYM_FUNC_START(aegis128_aesni_ad)
237
+
.set STATEP, %rdi
238
+
.set SRC, %rsi
239
+
.set LEN, %edx
194
240
195
-
cmp $0x10, LEN
196
-
jb .Lad_out
241
+
test LEN, LEN
242
+
jz .Lad_out
197
243
198
244
/* load the state: */
199
245
movdqu 0x00(STATEP), STATE0
···
204
246
movdqu 0x30(STATEP), STATE3
205
247
movdqu 0x40(STATEP), STATE4
206
248
207
-
mov SRC, %r8
208
-
and $0xF, %r8
209
-
jnz .Lad_u_loop
210
-
211
249
.align 8
212
-
.Lad_a_loop:
213
-
movdqa 0x00(SRC), MSG
214
-
aegis128_update
215
-
pxor MSG, STATE4
216
-
sub $0x10, LEN
217
-
cmp $0x10, LEN
218
-
jl .Lad_out_1
219
-
220
-
movdqa 0x10(SRC), MSG
221
-
aegis128_update
222
-
pxor MSG, STATE3
223
-
sub $0x10, LEN
224
-
cmp $0x10, LEN
225
-
jl .Lad_out_2
226
-
227
-
movdqa 0x20(SRC), MSG
228
-
aegis128_update
229
-
pxor MSG, STATE2
230
-
sub $0x10, LEN
231
-
cmp $0x10, LEN
232
-
jl .Lad_out_3
233
-
234
-
movdqa 0x30(SRC), MSG
235
-
aegis128_update
236
-
pxor MSG, STATE1
237
-
sub $0x10, LEN
238
-
cmp $0x10, LEN
239
-
jl .Lad_out_4
240
-
241
-
movdqa 0x40(SRC), MSG
242
-
aegis128_update
243
-
pxor MSG, STATE0
244
-
sub $0x10, LEN
245
-
cmp $0x10, LEN
246
-
jl .Lad_out_0
247
-
248
-
add $0x50, SRC
249
-
jmp .Lad_a_loop
250
-
251
-
.align 8
252
-
.Lad_u_loop:
250
+
.Lad_loop:
253
251
movdqu 0x00(SRC), MSG
254
252
aegis128_update
255
253
pxor MSG, STATE4
256
254
sub $0x10, LEN
257
-
cmp $0x10, LEN
258
-
jl .Lad_out_1
255
+
jz .Lad_out_1
259
256
260
257
movdqu 0x10(SRC), MSG
261
258
aegis128_update
262
259
pxor MSG, STATE3
263
260
sub $0x10, LEN
264
-
cmp $0x10, LEN
265
-
jl .Lad_out_2
261
+
jz .Lad_out_2
266
262
267
263
movdqu 0x20(SRC), MSG
268
264
aegis128_update
269
265
pxor MSG, STATE2
270
266
sub $0x10, LEN
271
-
cmp $0x10, LEN
272
-
jl .Lad_out_3
267
+
jz .Lad_out_3
273
268
274
269
movdqu 0x30(SRC), MSG
275
270
aegis128_update
276
271
pxor MSG, STATE1
277
272
sub $0x10, LEN
278
-
cmp $0x10, LEN
279
-
jl .Lad_out_4
273
+
jz .Lad_out_4
280
274
281
275
movdqu 0x40(SRC), MSG
282
276
aegis128_update
283
277
pxor MSG, STATE0
284
278
sub $0x10, LEN
285
-
cmp $0x10, LEN
286
-
jl .Lad_out_0
279
+
jz .Lad_out_0
287
280
288
281
add $0x50, SRC
289
-
jmp .Lad_u_loop
282
+
jmp .Lad_loop
290
283
291
284
/* store the state: */
292
285
.Lad_out_0:
···
246
337
movdqu STATE2, 0x20(STATEP)
247
338
movdqu STATE3, 0x30(STATEP)
248
339
movdqu STATE4, 0x40(STATEP)
249
-
FRAME_END
250
340
RET
251
341
252
342
.Lad_out_1:
···
254
346
movdqu STATE1, 0x20(STATEP)
255
347
movdqu STATE2, 0x30(STATEP)
256
348
movdqu STATE3, 0x40(STATEP)
257
-
FRAME_END
258
349
RET
259
350
260
351
.Lad_out_2:
···
262
355
movdqu STATE0, 0x20(STATEP)
263
356
movdqu STATE1, 0x30(STATEP)
264
357
movdqu STATE2, 0x40(STATEP)
265
-
FRAME_END
266
358
RET
267
359
268
360
.Lad_out_3:
···
270
364
movdqu STATE4, 0x20(STATEP)
271
365
movdqu STATE0, 0x30(STATEP)
272
366
movdqu STATE1, 0x40(STATEP)
273
-
FRAME_END
274
367
RET
275
368
276
369
.Lad_out_4:
···
278
373
movdqu STATE3, 0x20(STATEP)
279
374
movdqu STATE4, 0x30(STATEP)
280
375
movdqu STATE0, 0x40(STATEP)
281
-
FRAME_END
282
-
RET
283
-
284
376
.Lad_out:
285
-
FRAME_END
286
377
RET
287
-
SYM_FUNC_END(crypto_aegis128_aesni_ad)
378
+
SYM_FUNC_END(aegis128_aesni_ad)
288
379
289
-
.macro encrypt_block a s0 s1 s2 s3 s4 i
290
-
movdq\a (\i * 0x10)(SRC), MSG
380
+
.macro encrypt_block s0 s1 s2 s3 s4 i
381
+
movdqu (\i * 0x10)(SRC), MSG
291
382
movdqa MSG, T0
292
383
pxor \s1, T0
293
384
pxor \s4, T0
294
385
movdqa \s2, T1
295
386
pand \s3, T1
296
387
pxor T1, T0
297
-
movdq\a T0, (\i * 0x10)(DST)
388
+
movdqu T0, (\i * 0x10)(DST)
298
389
299
390
aegis128_update
300
391
pxor MSG, \s4
301
392
302
393
sub $0x10, LEN
303
-
cmp $0x10, LEN
304
-
jl .Lenc_out_\i
394
+
jz .Lenc_out_\i
305
395
.endm
306
396
307
397
/*
308
-
* void crypto_aegis128_aesni_enc(void *state, unsigned int length,
309
-
* const void *src, void *dst);
398
+
* void aegis128_aesni_enc(struct aegis_state *state, const u8 *src, u8 *dst,
399
+
* unsigned int len);
400
+
*
401
+
* len must be nonzero and a multiple of 16.
310
402
*/
311
-
SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc)
312
-
FRAME_BEGIN
313
-
314
-
cmp $0x10, LEN
315
-
jb .Lenc_out
403
+
SYM_FUNC_START(aegis128_aesni_enc)
404
+
.set STATEP, %rdi
405
+
.set SRC, %rsi
406
+
.set DST, %rdx
407
+
.set LEN, %ecx
316
408
317
409
/* load the state: */
318
410
movdqu 0x00(STATEP), STATE0
···
318
416
movdqu 0x30(STATEP), STATE3
319
417
movdqu 0x40(STATEP), STATE4
320
418
321
-
mov SRC, %r8
322
-
or DST, %r8
323
-
and $0xF, %r8
324
-
jnz .Lenc_u_loop
325
-
326
419
.align 8
327
-
.Lenc_a_loop:
328
-
encrypt_block a STATE0 STATE1 STATE2 STATE3 STATE4 0
329
-
encrypt_block a STATE4 STATE0 STATE1 STATE2 STATE3 1
330
-
encrypt_block a STATE3 STATE4 STATE0 STATE1 STATE2 2
331
-
encrypt_block a STATE2 STATE3 STATE4 STATE0 STATE1 3
332
-
encrypt_block a STATE1 STATE2 STATE3 STATE4 STATE0 4
420
+
.Lenc_loop:
421
+
encrypt_block STATE0 STATE1 STATE2 STATE3 STATE4 0
422
+
encrypt_block STATE4 STATE0 STATE1 STATE2 STATE3 1
423
+
encrypt_block STATE3 STATE4 STATE0 STATE1 STATE2 2
424
+
encrypt_block STATE2 STATE3 STATE4 STATE0 STATE1 3
425
+
encrypt_block STATE1 STATE2 STATE3 STATE4 STATE0 4
333
426
334
427
add $0x50, SRC
335
428
add $0x50, DST
336
-
jmp .Lenc_a_loop
337
-
338
-
.align 8
339
-
.Lenc_u_loop:
340
-
encrypt_block u STATE0 STATE1 STATE2 STATE3 STATE4 0
341
-
encrypt_block u STATE4 STATE0 STATE1 STATE2 STATE3 1
342
-
encrypt_block u STATE3 STATE4 STATE0 STATE1 STATE2 2
343
-
encrypt_block u STATE2 STATE3 STATE4 STATE0 STATE1 3
344
-
encrypt_block u STATE1 STATE2 STATE3 STATE4 STATE0 4
345
-
346
-
add $0x50, SRC
347
-
add $0x50, DST
348
-
jmp .Lenc_u_loop
429
+
jmp .Lenc_loop
349
430
350
431
/* store the state: */
351
432
.Lenc_out_0:
···
337
452
movdqu STATE1, 0x20(STATEP)
338
453
movdqu STATE2, 0x30(STATEP)
339
454
movdqu STATE3, 0x40(STATEP)
340
-
FRAME_END
341
455
RET
342
456
343
457
.Lenc_out_1:
···
345
461
movdqu STATE0, 0x20(STATEP)
346
462
movdqu STATE1, 0x30(STATEP)
347
463
movdqu STATE2, 0x40(STATEP)
348
-
FRAME_END
349
464
RET
350
465
351
466
.Lenc_out_2:
···
353
470
movdqu STATE4, 0x20(STATEP)
354
471
movdqu STATE0, 0x30(STATEP)
355
472
movdqu STATE1, 0x40(STATEP)
356
-
FRAME_END
357
473
RET
358
474
359
475
.Lenc_out_3:
···
361
479
movdqu STATE3, 0x20(STATEP)
362
480
movdqu STATE4, 0x30(STATEP)
363
481
movdqu STATE0, 0x40(STATEP)
364
-
FRAME_END
365
482
RET
366
483
367
484
.Lenc_out_4:
···
369
488
movdqu STATE2, 0x20(STATEP)
370
489
movdqu STATE3, 0x30(STATEP)
371
490
movdqu STATE4, 0x40(STATEP)
372
-
FRAME_END
373
-
RET
374
-
375
491
.Lenc_out:
376
-
FRAME_END
377
492
RET
378
-
SYM_FUNC_END(crypto_aegis128_aesni_enc)
493
+
SYM_FUNC_END(aegis128_aesni_enc)
379
494
380
495
/*
381
-
* void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length,
382
-
* const void *src, void *dst);
496
+
* void aegis128_aesni_enc_tail(struct aegis_state *state, const u8 *src,
497
+
* u8 *dst, unsigned int len);
383
498
*/
384
-
SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail)
385
-
FRAME_BEGIN
499
+
SYM_FUNC_START(aegis128_aesni_enc_tail)
500
+
.set STATEP, %rdi
501
+
.set SRC, %rsi
502
+
.set DST, %rdx
503
+
.set LEN, %ecx /* {load,store}_partial rely on this being %ecx */
386
504
387
505
/* load the state: */
388
506
movdqu 0x00(STATEP), STATE0
···
391
511
movdqu 0x40(STATEP), STATE4
392
512
393
513
/* encrypt message: */
394
-
call __load_partial
514
+
mov LEN, %r9d
515
+
load_partial
395
516
396
517
movdqa MSG, T0
397
518
pxor STATE1, T0
···
401
520
pand STATE3, T1
402
521
pxor T1, T0
403
522
404
-
call __store_partial
523
+
mov %r9d, LEN
524
+
store_partial T0
405
525
406
526
aegis128_update
407
527
pxor MSG, STATE4
···
413
531
movdqu STATE1, 0x20(STATEP)
414
532
movdqu STATE2, 0x30(STATEP)
415
533
movdqu STATE3, 0x40(STATEP)
416
-
417
-
FRAME_END
418
534
RET
419
-
SYM_FUNC_END(crypto_aegis128_aesni_enc_tail)
535
+
SYM_FUNC_END(aegis128_aesni_enc_tail)
420
536
421
-
.macro decrypt_block a s0 s1 s2 s3 s4 i
422
-
movdq\a (\i * 0x10)(SRC), MSG
537
+
.macro decrypt_block s0 s1 s2 s3 s4 i
538
+
movdqu (\i * 0x10)(SRC), MSG
423
539
pxor \s1, MSG
424
540
pxor \s4, MSG
425
541
movdqa \s2, T1
426
542
pand \s3, T1
427
543
pxor T1, MSG
428
-
movdq\a MSG, (\i * 0x10)(DST)
544
+
movdqu MSG, (\i * 0x10)(DST)
429
545
430
546
aegis128_update
431
547
pxor MSG, \s4
432
548
433
549
sub $0x10, LEN
434
-
cmp $0x10, LEN
435
-
jl .Ldec_out_\i
550
+
jz .Ldec_out_\i
436
551
.endm
437
552
438
553
/*
439
-
* void crypto_aegis128_aesni_dec(void *state, unsigned int length,
440
-
* const void *src, void *dst);
554
+
* void aegis128_aesni_dec(struct aegis_state *state, const u8 *src, u8 *dst,
555
+
* unsigned int len);
556
+
*
557
+
* len must be nonzero and a multiple of 16.
441
558
*/
442
-
SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec)
443
-
FRAME_BEGIN
444
-
445
-
cmp $0x10, LEN
446
-
jb .Ldec_out
559
+
SYM_FUNC_START(aegis128_aesni_dec)
560
+
.set STATEP, %rdi
561
+
.set SRC, %rsi
562
+
.set DST, %rdx
563
+
.set LEN, %ecx
447
564
448
565
/* load the state: */
449
566
movdqu 0x00(STATEP), STATE0
···
451
570
movdqu 0x30(STATEP), STATE3
452
571
movdqu 0x40(STATEP), STATE4
453
572
454
-
mov SRC, %r8
455
-
or DST, %r8
456
-
and $0xF, %r8
457
-
jnz .Ldec_u_loop
458
-
459
573
.align 8
460
-
.Ldec_a_loop:
461
-
decrypt_block a STATE0 STATE1 STATE2 STATE3 STATE4 0
462
-
decrypt_block a STATE4 STATE0 STATE1 STATE2 STATE3 1
463
-
decrypt_block a STATE3 STATE4 STATE0 STATE1 STATE2 2
464
-
decrypt_block a STATE2 STATE3 STATE4 STATE0 STATE1 3
465
-
decrypt_block a STATE1 STATE2 STATE3 STATE4 STATE0 4
574
+
.Ldec_loop:
575
+
decrypt_block STATE0 STATE1 STATE2 STATE3 STATE4 0
576
+
decrypt_block STATE4 STATE0 STATE1 STATE2 STATE3 1
577
+
decrypt_block STATE3 STATE4 STATE0 STATE1 STATE2 2
578
+
decrypt_block STATE2 STATE3 STATE4 STATE0 STATE1 3
579
+
decrypt_block STATE1 STATE2 STATE3 STATE4 STATE0 4
466
580
467
581
add $0x50, SRC
468
582
add $0x50, DST
469
-
jmp .Ldec_a_loop
470
-
471
-
.align 8
472
-
.Ldec_u_loop:
473
-
decrypt_block u STATE0 STATE1 STATE2 STATE3 STATE4 0
474
-
decrypt_block u STATE4 STATE0 STATE1 STATE2 STATE3 1
475
-
decrypt_block u STATE3 STATE4 STATE0 STATE1 STATE2 2
476
-
decrypt_block u STATE2 STATE3 STATE4 STATE0 STATE1 3
477
-
decrypt_block u STATE1 STATE2 STATE3 STATE4 STATE0 4
478
-
479
-
add $0x50, SRC
480
-
add $0x50, DST
481
-
jmp .Ldec_u_loop
583
+
jmp .Ldec_loop
482
584
483
585
/* store the state: */
484
586
.Ldec_out_0:
···
470
606
movdqu STATE1, 0x20(STATEP)
471
607
movdqu STATE2, 0x30(STATEP)
472
608
movdqu STATE3, 0x40(STATEP)
473
-
FRAME_END
474
609
RET
475
610
476
611
.Ldec_out_1:
···
478
615
movdqu STATE0, 0x20(STATEP)
479
616
movdqu STATE1, 0x30(STATEP)
480
617
movdqu STATE2, 0x40(STATEP)
481
-
FRAME_END
482
618
RET
483
619
484
620
.Ldec_out_2:
···
486
624
movdqu STATE4, 0x20(STATEP)
487
625
movdqu STATE0, 0x30(STATEP)
488
626
movdqu STATE1, 0x40(STATEP)
489
-
FRAME_END
490
627
RET
491
628
492
629
.Ldec_out_3:
···
494
633
movdqu STATE3, 0x20(STATEP)
495
634
movdqu STATE4, 0x30(STATEP)
496
635
movdqu STATE0, 0x40(STATEP)
497
-
FRAME_END
498
636
RET
499
637
500
638
.Ldec_out_4:
···
502
642
movdqu STATE2, 0x20(STATEP)
503
643
movdqu STATE3, 0x30(STATEP)
504
644
movdqu STATE4, 0x40(STATEP)
505
-
FRAME_END
506
-
RET
507
-
508
645
.Ldec_out:
509
-
FRAME_END
510
646
RET
511
-
SYM_FUNC_END(crypto_aegis128_aesni_dec)
647
+
SYM_FUNC_END(aegis128_aesni_dec)
512
648
513
649
/*
514
-
* void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length,
515
-
* const void *src, void *dst);
650
+
* void aegis128_aesni_dec_tail(struct aegis_state *state, const u8 *src,
651
+
* u8 *dst, unsigned int len);
516
652
*/
517
-
SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail)
518
-
FRAME_BEGIN
653
+
SYM_FUNC_START(aegis128_aesni_dec_tail)
654
+
.set STATEP, %rdi
655
+
.set SRC, %rsi
656
+
.set DST, %rdx
657
+
.set LEN, %ecx /* {load,store}_partial rely on this being %ecx */
519
658
520
659
/* load the state: */
521
660
movdqu 0x00(STATEP), STATE0
···
524
665
movdqu 0x40(STATEP), STATE4
525
666
526
667
/* decrypt message: */
527
-
call __load_partial
668
+
mov LEN, %r9d
669
+
load_partial
528
670
529
671
pxor STATE1, MSG
530
672
pxor STATE4, MSG
···
533
673
pand STATE3, T1
534
674
pxor T1, MSG
535
675
536
-
movdqa MSG, T0
537
-
call __store_partial
676
+
mov %r9d, LEN
677
+
store_partial MSG
538
678
539
679
/* mask with byte count: */
540
-
movq LEN, T0
541
-
punpcklbw T0, T0
542
-
punpcklbw T0, T0
543
-
punpcklbw T0, T0
544
-
punpcklbw T0, T0
545
-
movdqa .Laegis128_counter(%rip), T1
546
-
pcmpgtb T1, T0
680
+
lea .Lzeropad_mask+16(%rip), %rax
681
+
sub %r9, %rax
682
+
movdqu (%rax), T0
547
683
pand T0, MSG
548
684
549
685
aegis128_update
···
551
695
movdqu STATE1, 0x20(STATEP)
552
696
movdqu STATE2, 0x30(STATEP)
553
697
movdqu STATE3, 0x40(STATEP)
554
-
555
-
FRAME_END
556
698
RET
557
-
SYM_FUNC_END(crypto_aegis128_aesni_dec_tail)
699
+
SYM_FUNC_END(aegis128_aesni_dec_tail)
558
700
559
701
/*
560
-
* void crypto_aegis128_aesni_final(void *state, void *tag_xor,
561
-
* u64 assoclen, u64 cryptlen);
702
+
* void aegis128_aesni_final(struct aegis_state *state,
703
+
* struct aegis_block *tag_xor,
704
+
* unsigned int assoclen, unsigned int cryptlen);
562
705
*/
563
-
SYM_FUNC_START(crypto_aegis128_aesni_final)
564
-
FRAME_BEGIN
706
+
SYM_FUNC_START(aegis128_aesni_final)
707
+
.set STATEP, %rdi
708
+
.set TAG_XOR, %rsi
709
+
.set ASSOCLEN, %edx
710
+
.set CRYPTLEN, %ecx
565
711
566
712
/* load the state: */
567
713
movdqu 0x00(STATEP), STATE0
···
573
715
movdqu 0x40(STATEP), STATE4
574
716
575
717
/* prepare length block: */
576
-
movq %rdx, MSG
577
-
movq %rcx, T0
578
-
pslldq $8, T0
579
-
pxor T0, MSG
718
+
movd ASSOCLEN, MSG
719
+
pinsrd $2, CRYPTLEN, MSG
580
720
psllq $3, MSG /* multiply by 8 (to get bit count) */
581
721
582
722
pxor STATE3, MSG
···
589
733
aegis128_update; pxor MSG, STATE3
590
734
591
735
/* xor tag: */
592
-
movdqu (%rsi), MSG
736
+
movdqu (TAG_XOR), MSG
593
737
594
738
pxor STATE0, MSG
595
739
pxor STATE1, MSG
···
597
741
pxor STATE3, MSG
598
742
pxor STATE4, MSG
599
743
600
-
movdqu MSG, (%rsi)
601
-
602
-
FRAME_END
744
+
movdqu MSG, (TAG_XOR)
603
745
RET
604
-
SYM_FUNC_END(crypto_aegis128_aesni_final)
746
+
SYM_FUNC_END(aegis128_aesni_final)
+66
-79
arch/x86/crypto/aegis128-aesni-glue.c
+66
-79
arch/x86/crypto/aegis128-aesni-glue.c
···
1
1
// SPDX-License-Identifier: GPL-2.0-or-later
2
2
/*
3
3
* The AEGIS-128 Authenticated-Encryption Algorithm
4
-
* Glue for AES-NI + SSE2 implementation
4
+
* Glue for AES-NI + SSE4.1 implementation
5
5
*
6
6
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
7
7
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
···
23
23
#define AEGIS128_MIN_AUTH_SIZE 8
24
24
#define AEGIS128_MAX_AUTH_SIZE 16
25
25
26
-
asmlinkage void crypto_aegis128_aesni_init(void *state, void *key, void *iv);
27
-
28
-
asmlinkage void crypto_aegis128_aesni_ad(
29
-
void *state, unsigned int length, const void *data);
30
-
31
-
asmlinkage void crypto_aegis128_aesni_enc(
32
-
void *state, unsigned int length, const void *src, void *dst);
33
-
34
-
asmlinkage void crypto_aegis128_aesni_dec(
35
-
void *state, unsigned int length, const void *src, void *dst);
36
-
37
-
asmlinkage void crypto_aegis128_aesni_enc_tail(
38
-
void *state, unsigned int length, const void *src, void *dst);
39
-
40
-
asmlinkage void crypto_aegis128_aesni_dec_tail(
41
-
void *state, unsigned int length, const void *src, void *dst);
42
-
43
-
asmlinkage void crypto_aegis128_aesni_final(
44
-
void *state, void *tag_xor, unsigned int cryptlen,
45
-
unsigned int assoclen);
46
-
47
26
struct aegis_block {
48
27
u8 bytes[AEGIS128_BLOCK_SIZE] __aligned(AEGIS128_BLOCK_ALIGN);
49
28
};
···
35
56
struct aegis_block key;
36
57
};
37
58
38
-
struct aegis_crypt_ops {
39
-
int (*skcipher_walk_init)(struct skcipher_walk *walk,
40
-
struct aead_request *req, bool atomic);
59
+
asmlinkage void aegis128_aesni_init(struct aegis_state *state,
60
+
const struct aegis_block *key,
61
+
const u8 iv[AEGIS128_NONCE_SIZE]);
41
62
42
-
void (*crypt_blocks)(void *state, unsigned int length, const void *src,
43
-
void *dst);
44
-
void (*crypt_tail)(void *state, unsigned int length, const void *src,
45
-
void *dst);
46
-
};
63
+
asmlinkage void aegis128_aesni_ad(struct aegis_state *state, const u8 *data,
64
+
unsigned int len);
65
+
66
+
asmlinkage void aegis128_aesni_enc(struct aegis_state *state, const u8 *src,
67
+
u8 *dst, unsigned int len);
68
+
69
+
asmlinkage void aegis128_aesni_dec(struct aegis_state *state, const u8 *src,
70
+
u8 *dst, unsigned int len);
71
+
72
+
asmlinkage void aegis128_aesni_enc_tail(struct aegis_state *state,
73
+
const u8 *src, u8 *dst,
74
+
unsigned int len);
75
+
76
+
asmlinkage void aegis128_aesni_dec_tail(struct aegis_state *state,
77
+
const u8 *src, u8 *dst,
78
+
unsigned int len);
79
+
80
+
asmlinkage void aegis128_aesni_final(struct aegis_state *state,
81
+
struct aegis_block *tag_xor,
82
+
unsigned int assoclen,
83
+
unsigned int cryptlen);
47
84
48
85
static void crypto_aegis128_aesni_process_ad(
49
86
struct aegis_state *state, struct scatterlist *sg_src,
···
80
85
if (pos > 0) {
81
86
unsigned int fill = AEGIS128_BLOCK_SIZE - pos;
82
87
memcpy(buf.bytes + pos, src, fill);
83
-
crypto_aegis128_aesni_ad(state,
84
-
AEGIS128_BLOCK_SIZE,
85
-
buf.bytes);
88
+
aegis128_aesni_ad(state, buf.bytes,
89
+
AEGIS128_BLOCK_SIZE);
86
90
pos = 0;
87
91
left -= fill;
88
92
src += fill;
89
93
}
90
94
91
-
crypto_aegis128_aesni_ad(state, left, src);
92
-
95
+
aegis128_aesni_ad(state, src,
96
+
left & ~(AEGIS128_BLOCK_SIZE - 1));
93
97
src += left & ~(AEGIS128_BLOCK_SIZE - 1);
94
98
left &= AEGIS128_BLOCK_SIZE - 1;
95
99
}
···
104
110
105
111
if (pos > 0) {
106
112
memset(buf.bytes + pos, 0, AEGIS128_BLOCK_SIZE - pos);
107
-
crypto_aegis128_aesni_ad(state, AEGIS128_BLOCK_SIZE, buf.bytes);
113
+
aegis128_aesni_ad(state, buf.bytes, AEGIS128_BLOCK_SIZE);
108
114
}
109
115
}
110
116
111
-
static void crypto_aegis128_aesni_process_crypt(
112
-
struct aegis_state *state, struct skcipher_walk *walk,
113
-
const struct aegis_crypt_ops *ops)
117
+
static __always_inline void
118
+
crypto_aegis128_aesni_process_crypt(struct aegis_state *state,
119
+
struct skcipher_walk *walk, bool enc)
114
120
{
115
121
while (walk->nbytes >= AEGIS128_BLOCK_SIZE) {
116
-
ops->crypt_blocks(state,
117
-
round_down(walk->nbytes, AEGIS128_BLOCK_SIZE),
118
-
walk->src.virt.addr, walk->dst.virt.addr);
122
+
if (enc)
123
+
aegis128_aesni_enc(state, walk->src.virt.addr,
124
+
walk->dst.virt.addr,
125
+
round_down(walk->nbytes,
126
+
AEGIS128_BLOCK_SIZE));
127
+
else
128
+
aegis128_aesni_dec(state, walk->src.virt.addr,
129
+
walk->dst.virt.addr,
130
+
round_down(walk->nbytes,
131
+
AEGIS128_BLOCK_SIZE));
119
132
skcipher_walk_done(walk, walk->nbytes % AEGIS128_BLOCK_SIZE);
120
133
}
121
134
122
135
if (walk->nbytes) {
123
-
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
124
-
walk->dst.virt.addr);
136
+
if (enc)
137
+
aegis128_aesni_enc_tail(state, walk->src.virt.addr,
138
+
walk->dst.virt.addr,
139
+
walk->nbytes);
140
+
else
141
+
aegis128_aesni_dec_tail(state, walk->src.virt.addr,
142
+
walk->dst.virt.addr,
143
+
walk->nbytes);
125
144
skcipher_walk_done(walk, 0);
126
145
}
127
146
}
···
169
162
return 0;
170
163
}
171
164
172
-
static void crypto_aegis128_aesni_crypt(struct aead_request *req,
173
-
struct aegis_block *tag_xor,
174
-
unsigned int cryptlen,
175
-
const struct aegis_crypt_ops *ops)
165
+
static __always_inline void
166
+
crypto_aegis128_aesni_crypt(struct aead_request *req,
167
+
struct aegis_block *tag_xor,
168
+
unsigned int cryptlen, bool enc)
176
169
{
177
170
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
178
171
struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm);
179
172
struct skcipher_walk walk;
180
173
struct aegis_state state;
181
174
182
-
ops->skcipher_walk_init(&walk, req, true);
175
+
if (enc)
176
+
skcipher_walk_aead_encrypt(&walk, req, true);
177
+
else
178
+
skcipher_walk_aead_decrypt(&walk, req, true);
183
179
184
180
kernel_fpu_begin();
185
181
186
-
crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv);
182
+
aegis128_aesni_init(&state, &ctx->key, req->iv);
187
183
crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen);
188
-
crypto_aegis128_aesni_process_crypt(&state, &walk, ops);
189
-
crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
184
+
crypto_aegis128_aesni_process_crypt(&state, &walk, enc);
185
+
aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
190
186
191
187
kernel_fpu_end();
192
188
}
193
189
194
190
static int crypto_aegis128_aesni_encrypt(struct aead_request *req)
195
191
{
196
-
static const struct aegis_crypt_ops OPS = {
197
-
.skcipher_walk_init = skcipher_walk_aead_encrypt,
198
-
.crypt_blocks = crypto_aegis128_aesni_enc,
199
-
.crypt_tail = crypto_aegis128_aesni_enc_tail,
200
-
};
201
-
202
192
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
203
193
struct aegis_block tag = {};
204
194
unsigned int authsize = crypto_aead_authsize(tfm);
205
195
unsigned int cryptlen = req->cryptlen;
206
196
207
-
crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS);
197
+
crypto_aegis128_aesni_crypt(req, &tag, cryptlen, true);
208
198
209
199
scatterwalk_map_and_copy(tag.bytes, req->dst,
210
200
req->assoclen + cryptlen, authsize, 1);
···
212
208
{
213
209
static const struct aegis_block zeros = {};
214
210
215
-
static const struct aegis_crypt_ops OPS = {
216
-
.skcipher_walk_init = skcipher_walk_aead_decrypt,
217
-
.crypt_blocks = crypto_aegis128_aesni_dec,
218
-
.crypt_tail = crypto_aegis128_aesni_dec_tail,
219
-
};
220
-
221
211
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
222
212
struct aegis_block tag;
223
213
unsigned int authsize = crypto_aead_authsize(tfm);
···
220
222
scatterwalk_map_and_copy(tag.bytes, req->src,
221
223
req->assoclen + cryptlen, authsize, 0);
222
224
223
-
crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS);
225
+
crypto_aegis128_aesni_crypt(req, &tag, cryptlen, false);
224
226
225
227
return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0;
226
-
}
227
-
228
-
static int crypto_aegis128_aesni_init_tfm(struct crypto_aead *aead)
229
-
{
230
-
return 0;
231
-
}
232
-
233
-
static void crypto_aegis128_aesni_exit_tfm(struct crypto_aead *aead)
234
-
{
235
228
}
236
229
237
230
static struct aead_alg crypto_aegis128_aesni_alg = {
···
230
241
.setauthsize = crypto_aegis128_aesni_setauthsize,
231
242
.encrypt = crypto_aegis128_aesni_encrypt,
232
243
.decrypt = crypto_aegis128_aesni_decrypt,
233
-
.init = crypto_aegis128_aesni_init_tfm,
234
-
.exit = crypto_aegis128_aesni_exit_tfm,
235
244
236
245
.ivsize = AEGIS128_NONCE_SIZE,
237
246
.maxauthsize = AEGIS128_MAX_AUTH_SIZE,
···
254
267
255
268
static int __init crypto_aegis128_aesni_module_init(void)
256
269
{
257
-
if (!boot_cpu_has(X86_FEATURE_XMM2) ||
270
+
if (!boot_cpu_has(X86_FEATURE_XMM4_1) ||
258
271
!boot_cpu_has(X86_FEATURE_AES) ||
259
272
!cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL))
260
273
return -ENODEV;
···
273
286
274
287
MODULE_LICENSE("GPL");
275
288
MODULE_AUTHOR("Ondrej Mosnacek <omosnacek@gmail.com>");
276
-
MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE2 implementation");
289
+
MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE4.1 implementation");
277
290
MODULE_ALIAS_CRYPTO("aegis128");
278
291
MODULE_ALIAS_CRYPTO("aegis128-aesni");
+1
-1
arch/x86/crypto/aesni-intel_glue.c
+1
-1
arch/x86/crypto/aesni-intel_glue.c
-76
arch/x86/crypto/cast5-avx-x86_64-asm_64.S
-76
arch/x86/crypto/cast5-avx-x86_64-asm_64.S
···
487
487
FRAME_END
488
488
RET;
489
489
SYM_FUNC_END(cast5_cbc_dec_16way)
490
-
491
-
SYM_FUNC_START(cast5_ctr_16way)
492
-
/* input:
493
-
* %rdi: ctx
494
-
* %rsi: dst
495
-
* %rdx: src
496
-
* %rcx: iv (big endian, 64bit)
497
-
*/
498
-
FRAME_BEGIN
499
-
pushq %r12;
500
-
pushq %r15;
501
-
502
-
movq %rdi, CTX;
503
-
movq %rsi, %r11;
504
-
movq %rdx, %r12;
505
-
506
-
vpcmpeqd RTMP, RTMP, RTMP;
507
-
vpsrldq $8, RTMP, RTMP; /* low: -1, high: 0 */
508
-
509
-
vpcmpeqd RKR, RKR, RKR;
510
-
vpaddq RKR, RKR, RKR; /* low: -2, high: -2 */
511
-
vmovdqa .Lbswap_iv_mask(%rip), R1ST;
512
-
vmovdqa .Lbswap128_mask(%rip), RKM;
513
-
514
-
/* load IV and byteswap */
515
-
vmovq (%rcx), RX;
516
-
vpshufb R1ST, RX, RX;
517
-
518
-
/* construct IVs */
519
-
vpsubq RTMP, RX, RX; /* le: IV1, IV0 */
520
-
vpshufb RKM, RX, RL1; /* be: IV0, IV1 */
521
-
vpsubq RKR, RX, RX;
522
-
vpshufb RKM, RX, RR1; /* be: IV2, IV3 */
523
-
vpsubq RKR, RX, RX;
524
-
vpshufb RKM, RX, RL2; /* be: IV4, IV5 */
525
-
vpsubq RKR, RX, RX;
526
-
vpshufb RKM, RX, RR2; /* be: IV6, IV7 */
527
-
vpsubq RKR, RX, RX;
528
-
vpshufb RKM, RX, RL3; /* be: IV8, IV9 */
529
-
vpsubq RKR, RX, RX;
530
-
vpshufb RKM, RX, RR3; /* be: IV10, IV11 */
531
-
vpsubq RKR, RX, RX;
532
-
vpshufb RKM, RX, RL4; /* be: IV12, IV13 */
533
-
vpsubq RKR, RX, RX;
534
-
vpshufb RKM, RX, RR4; /* be: IV14, IV15 */
535
-
536
-
/* store last IV */
537
-
vpsubq RTMP, RX, RX; /* le: IV16, IV14 */
538
-
vpshufb R1ST, RX, RX; /* be: IV16, IV16 */
539
-
vmovq RX, (%rcx);
540
-
541
-
call __cast5_enc_blk16;
542
-
543
-
/* dst = src ^ iv */
544
-
vpxor (0*16)(%r12), RR1, RR1;
545
-
vpxor (1*16)(%r12), RL1, RL1;
546
-
vpxor (2*16)(%r12), RR2, RR2;
547
-
vpxor (3*16)(%r12), RL2, RL2;
548
-
vpxor (4*16)(%r12), RR3, RR3;
549
-
vpxor (5*16)(%r12), RL3, RL3;
550
-
vpxor (6*16)(%r12), RR4, RR4;
551
-
vpxor (7*16)(%r12), RL4, RL4;
552
-
vmovdqu RR1, (0*16)(%r11);
553
-
vmovdqu RL1, (1*16)(%r11);
554
-
vmovdqu RR2, (2*16)(%r11);
555
-
vmovdqu RL2, (3*16)(%r11);
556
-
vmovdqu RR3, (4*16)(%r11);
557
-
vmovdqu RL3, (5*16)(%r11);
558
-
vmovdqu RR4, (6*16)(%r11);
559
-
vmovdqu RL4, (7*16)(%r11);
560
-
561
-
popq %r15;
562
-
popq %r12;
563
-
FRAME_END
564
-
RET;
565
-
SYM_FUNC_END(cast5_ctr_16way)
+1
-1
arch/x86/crypto/crc32c-intel_glue.c
+1
-1
arch/x86/crypto/crc32c-intel_glue.c
+122
-226
arch/x86/crypto/crc32c-pcl-intel-asm_64.S
+122
-226
arch/x86/crypto/crc32c-pcl-intel-asm_64.S
···
7
7
* http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-paper.pdf
8
8
*
9
9
* Copyright (C) 2012 Intel Corporation.
10
+
* Copyright 2024 Google LLC
10
11
*
11
12
* Authors:
12
13
* Wajdi Feghali <wajdi.k.feghali@intel.com>
···
45
44
*/
46
45
47
46
#include <linux/linkage.h>
48
-
#include <asm/nospec-branch.h>
49
47
50
48
## ISCSI CRC 32 Implementation with crc32 and pclmulqdq Instruction
51
49
52
-
.macro LABEL prefix n
53
-
.L\prefix\n\():
54
-
.endm
55
-
56
-
.macro JMPTBL_ENTRY i
57
-
.quad .Lcrc_\i
58
-
.endm
59
-
60
-
.macro JNC_LESS_THAN j
61
-
jnc .Lless_than_\j
62
-
.endm
63
-
64
-
# Define threshold where buffers are considered "small" and routed to more
65
-
# efficient "by-1" code. This "by-1" code only handles up to 255 bytes, so
66
-
# SMALL_SIZE can be no larger than 255.
67
-
50
+
# Define threshold below which buffers are considered "small" and routed to
51
+
# regular CRC code that does not interleave the CRC instructions.
68
52
#define SMALL_SIZE 200
69
53
70
-
.if (SMALL_SIZE > 255)
71
-
.error "SMALL_ SIZE must be < 256"
72
-
.endif
73
-
74
-
# unsigned int crc_pcl(u8 *buffer, int len, unsigned int crc_init);
54
+
# unsigned int crc_pcl(const u8 *buffer, unsigned int len, unsigned int crc_init);
75
55
76
56
.text
77
57
SYM_FUNC_START(crc_pcl)
78
-
#define bufp rdi
79
-
#define bufp_dw %edi
80
-
#define bufp_w %di
81
-
#define bufp_b %dil
82
-
#define bufptmp %rcx
83
-
#define block_0 %rcx
84
-
#define block_1 %rdx
85
-
#define block_2 %r11
86
-
#define len %rsi
87
-
#define len_dw %esi
88
-
#define len_w %si
89
-
#define len_b %sil
90
-
#define crc_init_arg %rdx
91
-
#define tmp %rbx
92
-
#define crc_init %r8
93
-
#define crc_init_dw %r8d
94
-
#define crc1 %r9
95
-
#define crc2 %r10
58
+
#define bufp %rdi
59
+
#define bufp_d %edi
60
+
#define len %esi
61
+
#define crc_init %edx
62
+
#define crc_init_q %rdx
63
+
#define n_misaligned %ecx /* overlaps chunk_bytes! */
64
+
#define n_misaligned_q %rcx
65
+
#define chunk_bytes %ecx /* overlaps n_misaligned! */
66
+
#define chunk_bytes_q %rcx
67
+
#define crc1 %r8
68
+
#define crc2 %r9
96
69
97
-
pushq %rbx
98
-
pushq %rdi
99
-
pushq %rsi
100
-
101
-
## Move crc_init for Linux to a different
102
-
mov crc_init_arg, crc_init
70
+
cmp $SMALL_SIZE, len
71
+
jb .Lsmall
103
72
104
73
################################################################
105
74
## 1) ALIGN:
106
75
################################################################
107
-
108
-
mov %bufp, bufptmp # rdi = *buf
109
-
neg %bufp
110
-
and $7, %bufp # calculate the unalignment amount of
76
+
mov bufp_d, n_misaligned
77
+
neg n_misaligned
78
+
and $7, n_misaligned # calculate the misalignment amount of
111
79
# the address
112
-
je .Lproc_block # Skip if aligned
80
+
je .Laligned # Skip if aligned
113
81
114
-
## If len is less than 8 and we're unaligned, we need to jump
115
-
## to special code to avoid reading beyond the end of the buffer
116
-
cmp $8, len
117
-
jae .Ldo_align
118
-
# less_than_8 expects length in upper 3 bits of len_dw
119
-
# less_than_8_post_shl1 expects length = carryflag * 8 + len_dw[31:30]
120
-
shl $32-3+1, len_dw
121
-
jmp .Lless_than_8_post_shl1
122
-
82
+
# Process 1 <= n_misaligned <= 7 bytes individually in order to align
83
+
# the remaining data to an 8-byte boundary.
123
84
.Ldo_align:
124
-
#### Calculate CRC of unaligned bytes of the buffer (if any)
125
-
movq (bufptmp), tmp # load a quadward from the buffer
126
-
add %bufp, bufptmp # align buffer pointer for quadword
127
-
# processing
128
-
sub %bufp, len # update buffer length
85
+
movq (bufp), %rax
86
+
add n_misaligned_q, bufp
87
+
sub n_misaligned, len
129
88
.Lalign_loop:
130
-
crc32b %bl, crc_init_dw # compute crc32 of 1-byte
131
-
shr $8, tmp # get next byte
132
-
dec %bufp
89
+
crc32b %al, crc_init # compute crc32 of 1-byte
90
+
shr $8, %rax # get next byte
91
+
dec n_misaligned
133
92
jne .Lalign_loop
134
-
135
-
.Lproc_block:
93
+
.Laligned:
136
94
137
95
################################################################
138
-
## 2) PROCESS BLOCKS:
96
+
## 2) PROCESS BLOCK:
139
97
################################################################
140
98
141
-
## compute num of bytes to be processed
142
-
movq len, tmp # save num bytes in tmp
143
-
144
-
cmpq $128*24, len
99
+
cmp $128*24, len
145
100
jae .Lfull_block
146
101
147
-
.Lcontinue_block:
148
-
cmpq $SMALL_SIZE, len
149
-
jb .Lsmall
102
+
.Lpartial_block:
103
+
# Compute floor(len / 24) to get num qwords to process from each lane.
104
+
imul $2731, len, %eax # 2731 = ceil(2^16 / 24)
105
+
shr $16, %eax
106
+
jmp .Lcrc_3lanes
150
107
151
-
## len < 128*24
152
-
movq $2731, %rax # 2731 = ceil(2^16 / 24)
153
-
mul len_dw
154
-
shrq $16, %rax
155
-
156
-
## eax contains floor(bytes / 24) = num 24-byte chunks to do
157
-
158
-
## process rax 24-byte chunks (128 >= rax >= 0)
159
-
160
-
## compute end address of each block
161
-
## block 0 (base addr + RAX * 8)
162
-
## block 1 (base addr + RAX * 16)
163
-
## block 2 (base addr + RAX * 24)
164
-
lea (bufptmp, %rax, 8), block_0
165
-
lea (block_0, %rax, 8), block_1
166
-
lea (block_1, %rax, 8), block_2
167
-
168
-
xor crc1, crc1
169
-
xor crc2, crc2
170
-
171
-
## branch into array
172
-
leaq jump_table(%rip), %bufp
173
-
mov (%bufp,%rax,8), %bufp
174
-
JMP_NOSPEC bufp
175
-
176
-
################################################################
177
-
## 2a) PROCESS FULL BLOCKS:
178
-
################################################################
179
108
.Lfull_block:
180
-
movl $128,%eax
181
-
lea 128*8*2(block_0), block_1
182
-
lea 128*8*3(block_0), block_2
183
-
add $128*8*1, block_0
109
+
# Processing 128 qwords from each lane.
110
+
mov $128, %eax
184
111
185
-
xor crc1,crc1
112
+
################################################################
113
+
## 3) CRC each of three lanes:
114
+
################################################################
115
+
116
+
.Lcrc_3lanes:
117
+
xor crc1,crc1
186
118
xor crc2,crc2
119
+
mov %eax, chunk_bytes
120
+
shl $3, chunk_bytes # num bytes to process from each lane
121
+
sub $5, %eax # 4 for 4x_loop, 1 for special last iter
122
+
jl .Lcrc_3lanes_4x_done
187
123
188
-
# Fall through into top of crc array (crc_128)
124
+
# Unroll the loop by a factor of 4 to reduce the overhead of the loop
125
+
# bookkeeping instructions, which can compete with crc32q for the ALUs.
126
+
.Lcrc_3lanes_4x_loop:
127
+
crc32q (bufp), crc_init_q
128
+
crc32q (bufp,chunk_bytes_q), crc1
129
+
crc32q (bufp,chunk_bytes_q,2), crc2
130
+
crc32q 8(bufp), crc_init_q
131
+
crc32q 8(bufp,chunk_bytes_q), crc1
132
+
crc32q 8(bufp,chunk_bytes_q,2), crc2
133
+
crc32q 16(bufp), crc_init_q
134
+
crc32q 16(bufp,chunk_bytes_q), crc1
135
+
crc32q 16(bufp,chunk_bytes_q,2), crc2
136
+
crc32q 24(bufp), crc_init_q
137
+
crc32q 24(bufp,chunk_bytes_q), crc1
138
+
crc32q 24(bufp,chunk_bytes_q,2), crc2
139
+
add $32, bufp
140
+
sub $4, %eax
141
+
jge .Lcrc_3lanes_4x_loop
189
142
190
-
################################################################
191
-
## 3) CRC Array:
192
-
################################################################
143
+
.Lcrc_3lanes_4x_done:
144
+
add $4, %eax
145
+
jz .Lcrc_3lanes_last_qword
193
146
194
-
i=128
195
-
.rept 128-1
196
-
.altmacro
197
-
LABEL crc_ %i
198
-
.noaltmacro
199
-
ENDBR
200
-
crc32q -i*8(block_0), crc_init
201
-
crc32q -i*8(block_1), crc1
202
-
crc32q -i*8(block_2), crc2
203
-
i=(i-1)
204
-
.endr
147
+
.Lcrc_3lanes_1x_loop:
148
+
crc32q (bufp), crc_init_q
149
+
crc32q (bufp,chunk_bytes_q), crc1
150
+
crc32q (bufp,chunk_bytes_q,2), crc2
151
+
add $8, bufp
152
+
dec %eax
153
+
jnz .Lcrc_3lanes_1x_loop
205
154
206
-
.altmacro
207
-
LABEL crc_ %i
208
-
.noaltmacro
209
-
ENDBR
210
-
crc32q -i*8(block_0), crc_init
211
-
crc32q -i*8(block_1), crc1
212
-
# SKIP crc32 -i*8(block_2), crc2 ; Don't do this one yet
213
-
214
-
mov block_2, block_0
155
+
.Lcrc_3lanes_last_qword:
156
+
crc32q (bufp), crc_init_q
157
+
crc32q (bufp,chunk_bytes_q), crc1
158
+
# SKIP crc32q (bufp,chunk_bytes_q,2), crc2 ; Don't do this one yet
215
159
216
160
################################################################
217
161
## 4) Combine three results:
218
162
################################################################
219
163
220
-
lea (K_table-8)(%rip), %bufp # first entry is for idx 1
221
-
shlq $3, %rax # rax *= 8
222
-
pmovzxdq (%bufp,%rax), %xmm0 # 2 consts: K1:K2
223
-
leal (%eax,%eax,2), %eax # rax *= 3 (total *24)
224
-
subq %rax, tmp # tmp -= rax*24
164
+
lea (K_table-8)(%rip), %rax # first entry is for idx 1
165
+
pmovzxdq (%rax,chunk_bytes_q), %xmm0 # 2 consts: K1:K2
166
+
lea (chunk_bytes,chunk_bytes,2), %eax # chunk_bytes * 3
167
+
sub %eax, len # len -= chunk_bytes * 3
225
168
226
-
movq crc_init, %xmm1 # CRC for block 1
169
+
movq crc_init_q, %xmm1 # CRC for block 1
227
170
pclmulqdq $0x00, %xmm0, %xmm1 # Multiply by K2
228
171
229
172
movq crc1, %xmm2 # CRC for block 2
···
175
230
176
231
pxor %xmm2,%xmm1
177
232
movq %xmm1, %rax
178
-
xor -i*8(block_2), %rax
179
-
mov crc2, crc_init
180
-
crc32 %rax, crc_init
233
+
xor (bufp,chunk_bytes_q,2), %rax
234
+
mov crc2, crc_init_q
235
+
crc32 %rax, crc_init_q
236
+
lea 8(bufp,chunk_bytes_q,2), bufp
181
237
182
238
################################################################
183
-
## 5) Check for end:
239
+
## 5) If more blocks remain, goto (2):
184
240
################################################################
185
241
186
-
LABEL crc_ 0
187
-
ENDBR
188
-
mov tmp, len
189
-
cmp $128*24, tmp
190
-
jae .Lfull_block
191
-
cmp $24, tmp
192
-
jae .Lcontinue_block
193
-
194
-
.Lless_than_24:
195
-
shl $32-4, len_dw # less_than_16 expects length
196
-
# in upper 4 bits of len_dw
197
-
jnc .Lless_than_16
198
-
crc32q (bufptmp), crc_init
199
-
crc32q 8(bufptmp), crc_init
200
-
jz .Ldo_return
201
-
add $16, bufptmp
202
-
# len is less than 8 if we got here
203
-
# less_than_8 expects length in upper 3 bits of len_dw
204
-
# less_than_8_post_shl1 expects length = carryflag * 8 + len_dw[31:30]
205
-
shl $2, len_dw
206
-
jmp .Lless_than_8_post_shl1
242
+
cmp $128*24, len
243
+
jae .Lfull_block
244
+
cmp $SMALL_SIZE, len
245
+
jae .Lpartial_block
207
246
208
247
#######################################################################
209
-
## 6) LESS THAN 256-bytes REMAIN AT THIS POINT (8-bits of len are full)
248
+
## 6) Process any remainder without interleaving:
210
249
#######################################################################
211
250
.Lsmall:
212
-
shl $32-8, len_dw # Prepare len_dw for less_than_256
213
-
j=256
214
-
.rept 5 # j = {256, 128, 64, 32, 16}
215
-
.altmacro
216
-
LABEL less_than_ %j # less_than_j: Length should be in
217
-
# upper lg(j) bits of len_dw
218
-
j=(j/2)
219
-
shl $1, len_dw # Get next MSB
220
-
JNC_LESS_THAN %j
221
-
.noaltmacro
222
-
i=0
223
-
.rept (j/8)
224
-
crc32q i(bufptmp), crc_init # Compute crc32 of 8-byte data
225
-
i=i+8
226
-
.endr
227
-
jz .Ldo_return # Return if remaining length is zero
228
-
add $j, bufptmp # Advance buf
229
-
.endr
230
-
231
-
.Lless_than_8: # Length should be stored in
232
-
# upper 3 bits of len_dw
233
-
shl $1, len_dw
234
-
.Lless_than_8_post_shl1:
235
-
jnc .Lless_than_4
236
-
crc32l (bufptmp), crc_init_dw # CRC of 4 bytes
237
-
jz .Ldo_return # return if remaining data is zero
238
-
add $4, bufptmp
239
-
.Lless_than_4: # Length should be stored in
240
-
# upper 2 bits of len_dw
241
-
shl $1, len_dw
242
-
jnc .Lless_than_2
243
-
crc32w (bufptmp), crc_init_dw # CRC of 2 bytes
244
-
jz .Ldo_return # return if remaining data is zero
245
-
add $2, bufptmp
246
-
.Lless_than_2: # Length should be stored in the MSB
247
-
# of len_dw
248
-
shl $1, len_dw
249
-
jnc .Lless_than_1
250
-
crc32b (bufptmp), crc_init_dw # CRC of 1 byte
251
-
.Lless_than_1: # Length should be zero
252
-
.Ldo_return:
253
-
movq crc_init, %rax
254
-
popq %rsi
255
-
popq %rdi
256
-
popq %rbx
251
+
test len, len
252
+
jz .Ldone
253
+
mov len, %eax
254
+
shr $3, %eax
255
+
jz .Ldo_dword
256
+
.Ldo_qwords:
257
+
crc32q (bufp), crc_init_q
258
+
add $8, bufp
259
+
dec %eax
260
+
jnz .Ldo_qwords
261
+
.Ldo_dword:
262
+
test $4, len
263
+
jz .Ldo_word
264
+
crc32l (bufp), crc_init
265
+
add $4, bufp
266
+
.Ldo_word:
267
+
test $2, len
268
+
jz .Ldo_byte
269
+
crc32w (bufp), crc_init
270
+
add $2, bufp
271
+
.Ldo_byte:
272
+
test $1, len
273
+
jz .Ldone
274
+
crc32b (bufp), crc_init
275
+
.Ldone:
276
+
mov crc_init, %eax
257
277
RET
258
278
SYM_FUNC_END(crc_pcl)
259
279
260
280
.section .rodata, "a", @progbits
261
-
################################################################
262
-
## jump table Table is 129 entries x 2 bytes each
263
-
################################################################
264
-
.align 4
265
-
jump_table:
266
-
i=0
267
-
.rept 129
268
-
.altmacro
269
-
JMPTBL_ENTRY %i
270
-
.noaltmacro
271
-
i=i+1
272
-
.endr
273
-
274
-
275
281
################################################################
276
282
## PCLMULQDQ tables
277
283
## Table is 128 entries x 2 words (8 bytes) each
+4
-3
crypto/Kconfig
+4
-3
crypto/Kconfig
···
250
250
tristate "RSA (Rivest-Shamir-Adleman)"
251
251
select CRYPTO_AKCIPHER
252
252
select CRYPTO_MANAGER
253
+
select CRYPTO_SIG
253
254
select MPILIB
254
255
select ASN1
255
256
help
···
291
290
config CRYPTO_ECDSA
292
291
tristate "ECDSA (Elliptic Curve Digital Signature Algorithm)"
293
292
select CRYPTO_ECC
294
-
select CRYPTO_AKCIPHER
293
+
select CRYPTO_SIG
295
294
select ASN1
296
295
help
297
296
ECDSA (Elliptic Curve Digital Signature Algorithm) (FIPS 186,
298
297
ISO/IEC 14888-3)
299
-
using curves P-192, P-256, and P-384
298
+
using curves P-192, P-256, P-384 and P-521
300
299
301
300
Only signature verification is implemented.
302
301
303
302
config CRYPTO_ECRDSA
304
303
tristate "EC-RDSA (Elliptic Curve Russian Digital Signature Algorithm)"
305
304
select CRYPTO_ECC
306
-
select CRYPTO_AKCIPHER
305
+
select CRYPTO_SIG
307
306
select CRYPTO_STREEBOG
308
307
select OID_REGISTRY
309
308
select ASN1
+6
-1
crypto/Makefile
+6
-1
crypto/Makefile
···
48
48
rsa_generic-y += rsa.o
49
49
rsa_generic-y += rsa_helper.o
50
50
rsa_generic-y += rsa-pkcs1pad.o
51
+
rsa_generic-y += rsassa-pkcs1.o
51
52
obj-$(CONFIG_CRYPTO_RSA) += rsa_generic.o
52
53
53
54
$(obj)/ecdsasignature.asn1.o: $(obj)/ecdsasignature.asn1.c $(obj)/ecdsasignature.asn1.h
54
-
$(obj)/ecdsa.o: $(obj)/ecdsasignature.asn1.h
55
+
$(obj)/ecdsa-x962.o: $(obj)/ecdsasignature.asn1.h
55
56
ecdsa_generic-y += ecdsa.o
57
+
ecdsa_generic-y += ecdsa-x962.o
58
+
ecdsa_generic-y += ecdsa-p1363.o
56
59
ecdsa_generic-y += ecdsasignature.asn1.o
57
60
obj-$(CONFIG_CRYPTO_ECDSA) += ecdsa_generic.o
58
61
···
155
152
obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
156
153
obj-$(CONFIG_CRYPTO_CRC32C) += crc32c_generic.o
157
154
obj-$(CONFIG_CRYPTO_CRC32) += crc32_generic.o
155
+
CFLAGS_crc32c_generic.o += -DARCH=$(ARCH)
156
+
CFLAGS_crc32_generic.o += -DARCH=$(ARCH)
158
157
obj-$(CONFIG_CRYPTO_CRCT10DIF) += crct10dif_common.o crct10dif_generic.o
159
158
obj-$(CONFIG_CRYPTO_CRC64_ROCKSOFT) += crc64_rocksoft_generic.o
160
159
obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o authencesn.o
+19
-45
crypto/akcipher.c
+19
-45
crypto/akcipher.c
···
20
20
21
21
#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000e
22
22
23
+
struct crypto_akcipher_sync_data {
24
+
struct crypto_akcipher *tfm;
25
+
const void *src;
26
+
void *dst;
27
+
unsigned int slen;
28
+
unsigned int dlen;
29
+
30
+
struct akcipher_request *req;
31
+
struct crypto_wait cwait;
32
+
struct scatterlist sg;
33
+
u8 *buf;
34
+
};
35
+
23
36
static int __maybe_unused crypto_akcipher_report(
24
37
struct sk_buff *skb, struct crypto_alg *alg)
25
38
{
···
139
126
{
140
127
struct crypto_alg *base = &alg->base;
141
128
142
-
if (!alg->sign)
143
-
alg->sign = akcipher_default_op;
144
-
if (!alg->verify)
145
-
alg->verify = akcipher_default_op;
146
129
if (!alg->encrypt)
147
130
alg->encrypt = akcipher_default_op;
148
131
if (!alg->decrypt)
···
167
158
}
168
159
EXPORT_SYMBOL_GPL(akcipher_register_instance);
169
160
170
-
int crypto_akcipher_sync_prep(struct crypto_akcipher_sync_data *data)
161
+
static int crypto_akcipher_sync_prep(struct crypto_akcipher_sync_data *data)
171
162
{
172
163
unsigned int reqsize = crypto_akcipher_reqsize(data->tfm);
173
164
struct akcipher_request *req;
···
176
167
unsigned int len;
177
168
u8 *buf;
178
169
179
-
if (data->dst)
180
-
mlen = max(data->slen, data->dlen);
181
-
else
182
-
mlen = data->slen + data->dlen;
170
+
mlen = max(data->slen, data->dlen);
183
171
184
172
len = sizeof(*req) + reqsize + mlen;
185
173
if (len < mlen)
···
195
189
196
190
sg = &data->sg;
197
191
sg_init_one(sg, buf, mlen);
198
-
akcipher_request_set_crypt(req, sg, data->dst ? sg : NULL,
199
-
data->slen, data->dlen);
192
+
akcipher_request_set_crypt(req, sg, sg, data->slen, data->dlen);
200
193
201
194
crypto_init_wait(&data->cwait);
202
195
akcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
···
203
198
204
199
return 0;
205
200
}
206
-
EXPORT_SYMBOL_GPL(crypto_akcipher_sync_prep);
207
201
208
-
int crypto_akcipher_sync_post(struct crypto_akcipher_sync_data *data, int err)
202
+
static int crypto_akcipher_sync_post(struct crypto_akcipher_sync_data *data,
203
+
int err)
209
204
{
210
205
err = crypto_wait_req(err, &data->cwait);
211
-
if (data->dst)
212
-
memcpy(data->dst, data->buf, data->dlen);
206
+
memcpy(data->dst, data->buf, data->dlen);
213
207
data->dlen = data->req->dst_len;
214
208
kfree_sensitive(data->req);
215
209
return err;
216
210
}
217
-
EXPORT_SYMBOL_GPL(crypto_akcipher_sync_post);
218
211
219
212
int crypto_akcipher_sync_encrypt(struct crypto_akcipher *tfm,
220
213
const void *src, unsigned int slen,
···
250
247
data.dlen;
251
248
}
252
249
EXPORT_SYMBOL_GPL(crypto_akcipher_sync_decrypt);
253
-
254
-
static void crypto_exit_akcipher_ops_sig(struct crypto_tfm *tfm)
255
-
{
256
-
struct crypto_akcipher **ctx = crypto_tfm_ctx(tfm);
257
-
258
-
crypto_free_akcipher(*ctx);
259
-
}
260
-
261
-
int crypto_init_akcipher_ops_sig(struct crypto_tfm *tfm)
262
-
{
263
-
struct crypto_akcipher **ctx = crypto_tfm_ctx(tfm);
264
-
struct crypto_alg *calg = tfm->__crt_alg;
265
-
struct crypto_akcipher *akcipher;
266
-
267
-
if (!crypto_mod_get(calg))
268
-
return -EAGAIN;
269
-
270
-
akcipher = crypto_create_tfm(calg, &crypto_akcipher_type);
271
-
if (IS_ERR(akcipher)) {
272
-
crypto_mod_put(calg);
273
-
return PTR_ERR(akcipher);
274
-
}
275
-
276
-
*ctx = akcipher;
277
-
tfm->exit = crypto_exit_akcipher_ops_sig;
278
-
279
-
return 0;
280
-
}
281
-
EXPORT_SYMBOL_GPL(crypto_init_akcipher_ops_sig);
282
250
283
251
MODULE_LICENSE("GPL");
284
252
MODULE_DESCRIPTION("Generic public key cipher type");
-6
crypto/algapi.c
-6
crypto/algapi.c
···
6
6
*/
7
7
8
8
#include <crypto/algapi.h>
9
-
#include <crypto/internal/simd.h>
10
9
#include <linux/err.h>
11
10
#include <linux/errno.h>
12
11
#include <linux/fips.h>
···
21
22
#include "internal.h"
22
23
23
24
static LIST_HEAD(crypto_template_list);
24
-
25
-
#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS
26
-
DEFINE_PER_CPU(bool, crypto_simd_disabled_for_test);
27
-
EXPORT_PER_CPU_SYMBOL_GPL(crypto_simd_disabled_for_test);
28
-
#endif
29
25
30
26
static inline void crypto_check_module_sig(struct module *mod)
31
27
{
+19
-39
crypto/asymmetric_keys/public_key.c
+19
-39
crypto/asymmetric_keys/public_key.c
···
83
83
if (strcmp(encoding, "pkcs1") == 0) {
84
84
*sig = op == kernel_pkey_sign ||
85
85
op == kernel_pkey_verify;
86
-
if (!hash_algo) {
86
+
if (!*sig) {
87
+
/*
88
+
* For encrypt/decrypt, hash_algo is not used
89
+
* but allowed to be set for historic reasons.
90
+
*/
87
91
n = snprintf(alg_name, CRYPTO_MAX_ALG_NAME,
88
92
"pkcs1pad(%s)",
89
93
pkey->pkey_algo);
90
94
} else {
95
+
if (!hash_algo)
96
+
hash_algo = "none";
91
97
n = snprintf(alg_name, CRYPTO_MAX_ALG_NAME,
92
-
"pkcs1pad(%s,%s)",
98
+
"pkcs1(%s,%s)",
93
99
pkey->pkey_algo, hash_algo);
94
100
}
95
101
return n >= CRYPTO_MAX_ALG_NAME ? -EINVAL : 0;
···
110
104
return -EINVAL;
111
105
*sig = false;
112
106
} else if (strncmp(pkey->pkey_algo, "ecdsa", 5) == 0) {
113
-
if (strcmp(encoding, "x962") != 0)
107
+
if (strcmp(encoding, "x962") != 0 &&
108
+
strcmp(encoding, "p1363") != 0)
114
109
return -EINVAL;
115
110
/*
116
111
* ECDSA signatures are taken over a raw hash, so they don't
···
131
124
strcmp(hash_algo, "sha3-384") != 0 &&
132
125
strcmp(hash_algo, "sha3-512") != 0)
133
126
return -EINVAL;
127
+
n = snprintf(alg_name, CRYPTO_MAX_ALG_NAME, "%s(%s)",
128
+
encoding, pkey->pkey_algo);
129
+
return n >= CRYPTO_MAX_ALG_NAME ? -EINVAL : 0;
134
130
} else if (strcmp(pkey->pkey_algo, "ecrdsa") == 0) {
135
131
if (strcmp(encoding, "raw") != 0)
136
132
return -EINVAL;
···
202
192
if (ret < 0)
203
193
goto error_free_tfm;
204
194
205
-
len = crypto_sig_maxsize(sig);
195
+
len = crypto_sig_keysize(sig);
196
+
info->max_sig_size = crypto_sig_maxsize(sig);
197
+
info->max_data_size = crypto_sig_digestsize(sig);
206
198
207
199
info->supported_ops = KEYCTL_SUPPORTS_VERIFY;
208
200
if (pkey->key_is_private)
···
230
218
goto error_free_tfm;
231
219
232
220
len = crypto_akcipher_maxsize(tfm);
221
+
info->max_sig_size = len;
222
+
info->max_data_size = len;
233
223
234
224
info->supported_ops = KEYCTL_SUPPORTS_ENCRYPT;
235
225
if (pkey->key_is_private)
···
239
225
}
240
226
241
227
info->key_size = len * 8;
242
-
243
-
if (strncmp(pkey->pkey_algo, "ecdsa", 5) == 0) {
244
-
int slen = len;
245
-
/*
246
-
* ECDSA key sizes are much smaller than RSA, and thus could
247
-
* operate on (hashed) inputs that are larger than key size.
248
-
* For example SHA384-hashed input used with secp256r1
249
-
* based keys. Set max_data_size to be at least as large as
250
-
* the largest supported hash size (SHA512)
251
-
*/
252
-
info->max_data_size = 64;
253
-
254
-
/*
255
-
* Verify takes ECDSA-Sig (described in RFC 5480) as input,
256
-
* which is actually 2 'key_size'-bit integers encoded in
257
-
* ASN.1. Account for the ASN.1 encoding overhead here.
258
-
*
259
-
* NIST P192/256/384 may prepend a '0' to a coordinate to
260
-
* indicate a positive integer. NIST P521 never needs it.
261
-
*/
262
-
if (strcmp(pkey->pkey_algo, "ecdsa-nist-p521") != 0)
263
-
slen += 1;
264
-
/* Length of encoding the x & y coordinates */
265
-
slen = 2 * (slen + 2);
266
-
/*
267
-
* If coordinate encoding takes at least 128 bytes then an
268
-
* additional byte for length encoding is needed.
269
-
*/
270
-
info->max_sig_size = 1 + (slen >= 128) + 1 + slen;
271
-
} else {
272
-
info->max_data_size = len;
273
-
info->max_sig_size = len;
274
-
}
275
-
276
228
info->max_enc_size = len;
277
229
info->max_dec_size = len;
278
230
···
303
323
if (ret)
304
324
goto error_free_tfm;
305
325
306
-
ksz = crypto_sig_maxsize(sig);
326
+
ksz = crypto_sig_keysize(sig);
307
327
} else {
308
328
tfm = crypto_alloc_akcipher(alg_name, 0, 0);
309
329
if (IS_ERR(tfm)) {
-63
crypto/asymmetric_keys/signature.c
-63
crypto/asymmetric_keys/signature.c
···
65
65
EXPORT_SYMBOL_GPL(query_asymmetric_key);
66
66
67
67
/**
68
-
* encrypt_blob - Encrypt data using an asymmetric key
69
-
* @params: Various parameters
70
-
* @data: Data blob to be encrypted, length params->data_len
71
-
* @enc: Encrypted data buffer, length params->enc_len
72
-
*
73
-
* Encrypt the specified data blob using the private key specified by
74
-
* params->key. The encrypted data is wrapped in an encoding if
75
-
* params->encoding is specified (eg. "pkcs1").
76
-
*
77
-
* Returns the length of the data placed in the encrypted data buffer or an
78
-
* error.
79
-
*/
80
-
int encrypt_blob(struct kernel_pkey_params *params,
81
-
const void *data, void *enc)
82
-
{
83
-
params->op = kernel_pkey_encrypt;
84
-
return asymmetric_key_eds_op(params, data, enc);
85
-
}
86
-
EXPORT_SYMBOL_GPL(encrypt_blob);
87
-
88
-
/**
89
-
* decrypt_blob - Decrypt data using an asymmetric key
90
-
* @params: Various parameters
91
-
* @enc: Encrypted data to be decrypted, length params->enc_len
92
-
* @data: Decrypted data buffer, length params->data_len
93
-
*
94
-
* Decrypt the specified data blob using the private key specified by
95
-
* params->key. The decrypted data is wrapped in an encoding if
96
-
* params->encoding is specified (eg. "pkcs1").
97
-
*
98
-
* Returns the length of the data placed in the decrypted data buffer or an
99
-
* error.
100
-
*/
101
-
int decrypt_blob(struct kernel_pkey_params *params,
102
-
const void *enc, void *data)
103
-
{
104
-
params->op = kernel_pkey_decrypt;
105
-
return asymmetric_key_eds_op(params, enc, data);
106
-
}
107
-
EXPORT_SYMBOL_GPL(decrypt_blob);
108
-
109
-
/**
110
-
* create_signature - Sign some data using an asymmetric key
111
-
* @params: Various parameters
112
-
* @data: Data blob to be signed, length params->data_len
113
-
* @enc: Signature buffer, length params->enc_len
114
-
*
115
-
* Sign the specified data blob using the private key specified by params->key.
116
-
* The signature is wrapped in an encoding if params->encoding is specified
117
-
* (eg. "pkcs1"). If the encoding needs to know the digest type, this can be
118
-
* passed through params->hash_algo (eg. "sha1").
119
-
*
120
-
* Returns the length of the data placed in the signature buffer or an error.
121
-
*/
122
-
int create_signature(struct kernel_pkey_params *params,
123
-
const void *data, void *enc)
124
-
{
125
-
params->op = kernel_pkey_sign;
126
-
return asymmetric_key_eds_op(params, data, enc);
127
-
}
128
-
EXPORT_SYMBOL_GPL(create_signature);
129
-
130
-
/**
131
68
* verify_signature - Initiate the use of an asymmetric key to verify a signature
132
69
* @key: The asymmetric key to verify against
133
70
* @sig: The signature to check
+70
-24
crypto/crc32_generic.c
+70
-24
crypto/crc32_generic.c
···
59
59
{
60
60
u32 *crcp = shash_desc_ctx(desc);
61
61
62
+
*crcp = crc32_le_base(*crcp, data, len);
63
+
return 0;
64
+
}
65
+
66
+
static int crc32_update_arch(struct shash_desc *desc, const u8 *data,
67
+
unsigned int len)
68
+
{
69
+
u32 *crcp = shash_desc_ctx(desc);
70
+
62
71
*crcp = crc32_le(*crcp, data, len);
63
72
return 0;
64
73
}
···
75
66
/* No final XOR 0xFFFFFFFF, like crc32_le */
76
67
static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len,
77
68
u8 *out)
69
+
{
70
+
put_unaligned_le32(crc32_le_base(*crcp, data, len), out);
71
+
return 0;
72
+
}
73
+
74
+
static int __crc32_finup_arch(u32 *crcp, const u8 *data, unsigned int len,
75
+
u8 *out)
78
76
{
79
77
put_unaligned_le32(crc32_le(*crcp, data, len), out);
80
78
return 0;
···
91
75
unsigned int len, u8 *out)
92
76
{
93
77
return __crc32_finup(shash_desc_ctx(desc), data, len, out);
78
+
}
79
+
80
+
static int crc32_finup_arch(struct shash_desc *desc, const u8 *data,
81
+
unsigned int len, u8 *out)
82
+
{
83
+
return __crc32_finup_arch(shash_desc_ctx(desc), data, len, out);
94
84
}
95
85
96
86
static int crc32_final(struct shash_desc *desc, u8 *out)
···
110
88
static int crc32_digest(struct shash_desc *desc, const u8 *data,
111
89
unsigned int len, u8 *out)
112
90
{
113
-
return __crc32_finup(crypto_shash_ctx(desc->tfm), data, len,
114
-
out);
91
+
return __crc32_finup(crypto_shash_ctx(desc->tfm), data, len, out);
115
92
}
116
-
static struct shash_alg alg = {
117
-
.setkey = crc32_setkey,
118
-
.init = crc32_init,
119
-
.update = crc32_update,
120
-
.final = crc32_final,
121
-
.finup = crc32_finup,
122
-
.digest = crc32_digest,
123
-
.descsize = sizeof(u32),
124
-
.digestsize = CHKSUM_DIGEST_SIZE,
125
-
.base = {
126
-
.cra_name = "crc32",
127
-
.cra_driver_name = "crc32-generic",
128
-
.cra_priority = 100,
129
-
.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
130
-
.cra_blocksize = CHKSUM_BLOCK_SIZE,
131
-
.cra_ctxsize = sizeof(u32),
132
-
.cra_module = THIS_MODULE,
133
-
.cra_init = crc32_cra_init,
134
-
}
135
-
};
93
+
94
+
static int crc32_digest_arch(struct shash_desc *desc, const u8 *data,
95
+
unsigned int len, u8 *out)
96
+
{
97
+
return __crc32_finup_arch(crypto_shash_ctx(desc->tfm), data, len, out);
98
+
}
99
+
100
+
static struct shash_alg algs[] = {{
101
+
.setkey = crc32_setkey,
102
+
.init = crc32_init,
103
+
.update = crc32_update,
104
+
.final = crc32_final,
105
+
.finup = crc32_finup,
106
+
.digest = crc32_digest,
107
+
.descsize = sizeof(u32),
108
+
.digestsize = CHKSUM_DIGEST_SIZE,
109
+
110
+
.base.cra_name = "crc32",
111
+
.base.cra_driver_name = "crc32-generic",
112
+
.base.cra_priority = 100,
113
+
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
114
+
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
115
+
.base.cra_ctxsize = sizeof(u32),
116
+
.base.cra_module = THIS_MODULE,
117
+
.base.cra_init = crc32_cra_init,
118
+
}, {
119
+
.setkey = crc32_setkey,
120
+
.init = crc32_init,
121
+
.update = crc32_update_arch,
122
+
.final = crc32_final,
123
+
.finup = crc32_finup_arch,
124
+
.digest = crc32_digest_arch,
125
+
.descsize = sizeof(u32),
126
+
.digestsize = CHKSUM_DIGEST_SIZE,
127
+
128
+
.base.cra_name = "crc32",
129
+
.base.cra_driver_name = "crc32-" __stringify(ARCH),
130
+
.base.cra_priority = 150,
131
+
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
132
+
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
133
+
.base.cra_ctxsize = sizeof(u32),
134
+
.base.cra_module = THIS_MODULE,
135
+
.base.cra_init = crc32_cra_init,
136
+
}};
136
137
137
138
static int __init crc32_mod_init(void)
138
139
{
139
-
return crypto_register_shash(&alg);
140
+
/* register the arch flavor only if it differs from the generic one */
141
+
return crypto_register_shashes(algs, 1 + (&crc32_le != &crc32_le_base));
140
142
}
141
143
142
144
static void __exit crc32_mod_fini(void)
143
145
{
144
-
crypto_unregister_shash(&alg);
146
+
crypto_unregister_shashes(algs, 1 + (&crc32_le != &crc32_le_base));
145
147
}
146
148
147
149
subsys_initcall(crc32_mod_init);
+72
-22
crypto/crc32c_generic.c
+72
-22
crypto/crc32c_generic.c
···
85
85
{
86
86
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
87
87
88
+
ctx->crc = __crc32c_le_base(ctx->crc, data, length);
89
+
return 0;
90
+
}
91
+
92
+
static int chksum_update_arch(struct shash_desc *desc, const u8 *data,
93
+
unsigned int length)
94
+
{
95
+
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
96
+
88
97
ctx->crc = __crc32c_le(ctx->crc, data, length);
89
98
return 0;
90
99
}
···
108
99
109
100
static int __chksum_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out)
110
101
{
102
+
put_unaligned_le32(~__crc32c_le_base(*crcp, data, len), out);
103
+
return 0;
104
+
}
105
+
106
+
static int __chksum_finup_arch(u32 *crcp, const u8 *data, unsigned int len,
107
+
u8 *out)
108
+
{
111
109
put_unaligned_le32(~__crc32c_le(*crcp, data, len), out);
112
110
return 0;
113
111
}
···
127
111
return __chksum_finup(&ctx->crc, data, len, out);
128
112
}
129
113
114
+
static int chksum_finup_arch(struct shash_desc *desc, const u8 *data,
115
+
unsigned int len, u8 *out)
116
+
{
117
+
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
118
+
119
+
return __chksum_finup_arch(&ctx->crc, data, len, out);
120
+
}
121
+
130
122
static int chksum_digest(struct shash_desc *desc, const u8 *data,
131
123
unsigned int length, u8 *out)
132
124
{
133
125
struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm);
134
126
135
127
return __chksum_finup(&mctx->key, data, length, out);
128
+
}
129
+
130
+
static int chksum_digest_arch(struct shash_desc *desc, const u8 *data,
131
+
unsigned int length, u8 *out)
132
+
{
133
+
struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm);
134
+
135
+
return __chksum_finup_arch(&mctx->key, data, length, out);
136
136
}
137
137
138
138
static int crc32c_cra_init(struct crypto_tfm *tfm)
···
159
127
return 0;
160
128
}
161
129
162
-
static struct shash_alg alg = {
163
-
.digestsize = CHKSUM_DIGEST_SIZE,
164
-
.setkey = chksum_setkey,
165
-
.init = chksum_init,
166
-
.update = chksum_update,
167
-
.final = chksum_final,
168
-
.finup = chksum_finup,
169
-
.digest = chksum_digest,
170
-
.descsize = sizeof(struct chksum_desc_ctx),
171
-
.base = {
172
-
.cra_name = "crc32c",
173
-
.cra_driver_name = "crc32c-generic",
174
-
.cra_priority = 100,
175
-
.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
176
-
.cra_blocksize = CHKSUM_BLOCK_SIZE,
177
-
.cra_ctxsize = sizeof(struct chksum_ctx),
178
-
.cra_module = THIS_MODULE,
179
-
.cra_init = crc32c_cra_init,
180
-
}
181
-
};
130
+
static struct shash_alg algs[] = {{
131
+
.digestsize = CHKSUM_DIGEST_SIZE,
132
+
.setkey = chksum_setkey,
133
+
.init = chksum_init,
134
+
.update = chksum_update,
135
+
.final = chksum_final,
136
+
.finup = chksum_finup,
137
+
.digest = chksum_digest,
138
+
.descsize = sizeof(struct chksum_desc_ctx),
139
+
140
+
.base.cra_name = "crc32c",
141
+
.base.cra_driver_name = "crc32c-generic",
142
+
.base.cra_priority = 100,
143
+
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
144
+
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
145
+
.base.cra_ctxsize = sizeof(struct chksum_ctx),
146
+
.base.cra_module = THIS_MODULE,
147
+
.base.cra_init = crc32c_cra_init,
148
+
}, {
149
+
.digestsize = CHKSUM_DIGEST_SIZE,
150
+
.setkey = chksum_setkey,
151
+
.init = chksum_init,
152
+
.update = chksum_update_arch,
153
+
.final = chksum_final,
154
+
.finup = chksum_finup_arch,
155
+
.digest = chksum_digest_arch,
156
+
.descsize = sizeof(struct chksum_desc_ctx),
157
+
158
+
.base.cra_name = "crc32c",
159
+
.base.cra_driver_name = "crc32c-" __stringify(ARCH),
160
+
.base.cra_priority = 150,
161
+
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
162
+
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
163
+
.base.cra_ctxsize = sizeof(struct chksum_ctx),
164
+
.base.cra_module = THIS_MODULE,
165
+
.base.cra_init = crc32c_cra_init,
166
+
}};
182
167
183
168
static int __init crc32c_mod_init(void)
184
169
{
185
-
return crypto_register_shash(&alg);
170
+
/* register the arch flavor only if it differs from the generic one */
171
+
return crypto_register_shashes(algs, 1 + (&__crc32c_le != &__crc32c_le_base));
186
172
}
187
173
188
174
static void __exit crc32c_mod_fini(void)
189
175
{
190
-
crypto_unregister_shash(&alg);
176
+
crypto_unregister_shashes(algs, 1 + (&__crc32c_le != &__crc32c_le_base));
191
177
}
192
178
193
179
subsys_initcall(crc32c_mod_init);
+3
-2
crypto/drbg.c
+3
-2
crypto/drbg.c
···
101
101
#include <crypto/internal/cipher.h>
102
102
#include <linux/kernel.h>
103
103
#include <linux/jiffies.h>
104
+
#include <linux/string_choices.h>
104
105
105
106
/***************************************************************
106
107
* Backend cipher definitions available to DRBG
···
1413
1412
if (drbg->pr || drbg->seeded == DRBG_SEED_STATE_UNSEEDED) {
1414
1413
pr_devel("DRBG: reseeding before generation (prediction "
1415
1414
"resistance: %s, state %s)\n",
1416
-
drbg->pr ? "true" : "false",
1415
+
str_true_false(drbg->pr),
1417
1416
(drbg->seeded == DRBG_SEED_STATE_FULL ?
1418
1417
"seeded" : "unseeded"));
1419
1418
/* 9.3.1 steps 7.1 through 7.3 */
···
1563
1562
bool reseed = true;
1564
1563
1565
1564
pr_devel("DRBG: Initializing DRBG core %d with prediction resistance "
1566
-
"%s\n", coreref, pr ? "enabled" : "disabled");
1565
+
"%s\n", coreref, str_enabled_disabled(pr));
1567
1566
mutex_lock(&drbg->drbg_mutex);
1568
1567
1569
1568
/* 9.1 step 1 is implicit with the selected DRBG type */
+159
crypto/ecdsa-p1363.c
+159
crypto/ecdsa-p1363.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/*
3
+
* ECDSA P1363 signature encoding
4
+
*
5
+
* Copyright (c) 2024 Intel Corporation
6
+
*/
7
+
8
+
#include <linux/err.h>
9
+
#include <linux/module.h>
10
+
#include <crypto/algapi.h>
11
+
#include <crypto/sig.h>
12
+
#include <crypto/internal/ecc.h>
13
+
#include <crypto/internal/sig.h>
14
+
15
+
struct ecdsa_p1363_ctx {
16
+
struct crypto_sig *child;
17
+
};
18
+
19
+
static int ecdsa_p1363_verify(struct crypto_sig *tfm,
20
+
const void *src, unsigned int slen,
21
+
const void *digest, unsigned int dlen)
22
+
{
23
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
24
+
unsigned int keylen = crypto_sig_keysize(ctx->child);
25
+
unsigned int ndigits = DIV_ROUND_UP(keylen, sizeof(u64));
26
+
struct ecdsa_raw_sig sig;
27
+
28
+
if (slen != 2 * keylen)
29
+
return -EINVAL;
30
+
31
+
ecc_digits_from_bytes(src, keylen, sig.r, ndigits);
32
+
ecc_digits_from_bytes(src + keylen, keylen, sig.s, ndigits);
33
+
34
+
return crypto_sig_verify(ctx->child, &sig, sizeof(sig), digest, dlen);
35
+
}
36
+
37
+
static unsigned int ecdsa_p1363_key_size(struct crypto_sig *tfm)
38
+
{
39
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
40
+
41
+
return crypto_sig_keysize(ctx->child);
42
+
}
43
+
44
+
static unsigned int ecdsa_p1363_max_size(struct crypto_sig *tfm)
45
+
{
46
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
47
+
48
+
return 2 * crypto_sig_keysize(ctx->child);
49
+
}
50
+
51
+
static unsigned int ecdsa_p1363_digest_size(struct crypto_sig *tfm)
52
+
{
53
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
54
+
55
+
return crypto_sig_digestsize(ctx->child);
56
+
}
57
+
58
+
static int ecdsa_p1363_set_pub_key(struct crypto_sig *tfm,
59
+
const void *key, unsigned int keylen)
60
+
{
61
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
62
+
63
+
return crypto_sig_set_pubkey(ctx->child, key, keylen);
64
+
}
65
+
66
+
static int ecdsa_p1363_init_tfm(struct crypto_sig *tfm)
67
+
{
68
+
struct sig_instance *inst = sig_alg_instance(tfm);
69
+
struct crypto_sig_spawn *spawn = sig_instance_ctx(inst);
70
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
71
+
struct crypto_sig *child_tfm;
72
+
73
+
child_tfm = crypto_spawn_sig(spawn);
74
+
if (IS_ERR(child_tfm))
75
+
return PTR_ERR(child_tfm);
76
+
77
+
ctx->child = child_tfm;
78
+
79
+
return 0;
80
+
}
81
+
82
+
static void ecdsa_p1363_exit_tfm(struct crypto_sig *tfm)
83
+
{
84
+
struct ecdsa_p1363_ctx *ctx = crypto_sig_ctx(tfm);
85
+
86
+
crypto_free_sig(ctx->child);
87
+
}
88
+
89
+
static void ecdsa_p1363_free(struct sig_instance *inst)
90
+
{
91
+
struct crypto_sig_spawn *spawn = sig_instance_ctx(inst);
92
+
93
+
crypto_drop_sig(spawn);
94
+
kfree(inst);
95
+
}
96
+
97
+
static int ecdsa_p1363_create(struct crypto_template *tmpl, struct rtattr **tb)
98
+
{
99
+
struct crypto_sig_spawn *spawn;
100
+
struct sig_instance *inst;
101
+
struct sig_alg *ecdsa_alg;
102
+
u32 mask;
103
+
int err;
104
+
105
+
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SIG, &mask);
106
+
if (err)
107
+
return err;
108
+
109
+
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
110
+
if (!inst)
111
+
return -ENOMEM;
112
+
113
+
spawn = sig_instance_ctx(inst);
114
+
115
+
err = crypto_grab_sig(spawn, sig_crypto_instance(inst),
116
+
crypto_attr_alg_name(tb[1]), 0, mask);
117
+
if (err)
118
+
goto err_free_inst;
119
+
120
+
ecdsa_alg = crypto_spawn_sig_alg(spawn);
121
+
122
+
err = -EINVAL;
123
+
if (strncmp(ecdsa_alg->base.cra_name, "ecdsa", 5) != 0)
124
+
goto err_free_inst;
125
+
126
+
err = crypto_inst_setname(sig_crypto_instance(inst), tmpl->name,
127
+
&ecdsa_alg->base);
128
+
if (err)
129
+
goto err_free_inst;
130
+
131
+
inst->alg.base.cra_priority = ecdsa_alg->base.cra_priority;
132
+
inst->alg.base.cra_ctxsize = sizeof(struct ecdsa_p1363_ctx);
133
+
134
+
inst->alg.init = ecdsa_p1363_init_tfm;
135
+
inst->alg.exit = ecdsa_p1363_exit_tfm;
136
+
137
+
inst->alg.verify = ecdsa_p1363_verify;
138
+
inst->alg.key_size = ecdsa_p1363_key_size;
139
+
inst->alg.max_size = ecdsa_p1363_max_size;
140
+
inst->alg.digest_size = ecdsa_p1363_digest_size;
141
+
inst->alg.set_pub_key = ecdsa_p1363_set_pub_key;
142
+
143
+
inst->free = ecdsa_p1363_free;
144
+
145
+
err = sig_register_instance(tmpl, inst);
146
+
if (err) {
147
+
err_free_inst:
148
+
ecdsa_p1363_free(inst);
149
+
}
150
+
return err;
151
+
}
152
+
153
+
struct crypto_template ecdsa_p1363_tmpl = {
154
+
.name = "p1363",
155
+
.create = ecdsa_p1363_create,
156
+
.module = THIS_MODULE,
157
+
};
158
+
159
+
MODULE_ALIAS_CRYPTO("p1363");
+237
crypto/ecdsa-x962.c
+237
crypto/ecdsa-x962.c
···
1
+
// SPDX-License-Identifier: GPL-2.0+
2
+
/*
3
+
* ECDSA X9.62 signature encoding
4
+
*
5
+
* Copyright (c) 2021 IBM Corporation
6
+
* Copyright (c) 2024 Intel Corporation
7
+
*/
8
+
9
+
#include <linux/asn1_decoder.h>
10
+
#include <linux/err.h>
11
+
#include <linux/module.h>
12
+
#include <crypto/algapi.h>
13
+
#include <crypto/sig.h>
14
+
#include <crypto/internal/ecc.h>
15
+
#include <crypto/internal/sig.h>
16
+
17
+
#include "ecdsasignature.asn1.h"
18
+
19
+
struct ecdsa_x962_ctx {
20
+
struct crypto_sig *child;
21
+
};
22
+
23
+
struct ecdsa_x962_signature_ctx {
24
+
struct ecdsa_raw_sig sig;
25
+
unsigned int ndigits;
26
+
};
27
+
28
+
/* Get the r and s components of a signature from the X.509 certificate. */
29
+
static int ecdsa_get_signature_rs(u64 *dest, size_t hdrlen, unsigned char tag,
30
+
const void *value, size_t vlen,
31
+
unsigned int ndigits)
32
+
{
33
+
size_t bufsize = ndigits * sizeof(u64);
34
+
const char *d = value;
35
+
36
+
if (!value || !vlen || vlen > bufsize + 1)
37
+
return -EINVAL;
38
+
39
+
/*
40
+
* vlen may be 1 byte larger than bufsize due to a leading zero byte
41
+
* (necessary if the most significant bit of the integer is set).
42
+
*/
43
+
if (vlen > bufsize) {
44
+
/* skip over leading zeros that make 'value' a positive int */
45
+
if (*d == 0) {
46
+
vlen -= 1;
47
+
d++;
48
+
} else {
49
+
return -EINVAL;
50
+
}
51
+
}
52
+
53
+
ecc_digits_from_bytes(d, vlen, dest, ndigits);
54
+
55
+
return 0;
56
+
}
57
+
58
+
int ecdsa_get_signature_r(void *context, size_t hdrlen, unsigned char tag,
59
+
const void *value, size_t vlen)
60
+
{
61
+
struct ecdsa_x962_signature_ctx *sig_ctx = context;
62
+
63
+
return ecdsa_get_signature_rs(sig_ctx->sig.r, hdrlen, tag, value, vlen,
64
+
sig_ctx->ndigits);
65
+
}
66
+
67
+
int ecdsa_get_signature_s(void *context, size_t hdrlen, unsigned char tag,
68
+
const void *value, size_t vlen)
69
+
{
70
+
struct ecdsa_x962_signature_ctx *sig_ctx = context;
71
+
72
+
return ecdsa_get_signature_rs(sig_ctx->sig.s, hdrlen, tag, value, vlen,
73
+
sig_ctx->ndigits);
74
+
}
75
+
76
+
static int ecdsa_x962_verify(struct crypto_sig *tfm,
77
+
const void *src, unsigned int slen,
78
+
const void *digest, unsigned int dlen)
79
+
{
80
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
81
+
struct ecdsa_x962_signature_ctx sig_ctx;
82
+
int err;
83
+
84
+
sig_ctx.ndigits = DIV_ROUND_UP(crypto_sig_keysize(ctx->child),
85
+
sizeof(u64));
86
+
87
+
err = asn1_ber_decoder(&ecdsasignature_decoder, &sig_ctx, src, slen);
88
+
if (err < 0)
89
+
return err;
90
+
91
+
return crypto_sig_verify(ctx->child, &sig_ctx.sig, sizeof(sig_ctx.sig),
92
+
digest, dlen);
93
+
}
94
+
95
+
static unsigned int ecdsa_x962_key_size(struct crypto_sig *tfm)
96
+
{
97
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
98
+
99
+
return crypto_sig_keysize(ctx->child);
100
+
}
101
+
102
+
static unsigned int ecdsa_x962_max_size(struct crypto_sig *tfm)
103
+
{
104
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
105
+
struct sig_alg *alg = crypto_sig_alg(ctx->child);
106
+
int slen = crypto_sig_keysize(ctx->child);
107
+
108
+
/*
109
+
* Verify takes ECDSA-Sig-Value (described in RFC 5480) as input,
110
+
* which is actually 2 'key_size'-bit integers encoded in ASN.1.
111
+
* Account for the ASN.1 encoding overhead here.
112
+
*
113
+
* NIST P192/256/384 may prepend a '0' to a coordinate to indicate
114
+
* a positive integer. NIST P521 never needs it.
115
+
*/
116
+
if (strcmp(alg->base.cra_name, "ecdsa-nist-p521") != 0)
117
+
slen += 1;
118
+
119
+
/* Length of encoding the x & y coordinates */
120
+
slen = 2 * (slen + 2);
121
+
122
+
/*
123
+
* If coordinate encoding takes at least 128 bytes then an
124
+
* additional byte for length encoding is needed.
125
+
*/
126
+
return 1 + (slen >= 128) + 1 + slen;
127
+
}
128
+
129
+
static unsigned int ecdsa_x962_digest_size(struct crypto_sig *tfm)
130
+
{
131
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
132
+
133
+
return crypto_sig_digestsize(ctx->child);
134
+
}
135
+
136
+
static int ecdsa_x962_set_pub_key(struct crypto_sig *tfm,
137
+
const void *key, unsigned int keylen)
138
+
{
139
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
140
+
141
+
return crypto_sig_set_pubkey(ctx->child, key, keylen);
142
+
}
143
+
144
+
static int ecdsa_x962_init_tfm(struct crypto_sig *tfm)
145
+
{
146
+
struct sig_instance *inst = sig_alg_instance(tfm);
147
+
struct crypto_sig_spawn *spawn = sig_instance_ctx(inst);
148
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
149
+
struct crypto_sig *child_tfm;
150
+
151
+
child_tfm = crypto_spawn_sig(spawn);
152
+
if (IS_ERR(child_tfm))
153
+
return PTR_ERR(child_tfm);
154
+
155
+
ctx->child = child_tfm;
156
+
157
+
return 0;
158
+
}
159
+
160
+
static void ecdsa_x962_exit_tfm(struct crypto_sig *tfm)
161
+
{
162
+
struct ecdsa_x962_ctx *ctx = crypto_sig_ctx(tfm);
163
+
164
+
crypto_free_sig(ctx->child);
165
+
}
166
+
167
+
static void ecdsa_x962_free(struct sig_instance *inst)
168
+
{
169
+
struct crypto_sig_spawn *spawn = sig_instance_ctx(inst);
170
+
171
+
crypto_drop_sig(spawn);
172
+
kfree(inst);
173
+
}
174
+
175
+
static int ecdsa_x962_create(struct crypto_template *tmpl, struct rtattr **tb)
176
+
{
177
+
struct crypto_sig_spawn *spawn;
178
+
struct sig_instance *inst;
179
+
struct sig_alg *ecdsa_alg;
180
+
u32 mask;
181
+
int err;
182
+
183
+
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SIG, &mask);
184
+
if (err)
185
+
return err;
186
+
187
+
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
188
+
if (!inst)
189
+
return -ENOMEM;
190
+
191
+
spawn = sig_instance_ctx(inst);
192
+
193
+
err = crypto_grab_sig(spawn, sig_crypto_instance(inst),
194
+
crypto_attr_alg_name(tb[1]), 0, mask);
195
+
if (err)
196
+
goto err_free_inst;
197
+
198
+
ecdsa_alg = crypto_spawn_sig_alg(spawn);
199
+
200
+
err = -EINVAL;
201
+
if (strncmp(ecdsa_alg->base.cra_name, "ecdsa", 5) != 0)
202
+
goto err_free_inst;
203
+
204
+
err = crypto_inst_setname(sig_crypto_instance(inst), tmpl->name,
205
+
&ecdsa_alg->base);
206
+
if (err)
207
+
goto err_free_inst;
208
+
209
+
inst->alg.base.cra_priority = ecdsa_alg->base.cra_priority;
210
+
inst->alg.base.cra_ctxsize = sizeof(struct ecdsa_x962_ctx);
211
+
212
+
inst->alg.init = ecdsa_x962_init_tfm;
213
+
inst->alg.exit = ecdsa_x962_exit_tfm;
214
+
215
+
inst->alg.verify = ecdsa_x962_verify;
216
+
inst->alg.key_size = ecdsa_x962_key_size;
217
+
inst->alg.max_size = ecdsa_x962_max_size;
218
+
inst->alg.digest_size = ecdsa_x962_digest_size;
219
+
inst->alg.set_pub_key = ecdsa_x962_set_pub_key;
220
+
221
+
inst->free = ecdsa_x962_free;
222
+
223
+
err = sig_register_instance(tmpl, inst);
224
+
if (err) {
225
+
err_free_inst:
226
+
ecdsa_x962_free(inst);
227
+
}
228
+
return err;
229
+
}
230
+
231
+
struct crypto_template ecdsa_x962_tmpl = {
232
+
.name = "x962",
233
+
.create = ecdsa_x962_create,
234
+
.module = THIS_MODULE,
235
+
};
236
+
237
+
MODULE_ALIAS_CRYPTO("x962");
+80
-129
crypto/ecdsa.c
+80
-129
crypto/ecdsa.c
···
4
4
*/
5
5
6
6
#include <linux/module.h>
7
-
#include <crypto/internal/akcipher.h>
8
7
#include <crypto/internal/ecc.h>
9
-
#include <crypto/akcipher.h>
8
+
#include <crypto/internal/sig.h>
10
9
#include <crypto/ecdh.h>
11
-
#include <linux/asn1_decoder.h>
12
-
#include <linux/scatterlist.h>
13
-
14
-
#include "ecdsasignature.asn1.h"
10
+
#include <crypto/sha2.h>
11
+
#include <crypto/sig.h>
15
12
16
13
struct ecc_ctx {
17
14
unsigned int curve_id;
···
19
22
u64 y[ECC_MAX_DIGITS];
20
23
struct ecc_point pub_key;
21
24
};
22
-
23
-
struct ecdsa_signature_ctx {
24
-
const struct ecc_curve *curve;
25
-
u64 r[ECC_MAX_DIGITS];
26
-
u64 s[ECC_MAX_DIGITS];
27
-
};
28
-
29
-
/*
30
-
* Get the r and s components of a signature from the X509 certificate.
31
-
*/
32
-
static int ecdsa_get_signature_rs(u64 *dest, size_t hdrlen, unsigned char tag,
33
-
const void *value, size_t vlen, unsigned int ndigits)
34
-
{
35
-
size_t bufsize = ndigits * sizeof(u64);
36
-
ssize_t diff = vlen - bufsize;
37
-
const char *d = value;
38
-
39
-
if (!value || !vlen)
40
-
return -EINVAL;
41
-
42
-
/* diff = 0: 'value' has exacly the right size
43
-
* diff > 0: 'value' has too many bytes; one leading zero is allowed that
44
-
* makes the value a positive integer; error on more
45
-
* diff < 0: 'value' is missing leading zeros
46
-
*/
47
-
if (diff > 0) {
48
-
/* skip over leading zeros that make 'value' a positive int */
49
-
if (*d == 0) {
50
-
vlen -= 1;
51
-
diff--;
52
-
d++;
53
-
}
54
-
if (diff)
55
-
return -EINVAL;
56
-
}
57
-
if (-diff >= bufsize)
58
-
return -EINVAL;
59
-
60
-
ecc_digits_from_bytes(d, vlen, dest, ndigits);
61
-
62
-
return 0;
63
-
}
64
-
65
-
int ecdsa_get_signature_r(void *context, size_t hdrlen, unsigned char tag,
66
-
const void *value, size_t vlen)
67
-
{
68
-
struct ecdsa_signature_ctx *sig = context;
69
-
70
-
return ecdsa_get_signature_rs(sig->r, hdrlen, tag, value, vlen,
71
-
sig->curve->g.ndigits);
72
-
}
73
-
74
-
int ecdsa_get_signature_s(void *context, size_t hdrlen, unsigned char tag,
75
-
const void *value, size_t vlen)
76
-
{
77
-
struct ecdsa_signature_ctx *sig = context;
78
-
79
-
return ecdsa_get_signature_rs(sig->s, hdrlen, tag, value, vlen,
80
-
sig->curve->g.ndigits);
81
-
}
82
25
83
26
static int _ecdsa_verify(struct ecc_ctx *ctx, const u64 *hash, const u64 *r, const u64 *s)
84
27
{
···
63
126
/*
64
127
* Verify an ECDSA signature.
65
128
*/
66
-
static int ecdsa_verify(struct akcipher_request *req)
129
+
static int ecdsa_verify(struct crypto_sig *tfm,
130
+
const void *src, unsigned int slen,
131
+
const void *digest, unsigned int dlen)
67
132
{
68
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
69
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
133
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
70
134
size_t bufsize = ctx->curve->g.ndigits * sizeof(u64);
71
-
struct ecdsa_signature_ctx sig_ctx = {
72
-
.curve = ctx->curve,
73
-
};
135
+
const struct ecdsa_raw_sig *sig = src;
74
136
u64 hash[ECC_MAX_DIGITS];
75
-
unsigned char *buffer;
76
-
int ret;
77
137
78
138
if (unlikely(!ctx->pub_key_set))
79
139
return -EINVAL;
80
140
81
-
buffer = kmalloc(req->src_len + req->dst_len, GFP_KERNEL);
82
-
if (!buffer)
83
-
return -ENOMEM;
141
+
if (slen != sizeof(*sig))
142
+
return -EINVAL;
84
143
85
-
sg_pcopy_to_buffer(req->src,
86
-
sg_nents_for_len(req->src, req->src_len + req->dst_len),
87
-
buffer, req->src_len + req->dst_len, 0);
144
+
if (bufsize > dlen)
145
+
bufsize = dlen;
88
146
89
-
ret = asn1_ber_decoder(&ecdsasignature_decoder, &sig_ctx,
90
-
buffer, req->src_len);
91
-
if (ret < 0)
92
-
goto error;
147
+
ecc_digits_from_bytes(digest, bufsize, hash, ctx->curve->g.ndigits);
93
148
94
-
if (bufsize > req->dst_len)
95
-
bufsize = req->dst_len;
96
-
97
-
ecc_digits_from_bytes(buffer + req->src_len, bufsize,
98
-
hash, ctx->curve->g.ndigits);
99
-
100
-
ret = _ecdsa_verify(ctx, hash, sig_ctx.r, sig_ctx.s);
101
-
102
-
error:
103
-
kfree(buffer);
104
-
105
-
return ret;
149
+
return _ecdsa_verify(ctx, hash, sig->r, sig->s);
106
150
}
107
151
108
152
static int ecdsa_ecc_ctx_init(struct ecc_ctx *ctx, unsigned int curve_id)
···
119
201
* Set the public ECC key as defined by RFC5480 section 2.2 "Subject Public
120
202
* Key". Only the uncompressed format is supported.
121
203
*/
122
-
static int ecdsa_set_pub_key(struct crypto_akcipher *tfm, const void *key, unsigned int keylen)
204
+
static int ecdsa_set_pub_key(struct crypto_sig *tfm, const void *key,
205
+
unsigned int keylen)
123
206
{
124
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
207
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
125
208
unsigned int digitlen, ndigits;
126
209
const unsigned char *d = key;
127
210
int ret;
···
156
237
return ret;
157
238
}
158
239
159
-
static void ecdsa_exit_tfm(struct crypto_akcipher *tfm)
240
+
static void ecdsa_exit_tfm(struct crypto_sig *tfm)
160
241
{
161
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
242
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
162
243
163
244
ecdsa_ecc_ctx_deinit(ctx);
164
245
}
165
246
166
-
static unsigned int ecdsa_max_size(struct crypto_akcipher *tfm)
247
+
static unsigned int ecdsa_key_size(struct crypto_sig *tfm)
167
248
{
168
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
249
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
169
250
170
251
return DIV_ROUND_UP(ctx->curve->nbits, 8);
171
252
}
172
253
173
-
static int ecdsa_nist_p521_init_tfm(struct crypto_akcipher *tfm)
254
+
static unsigned int ecdsa_digest_size(struct crypto_sig *tfm)
174
255
{
175
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
256
+
/*
257
+
* ECDSA key sizes are much smaller than RSA, and thus could
258
+
* operate on (hashed) inputs that are larger than the key size.
259
+
* E.g. SHA384-hashed input used with secp256r1 based keys.
260
+
* Return the largest supported hash size (SHA512).
261
+
*/
262
+
return SHA512_DIGEST_SIZE;
263
+
}
264
+
265
+
static int ecdsa_nist_p521_init_tfm(struct crypto_sig *tfm)
266
+
{
267
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
176
268
177
269
return ecdsa_ecc_ctx_init(ctx, ECC_CURVE_NIST_P521);
178
270
}
179
271
180
-
static struct akcipher_alg ecdsa_nist_p521 = {
272
+
static struct sig_alg ecdsa_nist_p521 = {
181
273
.verify = ecdsa_verify,
182
274
.set_pub_key = ecdsa_set_pub_key,
183
-
.max_size = ecdsa_max_size,
275
+
.key_size = ecdsa_key_size,
276
+
.digest_size = ecdsa_digest_size,
184
277
.init = ecdsa_nist_p521_init_tfm,
185
278
.exit = ecdsa_exit_tfm,
186
279
.base = {
···
204
273
},
205
274
};
206
275
207
-
static int ecdsa_nist_p384_init_tfm(struct crypto_akcipher *tfm)
276
+
static int ecdsa_nist_p384_init_tfm(struct crypto_sig *tfm)
208
277
{
209
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
278
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
210
279
211
280
return ecdsa_ecc_ctx_init(ctx, ECC_CURVE_NIST_P384);
212
281
}
213
282
214
-
static struct akcipher_alg ecdsa_nist_p384 = {
283
+
static struct sig_alg ecdsa_nist_p384 = {
215
284
.verify = ecdsa_verify,
216
285
.set_pub_key = ecdsa_set_pub_key,
217
-
.max_size = ecdsa_max_size,
286
+
.key_size = ecdsa_key_size,
287
+
.digest_size = ecdsa_digest_size,
218
288
.init = ecdsa_nist_p384_init_tfm,
219
289
.exit = ecdsa_exit_tfm,
220
290
.base = {
···
227
295
},
228
296
};
229
297
230
-
static int ecdsa_nist_p256_init_tfm(struct crypto_akcipher *tfm)
298
+
static int ecdsa_nist_p256_init_tfm(struct crypto_sig *tfm)
231
299
{
232
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
300
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
233
301
234
302
return ecdsa_ecc_ctx_init(ctx, ECC_CURVE_NIST_P256);
235
303
}
236
304
237
-
static struct akcipher_alg ecdsa_nist_p256 = {
305
+
static struct sig_alg ecdsa_nist_p256 = {
238
306
.verify = ecdsa_verify,
239
307
.set_pub_key = ecdsa_set_pub_key,
240
-
.max_size = ecdsa_max_size,
308
+
.key_size = ecdsa_key_size,
309
+
.digest_size = ecdsa_digest_size,
241
310
.init = ecdsa_nist_p256_init_tfm,
242
311
.exit = ecdsa_exit_tfm,
243
312
.base = {
···
250
317
},
251
318
};
252
319
253
-
static int ecdsa_nist_p192_init_tfm(struct crypto_akcipher *tfm)
320
+
static int ecdsa_nist_p192_init_tfm(struct crypto_sig *tfm)
254
321
{
255
-
struct ecc_ctx *ctx = akcipher_tfm_ctx(tfm);
322
+
struct ecc_ctx *ctx = crypto_sig_ctx(tfm);
256
323
257
324
return ecdsa_ecc_ctx_init(ctx, ECC_CURVE_NIST_P192);
258
325
}
259
326
260
-
static struct akcipher_alg ecdsa_nist_p192 = {
327
+
static struct sig_alg ecdsa_nist_p192 = {
261
328
.verify = ecdsa_verify,
262
329
.set_pub_key = ecdsa_set_pub_key,
263
-
.max_size = ecdsa_max_size,
330
+
.key_size = ecdsa_key_size,
331
+
.digest_size = ecdsa_digest_size,
264
332
.init = ecdsa_nist_p192_init_tfm,
265
333
.exit = ecdsa_exit_tfm,
266
334
.base = {
···
279
345
int ret;
280
346
281
347
/* NIST p192 may not be available in FIPS mode */
282
-
ret = crypto_register_akcipher(&ecdsa_nist_p192);
348
+
ret = crypto_register_sig(&ecdsa_nist_p192);
283
349
ecdsa_nist_p192_registered = ret == 0;
284
350
285
-
ret = crypto_register_akcipher(&ecdsa_nist_p256);
351
+
ret = crypto_register_sig(&ecdsa_nist_p256);
286
352
if (ret)
287
353
goto nist_p256_error;
288
354
289
-
ret = crypto_register_akcipher(&ecdsa_nist_p384);
355
+
ret = crypto_register_sig(&ecdsa_nist_p384);
290
356
if (ret)
291
357
goto nist_p384_error;
292
358
293
-
ret = crypto_register_akcipher(&ecdsa_nist_p521);
359
+
ret = crypto_register_sig(&ecdsa_nist_p521);
294
360
if (ret)
295
361
goto nist_p521_error;
296
362
363
+
ret = crypto_register_template(&ecdsa_x962_tmpl);
364
+
if (ret)
365
+
goto x962_tmpl_error;
366
+
367
+
ret = crypto_register_template(&ecdsa_p1363_tmpl);
368
+
if (ret)
369
+
goto p1363_tmpl_error;
370
+
297
371
return 0;
298
372
373
+
p1363_tmpl_error:
374
+
crypto_unregister_template(&ecdsa_x962_tmpl);
375
+
376
+
x962_tmpl_error:
377
+
crypto_unregister_sig(&ecdsa_nist_p521);
378
+
299
379
nist_p521_error:
300
-
crypto_unregister_akcipher(&ecdsa_nist_p384);
380
+
crypto_unregister_sig(&ecdsa_nist_p384);
301
381
302
382
nist_p384_error:
303
-
crypto_unregister_akcipher(&ecdsa_nist_p256);
383
+
crypto_unregister_sig(&ecdsa_nist_p256);
304
384
305
385
nist_p256_error:
306
386
if (ecdsa_nist_p192_registered)
307
-
crypto_unregister_akcipher(&ecdsa_nist_p192);
387
+
crypto_unregister_sig(&ecdsa_nist_p192);
308
388
return ret;
309
389
}
310
390
311
391
static void __exit ecdsa_exit(void)
312
392
{
393
+
crypto_unregister_template(&ecdsa_x962_tmpl);
394
+
crypto_unregister_template(&ecdsa_p1363_tmpl);
395
+
313
396
if (ecdsa_nist_p192_registered)
314
-
crypto_unregister_akcipher(&ecdsa_nist_p192);
315
-
crypto_unregister_akcipher(&ecdsa_nist_p256);
316
-
crypto_unregister_akcipher(&ecdsa_nist_p384);
317
-
crypto_unregister_akcipher(&ecdsa_nist_p521);
397
+
crypto_unregister_sig(&ecdsa_nist_p192);
398
+
crypto_unregister_sig(&ecdsa_nist_p256);
399
+
crypto_unregister_sig(&ecdsa_nist_p384);
400
+
crypto_unregister_sig(&ecdsa_nist_p521);
318
401
}
319
402
320
403
subsys_initcall(ecdsa_init);
+32
-32
crypto/ecrdsa.c
+32
-32
crypto/ecrdsa.c
···
18
18
19
19
#include <linux/module.h>
20
20
#include <linux/crypto.h>
21
+
#include <crypto/sig.h>
21
22
#include <crypto/streebog.h>
22
-
#include <crypto/internal/akcipher.h>
23
23
#include <crypto/internal/ecc.h>
24
-
#include <crypto/akcipher.h>
24
+
#include <crypto/internal/sig.h>
25
25
#include <linux/oid_registry.h>
26
-
#include <linux/scatterlist.h>
27
26
#include "ecrdsa_params.asn1.h"
28
27
#include "ecrdsa_pub_key.asn1.h"
29
28
#include "ecrdsa_defs.h"
···
67
68
}
68
69
}
69
70
70
-
static int ecrdsa_verify(struct akcipher_request *req)
71
+
static int ecrdsa_verify(struct crypto_sig *tfm,
72
+
const void *src, unsigned int slen,
73
+
const void *digest, unsigned int dlen)
71
74
{
72
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
73
-
struct ecrdsa_ctx *ctx = akcipher_tfm_ctx(tfm);
74
-
unsigned char sig[ECRDSA_MAX_SIG_SIZE];
75
-
unsigned char digest[STREEBOG512_DIGEST_SIZE];
76
-
unsigned int ndigits = req->dst_len / sizeof(u64);
75
+
struct ecrdsa_ctx *ctx = crypto_sig_ctx(tfm);
76
+
unsigned int ndigits = dlen / sizeof(u64);
77
77
u64 r[ECRDSA_MAX_DIGITS]; /* witness (r) */
78
78
u64 _r[ECRDSA_MAX_DIGITS]; /* -r */
79
79
u64 s[ECRDSA_MAX_DIGITS]; /* second part of sig (s) */
···
89
91
*/
90
92
if (!ctx->curve ||
91
93
!ctx->digest ||
92
-
!req->src ||
94
+
!src ||
95
+
!digest ||
93
96
!ctx->pub_key.x ||
94
-
req->dst_len != ctx->digest_len ||
95
-
req->dst_len != ctx->curve->g.ndigits * sizeof(u64) ||
97
+
dlen != ctx->digest_len ||
98
+
dlen != ctx->curve->g.ndigits * sizeof(u64) ||
96
99
ctx->pub_key.ndigits != ctx->curve->g.ndigits ||
97
-
req->dst_len * 2 != req->src_len ||
98
-
WARN_ON(req->src_len > sizeof(sig)) ||
99
-
WARN_ON(req->dst_len > sizeof(digest)))
100
+
dlen * 2 != slen ||
101
+
WARN_ON(slen > ECRDSA_MAX_SIG_SIZE) ||
102
+
WARN_ON(dlen > STREEBOG512_DIGEST_SIZE))
100
103
return -EBADMSG;
101
104
102
-
sg_copy_to_buffer(req->src, sg_nents_for_len(req->src, req->src_len),
103
-
sig, req->src_len);
104
-
sg_pcopy_to_buffer(req->src,
105
-
sg_nents_for_len(req->src,
106
-
req->src_len + req->dst_len),
107
-
digest, req->dst_len, req->src_len);
108
-
109
-
vli_from_be64(s, sig, ndigits);
110
-
vli_from_be64(r, sig + ndigits * sizeof(u64), ndigits);
105
+
vli_from_be64(s, src, ndigits);
106
+
vli_from_be64(r, src + ndigits * sizeof(u64), ndigits);
111
107
112
108
/* Step 1: verify that 0 < r < q, 0 < s < q */
113
109
if (vli_is_zero(r, ndigits) ||
···
180
188
}
181
189
182
190
/* Parse BER encoded subjectPublicKey. */
183
-
static int ecrdsa_set_pub_key(struct crypto_akcipher *tfm, const void *key,
191
+
static int ecrdsa_set_pub_key(struct crypto_sig *tfm, const void *key,
184
192
unsigned int keylen)
185
193
{
186
-
struct ecrdsa_ctx *ctx = akcipher_tfm_ctx(tfm);
194
+
struct ecrdsa_ctx *ctx = crypto_sig_ctx(tfm);
187
195
unsigned int ndigits;
188
196
u32 algo, paramlen;
189
197
u8 *params;
···
241
249
return 0;
242
250
}
243
251
244
-
static unsigned int ecrdsa_max_size(struct crypto_akcipher *tfm)
252
+
static unsigned int ecrdsa_key_size(struct crypto_sig *tfm)
245
253
{
246
-
struct ecrdsa_ctx *ctx = akcipher_tfm_ctx(tfm);
254
+
struct ecrdsa_ctx *ctx = crypto_sig_ctx(tfm);
247
255
248
256
/*
249
257
* Verify doesn't need any output, so it's just informational
···
252
260
return ctx->pub_key.ndigits * sizeof(u64);
253
261
}
254
262
255
-
static void ecrdsa_exit_tfm(struct crypto_akcipher *tfm)
263
+
static unsigned int ecrdsa_max_size(struct crypto_sig *tfm)
264
+
{
265
+
struct ecrdsa_ctx *ctx = crypto_sig_ctx(tfm);
266
+
267
+
return 2 * ctx->pub_key.ndigits * sizeof(u64);
268
+
}
269
+
270
+
static void ecrdsa_exit_tfm(struct crypto_sig *tfm)
256
271
{
257
272
}
258
273
259
-
static struct akcipher_alg ecrdsa_alg = {
274
+
static struct sig_alg ecrdsa_alg = {
260
275
.verify = ecrdsa_verify,
261
276
.set_pub_key = ecrdsa_set_pub_key,
277
+
.key_size = ecrdsa_key_size,
262
278
.max_size = ecrdsa_max_size,
263
279
.exit = ecrdsa_exit_tfm,
264
280
.base = {
···
280
280
281
281
static int __init ecrdsa_mod_init(void)
282
282
{
283
-
return crypto_register_akcipher(&ecrdsa_alg);
283
+
return crypto_register_sig(&ecrdsa_alg);
284
284
}
285
285
286
286
static void __exit ecrdsa_mod_fini(void)
287
287
{
288
-
crypto_unregister_akcipher(&ecrdsa_alg);
288
+
crypto_unregister_sig(&ecrdsa_alg);
289
289
}
290
290
291
291
module_init(ecrdsa_mod_init);
-19
crypto/internal.h
-19
crypto/internal.h
···
22
22
#include <linux/sched.h>
23
23
#include <linux/types.h>
24
24
25
-
struct akcipher_request;
26
-
struct crypto_akcipher;
27
25
struct crypto_instance;
28
26
struct crypto_template;
29
27
···
31
33
struct completion completion;
32
34
u32 mask;
33
35
bool test_started;
34
-
};
35
-
36
-
struct crypto_akcipher_sync_data {
37
-
struct crypto_akcipher *tfm;
38
-
const void *src;
39
-
void *dst;
40
-
unsigned int slen;
41
-
unsigned int dlen;
42
-
43
-
struct akcipher_request *req;
44
-
struct crypto_wait cwait;
45
-
struct scatterlist sg;
46
-
u8 *buf;
47
36
};
48
37
49
38
enum {
···
113
128
const struct crypto_type *frontend, int node);
114
129
void *crypto_clone_tfm(const struct crypto_type *frontend,
115
130
struct crypto_tfm *otfm);
116
-
117
-
int crypto_akcipher_sync_prep(struct crypto_akcipher_sync_data *data);
118
-
int crypto_akcipher_sync_post(struct crypto_akcipher_sync_data *data, int err);
119
-
int crypto_init_akcipher_ops_sig(struct crypto_tfm *tfm);
120
131
121
132
static inline void *crypto_create_tfm(struct crypto_alg *alg,
122
133
const struct crypto_type *frontend)
+16
-15
crypto/jitterentropy-testing.c
+16
-15
crypto/jitterentropy-testing.c
···
15
15
#define JENT_TEST_RINGBUFFER_MASK (JENT_TEST_RINGBUFFER_SIZE - 1)
16
16
17
17
struct jent_testing {
18
-
u32 jent_testing_rb[JENT_TEST_RINGBUFFER_SIZE];
18
+
u64 jent_testing_rb[JENT_TEST_RINGBUFFER_SIZE];
19
19
u32 rb_reader;
20
20
atomic_t rb_writer;
21
21
atomic_t jent_testing_enabled;
···
72
72
pr_warn("Disabling data collection\n");
73
73
}
74
74
75
-
static bool jent_testing_store(struct jent_testing *data, u32 value,
75
+
static bool jent_testing_store(struct jent_testing *data, u64 value,
76
76
u32 *boot)
77
77
{
78
78
unsigned long flags;
···
156
156
}
157
157
158
158
/* We copy out word-wise */
159
-
if (outbuflen < sizeof(u32)) {
159
+
if (outbuflen < sizeof(u64)) {
160
160
spin_unlock_irqrestore(&data->lock, flags);
161
161
goto out;
162
162
}
163
163
164
164
memcpy(outbuf, &data->jent_testing_rb[data->rb_reader],
165
-
sizeof(u32));
165
+
sizeof(u64));
166
166
data->rb_reader++;
167
167
168
168
spin_unlock_irqrestore(&data->lock, flags);
169
169
170
-
outbuf += sizeof(u32);
171
-
outbuflen -= sizeof(u32);
172
-
collected_data += sizeof(u32);
170
+
outbuf += sizeof(u64);
171
+
outbuflen -= sizeof(u64);
172
+
collected_data += sizeof(u64);
173
173
}
174
174
175
175
out:
···
189
189
190
190
/*
191
191
* The intention of this interface is for collecting at least
192
-
* 1000 samples due to the SP800-90B requirements. So, we make no
193
-
* effort in avoiding allocating more memory that actually needed
194
-
* by the user. Hence, we allocate sufficient memory to always hold
195
-
* that amount of data.
192
+
* 1000 samples due to the SP800-90B requirements. However, due to
193
+
* memory and performance constraints, it is not desirable to allocate
194
+
* 8000 bytes of memory. Instead, we allocate space for only 125
195
+
* samples, which will allow the user to collect all 1000 samples using
196
+
* 8 calls to this interface.
196
197
*/
197
-
tmp = kmalloc(JENT_TEST_RINGBUFFER_SIZE + sizeof(u32), GFP_KERNEL);
198
+
tmp = kmalloc(125 * sizeof(u64) + sizeof(u64), GFP_KERNEL);
198
199
if (!tmp)
199
200
return -ENOMEM;
200
201
201
-
tmp_aligned = PTR_ALIGN(tmp, sizeof(u32));
202
+
tmp_aligned = PTR_ALIGN(tmp, sizeof(u64));
202
203
203
204
while (nbytes) {
204
205
int i;
···
213
212
schedule();
214
213
}
215
214
216
-
i = min_t(int, nbytes, JENT_TEST_RINGBUFFER_SIZE);
215
+
i = min_t(int, nbytes, 125 * sizeof(u64));
217
216
i = reader(tmp_aligned, i);
218
217
if (i <= 0) {
219
218
if (i < 0)
···
252
251
.read_wait = __WAIT_QUEUE_HEAD_INITIALIZER(jent_raw_hires.read_wait)
253
252
};
254
253
255
-
int jent_raw_hires_entropy_store(__u32 value)
254
+
int jent_raw_hires_entropy_store(__u64 value)
256
255
{
257
256
return jent_testing_store(&jent_raw_hires, value, &boot_raw_hires_test);
258
257
}
+2
-2
crypto/jitterentropy.h
+2
-2
crypto/jitterentropy.h
···
22
22
extern void jent_entropy_collector_free(struct rand_data *entropy_collector);
23
23
24
24
#ifdef CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE
25
-
int jent_raw_hires_entropy_store(__u32 value);
25
+
int jent_raw_hires_entropy_store(__u64 value);
26
26
void jent_testing_init(void);
27
27
void jent_testing_exit(void);
28
28
#else /* CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE */
29
-
static inline int jent_raw_hires_entropy_store(__u32 value) { return 0; }
29
+
static inline int jent_raw_hires_entropy_store(__u64 value) { return 0; }
30
30
static inline void jent_testing_init(void) { }
31
31
static inline void jent_testing_exit(void) { }
32
32
#endif /* CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE */
+8
-4
crypto/pcrypt.c
+8
-4
crypto/pcrypt.c
···
117
117
err = padata_do_parallel(ictx->psenc, padata, &ctx->cb_cpu);
118
118
if (!err)
119
119
return -EINPROGRESS;
120
-
if (err == -EBUSY)
121
-
return -EAGAIN;
120
+
if (err == -EBUSY) {
121
+
/* try non-parallel mode */
122
+
return crypto_aead_encrypt(creq);
123
+
}
122
124
123
125
return err;
124
126
}
···
168
166
err = padata_do_parallel(ictx->psdec, padata, &ctx->cb_cpu);
169
167
if (!err)
170
168
return -EINPROGRESS;
171
-
if (err == -EBUSY)
172
-
return -EAGAIN;
169
+
if (err == -EBUSY) {
170
+
/* try non-parallel mode */
171
+
return crypto_aead_decrypt(creq);
172
+
}
173
173
174
174
return err;
175
175
}
+17
-352
crypto/rsa-pkcs1pad.c
+17
-352
crypto/rsa-pkcs1pad.c
···
16
16
#include <linux/random.h>
17
17
#include <linux/scatterlist.h>
18
18
19
-
/*
20
-
* Hash algorithm OIDs plus ASN.1 DER wrappings [RFC4880 sec 5.2.2].
21
-
*/
22
-
static const u8 rsa_digest_info_md5[] = {
23
-
0x30, 0x20, 0x30, 0x0c, 0x06, 0x08,
24
-
0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x02, 0x05, /* OID */
25
-
0x05, 0x00, 0x04, 0x10
26
-
};
27
-
28
-
static const u8 rsa_digest_info_sha1[] = {
29
-
0x30, 0x21, 0x30, 0x09, 0x06, 0x05,
30
-
0x2b, 0x0e, 0x03, 0x02, 0x1a,
31
-
0x05, 0x00, 0x04, 0x14
32
-
};
33
-
34
-
static const u8 rsa_digest_info_rmd160[] = {
35
-
0x30, 0x21, 0x30, 0x09, 0x06, 0x05,
36
-
0x2b, 0x24, 0x03, 0x02, 0x01,
37
-
0x05, 0x00, 0x04, 0x14
38
-
};
39
-
40
-
static const u8 rsa_digest_info_sha224[] = {
41
-
0x30, 0x2d, 0x30, 0x0d, 0x06, 0x09,
42
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x04,
43
-
0x05, 0x00, 0x04, 0x1c
44
-
};
45
-
46
-
static const u8 rsa_digest_info_sha256[] = {
47
-
0x30, 0x31, 0x30, 0x0d, 0x06, 0x09,
48
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01,
49
-
0x05, 0x00, 0x04, 0x20
50
-
};
51
-
52
-
static const u8 rsa_digest_info_sha384[] = {
53
-
0x30, 0x41, 0x30, 0x0d, 0x06, 0x09,
54
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x02,
55
-
0x05, 0x00, 0x04, 0x30
56
-
};
57
-
58
-
static const u8 rsa_digest_info_sha512[] = {
59
-
0x30, 0x51, 0x30, 0x0d, 0x06, 0x09,
60
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x03,
61
-
0x05, 0x00, 0x04, 0x40
62
-
};
63
-
64
-
static const u8 rsa_digest_info_sha3_256[] = {
65
-
0x30, 0x31, 0x30, 0x0d, 0x06, 0x09,
66
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x08,
67
-
0x05, 0x00, 0x04, 0x20
68
-
};
69
-
70
-
static const u8 rsa_digest_info_sha3_384[] = {
71
-
0x30, 0x41, 0x30, 0x0d, 0x06, 0x09,
72
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x09,
73
-
0x05, 0x00, 0x04, 0x30
74
-
};
75
-
76
-
static const u8 rsa_digest_info_sha3_512[] = {
77
-
0x30, 0x51, 0x30, 0x0d, 0x06, 0x09,
78
-
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0A,
79
-
0x05, 0x00, 0x04, 0x40
80
-
};
81
-
82
-
static const struct rsa_asn1_template {
83
-
const char *name;
84
-
const u8 *data;
85
-
size_t size;
86
-
} rsa_asn1_templates[] = {
87
-
#define _(X) { #X, rsa_digest_info_##X, sizeof(rsa_digest_info_##X) }
88
-
_(md5),
89
-
_(sha1),
90
-
_(rmd160),
91
-
_(sha256),
92
-
_(sha384),
93
-
_(sha512),
94
-
_(sha224),
95
-
#undef _
96
-
#define _(X) { "sha3-" #X, rsa_digest_info_sha3_##X, sizeof(rsa_digest_info_sha3_##X) }
97
-
_(256),
98
-
_(384),
99
-
_(512),
100
-
#undef _
101
-
{ NULL }
102
-
};
103
-
104
-
static const struct rsa_asn1_template *rsa_lookup_asn1(const char *name)
105
-
{
106
-
const struct rsa_asn1_template *p;
107
-
108
-
for (p = rsa_asn1_templates; p->name; p++)
109
-
if (strcmp(name, p->name) == 0)
110
-
return p;
111
-
return NULL;
112
-
}
113
-
114
19
struct pkcs1pad_ctx {
115
20
struct crypto_akcipher *child;
116
21
unsigned int key_size;
···
23
118
24
119
struct pkcs1pad_inst_ctx {
25
120
struct crypto_akcipher_spawn spawn;
26
-
const struct rsa_asn1_template *digest_info;
27
121
};
28
122
29
123
struct pkcs1pad_request {
···
35
131
unsigned int keylen)
36
132
{
37
133
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
38
-
int err;
39
134
40
-
ctx->key_size = 0;
41
-
42
-
err = crypto_akcipher_set_pub_key(ctx->child, key, keylen);
43
-
if (err)
44
-
return err;
45
-
46
-
/* Find out new modulus size from rsa implementation */
47
-
err = crypto_akcipher_maxsize(ctx->child);
48
-
if (err > PAGE_SIZE)
49
-
return -ENOTSUPP;
50
-
51
-
ctx->key_size = err;
52
-
return 0;
135
+
return rsa_set_key(ctx->child, &ctx->key_size, RSA_PUB, key, keylen);
53
136
}
54
137
55
138
static int pkcs1pad_set_priv_key(struct crypto_akcipher *tfm, const void *key,
56
139
unsigned int keylen)
57
140
{
58
141
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
59
-
int err;
60
142
61
-
ctx->key_size = 0;
62
-
63
-
err = crypto_akcipher_set_priv_key(ctx->child, key, keylen);
64
-
if (err)
65
-
return err;
66
-
67
-
/* Find out new modulus size from rsa implementation */
68
-
err = crypto_akcipher_maxsize(ctx->child);
69
-
if (err > PAGE_SIZE)
70
-
return -ENOTSUPP;
71
-
72
-
ctx->key_size = err;
73
-
return 0;
143
+
return rsa_set_key(ctx->child, &ctx->key_size, RSA_PRIV, key, keylen);
74
144
}
75
145
76
146
static unsigned int pkcs1pad_get_max_size(struct crypto_akcipher *tfm)
···
52
174
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
53
175
54
176
/*
55
-
* The maximum destination buffer size for the encrypt/sign operations
177
+
* The maximum destination buffer size for the encrypt operation
56
178
* will be the same as for RSA, even though it's smaller for
57
-
* decrypt/verify.
179
+
* decrypt.
58
180
*/
59
181
60
182
return ctx->key_size;
···
72
194
sg_chain(sg, nsegs, next);
73
195
}
74
196
75
-
static int pkcs1pad_encrypt_sign_complete(struct akcipher_request *req, int err)
197
+
static int pkcs1pad_encrypt_complete(struct akcipher_request *req, int err)
76
198
{
77
199
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
78
200
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
···
111
233
return err;
112
234
}
113
235
114
-
static void pkcs1pad_encrypt_sign_complete_cb(void *data, int err)
236
+
static void pkcs1pad_encrypt_complete_cb(void *data, int err)
115
237
{
116
238
struct akcipher_request *req = data;
117
239
118
240
if (err == -EINPROGRESS)
119
241
goto out;
120
242
121
-
err = pkcs1pad_encrypt_sign_complete(req, err);
243
+
err = pkcs1pad_encrypt_complete(req, err);
122
244
123
245
out:
124
246
akcipher_request_complete(req, err);
···
159
281
160
282
akcipher_request_set_tfm(&req_ctx->child_req, ctx->child);
161
283
akcipher_request_set_callback(&req_ctx->child_req, req->base.flags,
162
-
pkcs1pad_encrypt_sign_complete_cb, req);
284
+
pkcs1pad_encrypt_complete_cb, req);
163
285
164
286
/* Reuse output buffer */
165
287
akcipher_request_set_crypt(&req_ctx->child_req, req_ctx->in_sg,
···
167
289
168
290
err = crypto_akcipher_encrypt(&req_ctx->child_req);
169
291
if (err != -EINPROGRESS && err != -EBUSY)
170
-
return pkcs1pad_encrypt_sign_complete(req, err);
292
+
return pkcs1pad_encrypt_complete(req, err);
171
293
172
294
return err;
173
295
}
···
272
394
return err;
273
395
}
274
396
275
-
static int pkcs1pad_sign(struct akcipher_request *req)
276
-
{
277
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
278
-
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
279
-
struct pkcs1pad_request *req_ctx = akcipher_request_ctx(req);
280
-
struct akcipher_instance *inst = akcipher_alg_instance(tfm);
281
-
struct pkcs1pad_inst_ctx *ictx = akcipher_instance_ctx(inst);
282
-
const struct rsa_asn1_template *digest_info = ictx->digest_info;
283
-
int err;
284
-
unsigned int ps_end, digest_info_size = 0;
285
-
286
-
if (!ctx->key_size)
287
-
return -EINVAL;
288
-
289
-
if (digest_info)
290
-
digest_info_size = digest_info->size;
291
-
292
-
if (req->src_len + digest_info_size > ctx->key_size - 11)
293
-
return -EOVERFLOW;
294
-
295
-
if (req->dst_len < ctx->key_size) {
296
-
req->dst_len = ctx->key_size;
297
-
return -EOVERFLOW;
298
-
}
299
-
300
-
req_ctx->in_buf = kmalloc(ctx->key_size - 1 - req->src_len,
301
-
GFP_KERNEL);
302
-
if (!req_ctx->in_buf)
303
-
return -ENOMEM;
304
-
305
-
ps_end = ctx->key_size - digest_info_size - req->src_len - 2;
306
-
req_ctx->in_buf[0] = 0x01;
307
-
memset(req_ctx->in_buf + 1, 0xff, ps_end - 1);
308
-
req_ctx->in_buf[ps_end] = 0x00;
309
-
310
-
if (digest_info)
311
-
memcpy(req_ctx->in_buf + ps_end + 1, digest_info->data,
312
-
digest_info->size);
313
-
314
-
pkcs1pad_sg_set_buf(req_ctx->in_sg, req_ctx->in_buf,
315
-
ctx->key_size - 1 - req->src_len, req->src);
316
-
317
-
akcipher_request_set_tfm(&req_ctx->child_req, ctx->child);
318
-
akcipher_request_set_callback(&req_ctx->child_req, req->base.flags,
319
-
pkcs1pad_encrypt_sign_complete_cb, req);
320
-
321
-
/* Reuse output buffer */
322
-
akcipher_request_set_crypt(&req_ctx->child_req, req_ctx->in_sg,
323
-
req->dst, ctx->key_size - 1, req->dst_len);
324
-
325
-
err = crypto_akcipher_decrypt(&req_ctx->child_req);
326
-
if (err != -EINPROGRESS && err != -EBUSY)
327
-
return pkcs1pad_encrypt_sign_complete(req, err);
328
-
329
-
return err;
330
-
}
331
-
332
-
static int pkcs1pad_verify_complete(struct akcipher_request *req, int err)
333
-
{
334
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
335
-
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
336
-
struct pkcs1pad_request *req_ctx = akcipher_request_ctx(req);
337
-
struct akcipher_instance *inst = akcipher_alg_instance(tfm);
338
-
struct pkcs1pad_inst_ctx *ictx = akcipher_instance_ctx(inst);
339
-
const struct rsa_asn1_template *digest_info = ictx->digest_info;
340
-
const unsigned int sig_size = req->src_len;
341
-
const unsigned int digest_size = req->dst_len;
342
-
unsigned int dst_len;
343
-
unsigned int pos;
344
-
u8 *out_buf;
345
-
346
-
if (err)
347
-
goto done;
348
-
349
-
err = -EINVAL;
350
-
dst_len = req_ctx->child_req.dst_len;
351
-
if (dst_len < ctx->key_size - 1)
352
-
goto done;
353
-
354
-
out_buf = req_ctx->out_buf;
355
-
if (dst_len == ctx->key_size) {
356
-
if (out_buf[0] != 0x00)
357
-
/* Decrypted value had no leading 0 byte */
358
-
goto done;
359
-
360
-
dst_len--;
361
-
out_buf++;
362
-
}
363
-
364
-
err = -EBADMSG;
365
-
if (out_buf[0] != 0x01)
366
-
goto done;
367
-
368
-
for (pos = 1; pos < dst_len; pos++)
369
-
if (out_buf[pos] != 0xff)
370
-
break;
371
-
372
-
if (pos < 9 || pos == dst_len || out_buf[pos] != 0x00)
373
-
goto done;
374
-
pos++;
375
-
376
-
if (digest_info) {
377
-
if (digest_info->size > dst_len - pos)
378
-
goto done;
379
-
if (crypto_memneq(out_buf + pos, digest_info->data,
380
-
digest_info->size))
381
-
goto done;
382
-
383
-
pos += digest_info->size;
384
-
}
385
-
386
-
err = 0;
387
-
388
-
if (digest_size != dst_len - pos) {
389
-
err = -EKEYREJECTED;
390
-
req->dst_len = dst_len - pos;
391
-
goto done;
392
-
}
393
-
/* Extract appended digest. */
394
-
sg_pcopy_to_buffer(req->src,
395
-
sg_nents_for_len(req->src, sig_size + digest_size),
396
-
req_ctx->out_buf + ctx->key_size,
397
-
digest_size, sig_size);
398
-
/* Do the actual verification step. */
399
-
if (memcmp(req_ctx->out_buf + ctx->key_size, out_buf + pos,
400
-
digest_size) != 0)
401
-
err = -EKEYREJECTED;
402
-
done:
403
-
kfree_sensitive(req_ctx->out_buf);
404
-
405
-
return err;
406
-
}
407
-
408
-
static void pkcs1pad_verify_complete_cb(void *data, int err)
409
-
{
410
-
struct akcipher_request *req = data;
411
-
412
-
if (err == -EINPROGRESS)
413
-
goto out;
414
-
415
-
err = pkcs1pad_verify_complete(req, err);
416
-
417
-
out:
418
-
akcipher_request_complete(req, err);
419
-
}
420
-
421
-
/*
422
-
* The verify operation is here for completeness similar to the verification
423
-
* defined in RFC2313 section 10.2 except that block type 0 is not accepted,
424
-
* as in RFC2437. RFC2437 section 9.2 doesn't define any operation to
425
-
* retrieve the DigestInfo from a signature, instead the user is expected
426
-
* to call the sign operation to generate the expected signature and compare
427
-
* signatures instead of the message-digests.
428
-
*/
429
-
static int pkcs1pad_verify(struct akcipher_request *req)
430
-
{
431
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
432
-
struct pkcs1pad_ctx *ctx = akcipher_tfm_ctx(tfm);
433
-
struct pkcs1pad_request *req_ctx = akcipher_request_ctx(req);
434
-
const unsigned int sig_size = req->src_len;
435
-
const unsigned int digest_size = req->dst_len;
436
-
int err;
437
-
438
-
if (WARN_ON(req->dst) || WARN_ON(!digest_size) ||
439
-
!ctx->key_size || sig_size != ctx->key_size)
440
-
return -EINVAL;
441
-
442
-
req_ctx->out_buf = kmalloc(ctx->key_size + digest_size, GFP_KERNEL);
443
-
if (!req_ctx->out_buf)
444
-
return -ENOMEM;
445
-
446
-
pkcs1pad_sg_set_buf(req_ctx->out_sg, req_ctx->out_buf,
447
-
ctx->key_size, NULL);
448
-
449
-
akcipher_request_set_tfm(&req_ctx->child_req, ctx->child);
450
-
akcipher_request_set_callback(&req_ctx->child_req, req->base.flags,
451
-
pkcs1pad_verify_complete_cb, req);
452
-
453
-
/* Reuse input buffer, output to a new buffer */
454
-
akcipher_request_set_crypt(&req_ctx->child_req, req->src,
455
-
req_ctx->out_sg, sig_size, ctx->key_size);
456
-
457
-
err = crypto_akcipher_encrypt(&req_ctx->child_req);
458
-
if (err != -EINPROGRESS && err != -EBUSY)
459
-
return pkcs1pad_verify_complete(req, err);
460
-
461
-
return err;
462
-
}
463
-
464
397
static int pkcs1pad_init_tfm(struct crypto_akcipher *tfm)
465
398
{
466
399
struct akcipher_instance *inst = akcipher_alg_instance(tfm);
···
313
624
struct akcipher_instance *inst;
314
625
struct pkcs1pad_inst_ctx *ctx;
315
626
struct akcipher_alg *rsa_alg;
316
-
const char *hash_name;
317
627
int err;
318
628
319
629
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_AKCIPHER, &mask);
···
338
650
}
339
651
340
652
err = -ENAMETOOLONG;
341
-
hash_name = crypto_attr_alg_name(tb[2]);
342
-
if (IS_ERR(hash_name)) {
343
-
if (snprintf(inst->alg.base.cra_name,
344
-
CRYPTO_MAX_ALG_NAME, "pkcs1pad(%s)",
345
-
rsa_alg->base.cra_name) >= CRYPTO_MAX_ALG_NAME)
346
-
goto err_free_inst;
653
+
if (snprintf(inst->alg.base.cra_name,
654
+
CRYPTO_MAX_ALG_NAME, "pkcs1pad(%s)",
655
+
rsa_alg->base.cra_name) >= CRYPTO_MAX_ALG_NAME)
656
+
goto err_free_inst;
347
657
348
-
if (snprintf(inst->alg.base.cra_driver_name,
349
-
CRYPTO_MAX_ALG_NAME, "pkcs1pad(%s)",
350
-
rsa_alg->base.cra_driver_name) >=
351
-
CRYPTO_MAX_ALG_NAME)
352
-
goto err_free_inst;
353
-
} else {
354
-
ctx->digest_info = rsa_lookup_asn1(hash_name);
355
-
if (!ctx->digest_info) {
356
-
err = -EINVAL;
357
-
goto err_free_inst;
358
-
}
359
-
360
-
if (snprintf(inst->alg.base.cra_name, CRYPTO_MAX_ALG_NAME,
361
-
"pkcs1pad(%s,%s)", rsa_alg->base.cra_name,
362
-
hash_name) >= CRYPTO_MAX_ALG_NAME)
363
-
goto err_free_inst;
364
-
365
-
if (snprintf(inst->alg.base.cra_driver_name,
366
-
CRYPTO_MAX_ALG_NAME, "pkcs1pad(%s,%s)",
367
-
rsa_alg->base.cra_driver_name,
368
-
hash_name) >= CRYPTO_MAX_ALG_NAME)
369
-
goto err_free_inst;
370
-
}
658
+
if (snprintf(inst->alg.base.cra_driver_name,
659
+
CRYPTO_MAX_ALG_NAME, "pkcs1pad(%s)",
660
+
rsa_alg->base.cra_driver_name) >= CRYPTO_MAX_ALG_NAME)
661
+
goto err_free_inst;
371
662
372
663
inst->alg.base.cra_priority = rsa_alg->base.cra_priority;
373
664
inst->alg.base.cra_ctxsize = sizeof(struct pkcs1pad_ctx);
···
356
689
357
690
inst->alg.encrypt = pkcs1pad_encrypt;
358
691
inst->alg.decrypt = pkcs1pad_decrypt;
359
-
inst->alg.sign = pkcs1pad_sign;
360
-
inst->alg.verify = pkcs1pad_verify;
361
692
inst->alg.set_pub_key = pkcs1pad_set_pub_key;
362
693
inst->alg.set_priv_key = pkcs1pad_set_priv_key;
363
694
inst->alg.max_size = pkcs1pad_get_max_size;
+13
-4
crypto/rsa.c
+13
-4
crypto/rsa.c
···
407
407
return err;
408
408
409
409
err = crypto_register_template(&rsa_pkcs1pad_tmpl);
410
-
if (err) {
411
-
crypto_unregister_akcipher(&rsa);
412
-
return err;
413
-
}
410
+
if (err)
411
+
goto err_unregister_rsa;
412
+
413
+
err = crypto_register_template(&rsassa_pkcs1_tmpl);
414
+
if (err)
415
+
goto err_unregister_rsa_pkcs1pad;
414
416
415
417
return 0;
418
+
419
+
err_unregister_rsa_pkcs1pad:
420
+
crypto_unregister_template(&rsa_pkcs1pad_tmpl);
421
+
err_unregister_rsa:
422
+
crypto_unregister_akcipher(&rsa);
423
+
return err;
416
424
}
417
425
418
426
static void __exit rsa_exit(void)
419
427
{
428
+
crypto_unregister_template(&rsassa_pkcs1_tmpl);
420
429
crypto_unregister_template(&rsa_pkcs1pad_tmpl);
421
430
crypto_unregister_akcipher(&rsa);
422
431
}
+454
crypto/rsassa-pkcs1.c
+454
crypto/rsassa-pkcs1.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-or-later
2
+
/*
3
+
* RSA Signature Scheme with Appendix - PKCS #1 v1.5 (RFC 8017 sec 8.2)
4
+
*
5
+
* https://www.rfc-editor.org/rfc/rfc8017#section-8.2
6
+
*
7
+
* Copyright (c) 2015 - 2024 Intel Corporation
8
+
*/
9
+
10
+
#include <linux/module.h>
11
+
#include <linux/scatterlist.h>
12
+
#include <crypto/akcipher.h>
13
+
#include <crypto/algapi.h>
14
+
#include <crypto/hash.h>
15
+
#include <crypto/sig.h>
16
+
#include <crypto/internal/akcipher.h>
17
+
#include <crypto/internal/rsa.h>
18
+
#include <crypto/internal/sig.h>
19
+
20
+
/*
21
+
* Full Hash Prefix for EMSA-PKCS1-v1_5 encoding method (RFC 9580 table 24)
22
+
*
23
+
* RSA keys are usually much larger than the hash of the message to be signed.
24
+
* The hash is therefore prepended by the Full Hash Prefix and a 0xff padding.
25
+
* The Full Hash Prefix is an ASN.1 SEQUENCE containing the hash algorithm OID.
26
+
*
27
+
* https://www.rfc-editor.org/rfc/rfc9580#table-24
28
+
*/
29
+
30
+
static const u8 hash_prefix_none[] = { };
31
+
32
+
static const u8 hash_prefix_md5[] = {
33
+
0x30, 0x20, 0x30, 0x0c, 0x06, 0x08, /* SEQUENCE (SEQUENCE (OID */
34
+
0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x02, 0x05, /* <algorithm>, */
35
+
0x05, 0x00, 0x04, 0x10 /* NULL), OCTET STRING <hash>) */
36
+
};
37
+
38
+
static const u8 hash_prefix_sha1[] = {
39
+
0x30, 0x21, 0x30, 0x09, 0x06, 0x05,
40
+
0x2b, 0x0e, 0x03, 0x02, 0x1a,
41
+
0x05, 0x00, 0x04, 0x14
42
+
};
43
+
44
+
static const u8 hash_prefix_rmd160[] = {
45
+
0x30, 0x21, 0x30, 0x09, 0x06, 0x05,
46
+
0x2b, 0x24, 0x03, 0x02, 0x01,
47
+
0x05, 0x00, 0x04, 0x14
48
+
};
49
+
50
+
static const u8 hash_prefix_sha224[] = {
51
+
0x30, 0x2d, 0x30, 0x0d, 0x06, 0x09,
52
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x04,
53
+
0x05, 0x00, 0x04, 0x1c
54
+
};
55
+
56
+
static const u8 hash_prefix_sha256[] = {
57
+
0x30, 0x31, 0x30, 0x0d, 0x06, 0x09,
58
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01,
59
+
0x05, 0x00, 0x04, 0x20
60
+
};
61
+
62
+
static const u8 hash_prefix_sha384[] = {
63
+
0x30, 0x41, 0x30, 0x0d, 0x06, 0x09,
64
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x02,
65
+
0x05, 0x00, 0x04, 0x30
66
+
};
67
+
68
+
static const u8 hash_prefix_sha512[] = {
69
+
0x30, 0x51, 0x30, 0x0d, 0x06, 0x09,
70
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x03,
71
+
0x05, 0x00, 0x04, 0x40
72
+
};
73
+
74
+
static const u8 hash_prefix_sha3_256[] = {
75
+
0x30, 0x31, 0x30, 0x0d, 0x06, 0x09,
76
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x08,
77
+
0x05, 0x00, 0x04, 0x20
78
+
};
79
+
80
+
static const u8 hash_prefix_sha3_384[] = {
81
+
0x30, 0x41, 0x30, 0x0d, 0x06, 0x09,
82
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x09,
83
+
0x05, 0x00, 0x04, 0x30
84
+
};
85
+
86
+
static const u8 hash_prefix_sha3_512[] = {
87
+
0x30, 0x51, 0x30, 0x0d, 0x06, 0x09,
88
+
0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0a,
89
+
0x05, 0x00, 0x04, 0x40
90
+
};
91
+
92
+
static const struct hash_prefix {
93
+
const char *name;
94
+
const u8 *data;
95
+
size_t size;
96
+
} hash_prefixes[] = {
97
+
#define _(X) { #X, hash_prefix_##X, sizeof(hash_prefix_##X) }
98
+
_(none),
99
+
_(md5),
100
+
_(sha1),
101
+
_(rmd160),
102
+
_(sha256),
103
+
_(sha384),
104
+
_(sha512),
105
+
_(sha224),
106
+
#undef _
107
+
#define _(X) { "sha3-" #X, hash_prefix_sha3_##X, sizeof(hash_prefix_sha3_##X) }
108
+
_(256),
109
+
_(384),
110
+
_(512),
111
+
#undef _
112
+
{ NULL }
113
+
};
114
+
115
+
static const struct hash_prefix *rsassa_pkcs1_find_hash_prefix(const char *name)
116
+
{
117
+
const struct hash_prefix *p;
118
+
119
+
for (p = hash_prefixes; p->name; p++)
120
+
if (strcmp(name, p->name) == 0)
121
+
return p;
122
+
return NULL;
123
+
}
124
+
125
+
static bool rsassa_pkcs1_invalid_hash_len(unsigned int len,
126
+
const struct hash_prefix *p)
127
+
{
128
+
/*
129
+
* Legacy protocols such as TLS 1.1 or earlier and IKE version 1
130
+
* do not prepend a Full Hash Prefix to the hash. In that case,
131
+
* the size of the Full Hash Prefix is zero.
132
+
*/
133
+
if (p->data == hash_prefix_none)
134
+
return false;
135
+
136
+
/*
137
+
* The final byte of the Full Hash Prefix encodes the hash length.
138
+
*
139
+
* This needs to be revisited should hash algorithms with more than
140
+
* 1016 bits (127 bytes * 8) ever be added. The length would then
141
+
* be encoded into more than one byte by ASN.1.
142
+
*/
143
+
static_assert(HASH_MAX_DIGESTSIZE <= 127);
144
+
145
+
return len != p->data[p->size - 1];
146
+
}
147
+
148
+
struct rsassa_pkcs1_ctx {
149
+
struct crypto_akcipher *child;
150
+
unsigned int key_size;
151
+
};
152
+
153
+
struct rsassa_pkcs1_inst_ctx {
154
+
struct crypto_akcipher_spawn spawn;
155
+
const struct hash_prefix *hash_prefix;
156
+
};
157
+
158
+
static int rsassa_pkcs1_sign(struct crypto_sig *tfm,
159
+
const void *src, unsigned int slen,
160
+
void *dst, unsigned int dlen)
161
+
{
162
+
struct sig_instance *inst = sig_alg_instance(tfm);
163
+
struct rsassa_pkcs1_inst_ctx *ictx = sig_instance_ctx(inst);
164
+
const struct hash_prefix *hash_prefix = ictx->hash_prefix;
165
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
166
+
unsigned int child_reqsize = crypto_akcipher_reqsize(ctx->child);
167
+
struct akcipher_request *child_req __free(kfree_sensitive) = NULL;
168
+
struct scatterlist in_sg[3], out_sg;
169
+
struct crypto_wait cwait;
170
+
unsigned int pad_len;
171
+
unsigned int ps_end;
172
+
unsigned int len;
173
+
u8 *in_buf;
174
+
int err;
175
+
176
+
if (!ctx->key_size)
177
+
return -EINVAL;
178
+
179
+
if (dlen < ctx->key_size)
180
+
return -EOVERFLOW;
181
+
182
+
if (rsassa_pkcs1_invalid_hash_len(slen, hash_prefix))
183
+
return -EINVAL;
184
+
185
+
if (slen + hash_prefix->size > ctx->key_size - 11)
186
+
return -EOVERFLOW;
187
+
188
+
pad_len = ctx->key_size - slen - hash_prefix->size - 1;
189
+
190
+
child_req = kmalloc(sizeof(*child_req) + child_reqsize + pad_len,
191
+
GFP_KERNEL);
192
+
if (!child_req)
193
+
return -ENOMEM;
194
+
195
+
/* RFC 8017 sec 8.2.1 step 1 - EMSA-PKCS1-v1_5 encoding generation */
196
+
in_buf = (u8 *)(child_req + 1) + child_reqsize;
197
+
ps_end = pad_len - 1;
198
+
in_buf[0] = 0x01;
199
+
memset(in_buf + 1, 0xff, ps_end - 1);
200
+
in_buf[ps_end] = 0x00;
201
+
202
+
/* RFC 8017 sec 8.2.1 step 2 - RSA signature */
203
+
crypto_init_wait(&cwait);
204
+
sg_init_table(in_sg, 3);
205
+
sg_set_buf(&in_sg[0], in_buf, pad_len);
206
+
sg_set_buf(&in_sg[1], hash_prefix->data, hash_prefix->size);
207
+
sg_set_buf(&in_sg[2], src, slen);
208
+
sg_init_one(&out_sg, dst, dlen);
209
+
akcipher_request_set_tfm(child_req, ctx->child);
210
+
akcipher_request_set_crypt(child_req, in_sg, &out_sg,
211
+
ctx->key_size - 1, dlen);
212
+
akcipher_request_set_callback(child_req, CRYPTO_TFM_REQ_MAY_SLEEP,
213
+
crypto_req_done, &cwait);
214
+
215
+
err = crypto_akcipher_decrypt(child_req);
216
+
err = crypto_wait_req(err, &cwait);
217
+
if (err)
218
+
return err;
219
+
220
+
len = child_req->dst_len;
221
+
pad_len = ctx->key_size - len;
222
+
223
+
/* Four billion to one */
224
+
if (unlikely(pad_len)) {
225
+
memmove(dst + pad_len, dst, len);
226
+
memset(dst, 0, pad_len);
227
+
}
228
+
229
+
return 0;
230
+
}
231
+
232
+
static int rsassa_pkcs1_verify(struct crypto_sig *tfm,
233
+
const void *src, unsigned int slen,
234
+
const void *digest, unsigned int dlen)
235
+
{
236
+
struct sig_instance *inst = sig_alg_instance(tfm);
237
+
struct rsassa_pkcs1_inst_ctx *ictx = sig_instance_ctx(inst);
238
+
const struct hash_prefix *hash_prefix = ictx->hash_prefix;
239
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
240
+
unsigned int child_reqsize = crypto_akcipher_reqsize(ctx->child);
241
+
struct akcipher_request *child_req __free(kfree_sensitive) = NULL;
242
+
struct scatterlist in_sg, out_sg;
243
+
struct crypto_wait cwait;
244
+
unsigned int dst_len;
245
+
unsigned int pos;
246
+
u8 *out_buf;
247
+
int err;
248
+
249
+
/* RFC 8017 sec 8.2.2 step 1 - length checking */
250
+
if (!ctx->key_size ||
251
+
slen != ctx->key_size ||
252
+
rsassa_pkcs1_invalid_hash_len(dlen, hash_prefix))
253
+
return -EINVAL;
254
+
255
+
/* RFC 8017 sec 8.2.2 step 2 - RSA verification */
256
+
child_req = kmalloc(sizeof(*child_req) + child_reqsize + ctx->key_size,
257
+
GFP_KERNEL);
258
+
if (!child_req)
259
+
return -ENOMEM;
260
+
261
+
out_buf = (u8 *)(child_req + 1) + child_reqsize;
262
+
263
+
crypto_init_wait(&cwait);
264
+
sg_init_one(&in_sg, src, slen);
265
+
sg_init_one(&out_sg, out_buf, ctx->key_size);
266
+
akcipher_request_set_tfm(child_req, ctx->child);
267
+
akcipher_request_set_crypt(child_req, &in_sg, &out_sg,
268
+
slen, ctx->key_size);
269
+
akcipher_request_set_callback(child_req, CRYPTO_TFM_REQ_MAY_SLEEP,
270
+
crypto_req_done, &cwait);
271
+
272
+
err = crypto_akcipher_encrypt(child_req);
273
+
err = crypto_wait_req(err, &cwait);
274
+
if (err)
275
+
return err;
276
+
277
+
/* RFC 8017 sec 8.2.2 step 3 - EMSA-PKCS1-v1_5 encoding verification */
278
+
dst_len = child_req->dst_len;
279
+
if (dst_len < ctx->key_size - 1)
280
+
return -EINVAL;
281
+
282
+
if (dst_len == ctx->key_size) {
283
+
if (out_buf[0] != 0x00)
284
+
/* Encrypted value had no leading 0 byte */
285
+
return -EINVAL;
286
+
287
+
dst_len--;
288
+
out_buf++;
289
+
}
290
+
291
+
if (out_buf[0] != 0x01)
292
+
return -EBADMSG;
293
+
294
+
for (pos = 1; pos < dst_len; pos++)
295
+
if (out_buf[pos] != 0xff)
296
+
break;
297
+
298
+
if (pos < 9 || pos == dst_len || out_buf[pos] != 0x00)
299
+
return -EBADMSG;
300
+
pos++;
301
+
302
+
if (hash_prefix->size > dst_len - pos)
303
+
return -EBADMSG;
304
+
if (crypto_memneq(out_buf + pos, hash_prefix->data, hash_prefix->size))
305
+
return -EBADMSG;
306
+
pos += hash_prefix->size;
307
+
308
+
/* RFC 8017 sec 8.2.2 step 4 - comparison of digest with out_buf */
309
+
if (dlen != dst_len - pos)
310
+
return -EKEYREJECTED;
311
+
if (memcmp(digest, out_buf + pos, dlen) != 0)
312
+
return -EKEYREJECTED;
313
+
314
+
return 0;
315
+
}
316
+
317
+
static unsigned int rsassa_pkcs1_key_size(struct crypto_sig *tfm)
318
+
{
319
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
320
+
321
+
return ctx->key_size;
322
+
}
323
+
324
+
static int rsassa_pkcs1_set_pub_key(struct crypto_sig *tfm,
325
+
const void *key, unsigned int keylen)
326
+
{
327
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
328
+
329
+
return rsa_set_key(ctx->child, &ctx->key_size, RSA_PUB, key, keylen);
330
+
}
331
+
332
+
static int rsassa_pkcs1_set_priv_key(struct crypto_sig *tfm,
333
+
const void *key, unsigned int keylen)
334
+
{
335
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
336
+
337
+
return rsa_set_key(ctx->child, &ctx->key_size, RSA_PRIV, key, keylen);
338
+
}
339
+
340
+
static int rsassa_pkcs1_init_tfm(struct crypto_sig *tfm)
341
+
{
342
+
struct sig_instance *inst = sig_alg_instance(tfm);
343
+
struct rsassa_pkcs1_inst_ctx *ictx = sig_instance_ctx(inst);
344
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
345
+
struct crypto_akcipher *child_tfm;
346
+
347
+
child_tfm = crypto_spawn_akcipher(&ictx->spawn);
348
+
if (IS_ERR(child_tfm))
349
+
return PTR_ERR(child_tfm);
350
+
351
+
ctx->child = child_tfm;
352
+
353
+
return 0;
354
+
}
355
+
356
+
static void rsassa_pkcs1_exit_tfm(struct crypto_sig *tfm)
357
+
{
358
+
struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
359
+
360
+
crypto_free_akcipher(ctx->child);
361
+
}
362
+
363
+
static void rsassa_pkcs1_free(struct sig_instance *inst)
364
+
{
365
+
struct rsassa_pkcs1_inst_ctx *ctx = sig_instance_ctx(inst);
366
+
struct crypto_akcipher_spawn *spawn = &ctx->spawn;
367
+
368
+
crypto_drop_akcipher(spawn);
369
+
kfree(inst);
370
+
}
371
+
372
+
static int rsassa_pkcs1_create(struct crypto_template *tmpl, struct rtattr **tb)
373
+
{
374
+
struct rsassa_pkcs1_inst_ctx *ctx;
375
+
struct akcipher_alg *rsa_alg;
376
+
struct sig_instance *inst;
377
+
const char *hash_name;
378
+
u32 mask;
379
+
int err;
380
+
381
+
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SIG, &mask);
382
+
if (err)
383
+
return err;
384
+
385
+
inst = kzalloc(sizeof(*inst) + sizeof(*ctx), GFP_KERNEL);
386
+
if (!inst)
387
+
return -ENOMEM;
388
+
389
+
ctx = sig_instance_ctx(inst);
390
+
391
+
err = crypto_grab_akcipher(&ctx->spawn, sig_crypto_instance(inst),
392
+
crypto_attr_alg_name(tb[1]), 0, mask);
393
+
if (err)
394
+
goto err_free_inst;
395
+
396
+
rsa_alg = crypto_spawn_akcipher_alg(&ctx->spawn);
397
+
398
+
if (strcmp(rsa_alg->base.cra_name, "rsa") != 0) {
399
+
err = -EINVAL;
400
+
goto err_free_inst;
401
+
}
402
+
403
+
hash_name = crypto_attr_alg_name(tb[2]);
404
+
if (IS_ERR(hash_name)) {
405
+
err = PTR_ERR(hash_name);
406
+
goto err_free_inst;
407
+
}
408
+
409
+
ctx->hash_prefix = rsassa_pkcs1_find_hash_prefix(hash_name);
410
+
if (!ctx->hash_prefix) {
411
+
err = -EINVAL;
412
+
goto err_free_inst;
413
+
}
414
+
415
+
err = -ENAMETOOLONG;
416
+
if (snprintf(inst->alg.base.cra_name, CRYPTO_MAX_ALG_NAME,
417
+
"pkcs1(%s,%s)", rsa_alg->base.cra_name,
418
+
hash_name) >= CRYPTO_MAX_ALG_NAME)
419
+
goto err_free_inst;
420
+
421
+
if (snprintf(inst->alg.base.cra_driver_name, CRYPTO_MAX_ALG_NAME,
422
+
"pkcs1(%s,%s)", rsa_alg->base.cra_driver_name,
423
+
hash_name) >= CRYPTO_MAX_ALG_NAME)
424
+
goto err_free_inst;
425
+
426
+
inst->alg.base.cra_priority = rsa_alg->base.cra_priority;
427
+
inst->alg.base.cra_ctxsize = sizeof(struct rsassa_pkcs1_ctx);
428
+
429
+
inst->alg.init = rsassa_pkcs1_init_tfm;
430
+
inst->alg.exit = rsassa_pkcs1_exit_tfm;
431
+
432
+
inst->alg.sign = rsassa_pkcs1_sign;
433
+
inst->alg.verify = rsassa_pkcs1_verify;
434
+
inst->alg.key_size = rsassa_pkcs1_key_size;
435
+
inst->alg.set_pub_key = rsassa_pkcs1_set_pub_key;
436
+
inst->alg.set_priv_key = rsassa_pkcs1_set_priv_key;
437
+
438
+
inst->free = rsassa_pkcs1_free;
439
+
440
+
err = sig_register_instance(tmpl, inst);
441
+
if (err) {
442
+
err_free_inst:
443
+
rsassa_pkcs1_free(inst);
444
+
}
445
+
return err;
446
+
}
447
+
448
+
struct crypto_template rsassa_pkcs1_tmpl = {
449
+
.name = "pkcs1",
450
+
.create = rsassa_pkcs1_create,
451
+
.module = THIS_MODULE,
452
+
};
453
+
454
+
MODULE_ALIAS_CRYPTO("pkcs1");
+94
-55
crypto/sig.c
+94
-55
crypto/sig.c
···
5
5
* Copyright (c) 2023 Herbert Xu <herbert@gondor.apana.org.au>
6
6
*/
7
7
8
-
#include <crypto/akcipher.h>
9
8
#include <crypto/internal/sig.h>
10
9
#include <linux/cryptouser.h>
11
10
#include <linux/kernel.h>
12
11
#include <linux/module.h>
13
-
#include <linux/scatterlist.h>
14
12
#include <linux/seq_file.h>
15
13
#include <linux/string.h>
16
14
#include <net/netlink.h>
···
17
19
18
20
#define CRYPTO_ALG_TYPE_SIG_MASK 0x0000000e
19
21
20
-
static const struct crypto_type crypto_sig_type;
22
+
static void crypto_sig_exit_tfm(struct crypto_tfm *tfm)
23
+
{
24
+
struct crypto_sig *sig = __crypto_sig_tfm(tfm);
25
+
struct sig_alg *alg = crypto_sig_alg(sig);
26
+
27
+
alg->exit(sig);
28
+
}
21
29
22
30
static int crypto_sig_init_tfm(struct crypto_tfm *tfm)
23
31
{
24
-
if (tfm->__crt_alg->cra_type != &crypto_sig_type)
25
-
return crypto_init_akcipher_ops_sig(tfm);
32
+
struct crypto_sig *sig = __crypto_sig_tfm(tfm);
33
+
struct sig_alg *alg = crypto_sig_alg(sig);
34
+
35
+
if (alg->exit)
36
+
sig->base.exit = crypto_sig_exit_tfm;
37
+
38
+
if (alg->init)
39
+
return alg->init(sig);
26
40
27
41
return 0;
42
+
}
43
+
44
+
static void crypto_sig_free_instance(struct crypto_instance *inst)
45
+
{
46
+
struct sig_instance *sig = sig_instance(inst);
47
+
48
+
sig->free(sig);
28
49
}
29
50
30
51
static void __maybe_unused crypto_sig_show(struct seq_file *m,
···
55
38
static int __maybe_unused crypto_sig_report(struct sk_buff *skb,
56
39
struct crypto_alg *alg)
57
40
{
58
-
struct crypto_report_akcipher rsig = {};
41
+
struct crypto_report_sig rsig = {};
59
42
60
43
strscpy(rsig.type, "sig", sizeof(rsig.type));
61
44
62
-
return nla_put(skb, CRYPTOCFGA_REPORT_AKCIPHER, sizeof(rsig), &rsig);
45
+
return nla_put(skb, CRYPTOCFGA_REPORT_SIG, sizeof(rsig), &rsig);
63
46
}
64
47
65
48
static const struct crypto_type crypto_sig_type = {
66
49
.extsize = crypto_alg_extsize,
67
50
.init_tfm = crypto_sig_init_tfm,
51
+
.free = crypto_sig_free_instance,
68
52
#ifdef CONFIG_PROC_FS
69
53
.show = crypto_sig_show,
70
54
#endif
···
84
66
}
85
67
EXPORT_SYMBOL_GPL(crypto_alloc_sig);
86
68
87
-
int crypto_sig_maxsize(struct crypto_sig *tfm)
69
+
static int sig_default_sign(struct crypto_sig *tfm,
70
+
const void *src, unsigned int slen,
71
+
void *dst, unsigned int dlen)
88
72
{
89
-
struct crypto_akcipher **ctx = crypto_sig_ctx(tfm);
90
-
91
-
return crypto_akcipher_maxsize(*ctx);
73
+
return -ENOSYS;
92
74
}
93
-
EXPORT_SYMBOL_GPL(crypto_sig_maxsize);
94
75
95
-
int crypto_sig_sign(struct crypto_sig *tfm,
96
-
const void *src, unsigned int slen,
97
-
void *dst, unsigned int dlen)
76
+
static int sig_default_verify(struct crypto_sig *tfm,
77
+
const void *src, unsigned int slen,
78
+
const void *dst, unsigned int dlen)
98
79
{
99
-
struct crypto_akcipher **ctx = crypto_sig_ctx(tfm);
100
-
struct crypto_akcipher_sync_data data = {
101
-
.tfm = *ctx,
102
-
.src = src,
103
-
.dst = dst,
104
-
.slen = slen,
105
-
.dlen = dlen,
106
-
};
107
-
108
-
return crypto_akcipher_sync_prep(&data) ?:
109
-
crypto_akcipher_sync_post(&data,
110
-
crypto_akcipher_sign(data.req));
80
+
return -ENOSYS;
111
81
}
112
-
EXPORT_SYMBOL_GPL(crypto_sig_sign);
113
82
114
-
int crypto_sig_verify(struct crypto_sig *tfm,
115
-
const void *src, unsigned int slen,
116
-
const void *digest, unsigned int dlen)
83
+
static int sig_default_set_key(struct crypto_sig *tfm,
84
+
const void *key, unsigned int keylen)
117
85
{
118
-
struct crypto_akcipher **ctx = crypto_sig_ctx(tfm);
119
-
struct crypto_akcipher_sync_data data = {
120
-
.tfm = *ctx,
121
-
.src = src,
122
-
.slen = slen,
123
-
.dlen = dlen,
124
-
};
86
+
return -ENOSYS;
87
+
}
88
+
89
+
static int sig_prepare_alg(struct sig_alg *alg)
90
+
{
91
+
struct crypto_alg *base = &alg->base;
92
+
93
+
if (!alg->sign)
94
+
alg->sign = sig_default_sign;
95
+
if (!alg->verify)
96
+
alg->verify = sig_default_verify;
97
+
if (!alg->set_priv_key)
98
+
alg->set_priv_key = sig_default_set_key;
99
+
if (!alg->set_pub_key)
100
+
return -EINVAL;
101
+
if (!alg->key_size)
102
+
return -EINVAL;
103
+
if (!alg->max_size)
104
+
alg->max_size = alg->key_size;
105
+
if (!alg->digest_size)
106
+
alg->digest_size = alg->key_size;
107
+
108
+
base->cra_type = &crypto_sig_type;
109
+
base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK;
110
+
base->cra_flags |= CRYPTO_ALG_TYPE_SIG;
111
+
112
+
return 0;
113
+
}
114
+
115
+
int crypto_register_sig(struct sig_alg *alg)
116
+
{
117
+
struct crypto_alg *base = &alg->base;
125
118
int err;
126
119
127
-
err = crypto_akcipher_sync_prep(&data);
120
+
err = sig_prepare_alg(alg);
128
121
if (err)
129
122
return err;
130
123
131
-
memcpy(data.buf + slen, digest, dlen);
132
-
133
-
return crypto_akcipher_sync_post(&data,
134
-
crypto_akcipher_verify(data.req));
124
+
return crypto_register_alg(base);
135
125
}
136
-
EXPORT_SYMBOL_GPL(crypto_sig_verify);
126
+
EXPORT_SYMBOL_GPL(crypto_register_sig);
137
127
138
-
int crypto_sig_set_pubkey(struct crypto_sig *tfm,
139
-
const void *key, unsigned int keylen)
128
+
void crypto_unregister_sig(struct sig_alg *alg)
140
129
{
141
-
struct crypto_akcipher **ctx = crypto_sig_ctx(tfm);
142
-
143
-
return crypto_akcipher_set_pub_key(*ctx, key, keylen);
130
+
crypto_unregister_alg(&alg->base);
144
131
}
145
-
EXPORT_SYMBOL_GPL(crypto_sig_set_pubkey);
132
+
EXPORT_SYMBOL_GPL(crypto_unregister_sig);
146
133
147
-
int crypto_sig_set_privkey(struct crypto_sig *tfm,
148
-
const void *key, unsigned int keylen)
134
+
int sig_register_instance(struct crypto_template *tmpl,
135
+
struct sig_instance *inst)
149
136
{
150
-
struct crypto_akcipher **ctx = crypto_sig_ctx(tfm);
137
+
int err;
151
138
152
-
return crypto_akcipher_set_priv_key(*ctx, key, keylen);
139
+
if (WARN_ON(!inst->free))
140
+
return -EINVAL;
141
+
142
+
err = sig_prepare_alg(&inst->alg);
143
+
if (err)
144
+
return err;
145
+
146
+
return crypto_register_instance(tmpl, sig_crypto_instance(inst));
153
147
}
154
-
EXPORT_SYMBOL_GPL(crypto_sig_set_privkey);
148
+
EXPORT_SYMBOL_GPL(sig_register_instance);
149
+
150
+
int crypto_grab_sig(struct crypto_sig_spawn *spawn,
151
+
struct crypto_instance *inst,
152
+
const char *name, u32 type, u32 mask)
153
+
{
154
+
spawn->base.frontend = &crypto_sig_type;
155
+
return crypto_grab_spawn(&spawn->base, inst, name, type, mask);
156
+
}
157
+
EXPORT_SYMBOL_GPL(crypto_grab_sig);
155
158
156
159
MODULE_LICENSE("GPL");
157
160
MODULE_DESCRIPTION("Public Key Signature Algorithms");
+227
-105
crypto/testmgr.c
+227
-105
crypto/testmgr.c
···
33
33
#include <crypto/akcipher.h>
34
34
#include <crypto/kpp.h>
35
35
#include <crypto/acompress.h>
36
+
#include <crypto/sig.h>
36
37
#include <crypto/internal/cipher.h>
37
38
#include <crypto/internal/simd.h>
38
39
···
132
131
unsigned int count;
133
132
};
134
133
134
+
struct sig_test_suite {
135
+
const struct sig_testvec *vecs;
136
+
unsigned int count;
137
+
};
138
+
135
139
struct kpp_test_suite {
136
140
const struct kpp_testvec *vecs;
137
141
unsigned int count;
···
157
151
struct cprng_test_suite cprng;
158
152
struct drbg_test_suite drbg;
159
153
struct akcipher_test_suite akcipher;
154
+
struct sig_test_suite sig;
160
155
struct kpp_test_suite kpp;
161
156
} suite;
162
157
};
···
4130
4123
struct crypto_wait wait;
4131
4124
unsigned int out_len_max, out_len = 0;
4132
4125
int err = -ENOMEM;
4133
-
struct scatterlist src, dst, src_tab[3];
4134
-
const char *m, *c;
4135
-
unsigned int m_size, c_size;
4136
-
const char *op;
4137
-
u8 *key, *ptr;
4126
+
struct scatterlist src, dst, src_tab[2];
4127
+
const char *c;
4128
+
unsigned int c_size;
4138
4129
4139
4130
if (testmgr_alloc_buf(xbuf))
4140
4131
return err;
···
4143
4138
4144
4139
crypto_init_wait(&wait);
4145
4140
4146
-
key = kmalloc(vecs->key_len + sizeof(u32) * 2 + vecs->param_len,
4147
-
GFP_KERNEL);
4148
-
if (!key)
4149
-
goto free_req;
4150
-
memcpy(key, vecs->key, vecs->key_len);
4151
-
ptr = key + vecs->key_len;
4152
-
ptr = test_pack_u32(ptr, vecs->algo);
4153
-
ptr = test_pack_u32(ptr, vecs->param_len);
4154
-
memcpy(ptr, vecs->params, vecs->param_len);
4155
-
4156
4141
if (vecs->public_key_vec)
4157
-
err = crypto_akcipher_set_pub_key(tfm, key, vecs->key_len);
4142
+
err = crypto_akcipher_set_pub_key(tfm, vecs->key,
4143
+
vecs->key_len);
4158
4144
else
4159
-
err = crypto_akcipher_set_priv_key(tfm, key, vecs->key_len);
4145
+
err = crypto_akcipher_set_priv_key(tfm, vecs->key,
4146
+
vecs->key_len);
4160
4147
if (err)
4161
-
goto free_key;
4148
+
goto free_req;
4162
4149
4163
-
/*
4164
-
* First run test which do not require a private key, such as
4165
-
* encrypt or verify.
4166
-
*/
4150
+
/* First run encrypt test which does not require a private key */
4167
4151
err = -ENOMEM;
4168
4152
out_len_max = crypto_akcipher_maxsize(tfm);
4169
4153
outbuf_enc = kzalloc(out_len_max, GFP_KERNEL);
4170
4154
if (!outbuf_enc)
4171
-
goto free_key;
4155
+
goto free_req;
4172
4156
4173
-
if (!vecs->siggen_sigver_test) {
4174
-
m = vecs->m;
4175
-
m_size = vecs->m_size;
4176
-
c = vecs->c;
4177
-
c_size = vecs->c_size;
4178
-
op = "encrypt";
4179
-
} else {
4180
-
/* Swap args so we could keep plaintext (digest)
4181
-
* in vecs->m, and cooked signature in vecs->c.
4182
-
*/
4183
-
m = vecs->c; /* signature */
4184
-
m_size = vecs->c_size;
4185
-
c = vecs->m; /* digest */
4186
-
c_size = vecs->m_size;
4187
-
op = "verify";
4188
-
}
4157
+
c = vecs->c;
4158
+
c_size = vecs->c_size;
4189
4159
4190
4160
err = -E2BIG;
4191
-
if (WARN_ON(m_size > PAGE_SIZE))
4161
+
if (WARN_ON(vecs->m_size > PAGE_SIZE))
4192
4162
goto free_all;
4193
-
memcpy(xbuf[0], m, m_size);
4163
+
memcpy(xbuf[0], vecs->m, vecs->m_size);
4194
4164
4195
-
sg_init_table(src_tab, 3);
4165
+
sg_init_table(src_tab, 2);
4196
4166
sg_set_buf(&src_tab[0], xbuf[0], 8);
4197
-
sg_set_buf(&src_tab[1], xbuf[0] + 8, m_size - 8);
4198
-
if (vecs->siggen_sigver_test) {
4199
-
if (WARN_ON(c_size > PAGE_SIZE))
4200
-
goto free_all;
4201
-
memcpy(xbuf[1], c, c_size);
4202
-
sg_set_buf(&src_tab[2], xbuf[1], c_size);
4203
-
akcipher_request_set_crypt(req, src_tab, NULL, m_size, c_size);
4204
-
} else {
4205
-
sg_init_one(&dst, outbuf_enc, out_len_max);
4206
-
akcipher_request_set_crypt(req, src_tab, &dst, m_size,
4207
-
out_len_max);
4208
-
}
4167
+
sg_set_buf(&src_tab[1], xbuf[0] + 8, vecs->m_size - 8);
4168
+
sg_init_one(&dst, outbuf_enc, out_len_max);
4169
+
akcipher_request_set_crypt(req, src_tab, &dst, vecs->m_size,
4170
+
out_len_max);
4209
4171
akcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
4210
4172
crypto_req_done, &wait);
4211
4173
4212
-
err = crypto_wait_req(vecs->siggen_sigver_test ?
4213
-
/* Run asymmetric signature verification */
4214
-
crypto_akcipher_verify(req) :
4215
-
/* Run asymmetric encrypt */
4216
-
crypto_akcipher_encrypt(req), &wait);
4174
+
err = crypto_wait_req(crypto_akcipher_encrypt(req), &wait);
4217
4175
if (err) {
4218
-
pr_err("alg: akcipher: %s test failed. err %d\n", op, err);
4176
+
pr_err("alg: akcipher: encrypt test failed. err %d\n", err);
4219
4177
goto free_all;
4220
4178
}
4221
-
if (!vecs->siggen_sigver_test && c) {
4179
+
if (c) {
4222
4180
if (req->dst_len != c_size) {
4223
-
pr_err("alg: akcipher: %s test failed. Invalid output len\n",
4224
-
op);
4181
+
pr_err("alg: akcipher: encrypt test failed. Invalid output len\n");
4225
4182
err = -EINVAL;
4226
4183
goto free_all;
4227
4184
}
4228
4185
/* verify that encrypted message is equal to expected */
4229
4186
if (memcmp(c, outbuf_enc, c_size) != 0) {
4230
-
pr_err("alg: akcipher: %s test failed. Invalid output\n",
4231
-
op);
4187
+
pr_err("alg: akcipher: encrypt test failed. Invalid output\n");
4232
4188
hexdump(outbuf_enc, c_size);
4233
4189
err = -EINVAL;
4234
4190
goto free_all;
···
4197
4231
}
4198
4232
4199
4233
/*
4200
-
* Don't invoke (decrypt or sign) test which require a private key
4234
+
* Don't invoke decrypt test which requires a private key
4201
4235
* for vectors with only a public key.
4202
4236
*/
4203
4237
if (vecs->public_key_vec) {
···
4210
4244
goto free_all;
4211
4245
}
4212
4246
4213
-
if (!vecs->siggen_sigver_test && !c) {
4247
+
if (!c) {
4214
4248
c = outbuf_enc;
4215
4249
c_size = req->dst_len;
4216
4250
}
4217
4251
4218
4252
err = -E2BIG;
4219
-
op = vecs->siggen_sigver_test ? "sign" : "decrypt";
4220
4253
if (WARN_ON(c_size > PAGE_SIZE))
4221
4254
goto free_all;
4222
4255
memcpy(xbuf[0], c, c_size);
···
4225
4260
crypto_init_wait(&wait);
4226
4261
akcipher_request_set_crypt(req, &src, &dst, c_size, out_len_max);
4227
4262
4228
-
err = crypto_wait_req(vecs->siggen_sigver_test ?
4229
-
/* Run asymmetric signature generation */
4230
-
crypto_akcipher_sign(req) :
4231
-
/* Run asymmetric decrypt */
4232
-
crypto_akcipher_decrypt(req), &wait);
4263
+
err = crypto_wait_req(crypto_akcipher_decrypt(req), &wait);
4233
4264
if (err) {
4234
-
pr_err("alg: akcipher: %s test failed. err %d\n", op, err);
4265
+
pr_err("alg: akcipher: decrypt test failed. err %d\n", err);
4235
4266
goto free_all;
4236
4267
}
4237
4268
out_len = req->dst_len;
4238
-
if (out_len < m_size) {
4239
-
pr_err("alg: akcipher: %s test failed. Invalid output len %u\n",
4240
-
op, out_len);
4269
+
if (out_len < vecs->m_size) {
4270
+
pr_err("alg: akcipher: decrypt test failed. Invalid output len %u\n",
4271
+
out_len);
4241
4272
err = -EINVAL;
4242
4273
goto free_all;
4243
4274
}
4244
4275
/* verify that decrypted message is equal to the original msg */
4245
-
if (memchr_inv(outbuf_dec, 0, out_len - m_size) ||
4246
-
memcmp(m, outbuf_dec + out_len - m_size, m_size)) {
4247
-
pr_err("alg: akcipher: %s test failed. Invalid output\n", op);
4276
+
if (memchr_inv(outbuf_dec, 0, out_len - vecs->m_size) ||
4277
+
memcmp(vecs->m, outbuf_dec + out_len - vecs->m_size,
4278
+
vecs->m_size)) {
4279
+
pr_err("alg: akcipher: decrypt test failed. Invalid output\n");
4248
4280
hexdump(outbuf_dec, out_len);
4249
4281
err = -EINVAL;
4250
4282
}
4251
4283
free_all:
4252
4284
kfree(outbuf_dec);
4253
4285
kfree(outbuf_enc);
4254
-
free_key:
4255
-
kfree(key);
4256
4286
free_req:
4257
4287
akcipher_request_free(req);
4258
4288
free_xbuf:
···
4294
4334
desc->suite.akcipher.count);
4295
4335
4296
4336
crypto_free_akcipher(tfm);
4337
+
return err;
4338
+
}
4339
+
4340
+
static int test_sig_one(struct crypto_sig *tfm, const struct sig_testvec *vecs)
4341
+
{
4342
+
u8 *ptr, *key __free(kfree);
4343
+
int err, sig_size;
4344
+
4345
+
key = kmalloc(vecs->key_len + 2 * sizeof(u32) + vecs->param_len,
4346
+
GFP_KERNEL);
4347
+
if (!key)
4348
+
return -ENOMEM;
4349
+
4350
+
/* ecrdsa expects additional parameters appended to the key */
4351
+
memcpy(key, vecs->key, vecs->key_len);
4352
+
ptr = key + vecs->key_len;
4353
+
ptr = test_pack_u32(ptr, vecs->algo);
4354
+
ptr = test_pack_u32(ptr, vecs->param_len);
4355
+
memcpy(ptr, vecs->params, vecs->param_len);
4356
+
4357
+
if (vecs->public_key_vec)
4358
+
err = crypto_sig_set_pubkey(tfm, key, vecs->key_len);
4359
+
else
4360
+
err = crypto_sig_set_privkey(tfm, key, vecs->key_len);
4361
+
if (err)
4362
+
return err;
4363
+
4364
+
/*
4365
+
* Run asymmetric signature verification first
4366
+
* (which does not require a private key)
4367
+
*/
4368
+
err = crypto_sig_verify(tfm, vecs->c, vecs->c_size,
4369
+
vecs->m, vecs->m_size);
4370
+
if (err) {
4371
+
pr_err("alg: sig: verify test failed: err %d\n", err);
4372
+
return err;
4373
+
}
4374
+
4375
+
/*
4376
+
* Don't invoke sign test (which requires a private key)
4377
+
* for vectors with only a public key.
4378
+
*/
4379
+
if (vecs->public_key_vec)
4380
+
return 0;
4381
+
4382
+
sig_size = crypto_sig_keysize(tfm);
4383
+
if (sig_size < vecs->c_size) {
4384
+
pr_err("alg: sig: invalid maxsize %u\n", sig_size);
4385
+
return -EINVAL;
4386
+
}
4387
+
4388
+
u8 *sig __free(kfree) = kzalloc(sig_size, GFP_KERNEL);
4389
+
if (!sig)
4390
+
return -ENOMEM;
4391
+
4392
+
/* Run asymmetric signature generation */
4393
+
err = crypto_sig_sign(tfm, vecs->m, vecs->m_size, sig, sig_size);
4394
+
if (err) {
4395
+
pr_err("alg: sig: sign test failed: err %d\n", err);
4396
+
return err;
4397
+
}
4398
+
4399
+
/* Verify that generated signature equals cooked signature */
4400
+
if (memcmp(sig, vecs->c, vecs->c_size) ||
4401
+
memchr_inv(sig + vecs->c_size, 0, sig_size - vecs->c_size)) {
4402
+
pr_err("alg: sig: sign test failed: invalid output\n");
4403
+
hexdump(sig, sig_size);
4404
+
return -EINVAL;
4405
+
}
4406
+
4407
+
return 0;
4408
+
}
4409
+
4410
+
static int test_sig(struct crypto_sig *tfm, const char *alg,
4411
+
const struct sig_testvec *vecs, unsigned int tcount)
4412
+
{
4413
+
const char *algo = crypto_tfm_alg_driver_name(crypto_sig_tfm(tfm));
4414
+
int ret, i;
4415
+
4416
+
for (i = 0; i < tcount; i++) {
4417
+
ret = test_sig_one(tfm, vecs++);
4418
+
if (ret) {
4419
+
pr_err("alg: sig: test %d failed for %s: err %d\n",
4420
+
i + 1, algo, ret);
4421
+
return ret;
4422
+
}
4423
+
}
4424
+
return 0;
4425
+
}
4426
+
4427
+
static int alg_test_sig(const struct alg_test_desc *desc, const char *driver,
4428
+
u32 type, u32 mask)
4429
+
{
4430
+
struct crypto_sig *tfm;
4431
+
int err = 0;
4432
+
4433
+
tfm = crypto_alloc_sig(driver, type, mask);
4434
+
if (IS_ERR(tfm)) {
4435
+
pr_err("alg: sig: Failed to load tfm for %s: %ld\n",
4436
+
driver, PTR_ERR(tfm));
4437
+
return PTR_ERR(tfm);
4438
+
}
4439
+
if (desc->suite.sig.vecs)
4440
+
err = test_sig(tfm, desc->alg, desc->suite.sig.vecs,
4441
+
desc->suite.sig.count);
4442
+
4443
+
crypto_free_sig(tfm);
4297
4444
return err;
4298
4445
}
4299
4446
···
5193
5126
}
5194
5127
}, {
5195
5128
.alg = "ecdsa-nist-p192",
5196
-
.test = alg_test_akcipher,
5129
+
.test = alg_test_sig,
5197
5130
.suite = {
5198
-
.akcipher = __VECS(ecdsa_nist_p192_tv_template)
5131
+
.sig = __VECS(ecdsa_nist_p192_tv_template)
5199
5132
}
5200
5133
}, {
5201
5134
.alg = "ecdsa-nist-p256",
5202
-
.test = alg_test_akcipher,
5135
+
.test = alg_test_sig,
5203
5136
.fips_allowed = 1,
5204
5137
.suite = {
5205
-
.akcipher = __VECS(ecdsa_nist_p256_tv_template)
5138
+
.sig = __VECS(ecdsa_nist_p256_tv_template)
5206
5139
}
5207
5140
}, {
5208
5141
.alg = "ecdsa-nist-p384",
5209
-
.test = alg_test_akcipher,
5142
+
.test = alg_test_sig,
5210
5143
.fips_allowed = 1,
5211
5144
.suite = {
5212
-
.akcipher = __VECS(ecdsa_nist_p384_tv_template)
5145
+
.sig = __VECS(ecdsa_nist_p384_tv_template)
5213
5146
}
5214
5147
}, {
5215
5148
.alg = "ecdsa-nist-p521",
5216
-
.test = alg_test_akcipher,
5149
+
.test = alg_test_sig,
5217
5150
.fips_allowed = 1,
5218
5151
.suite = {
5219
-
.akcipher = __VECS(ecdsa_nist_p521_tv_template)
5152
+
.sig = __VECS(ecdsa_nist_p521_tv_template)
5220
5153
}
5221
5154
}, {
5222
5155
.alg = "ecrdsa",
5223
-
.test = alg_test_akcipher,
5156
+
.test = alg_test_sig,
5224
5157
.suite = {
5225
-
.akcipher = __VECS(ecrdsa_tv_template)
5158
+
.sig = __VECS(ecrdsa_tv_template)
5226
5159
}
5227
5160
}, {
5228
5161
.alg = "essiv(authenc(hmac(sha256),cbc(aes)),sha256)",
···
5515
5448
.hash = __VECS(nhpoly1305_tv_template)
5516
5449
}
5517
5450
}, {
5451
+
.alg = "p1363(ecdsa-nist-p192)",
5452
+
.test = alg_test_null,
5453
+
}, {
5454
+
.alg = "p1363(ecdsa-nist-p256)",
5455
+
.test = alg_test_sig,
5456
+
.fips_allowed = 1,
5457
+
.suite = {
5458
+
.sig = __VECS(p1363_ecdsa_nist_p256_tv_template)
5459
+
}
5460
+
}, {
5461
+
.alg = "p1363(ecdsa-nist-p384)",
5462
+
.test = alg_test_null,
5463
+
.fips_allowed = 1,
5464
+
}, {
5465
+
.alg = "p1363(ecdsa-nist-p521)",
5466
+
.test = alg_test_null,
5467
+
.fips_allowed = 1,
5468
+
}, {
5518
5469
.alg = "pcbc(fcrypt)",
5519
5470
.test = alg_test_skcipher,
5520
5471
.suite = {
5521
5472
.cipher = __VECS(fcrypt_pcbc_tv_template)
5522
5473
}
5523
5474
}, {
5524
-
.alg = "pkcs1pad(rsa,sha224)",
5525
-
.test = alg_test_null,
5526
-
.fips_allowed = 1,
5527
-
}, {
5528
-
.alg = "pkcs1pad(rsa,sha256)",
5529
-
.test = alg_test_akcipher,
5530
-
.fips_allowed = 1,
5475
+
.alg = "pkcs1(rsa,none)",
5476
+
.test = alg_test_sig,
5531
5477
.suite = {
5532
-
.akcipher = __VECS(pkcs1pad_rsa_tv_template)
5478
+
.sig = __VECS(pkcs1_rsa_none_tv_template)
5533
5479
}
5534
5480
}, {
5535
-
.alg = "pkcs1pad(rsa,sha3-256)",
5481
+
.alg = "pkcs1(rsa,sha224)",
5536
5482
.test = alg_test_null,
5537
5483
.fips_allowed = 1,
5538
5484
}, {
5539
-
.alg = "pkcs1pad(rsa,sha3-384)",
5485
+
.alg = "pkcs1(rsa,sha256)",
5486
+
.test = alg_test_sig,
5487
+
.fips_allowed = 1,
5488
+
.suite = {
5489
+
.sig = __VECS(pkcs1_rsa_tv_template)
5490
+
}
5491
+
}, {
5492
+
.alg = "pkcs1(rsa,sha3-256)",
5540
5493
.test = alg_test_null,
5541
5494
.fips_allowed = 1,
5542
5495
}, {
5543
-
.alg = "pkcs1pad(rsa,sha3-512)",
5496
+
.alg = "pkcs1(rsa,sha3-384)",
5544
5497
.test = alg_test_null,
5545
5498
.fips_allowed = 1,
5546
5499
}, {
5547
-
.alg = "pkcs1pad(rsa,sha384)",
5500
+
.alg = "pkcs1(rsa,sha3-512)",
5548
5501
.test = alg_test_null,
5549
5502
.fips_allowed = 1,
5550
5503
}, {
5551
-
.alg = "pkcs1pad(rsa,sha512)",
5504
+
.alg = "pkcs1(rsa,sha384)",
5505
+
.test = alg_test_null,
5506
+
.fips_allowed = 1,
5507
+
}, {
5508
+
.alg = "pkcs1(rsa,sha512)",
5509
+
.test = alg_test_null,
5510
+
.fips_allowed = 1,
5511
+
}, {
5512
+
.alg = "pkcs1pad(rsa)",
5552
5513
.test = alg_test_null,
5553
5514
.fips_allowed = 1,
5554
5515
}, {
···
5772
5677
.test = alg_test_hash,
5773
5678
.suite = {
5774
5679
.hash = __VECS(wp512_tv_template)
5680
+
}
5681
+
}, {
5682
+
.alg = "x962(ecdsa-nist-p192)",
5683
+
.test = alg_test_sig,
5684
+
.suite = {
5685
+
.sig = __VECS(x962_ecdsa_nist_p192_tv_template)
5686
+
}
5687
+
}, {
5688
+
.alg = "x962(ecdsa-nist-p256)",
5689
+
.test = alg_test_sig,
5690
+
.fips_allowed = 1,
5691
+
.suite = {
5692
+
.sig = __VECS(x962_ecdsa_nist_p256_tv_template)
5693
+
}
5694
+
}, {
5695
+
.alg = "x962(ecdsa-nist-p384)",
5696
+
.test = alg_test_sig,
5697
+
.fips_allowed = 1,
5698
+
.suite = {
5699
+
.sig = __VECS(x962_ecdsa_nist_p384_tv_template)
5700
+
}
5701
+
}, {
5702
+
.alg = "x962(ecdsa-nist-p521)",
5703
+
.test = alg_test_sig,
5704
+
.fips_allowed = 1,
5705
+
.suite = {
5706
+
.sig = __VECS(x962_ecdsa_nist_p521_tv_template)
5775
5707
}
5776
5708
}, {
5777
5709
.alg = "xcbc(aes)",
+945
-280
crypto/testmgr.h
+945
-280
crypto/testmgr.h
···
21
21
#define _CRYPTO_TESTMGR_H
22
22
23
23
#include <linux/oid_registry.h>
24
+
#include <crypto/internal/ecc.h>
24
25
25
26
#define MAX_IVLEN 32
26
27
···
151
150
152
151
struct akcipher_testvec {
153
152
const unsigned char *key;
153
+
const unsigned char *m;
154
+
const unsigned char *c;
155
+
unsigned int key_len;
156
+
unsigned int m_size;
157
+
unsigned int c_size;
158
+
bool public_key_vec;
159
+
};
160
+
161
+
struct sig_testvec {
162
+
const unsigned char *key;
154
163
const unsigned char *params;
155
164
const unsigned char *m;
156
165
const unsigned char *c;
···
169
158
unsigned int m_size;
170
159
unsigned int c_size;
171
160
bool public_key_vec;
172
-
bool siggen_sigver_test;
173
161
enum OID algo;
174
162
};
175
163
···
657
647
}
658
648
};
659
649
650
+
#ifdef CONFIG_CPU_BIG_ENDIAN
651
+
#define be64_to_cpua(b1, b2, b3, b4, b5, b6, b7, b8) \
652
+
0x##b1, 0x##b2, 0x##b3, 0x##b4, 0x##b5, 0x##b6, 0x##b7, 0x##b8
653
+
#else
654
+
#define be64_to_cpua(b1, b2, b3, b4, b5, b6, b7, b8) \
655
+
0x##b8, 0x##b7, 0x##b6, 0x##b5, 0x##b4, 0x##b3, 0x##b2, 0x##b1
656
+
#endif
657
+
660
658
/*
661
659
* ECDSA test vectors.
662
660
*/
663
-
static const struct akcipher_testvec ecdsa_nist_p192_tv_template[] = {
661
+
static const struct sig_testvec ecdsa_nist_p192_tv_template[] = {
664
662
{
665
-
.key =
663
+
.key = /* secp192r1(sha1) */
666
664
"\x04\xf7\x46\xf8\x2f\x15\xf6\x22\x8e\xd7\x57\x4f\xcc\xe7\xbb\xc1"
667
665
"\xd4\x09\x73\xcf\xea\xd0\x15\x07\x3d\xa5\x8a\x8a\x95\x43\xe4\x68"
668
666
"\xea\xc6\x25\xc1\xc1\x01\x25\x4c\x7e\xc3\x3c\xa6\x04\x0a\xe7\x08"
669
667
"\x98",
670
668
.key_len = 49,
671
-
.params =
672
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
673
-
"\xce\x3d\x03\x01\x01",
674
-
.param_len = 21,
675
669
.m =
676
670
"\xcd\xb9\xd2\x1c\xb7\x6f\xcd\x44\xb3\xfd\x63\xea\xa3\x66\x7f\xae"
677
671
"\x63\x85\xe7\x82",
678
672
.m_size = 20,
679
-
.algo = OID_id_ecdsa_with_sha1,
680
-
.c =
681
-
"\x30\x35\x02\x19\x00\xba\xe5\x93\x83\x6e\xb6\x3b\x63\xa0\x27\x91"
682
-
"\xc6\xf6\x7f\xc3\x09\xad\x59\xad\x88\x27\xd6\x92\x6b\x02\x18\x10"
683
-
"\x68\x01\x9d\xba\xce\x83\x08\xef\x95\x52\x7b\xa0\x0f\xe4\x18\x86"
684
-
"\x80\x6f\xa5\x79\x77\xda\xd0",
685
-
.c_size = 55,
673
+
.c = (const unsigned char[]){
674
+
be64_to_cpua(ad, 59, ad, 88, 27, d6, 92, 6b),
675
+
be64_to_cpua(a0, 27, 91, c6, f6, 7f, c3, 09),
676
+
be64_to_cpua(ba, e5, 93, 83, 6e, b6, 3b, 63),
677
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
678
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
679
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
680
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
681
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
682
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
683
+
be64_to_cpua(86, 80, 6f, a5, 79, 77, da, d0),
684
+
be64_to_cpua(ef, 95, 52, 7b, a0, 0f, e4, 18),
685
+
be64_to_cpua(10, 68, 01, 9d, ba, ce, 83, 08),
686
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
687
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
688
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
689
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
690
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
691
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
692
+
.c_size = ECC_MAX_BYTES * 2,
686
693
.public_key_vec = true,
687
-
.siggen_sigver_test = true,
688
694
}, {
689
-
.key =
695
+
.key = /* secp192r1(sha224) */
690
696
"\x04\xb6\x4b\xb1\xd1\xac\xba\x24\x8f\x65\xb2\x60\x00\x90\xbf\xbd"
691
697
"\x78\x05\x73\xe9\x79\x1d\x6f\x7c\x0b\xd2\xc3\x93\xa7\x28\xe1\x75"
692
698
"\xf7\xd5\x95\x1d\x28\x10\xc0\x75\x50\x5c\x1a\x4f\x3f\x8f\xa5\xee"
693
699
"\xa3",
694
700
.key_len = 49,
695
-
.params =
696
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
697
-
"\xce\x3d\x03\x01\x01",
698
-
.param_len = 21,
699
701
.m =
700
702
"\x8d\xd6\xb8\x3e\xe5\xff\x23\xf6\x25\xa2\x43\x42\x74\x45\xa7\x40"
701
703
"\x3a\xff\x2f\xe1\xd3\xf6\x9f\xe8\x33\xcb\x12\x11",
702
704
.m_size = 28,
703
-
.algo = OID_id_ecdsa_with_sha224,
704
-
.c =
705
-
"\x30\x34\x02\x18\x5a\x8b\x82\x69\x7e\x8a\x0a\x09\x14\xf8\x11\x2b"
706
-
"\x55\xdc\xae\x37\x83\x7b\x12\xe6\xb6\x5b\xcb\xd4\x02\x18\x6a\x14"
707
-
"\x4f\x53\x75\xc8\x02\x48\xeb\xc3\x92\x0f\x1e\x72\xee\xc4\xa3\xe3"
708
-
"\x5c\x99\xdb\x92\x5b\x36",
709
-
.c_size = 54,
705
+
.c = (const unsigned char[]){
706
+
be64_to_cpua(83, 7b, 12, e6, b6, 5b, cb, d4),
707
+
be64_to_cpua(14, f8, 11, 2b, 55, dc, ae, 37),
708
+
be64_to_cpua(5a, 8b, 82, 69, 7e, 8a, 0a, 09),
709
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
710
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
711
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
712
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
713
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
714
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
715
+
be64_to_cpua(a3, e3, 5c, 99, db, 92, 5b, 36),
716
+
be64_to_cpua(eb, c3, 92, 0f, 1e, 72, ee, c4),
717
+
be64_to_cpua(6a, 14, 4f, 53, 75, c8, 02, 48),
718
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
719
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
720
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
721
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
722
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
723
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
724
+
.c_size = ECC_MAX_BYTES * 2,
710
725
.public_key_vec = true,
711
-
.siggen_sigver_test = true,
712
726
}, {
713
-
.key =
727
+
.key = /* secp192r1(sha256) */
714
728
"\x04\xe2\x51\x24\x9b\xf7\xb6\x32\x82\x39\x66\x3d\x5b\xec\x3b\xae"
715
729
"\x0c\xd5\xf2\x67\xd1\xc7\xe1\x02\xe4\xbf\x90\x62\xb8\x55\x75\x56"
716
730
"\x69\x20\x5e\xcb\x4e\xca\x33\xd6\xcb\x62\x6b\x94\xa9\xa2\xe9\x58"
717
731
"\x91",
718
732
.key_len = 49,
719
-
.params =
720
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
721
-
"\xce\x3d\x03\x01\x01",
722
-
.param_len = 21,
723
733
.m =
724
734
"\x35\xec\xa1\xa0\x9e\x14\xde\x33\x03\xb6\xf6\xbd\x0c\x2f\xb2\xfd"
725
735
"\x1f\x27\x82\xa5\xd7\x70\x3f\xef\xa0\x82\x69\x8e\x73\x31\x8e\xd7",
726
736
.m_size = 32,
727
-
.algo = OID_id_ecdsa_with_sha256,
728
-
.c =
729
-
"\x30\x35\x02\x18\x3f\x72\x3f\x1f\x42\xd2\x3f\x1d\x6b\x1a\x58\x56"
730
-
"\xf1\x8f\xf7\xfd\x01\x48\xfb\x5f\x72\x2a\xd4\x8f\x02\x19\x00\xb3"
731
-
"\x69\x43\xfd\x48\x19\x86\xcf\x32\xdd\x41\x74\x6a\x51\xc7\xd9\x7d"
732
-
"\x3a\x97\xd9\xcd\x1a\x6a\x49",
733
-
.c_size = 55,
737
+
.c = (const unsigned char[]){
738
+
be64_to_cpua(01, 48, fb, 5f, 72, 2a, d4, 8f),
739
+
be64_to_cpua(6b, 1a, 58, 56, f1, 8f, f7, fd),
740
+
be64_to_cpua(3f, 72, 3f, 1f, 42, d2, 3f, 1d),
741
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
742
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
743
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
744
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
745
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
746
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
747
+
be64_to_cpua(7d, 3a, 97, d9, cd, 1a, 6a, 49),
748
+
be64_to_cpua(32, dd, 41, 74, 6a, 51, c7, d9),
749
+
be64_to_cpua(b3, 69, 43, fd, 48, 19, 86, cf),
750
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
751
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
752
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
753
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
754
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
755
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
756
+
.c_size = ECC_MAX_BYTES * 2,
734
757
.public_key_vec = true,
735
-
.siggen_sigver_test = true,
736
758
}, {
737
-
.key =
759
+
.key = /* secp192r1(sha384) */
738
760
"\x04\x5a\x13\xfe\x68\x86\x4d\xf4\x17\xc7\xa4\xe5\x8c\x65\x57\xb7"
739
761
"\x03\x73\x26\x57\xfb\xe5\x58\x40\xd8\xfd\x49\x05\xab\xf1\x66\x1f"
740
762
"\xe2\x9d\x93\x9e\xc2\x22\x5a\x8b\x4f\xf3\x77\x22\x59\x7e\xa6\x4e"
741
763
"\x8b",
742
764
.key_len = 49,
743
-
.params =
744
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
745
-
"\xce\x3d\x03\x01\x01",
746
-
.param_len = 21,
747
765
.m =
748
766
"\x9d\x2e\x1a\x8f\xed\x6c\x4b\x61\xae\xac\xd5\x19\x79\xce\x67\xf9"
749
767
"\xa0\x34\xeb\xb0\x81\xf9\xd9\xdc\x6e\xb3\x5c\xa8\x69\xfc\x8a\x61"
750
768
"\x39\x81\xfb\xfd\x5c\x30\x6b\xa8\xee\xed\x89\xaf\xa3\x05\xe4\x78",
751
769
.m_size = 48,
752
-
.algo = OID_id_ecdsa_with_sha384,
753
-
.c =
754
-
"\x30\x35\x02\x19\x00\xf0\xa3\x38\xce\x2b\xf8\x9d\x1a\xcf\x7f\x34"
755
-
"\xb4\xb4\xe5\xc5\x00\xdd\x15\xbb\xd6\x8c\xa7\x03\x78\x02\x18\x64"
756
-
"\xbc\x5a\x1f\x82\x96\x61\xd7\xd1\x01\x77\x44\x5d\x53\xa4\x7c\x93"
757
-
"\x12\x3b\x3b\x28\xfb\x6d\xe1",
758
-
.c_size = 55,
770
+
.c = (const unsigned char[]){
771
+
be64_to_cpua(dd, 15, bb, d6, 8c, a7, 03, 78),
772
+
be64_to_cpua(cf, 7f, 34, b4, b4, e5, c5, 00),
773
+
be64_to_cpua(f0, a3, 38, ce, 2b, f8, 9d, 1a),
774
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
775
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
776
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
777
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
778
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
779
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
780
+
be64_to_cpua(93, 12, 3b, 3b, 28, fb, 6d, e1),
781
+
be64_to_cpua(d1, 01, 77, 44, 5d, 53, a4, 7c),
782
+
be64_to_cpua(64, bc, 5a, 1f, 82, 96, 61, d7),
783
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
784
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
785
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
786
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
787
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
788
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
789
+
.c_size = ECC_MAX_BYTES * 2,
759
790
.public_key_vec = true,
760
-
.siggen_sigver_test = true,
761
791
}, {
762
-
.key =
792
+
.key = /* secp192r1(sha512) */
763
793
"\x04\xd5\xf2\x6e\xc3\x94\x5c\x52\xbc\xdf\x86\x6c\x14\xd1\xca\xea"
764
794
"\xcc\x72\x3a\x8a\xf6\x7a\x3a\x56\x36\x3b\xca\xc6\x94\x0e\x17\x1d"
765
795
"\x9e\xa0\x58\x28\xf9\x4b\xe6\xd1\xa5\x44\x91\x35\x0d\xe7\xf5\x11"
766
796
"\x57",
767
797
.key_len = 49,
768
-
.params =
769
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
770
-
"\xce\x3d\x03\x01\x01",
771
-
.param_len = 21,
772
798
.m =
773
799
"\xd5\x4b\xe9\x36\xda\xd8\x6e\xc0\x50\x03\xbe\x00\x43\xff\xf0\x23"
774
800
"\xac\xa2\x42\xe7\x37\x77\x79\x52\x8f\x3e\xc0\x16\xc1\xfc\x8c\x67"
775
801
"\x16\xbc\x8a\x5d\x3b\xd3\x13\xbb\xb6\xc0\x26\x1b\xeb\x33\xcc\x70"
776
802
"\x4a\xf2\x11\x37\xe8\x1b\xba\x55\xac\x69\xe1\x74\x62\x7c\x6e\xb5",
777
803
.m_size = 64,
778
-
.algo = OID_id_ecdsa_with_sha512,
779
-
.c =
780
-
"\x30\x35\x02\x19\x00\x88\x5b\x8f\x59\x43\xbf\xcf\xc6\xdd\x3f\x07"
781
-
"\x87\x12\xa0\xd4\xac\x2b\x11\x2d\x1c\xb6\x06\xc9\x6c\x02\x18\x73"
782
-
"\xb4\x22\x9a\x98\x73\x3c\x83\xa9\x14\x2a\x5e\xf5\xe5\xfb\x72\x28"
783
-
"\x6a\xdf\x97\xfd\x82\x76\x24",
784
-
.c_size = 55,
804
+
.c = (const unsigned char[]){
805
+
be64_to_cpua(2b, 11, 2d, 1c, b6, 06, c9, 6c),
806
+
be64_to_cpua(dd, 3f, 07, 87, 12, a0, d4, ac),
807
+
be64_to_cpua(88, 5b, 8f, 59, 43, bf, cf, c6),
808
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
809
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
810
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
811
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
812
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
813
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
814
+
be64_to_cpua(28, 6a, df, 97, fd, 82, 76, 24),
815
+
be64_to_cpua(a9, 14, 2a, 5e, f5, e5, fb, 72),
816
+
be64_to_cpua(73, b4, 22, 9a, 98, 73, 3c, 83),
817
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
818
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
819
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
820
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
821
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
822
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
823
+
.c_size = ECC_MAX_BYTES * 2,
785
824
.public_key_vec = true,
786
-
.siggen_sigver_test = true,
787
825
},
788
826
};
789
827
790
-
static const struct akcipher_testvec ecdsa_nist_p256_tv_template[] = {
828
+
static const struct sig_testvec ecdsa_nist_p256_tv_template[] = {
791
829
{
792
-
.key =
830
+
.key = /* secp256r1(sha1) */
793
831
"\x04\xb9\x7b\xbb\xd7\x17\x64\xd2\x7e\xfc\x81\x5d\x87\x06\x83\x41"
794
832
"\x22\xd6\x9a\xaa\x87\x17\xec\x4f\x63\x55\x2f\x94\xba\xdd\x83\xe9"
795
833
"\x34\x4b\xf3\xe9\x91\x13\x50\xb6\xcb\xca\x62\x08\xe7\x3b\x09\xdc"
796
834
"\xc3\x63\x4b\x2d\xb9\x73\x53\xe4\x45\xe6\x7c\xad\xe7\x6b\xb0\xe8"
797
835
"\xaf",
798
836
.key_len = 65,
799
-
.params =
800
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
801
-
"\xce\x3d\x03\x01\x07",
802
-
.param_len = 21,
803
837
.m =
804
838
"\xc2\x2b\x5f\x91\x78\x34\x26\x09\x42\x8d\x6f\x51\xb2\xc5\xaf\x4c"
805
839
"\x0b\xde\x6a\x42",
806
840
.m_size = 20,
807
-
.algo = OID_id_ecdsa_with_sha1,
808
-
.c =
809
-
"\x30\x46\x02\x21\x00\xf9\x25\xce\x9f\x3a\xa6\x35\x81\xcf\xd4\xe7"
810
-
"\xb7\xf0\x82\x56\x41\xf7\xd4\xad\x8d\x94\x5a\x69\x89\xee\xca\x6a"
811
-
"\x52\x0e\x48\x4d\xcc\x02\x21\x00\xd7\xe4\xef\x52\x66\xd3\x5b\x9d"
812
-
"\x8a\xfa\x54\x93\x29\xa7\x70\x86\xf1\x03\x03\xf3\x3b\xe2\x73\xf7"
813
-
"\xfb\x9d\x8b\xde\xd4\x8d\x6f\xad",
814
-
.c_size = 72,
841
+
.c = (const unsigned char[]){
842
+
be64_to_cpua(ee, ca, 6a, 52, 0e, 48, 4d, cc),
843
+
be64_to_cpua(f7, d4, ad, 8d, 94, 5a, 69, 89),
844
+
be64_to_cpua(cf, d4, e7, b7, f0, 82, 56, 41),
845
+
be64_to_cpua(f9, 25, ce, 9f, 3a, a6, 35, 81),
846
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
847
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
848
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
849
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
850
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
851
+
be64_to_cpua(fb, 9d, 8b, de, d4, 8d, 6f, ad),
852
+
be64_to_cpua(f1, 03, 03, f3, 3b, e2, 73, f7),
853
+
be64_to_cpua(8a, fa, 54, 93, 29, a7, 70, 86),
854
+
be64_to_cpua(d7, e4, ef, 52, 66, d3, 5b, 9d),
855
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
856
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
857
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
858
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
859
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
860
+
.c_size = ECC_MAX_BYTES * 2,
815
861
.public_key_vec = true,
816
-
.siggen_sigver_test = true,
817
862
}, {
818
-
.key =
863
+
.key = /* secp256r1(sha224) */
819
864
"\x04\x8b\x6d\xc0\x33\x8e\x2d\x8b\x67\xf5\xeb\xc4\x7f\xa0\xf5\xd9"
820
865
"\x7b\x03\xa5\x78\x9a\xb5\xea\x14\xe4\x23\xd0\xaf\xd7\x0e\x2e\xa0"
821
866
"\xc9\x8b\xdb\x95\xf8\xb3\xaf\xac\x00\x2c\x2c\x1f\x7a\xfd\x95\x88"
822
867
"\x43\x13\xbf\xf3\x1c\x05\x1a\x14\x18\x09\x3f\xd6\x28\x3e\xc5\xa0"
823
868
"\xd4",
824
869
.key_len = 65,
825
-
.params =
826
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
827
-
"\xce\x3d\x03\x01\x07",
828
-
.param_len = 21,
829
870
.m =
830
871
"\x1a\x15\xbc\xa3\xe4\xed\x3a\xb8\x23\x67\xc6\xc4\x34\xf8\x6c\x41"
831
872
"\x04\x0b\xda\xc5\x77\xfa\x1c\x2d\xe6\x2c\x3b\xe0",
832
873
.m_size = 28,
833
-
.algo = OID_id_ecdsa_with_sha224,
834
-
.c =
835
-
"\x30\x44\x02\x20\x20\x43\xfa\xc0\x9f\x9d\x7b\xe7\xae\xce\x77\x59"
836
-
"\x1a\xdb\x59\xd5\x34\x62\x79\xcb\x6a\x91\x67\x2e\x7d\x25\xd8\x25"
837
-
"\xf5\x81\xd2\x1e\x02\x20\x5f\xf8\x74\xf8\x57\xd0\x5e\x54\x76\x20"
838
-
"\x4a\x77\x22\xec\xc8\x66\xbf\x50\x05\x58\x39\x0e\x26\x92\xce\xd5"
839
-
"\x2e\x8b\xde\x5a\x04\x0e",
840
-
.c_size = 70,
874
+
.c = (const unsigned char[]){
875
+
be64_to_cpua(7d, 25, d8, 25, f5, 81, d2, 1e),
876
+
be64_to_cpua(34, 62, 79, cb, 6a, 91, 67, 2e),
877
+
be64_to_cpua(ae, ce, 77, 59, 1a, db, 59, d5),
878
+
be64_to_cpua(20, 43, fa, c0, 9f, 9d, 7b, e7),
879
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
880
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
881
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
882
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
883
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
884
+
be64_to_cpua(ce, d5, 2e, 8b, de, 5a, 04, 0e),
885
+
be64_to_cpua(bf, 50, 05, 58, 39, 0e, 26, 92),
886
+
be64_to_cpua(76, 20, 4a, 77, 22, ec, c8, 66),
887
+
be64_to_cpua(5f, f8, 74, f8, 57, d0, 5e, 54),
888
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
889
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
890
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
891
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
892
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
893
+
.c_size = ECC_MAX_BYTES * 2,
841
894
.public_key_vec = true,
842
-
.siggen_sigver_test = true,
843
895
}, {
844
-
.key =
896
+
.key = /* secp256r1(sha256) */
845
897
"\x04\xf1\xea\xc4\x53\xf3\xb9\x0e\x9f\x7e\xad\xe3\xea\xd7\x0e\x0f"
846
898
"\xd6\x98\x9a\xca\x92\x4d\x0a\x80\xdb\x2d\x45\xc7\xec\x4b\x97\x00"
847
899
"\x2f\xe9\x42\x6c\x29\xdc\x55\x0e\x0b\x53\x12\x9b\x2b\xad\x2c\xe9"
848
900
"\x80\xe6\xc5\x43\xc2\x1d\x5e\xbb\x65\x21\x50\xb6\x37\xb0\x03\x8e"
849
901
"\xb8",
850
902
.key_len = 65,
851
-
.params =
852
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
853
-
"\xce\x3d\x03\x01\x07",
854
-
.param_len = 21,
855
903
.m =
856
904
"\x8f\x43\x43\x46\x64\x8f\x6b\x96\xdf\x89\xdd\xa9\x01\xc5\x17\x6b"
857
905
"\x10\xa6\xd8\x39\x61\xdd\x3c\x1a\xc8\x8b\x59\xb2\xdc\x32\x7a\xa4",
858
906
.m_size = 32,
859
-
.algo = OID_id_ecdsa_with_sha256,
860
-
.c =
861
-
"\x30\x45\x02\x20\x08\x31\xfa\x74\x0d\x1d\x21\x5d\x09\xdc\x29\x63"
862
-
"\xa8\x1a\xad\xfc\xac\x44\xc3\xe8\x24\x11\x2d\xa4\x91\xdc\x02\x67"
863
-
"\xdc\x0c\xd0\x82\x02\x21\x00\xbd\xff\xce\xee\x42\xc3\x97\xff\xf9"
864
-
"\xa9\x81\xac\x4a\x50\xd0\x91\x0a\x6e\x1b\xc4\xaf\xe1\x83\xc3\x4f"
865
-
"\x2a\x65\x35\x23\xe3\x1d\xfa",
866
-
.c_size = 71,
907
+
.c = (const unsigned char[]){
908
+
be64_to_cpua(91, dc, 02, 67, dc, 0c, d0, 82),
909
+
be64_to_cpua(ac, 44, c3, e8, 24, 11, 2d, a4),
910
+
be64_to_cpua(09, dc, 29, 63, a8, 1a, ad, fc),
911
+
be64_to_cpua(08, 31, fa, 74, 0d, 1d, 21, 5d),
912
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
913
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
914
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
915
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
916
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
917
+
be64_to_cpua(4f, 2a, 65, 35, 23, e3, 1d, fa),
918
+
be64_to_cpua(0a, 6e, 1b, c4, af, e1, 83, c3),
919
+
be64_to_cpua(f9, a9, 81, ac, 4a, 50, d0, 91),
920
+
be64_to_cpua(bd, ff, ce, ee, 42, c3, 97, ff),
921
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
922
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
923
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
924
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
925
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
926
+
.c_size = ECC_MAX_BYTES * 2,
867
927
.public_key_vec = true,
868
-
.siggen_sigver_test = true,
869
928
}, {
870
-
.key =
929
+
.key = /* secp256r1(sha384) */
871
930
"\x04\xc5\xc6\xea\x60\xc9\xce\xad\x02\x8d\xf5\x3e\x24\xe3\x52\x1d"
872
931
"\x28\x47\x3b\xc3\x6b\xa4\x99\x35\x99\x11\x88\x88\xc8\xf4\xee\x7e"
873
932
"\x8c\x33\x8f\x41\x03\x24\x46\x2b\x1a\x82\xf9\x9f\xe1\x97\x1b\x00"
874
933
"\xda\x3b\x24\x41\xf7\x66\x33\x58\x3d\x3a\x81\xad\xcf\x16\xe9\xe2"
875
934
"\x7c",
876
935
.key_len = 65,
877
-
.params =
878
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
879
-
"\xce\x3d\x03\x01\x07",
880
-
.param_len = 21,
881
936
.m =
882
937
"\x3e\x78\x70\xfb\xcd\x66\xba\x91\xa1\x79\xff\x1e\x1c\x6b\x78\xe6"
883
938
"\xc0\x81\x3a\x65\x97\x14\x84\x36\x14\x1a\x9a\xb7\xc5\xab\x84\x94"
884
939
"\x5e\xbb\x1b\x34\x71\xcb\x41\xe1\xf6\xfc\x92\x7b\x34\xbb\x86\xbb",
885
940
.m_size = 48,
886
-
.algo = OID_id_ecdsa_with_sha384,
887
-
.c =
888
-
"\x30\x46\x02\x21\x00\x8e\xf3\x6f\xdc\xf8\x69\xa6\x2e\xd0\x2e\x95"
889
-
"\x54\xd1\x95\x64\x93\x08\xb2\x6b\x24\x94\x48\x46\x5e\xf2\xe4\x6c"
890
-
"\xc7\x94\xb1\xd5\xfe\x02\x21\x00\xeb\xa7\x80\x26\xdc\xf9\x3a\x44"
891
-
"\x19\xfb\x5f\x92\xf4\xc9\x23\x37\x69\xf4\x3b\x4f\x47\xcf\x9b\x16"
892
-
"\xc0\x60\x11\x92\xdc\x17\x89\x12",
893
-
.c_size = 72,
941
+
.c = (const unsigned char[]){
942
+
be64_to_cpua(f2, e4, 6c, c7, 94, b1, d5, fe),
943
+
be64_to_cpua(08, b2, 6b, 24, 94, 48, 46, 5e),
944
+
be64_to_cpua(d0, 2e, 95, 54, d1, 95, 64, 93),
945
+
be64_to_cpua(8e, f3, 6f, dc, f8, 69, a6, 2e),
946
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
947
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
948
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
949
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
950
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
951
+
be64_to_cpua(c0, 60, 11, 92, dc, 17, 89, 12),
952
+
be64_to_cpua(69, f4, 3b, 4f, 47, cf, 9b, 16),
953
+
be64_to_cpua(19, fb, 5f, 92, f4, c9, 23, 37),
954
+
be64_to_cpua(eb, a7, 80, 26, dc, f9, 3a, 44),
955
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
956
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
957
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
958
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
959
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
960
+
.c_size = ECC_MAX_BYTES * 2,
894
961
.public_key_vec = true,
895
-
.siggen_sigver_test = true,
896
962
}, {
897
-
.key =
963
+
.key = /* secp256r1(sha512) */
898
964
"\x04\xd7\x27\x46\x49\xf6\x26\x85\x12\x40\x76\x8e\xe2\xe6\x2a\x7a"
899
965
"\x83\xb1\x4e\x7a\xeb\x3b\x5c\x67\x4a\xb5\xa4\x92\x8c\x69\xff\x38"
900
966
"\xee\xd9\x4e\x13\x29\x59\xad\xde\x6b\xbb\x45\x31\xee\xfd\xd1\x1b"
901
967
"\x64\xd3\xb5\xfc\xaf\x9b\x4b\x88\x3b\x0e\xb7\xd6\xdf\xf1\xd5\x92"
902
968
"\xbf",
903
969
.key_len = 65,
904
-
.params =
905
-
"\x30\x13\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x08\x2a\x86\x48"
906
-
"\xce\x3d\x03\x01\x07",
907
-
.param_len = 21,
908
970
.m =
909
971
"\x57\xb7\x9e\xe9\x05\x0a\x8c\x1b\xc9\x13\xe5\x4a\x24\xc7\xe2\xe9"
910
972
"\x43\xc3\xd1\x76\x62\xf4\x98\x1a\x9c\x13\xb0\x20\x1b\xe5\x39\xca"
911
973
"\x4f\xd9\x85\x34\x95\xa2\x31\xbc\xbb\xde\xdd\x76\xbb\x61\xe3\xcf"
912
974
"\x9d\xc0\x49\x7a\xf3\x7a\xc4\x7d\xa8\x04\x4b\x8d\xb4\x4d\x5b\xd6",
913
975
.m_size = 64,
914
-
.algo = OID_id_ecdsa_with_sha512,
915
-
.c =
916
-
"\x30\x45\x02\x21\x00\xb8\x6d\x87\x81\x43\xdf\xfb\x9f\x40\xea\x44"
917
-
"\x81\x00\x4e\x29\x08\xed\x8c\x73\x30\x6c\x22\xb3\x97\x76\xf6\x04"
918
-
"\x99\x09\x37\x4d\xfa\x02\x20\x1e\xb9\x75\x31\xf6\x04\xa5\x4d\xf8"
919
-
"\x00\xdd\xab\xd4\xc0\x2b\xe6\x5c\xad\xc3\x78\x1c\xc2\xc1\x19\x76"
920
-
"\x31\x79\x4a\xe9\x81\x6a\xee",
921
-
.c_size = 71,
976
+
.c = (const unsigned char[]){
977
+
be64_to_cpua(76, f6, 04, 99, 09, 37, 4d, fa),
978
+
be64_to_cpua(ed, 8c, 73, 30, 6c, 22, b3, 97),
979
+
be64_to_cpua(40, ea, 44, 81, 00, 4e, 29, 08),
980
+
be64_to_cpua(b8, 6d, 87, 81, 43, df, fb, 9f),
981
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
982
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
983
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
984
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
985
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
986
+
be64_to_cpua(76, 31, 79, 4a, e9, 81, 6a, ee),
987
+
be64_to_cpua(5c, ad, c3, 78, 1c, c2, c1, 19),
988
+
be64_to_cpua(f8, 00, dd, ab, d4, c0, 2b, e6),
989
+
be64_to_cpua(1e, b9, 75, 31, f6, 04, a5, 4d),
990
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
991
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
992
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
993
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
994
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
995
+
.c_size = ECC_MAX_BYTES * 2,
922
996
.public_key_vec = true,
923
-
.siggen_sigver_test = true,
924
997
},
925
998
};
926
999
927
-
static const struct akcipher_testvec ecdsa_nist_p384_tv_template[] = {
1000
+
static const struct sig_testvec ecdsa_nist_p384_tv_template[] = {
928
1001
{
929
1002
.key = /* secp384r1(sha1) */
930
1003
"\x04\x89\x25\xf3\x97\x88\xcb\xb0\x78\xc5\x72\x9a\x14\x6e\x7a\xb1"
···
1018
925
"\x0b\x25\xd6\x80\x5c\x3b\xe6\x1a\x98\x48\x91\x45\x7a\x73\xb0\xc3"
1019
926
"\xf1",
1020
927
.key_len = 97,
1021
-
.params =
1022
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1023
-
"\x00\x22",
1024
-
.param_len = 18,
1025
928
.m =
1026
929
"\x12\x55\x28\xf0\x77\xd5\xb6\x21\x71\x32\x48\xcd\x28\xa8\x25\x22"
1027
930
"\x3a\x69\xc1\x93",
1028
931
.m_size = 20,
1029
-
.algo = OID_id_ecdsa_with_sha1,
1030
-
.c =
1031
-
"\x30\x66\x02\x31\x00\xf5\x0f\x24\x4c\x07\x93\x6f\x21\x57\x55\x07"
1032
-
"\x20\x43\x30\xde\xa0\x8d\x26\x8e\xae\x63\x3f\xbc\x20\x3a\xc6\xf1"
1033
-
"\x32\x3c\xce\x70\x2b\x78\xf1\x4c\x26\xe6\x5b\x86\xcf\xec\x7c\x7e"
1034
-
"\xd0\x87\xd7\xd7\x6e\x02\x31\x00\xcd\xbb\x7e\x81\x5d\x8f\x63\xc0"
1035
-
"\x5f\x63\xb1\xbe\x5e\x4c\x0e\xa1\xdf\x28\x8c\x1b\xfa\xf9\x95\x88"
1036
-
"\x74\xa0\x0f\xbf\xaf\xc3\x36\x76\x4a\xa1\x59\xf1\x1c\xa4\x58\x26"
1037
-
"\x79\x12\x2a\xb7\xc5\x15\x92\xc5",
1038
-
.c_size = 104,
932
+
.c = (const unsigned char[]){
933
+
be64_to_cpua(ec, 7c, 7e, d0, 87, d7, d7, 6e),
934
+
be64_to_cpua(78, f1, 4c, 26, e6, 5b, 86, cf),
935
+
be64_to_cpua(3a, c6, f1, 32, 3c, ce, 70, 2b),
936
+
be64_to_cpua(8d, 26, 8e, ae, 63, 3f, bc, 20),
937
+
be64_to_cpua(57, 55, 07, 20, 43, 30, de, a0),
938
+
be64_to_cpua(f5, 0f, 24, 4c, 07, 93, 6f, 21),
939
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
940
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
941
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
942
+
be64_to_cpua(79, 12, 2a, b7, c5, 15, 92, c5),
943
+
be64_to_cpua(4a, a1, 59, f1, 1c, a4, 58, 26),
944
+
be64_to_cpua(74, a0, 0f, bf, af, c3, 36, 76),
945
+
be64_to_cpua(df, 28, 8c, 1b, fa, f9, 95, 88),
946
+
be64_to_cpua(5f, 63, b1, be, 5e, 4c, 0e, a1),
947
+
be64_to_cpua(cd, bb, 7e, 81, 5d, 8f, 63, c0),
948
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
949
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
950
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
951
+
.c_size = ECC_MAX_BYTES * 2,
1039
952
.public_key_vec = true,
1040
-
.siggen_sigver_test = true,
1041
953
}, {
1042
954
.key = /* secp384r1(sha224) */
1043
955
"\x04\x69\x6c\xcf\x62\xee\xd0\x0d\xe5\xb5\x2f\x70\x54\xcf\x26\xa0"
···
1053
955
"\x6b\x93\x99\x6c\x66\x4c\x42\x3f\x65\x60\x6c\x1c\x0b\x93\x9b\x9d"
1054
956
"\xe0",
1055
957
.key_len = 97,
1056
-
.params =
1057
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1058
-
"\x00\x22",
1059
-
.param_len = 18,
1060
958
.m =
1061
959
"\x12\x80\xb6\xeb\x25\xe2\x3d\xf0\x21\x32\x96\x17\x3a\x38\x39\xfd"
1062
960
"\x1f\x05\x34\x7b\xb8\xf9\x71\x66\x03\x4f\xd5\xe5",
1063
961
.m_size = 28,
1064
-
.algo = OID_id_ecdsa_with_sha224,
1065
-
.c =
1066
-
"\x30\x66\x02\x31\x00\x8a\x51\x84\xce\x13\x1e\xd2\xdc\xec\xcb\xe4"
1067
-
"\x89\x47\xb2\xf7\xbc\x97\xf1\xc8\x72\x26\xcf\x5a\x5e\xc5\xda\xb4"
1068
-
"\xe3\x93\x07\xe0\x99\xc9\x9c\x11\xb8\x10\x01\xc5\x41\x3f\xdd\x15"
1069
-
"\x1b\x68\x2b\x9d\x8b\x02\x31\x00\x8b\x03\x2c\xfc\x1f\xd1\xa9\xa4"
1070
-
"\x4b\x00\x08\x31\x6c\xf5\xd5\xf6\xdf\xd8\x68\xa2\x64\x42\x65\xf3"
1071
-
"\x4d\xd0\xc6\x6e\xb0\xe9\xfc\x14\x9f\x19\xd0\x42\x8b\x93\xc2\x11"
1072
-
"\x88\x2b\x82\x26\x5e\x1c\xda\xfb",
1073
-
.c_size = 104,
962
+
.c = (const unsigned char[]){
963
+
be64_to_cpua(3f, dd, 15, 1b, 68, 2b, 9d, 8b),
964
+
be64_to_cpua(c9, 9c, 11, b8, 10, 01, c5, 41),
965
+
be64_to_cpua(c5, da, b4, e3, 93, 07, e0, 99),
966
+
be64_to_cpua(97, f1, c8, 72, 26, cf, 5a, 5e),
967
+
be64_to_cpua(ec, cb, e4, 89, 47, b2, f7, bc),
968
+
be64_to_cpua(8a, 51, 84, ce, 13, 1e, d2, dc),
969
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
970
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
971
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
972
+
be64_to_cpua(88, 2b, 82, 26, 5e, 1c, da, fb),
973
+
be64_to_cpua(9f, 19, d0, 42, 8b, 93, c2, 11),
974
+
be64_to_cpua(4d, d0, c6, 6e, b0, e9, fc, 14),
975
+
be64_to_cpua(df, d8, 68, a2, 64, 42, 65, f3),
976
+
be64_to_cpua(4b, 00, 08, 31, 6c, f5, d5, f6),
977
+
be64_to_cpua(8b, 03, 2c, fc, 1f, d1, a9, a4),
978
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
979
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
980
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
981
+
.c_size = ECC_MAX_BYTES * 2,
1074
982
.public_key_vec = true,
1075
-
.siggen_sigver_test = true,
1076
983
}, {
1077
984
.key = /* secp384r1(sha256) */
1078
985
"\x04\xee\xd6\xda\x3e\x94\x90\x00\x27\xed\xf8\x64\x55\xd6\x51\x9a"
···
1088
985
"\x17\xc3\x34\x29\xd6\x40\xea\x5c\xb9\x3f\xfb\x32\x2e\x12\x33\xbc"
1089
986
"\xab",
1090
987
.key_len = 97,
1091
-
.params =
1092
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1093
-
"\x00\x22",
1094
-
.param_len = 18,
1095
988
.m =
1096
989
"\xaa\xe7\xfd\x03\x26\xcb\x94\x71\xe4\xce\x0f\xc5\xff\xa6\x29\xa3"
1097
990
"\xe1\xcc\x4c\x35\x4e\xde\xca\x80\xab\x26\x0c\x25\xe6\x68\x11\xc2",
1098
991
.m_size = 32,
1099
-
.algo = OID_id_ecdsa_with_sha256,
1100
-
.c =
1101
-
"\x30\x64\x02\x30\x08\x09\x12\x9d\x6e\x96\x64\xa6\x8e\x3f\x7e\xce"
1102
-
"\x0a\x9b\xaa\x59\xcc\x47\x53\x87\xbc\xbd\x83\x3f\xaf\x06\x3f\x84"
1103
-
"\x04\xe2\xf9\x67\xb6\xc6\xfc\x70\x2e\x66\x3c\x77\xc8\x8d\x2c\x79"
1104
-
"\x3a\x8e\x32\xc4\x02\x30\x40\x34\xb8\x90\xa9\x80\xab\x47\x26\xa2"
1105
-
"\xb0\x89\x42\x0a\xda\xd9\xdd\xce\xbc\xb2\x97\xf4\x9c\xf3\x15\x68"
1106
-
"\xc0\x75\x3e\x23\x5e\x36\x4f\x8d\xde\x1e\x93\x8d\x95\xbb\x10\x0e"
1107
-
"\xf4\x1f\x39\xca\x4d\x43",
1108
-
.c_size = 102,
992
+
.c = (const unsigned char[]){
993
+
be64_to_cpua(c8, 8d, 2c, 79, 3a, 8e, 32, c4),
994
+
be64_to_cpua(b6, c6, fc, 70, 2e, 66, 3c, 77),
995
+
be64_to_cpua(af, 06, 3f, 84, 04, e2, f9, 67),
996
+
be64_to_cpua(cc, 47, 53, 87, bc, bd, 83, 3f),
997
+
be64_to_cpua(8e, 3f, 7e, ce, 0a, 9b, aa, 59),
998
+
be64_to_cpua(08, 09, 12, 9d, 6e, 96, 64, a6),
999
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1000
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1001
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1002
+
be64_to_cpua(10, 0e, f4, 1f, 39, ca, 4d, 43),
1003
+
be64_to_cpua(4f, 8d, de, 1e, 93, 8d, 95, bb),
1004
+
be64_to_cpua(15, 68, c0, 75, 3e, 23, 5e, 36),
1005
+
be64_to_cpua(dd, ce, bc, b2, 97, f4, 9c, f3),
1006
+
be64_to_cpua(26, a2, b0, 89, 42, 0a, da, d9),
1007
+
be64_to_cpua(40, 34, b8, 90, a9, 80, ab, 47),
1008
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1009
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1010
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
1011
+
.c_size = ECC_MAX_BYTES * 2,
1109
1012
.public_key_vec = true,
1110
-
.siggen_sigver_test = true,
1111
1013
}, {
1112
1014
.key = /* secp384r1(sha384) */
1113
1015
"\x04\x3a\x2f\x62\xe7\x1a\xcf\x24\xd0\x0b\x7c\xe0\xed\x46\x0a\x4f"
···
1123
1015
"\x21\x67\xe5\x1b\x5a\x52\x31\x68\xd6\xee\xf0\x19\xb0\x55\xed\x89"
1124
1016
"\x9e",
1125
1017
.key_len = 97,
1126
-
.params =
1127
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1128
-
"\x00\x22",
1129
-
.param_len = 18,
1130
1018
.m =
1131
1019
"\x8d\xf2\xc0\xe9\xa8\xf3\x8e\x44\xc4\x8c\x1a\xa0\xb8\xd7\x17\xdf"
1132
1020
"\xf2\x37\x1b\xc6\xe3\xf5\x62\xcc\x68\xf5\xd5\x0b\xbf\x73\x2b\xb1"
1133
1021
"\xb0\x4c\x04\x00\x31\xab\xfe\xc8\xd6\x09\xc8\xf2\xea\xd3\x28\xff",
1134
1022
.m_size = 48,
1135
-
.algo = OID_id_ecdsa_with_sha384,
1136
-
.c =
1137
-
"\x30\x66\x02\x31\x00\x9b\x28\x68\xc0\xa1\xea\x8c\x50\xee\x2e\x62"
1138
-
"\x35\x46\xfa\x00\xd8\x2d\x7a\x91\x5f\x49\x2d\x22\x08\x29\xe6\xfb"
1139
-
"\xca\x8c\xd6\xb6\xb4\x3b\x1f\x07\x8f\x15\x02\xfe\x1d\xa2\xa4\xc8"
1140
-
"\xf2\xea\x9d\x11\x1f\x02\x31\x00\xfc\x50\xf6\x43\xbd\x50\x82\x0e"
1141
-
"\xbf\xe3\x75\x24\x49\xac\xfb\xc8\x71\xcd\x8f\x18\x99\xf0\x0f\x13"
1142
-
"\x44\x92\x8c\x86\x99\x65\xb3\x97\x96\x17\x04\xc9\x05\x77\xf1\x8e"
1143
-
"\xab\x8d\x4e\xde\xe6\x6d\x9b\x66",
1144
-
.c_size = 104,
1023
+
.c = (const unsigned char[]){
1024
+
be64_to_cpua(a2, a4, c8, f2, ea, 9d, 11, 1f),
1025
+
be64_to_cpua(3b, 1f, 07, 8f, 15, 02, fe, 1d),
1026
+
be64_to_cpua(29, e6, fb, ca, 8c, d6, b6, b4),
1027
+
be64_to_cpua(2d, 7a, 91, 5f, 49, 2d, 22, 08),
1028
+
be64_to_cpua(ee, 2e, 62, 35, 46, fa, 00, d8),
1029
+
be64_to_cpua(9b, 28, 68, c0, a1, ea, 8c, 50),
1030
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1031
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1032
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1033
+
be64_to_cpua(ab, 8d, 4e, de, e6, 6d, 9b, 66),
1034
+
be64_to_cpua(96, 17, 04, c9, 05, 77, f1, 8e),
1035
+
be64_to_cpua(44, 92, 8c, 86, 99, 65, b3, 97),
1036
+
be64_to_cpua(71, cd, 8f, 18, 99, f0, 0f, 13),
1037
+
be64_to_cpua(bf, e3, 75, 24, 49, ac, fb, c8),
1038
+
be64_to_cpua(fc, 50, f6, 43, bd, 50, 82, 0e),
1039
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1040
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1041
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
1042
+
.c_size = ECC_MAX_BYTES * 2,
1145
1043
.public_key_vec = true,
1146
-
.siggen_sigver_test = true,
1147
1044
}, {
1148
1045
.key = /* secp384r1(sha512) */
1149
1046
"\x04\xb4\xe7\xc1\xeb\x64\x25\x22\x46\xc3\x86\x61\x80\xbe\x1e\x46"
···
1159
1046
"\xdf\x42\x5c\xc2\x5a\xc7\x0c\xf4\x15\xf7\x1b\xa3\x2e\xd7\x00\xac"
1160
1047
"\xa3",
1161
1048
.key_len = 97,
1162
-
.params =
1163
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1164
-
"\x00\x22",
1165
-
.param_len = 18,
1166
1049
.m =
1167
1050
"\xe8\xb7\x52\x7d\x1a\x44\x20\x05\x53\x6b\x3a\x68\xf2\xe7\x6c\xa1"
1168
1051
"\xae\x9d\x84\xbb\xba\x52\x43\x3e\x2c\x42\x78\x49\xbf\x78\xb2\x71"
1169
1052
"\xeb\xe1\xe0\xe8\x42\x7b\x11\xad\x2b\x99\x05\x1d\x36\xe6\xac\xfc"
1170
1053
"\x55\x73\xf0\x15\x63\x39\xb8\x6a\x6a\xc5\x91\x5b\xca\x6a\xa8\x0e",
1171
1054
.m_size = 64,
1172
-
.algo = OID_id_ecdsa_with_sha512,
1173
-
.c =
1174
-
"\x30\x63\x02\x2f\x1d\x20\x94\x77\xfe\x31\xfa\x4d\xc6\xef\xda\x02"
1175
-
"\xe7\x0f\x52\x9a\x02\xde\x93\xe8\x83\xe4\x84\x4c\xfc\x6f\x80\xe3"
1176
-
"\xaf\xb3\xd9\xdc\x2b\x43\x0e\x6a\xb3\x53\x6f\x3e\xb3\xc7\xa8\xb3"
1177
-
"\x17\x77\xd1\x02\x30\x63\xf6\xf0\x3d\x5f\x5f\x99\x3f\xde\x3a\x3d"
1178
-
"\x16\xaf\xb4\x52\x6a\xec\x63\xe3\x0c\xec\x50\xdc\xcc\xc4\x6a\x03"
1179
-
"\x5f\x8d\x7a\xf9\xfb\x34\xe4\x8b\x80\xa5\xb6\xda\x2c\x4e\x45\xcf"
1180
-
"\x3c\x93\xff\x50\x5d",
1181
-
.c_size = 101,
1055
+
.c = (const unsigned char[]){
1056
+
be64_to_cpua(3e, b3, c7, a8, b3, 17, 77, d1),
1057
+
be64_to_cpua(dc, 2b, 43, 0e, 6a, b3, 53, 6f),
1058
+
be64_to_cpua(4c, fc, 6f, 80, e3, af, b3, d9),
1059
+
be64_to_cpua(9a, 02, de, 93, e8, 83, e4, 84),
1060
+
be64_to_cpua(4d, c6, ef, da, 02, e7, 0f, 52),
1061
+
be64_to_cpua(00, 1d, 20, 94, 77, fe, 31, fa),
1062
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1063
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1064
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1065
+
be64_to_cpua(4e, 45, cf, 3c, 93, ff, 50, 5d),
1066
+
be64_to_cpua(34, e4, 8b, 80, a5, b6, da, 2c),
1067
+
be64_to_cpua(c4, 6a, 03, 5f, 8d, 7a, f9, fb),
1068
+
be64_to_cpua(ec, 63, e3, 0c, ec, 50, dc, cc),
1069
+
be64_to_cpua(de, 3a, 3d, 16, af, b4, 52, 6a),
1070
+
be64_to_cpua(63, f6, f0, 3d, 5f, 5f, 99, 3f),
1071
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1072
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00),
1073
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 00) },
1074
+
.c_size = ECC_MAX_BYTES * 2,
1182
1075
.public_key_vec = true,
1183
-
.siggen_sigver_test = true,
1184
1076
},
1185
1077
};
1186
1078
1187
-
static const struct akcipher_testvec ecdsa_nist_p521_tv_template[] = {
1079
+
static const struct sig_testvec ecdsa_nist_p521_tv_template[] = {
1188
1080
{
1189
1081
.key = /* secp521r1(sha224) */
1190
1082
"\x04\x01\x4f\x43\x18\xb6\xa9\xc9\x5d\x68\xd3\xa9\x42\xf8\x98\xc0"
···
1202
1084
"\xed\x37\x0f\x99\x3f\x26\xba\xa3\x8e\xff\x79\x34\x7c\x3a\xfe\x1f"
1203
1085
"\x3b\x83\x82\x2f\x14",
1204
1086
.key_len = 133,
1205
-
.params =
1206
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1207
-
"\x00\x23",
1208
-
.param_len = 18,
1209
1087
.m =
1210
1088
"\xa2\x3a\x6a\x8c\x7b\x3c\xf2\x51\xf8\xbe\x5f\x4f\x3b\x15\x05\xc4"
1211
1089
"\xb5\xbc\x19\xe7\x21\x85\xe9\x23\x06\x33\x62\xfb",
1212
1090
.m_size = 28,
1213
-
.algo = OID_id_ecdsa_with_sha224,
1214
-
.c =
1215
-
"\x30\x81\x86\x02\x41\x01\xd6\x43\xe7\xff\x42\xb2\xba\x74\x35\xf6"
1216
-
"\xdc\x6d\x02\x7b\x22\xac\xe2\xef\x07\x92\xee\x60\x94\x06\xf8\x3f"
1217
-
"\x59\x0f\x74\xf0\x3f\xd8\x18\xc6\x37\x8a\xcb\xa7\xd8\x7d\x98\x85"
1218
-
"\x29\x88\xff\x0b\x94\x94\x6c\xa6\x9b\x89\x8b\x1e\xfd\x09\x46\x6b"
1219
-
"\xc7\xaf\x7a\xb9\x19\x0a\x02\x41\x3a\x26\x0d\x55\xcd\x23\x1e\x7d"
1220
-
"\xa0\x5e\xf9\x88\xf3\xd2\x32\x90\x57\x0f\xf8\x65\x97\x6b\x09\x4d"
1221
-
"\x22\x26\x0b\x5f\x49\x32\x6b\x91\x99\x30\x90\x0f\x1c\x8f\x78\xd3"
1222
-
"\x9f\x0e\x64\xcc\xc4\xe8\x43\xd9\x0e\x1c\xad\x22\xda\x82\x00\x35"
1223
-
"\xa3\x50\xb1\xa5\x98\x92\x2a\xa5\x52",
1224
-
.c_size = 137,
1091
+
.c = (const unsigned char[]){
1092
+
be64_to_cpua(46, 6b, c7, af, 7a, b9, 19, 0a),
1093
+
be64_to_cpua(6c, a6, 9b, 89, 8b, 1e, fd, 09),
1094
+
be64_to_cpua(98, 85, 29, 88, ff, 0b, 94, 94),
1095
+
be64_to_cpua(18, c6, 37, 8a, cb, a7, d8, 7d),
1096
+
be64_to_cpua(f8, 3f, 59, 0f, 74, f0, 3f, d8),
1097
+
be64_to_cpua(e2, ef, 07, 92, ee, 60, 94, 06),
1098
+
be64_to_cpua(35, f6, dc, 6d, 02, 7b, 22, ac),
1099
+
be64_to_cpua(d6, 43, e7, ff, 42, b2, ba, 74),
1100
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 01),
1101
+
be64_to_cpua(50, b1, a5, 98, 92, 2a, a5, 52),
1102
+
be64_to_cpua(1c, ad, 22, da, 82, 00, 35, a3),
1103
+
be64_to_cpua(0e, 64, cc, c4, e8, 43, d9, 0e),
1104
+
be64_to_cpua(30, 90, 0f, 1c, 8f, 78, d3, 9f),
1105
+
be64_to_cpua(26, 0b, 5f, 49, 32, 6b, 91, 99),
1106
+
be64_to_cpua(0f, f8, 65, 97, 6b, 09, 4d, 22),
1107
+
be64_to_cpua(5e, f9, 88, f3, d2, 32, 90, 57),
1108
+
be64_to_cpua(26, 0d, 55, cd, 23, 1e, 7d, a0),
1109
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 3a) },
1110
+
.c_size = ECC_MAX_BYTES * 2,
1225
1111
.public_key_vec = true,
1226
-
.siggen_sigver_test = true,
1227
1112
},
1228
1113
{
1229
1114
.key = /* secp521r1(sha256) */
···
1240
1119
"\x8a\xe9\x53\xa8\xcf\xce\x43\x0e\x82\x20\x86\xbc\x88\x9c\xb7\xe3"
1241
1120
"\xe6\x77\x1e\x1f\x8a",
1242
1121
.key_len = 133,
1243
-
.params =
1244
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1245
-
"\x00\x23",
1246
-
.param_len = 18,
1247
1122
.m =
1248
1123
"\xcc\x97\x73\x0c\x73\xa2\x53\x2b\xfa\xd7\x83\x1d\x0c\x72\x1b\x39"
1249
1124
"\x80\x71\x8d\xdd\xc5\x9b\xff\x55\x32\x98\x25\xa2\x58\x2e\xb7\x73",
1250
1125
.m_size = 32,
1251
-
.algo = OID_id_ecdsa_with_sha256,
1252
-
.c =
1253
-
"\x30\x81\x88\x02\x42\x00\xcd\xa5\x5f\x57\x52\x27\x78\x3a\xb5\x06"
1254
-
"\x0f\xfd\x83\xfc\x0e\xd9\xce\x50\x9f\x7d\x1f\xca\x8b\xa8\x2d\x56"
1255
-
"\x3c\xf6\xf0\xd8\xe1\xb7\x5d\x95\x35\x6f\x02\x0e\xaf\xe1\x4c\xae"
1256
-
"\xce\x54\x76\x9a\xc2\x8f\xb8\x38\x1f\x46\x0b\x04\x64\x34\x79\xde"
1257
-
"\x7e\xd7\x59\x10\xe9\xd9\xd5\x02\x42\x01\xcf\x50\x85\x38\xf9\x15"
1258
-
"\x83\x18\x04\x6b\x35\xae\x65\xb5\x99\x12\x0a\xa9\x79\x24\xb9\x37"
1259
-
"\x35\xdd\xa0\xe0\x87\x2c\x44\x4b\x5a\xee\xaf\xfa\x10\xdd\x9b\xfb"
1260
-
"\x36\x1a\x31\x03\x42\x02\x5f\x50\xf0\xa2\x0d\x1c\x57\x56\x8f\x12"
1261
-
"\xb7\x1d\x91\x55\x38\xb6\xf6\x34\x65\xc7\xbd",
1262
-
.c_size = 139,
1126
+
.c = (const unsigned char[]){
1127
+
be64_to_cpua(de, 7e, d7, 59, 10, e9, d9, d5),
1128
+
be64_to_cpua(38, 1f, 46, 0b, 04, 64, 34, 79),
1129
+
be64_to_cpua(ae, ce, 54, 76, 9a, c2, 8f, b8),
1130
+
be64_to_cpua(95, 35, 6f, 02, 0e, af, e1, 4c),
1131
+
be64_to_cpua(56, 3c, f6, f0, d8, e1, b7, 5d),
1132
+
be64_to_cpua(50, 9f, 7d, 1f, ca, 8b, a8, 2d),
1133
+
be64_to_cpua(06, 0f, fd, 83, fc, 0e, d9, ce),
1134
+
be64_to_cpua(a5, 5f, 57, 52, 27, 78, 3a, b5),
1135
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, cd),
1136
+
be64_to_cpua(55, 38, b6, f6, 34, 65, c7, bd),
1137
+
be64_to_cpua(1c, 57, 56, 8f, 12, b7, 1d, 91),
1138
+
be64_to_cpua(03, 42, 02, 5f, 50, f0, a2, 0d),
1139
+
be64_to_cpua(fa, 10, dd, 9b, fb, 36, 1a, 31),
1140
+
be64_to_cpua(e0, 87, 2c, 44, 4b, 5a, ee, af),
1141
+
be64_to_cpua(a9, 79, 24, b9, 37, 35, dd, a0),
1142
+
be64_to_cpua(6b, 35, ae, 65, b5, 99, 12, 0a),
1143
+
be64_to_cpua(50, 85, 38, f9, 15, 83, 18, 04),
1144
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 01, cf) },
1145
+
.c_size = ECC_MAX_BYTES * 2,
1263
1146
.public_key_vec = true,
1264
-
.siggen_sigver_test = true,
1265
1147
},
1266
1148
{
1267
1149
.key = /* secp521r1(sha384) */
···
1278
1154
"\x22\x6e\xd7\x35\xc7\x23\xb7\x13\xae\xb6\x34\xff\xd7\x80\xe5\x39"
1279
1155
"\xb3\x3b\x5b\x1b\x94",
1280
1156
.key_len = 133,
1281
-
.params =
1282
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1283
-
"\x00\x23",
1284
-
.param_len = 18,
1285
1157
.m =
1286
1158
"\x36\x98\xd6\x82\xfa\xad\xed\x3c\xb9\x40\xb6\x4d\x9e\xb7\x04\x26"
1287
1159
"\xad\x72\x34\x44\xd2\x81\xb4\x9b\xbe\x01\x04\x7a\xd8\x50\xf8\x59"
1288
1160
"\xba\xad\x23\x85\x6b\x59\xbe\xfb\xf6\x86\xd4\x67\xa8\x43\x28\x76",
1289
1161
.m_size = 48,
1290
-
.algo = OID_id_ecdsa_with_sha384,
1291
-
.c =
1292
-
"\x30\x81\x88\x02\x42\x00\x93\x96\x76\x3c\x27\xea\xaa\x9c\x26\xec"
1293
-
"\x51\xdc\xe8\x35\x5e\xae\x16\xf2\x4b\x64\x98\xf7\xec\xda\xc7\x7e"
1294
-
"\x42\x71\x86\x57\x2d\xf1\x7d\xe4\xdf\x9b\x7d\x9e\x47\xca\x33\x32"
1295
-
"\x76\x06\xd0\xf9\xc0\xe4\xe6\x84\x59\xfd\x1a\xc4\x40\xdd\x43\xb8"
1296
-
"\x6a\xdd\xfb\xe6\x63\x4e\x28\x02\x42\x00\xff\xc3\x6a\x87\x6e\xb5"
1297
-
"\x13\x1f\x20\x55\xce\x37\x97\xc9\x05\x51\xe5\xe4\x3c\xbc\x93\x65"
1298
-
"\x57\x1c\x30\xda\xa7\xcd\x26\x28\x76\x3b\x52\xdf\xc4\xc0\xdb\x54"
1299
-
"\xdb\x8a\x0d\x6a\xc3\xf3\x7a\xd1\xfa\xe7\xa7\xe5\x5a\x94\x56\xcf"
1300
-
"\x8f\xb4\x22\xc6\x4f\xab\x2b\x62\xc1\x42\xb1",
1301
-
.c_size = 139,
1162
+
.c = (const unsigned char[]){
1163
+
be64_to_cpua(b8, 6a, dd, fb, e6, 63, 4e, 28),
1164
+
be64_to_cpua(84, 59, fd, 1a, c4, 40, dd, 43),
1165
+
be64_to_cpua(32, 76, 06, d0, f9, c0, e4, e6),
1166
+
be64_to_cpua(e4, df, 9b, 7d, 9e, 47, ca, 33),
1167
+
be64_to_cpua(7e, 42, 71, 86, 57, 2d, f1, 7d),
1168
+
be64_to_cpua(f2, 4b, 64, 98, f7, ec, da, c7),
1169
+
be64_to_cpua(ec, 51, dc, e8, 35, 5e, ae, 16),
1170
+
be64_to_cpua(96, 76, 3c, 27, ea, aa, 9c, 26),
1171
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, 93),
1172
+
be64_to_cpua(c6, 4f, ab, 2b, 62, c1, 42, b1),
1173
+
be64_to_cpua(e5, 5a, 94, 56, cf, 8f, b4, 22),
1174
+
be64_to_cpua(6a, c3, f3, 7a, d1, fa, e7, a7),
1175
+
be64_to_cpua(df, c4, c0, db, 54, db, 8a, 0d),
1176
+
be64_to_cpua(da, a7, cd, 26, 28, 76, 3b, 52),
1177
+
be64_to_cpua(e4, 3c, bc, 93, 65, 57, 1c, 30),
1178
+
be64_to_cpua(55, ce, 37, 97, c9, 05, 51, e5),
1179
+
be64_to_cpua(c3, 6a, 87, 6e, b5, 13, 1f, 20),
1180
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 00, ff) },
1181
+
.c_size = ECC_MAX_BYTES * 2,
1302
1182
.public_key_vec = true,
1303
-
.siggen_sigver_test = true,
1304
1183
},
1305
1184
{
1306
1185
.key = /* secp521r1(sha512) */
···
1317
1190
"\xfe\x3a\x05\x1a\xdb\xa9\x0f\xc0\x6c\x76\x30\x8c\xd8\xde\x44\xae"
1318
1191
"\xd0\x17\xdf\x49\x6a",
1319
1192
.key_len = 133,
1320
-
.params =
1321
-
"\x30\x10\x06\x07\x2a\x86\x48\xce\x3d\x02\x01\x06\x05\x2b\x81\x04"
1322
-
"\x00\x23",
1323
-
.param_len = 18,
1324
1193
.m =
1325
1194
"\x5c\xa6\xbc\x79\xb8\xa0\x1e\x11\x83\xf7\xe9\x05\xdf\xba\xf7\x69"
1326
1195
"\x97\x22\x32\xe4\x94\x7c\x65\xbd\x74\xc6\x9a\x8b\xbd\x0d\xdc\xed"
1327
1196
"\xf5\x9c\xeb\xe1\xc5\x68\x40\xf2\xc7\x04\xde\x9e\x0d\x76\xc5\xa3"
1328
1197
"\xf9\x3c\x6c\x98\x08\x31\xbd\x39\xe8\x42\x7f\x80\x39\x6f\xfe\x68",
1329
1198
.m_size = 64,
1330
-
.algo = OID_id_ecdsa_with_sha512,
1199
+
.c = (const unsigned char[]){
1200
+
be64_to_cpua(28, b5, 04, b0, b6, 33, 1c, 7e),
1201
+
be64_to_cpua(80, a6, 13, fc, b6, 90, f7, bb),
1202
+
be64_to_cpua(27, 93, e8, 6c, 49, 7d, 28, fc),
1203
+
be64_to_cpua(1f, 12, 3e, b7, 7e, 51, ff, 7f),
1204
+
be64_to_cpua(fb, 62, 1e, 42, 03, 6c, 74, 8a),
1205
+
be64_to_cpua(63, 0e, 02, cc, 94, a9, 05, b9),
1206
+
be64_to_cpua(aa, 86, ec, a8, 05, 03, 52, 56),
1207
+
be64_to_cpua(71, 86, 96, ac, 21, 33, 7e, 4e),
1208
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 01, 5c),
1209
+
be64_to_cpua(46, 1e, 77, 44, 78, e0, d1, 04),
1210
+
be64_to_cpua(72, 74, 13, 63, 39, a6, e5, 25),
1211
+
be64_to_cpua(00, 55, bb, 6a, b4, 73, 00, d2),
1212
+
be64_to_cpua(71, d0, e9, ca, a7, c0, cb, aa),
1213
+
be64_to_cpua(7a, 76, 37, 51, 47, 49, 98, 12),
1214
+
be64_to_cpua(88, 05, 3e, 43, 39, 01, bd, b7),
1215
+
be64_to_cpua(95, 35, 89, 4f, 41, 5f, 9e, 19),
1216
+
be64_to_cpua(43, 52, 1d, e3, c6, bd, 5a, 40),
1217
+
be64_to_cpua(00, 00, 00, 00, 00, 00, 01, 70) },
1218
+
.c_size = ECC_MAX_BYTES * 2,
1219
+
.public_key_vec = true,
1220
+
},
1221
+
};
1222
+
1223
+
/*
1224
+
* ECDSA X9.62 test vectors.
1225
+
*
1226
+
* Identical to ECDSA test vectors, except signature in "c" is X9.62 encoded.
1227
+
*/
1228
+
static const struct sig_testvec x962_ecdsa_nist_p192_tv_template[] = {
1229
+
{
1230
+
.key = /* secp192r1(sha1) */
1231
+
"\x04\xf7\x46\xf8\x2f\x15\xf6\x22\x8e\xd7\x57\x4f\xcc\xe7\xbb\xc1"
1232
+
"\xd4\x09\x73\xcf\xea\xd0\x15\x07\x3d\xa5\x8a\x8a\x95\x43\xe4\x68"
1233
+
"\xea\xc6\x25\xc1\xc1\x01\x25\x4c\x7e\xc3\x3c\xa6\x04\x0a\xe7\x08"
1234
+
"\x98",
1235
+
.key_len = 49,
1236
+
.m =
1237
+
"\xcd\xb9\xd2\x1c\xb7\x6f\xcd\x44\xb3\xfd\x63\xea\xa3\x66\x7f\xae"
1238
+
"\x63\x85\xe7\x82",
1239
+
.m_size = 20,
1240
+
.c =
1241
+
"\x30\x35\x02\x19\x00\xba\xe5\x93\x83\x6e\xb6\x3b\x63\xa0\x27\x91"
1242
+
"\xc6\xf6\x7f\xc3\x09\xad\x59\xad\x88\x27\xd6\x92\x6b\x02\x18\x10"
1243
+
"\x68\x01\x9d\xba\xce\x83\x08\xef\x95\x52\x7b\xa0\x0f\xe4\x18\x86"
1244
+
"\x80\x6f\xa5\x79\x77\xda\xd0",
1245
+
.c_size = 55,
1246
+
.public_key_vec = true,
1247
+
}, {
1248
+
.key = /* secp192r1(sha224) */
1249
+
"\x04\xb6\x4b\xb1\xd1\xac\xba\x24\x8f\x65\xb2\x60\x00\x90\xbf\xbd"
1250
+
"\x78\x05\x73\xe9\x79\x1d\x6f\x7c\x0b\xd2\xc3\x93\xa7\x28\xe1\x75"
1251
+
"\xf7\xd5\x95\x1d\x28\x10\xc0\x75\x50\x5c\x1a\x4f\x3f\x8f\xa5\xee"
1252
+
"\xa3",
1253
+
.key_len = 49,
1254
+
.m =
1255
+
"\x8d\xd6\xb8\x3e\xe5\xff\x23\xf6\x25\xa2\x43\x42\x74\x45\xa7\x40"
1256
+
"\x3a\xff\x2f\xe1\xd3\xf6\x9f\xe8\x33\xcb\x12\x11",
1257
+
.m_size = 28,
1258
+
.c =
1259
+
"\x30\x34\x02\x18\x5a\x8b\x82\x69\x7e\x8a\x0a\x09\x14\xf8\x11\x2b"
1260
+
"\x55\xdc\xae\x37\x83\x7b\x12\xe6\xb6\x5b\xcb\xd4\x02\x18\x6a\x14"
1261
+
"\x4f\x53\x75\xc8\x02\x48\xeb\xc3\x92\x0f\x1e\x72\xee\xc4\xa3\xe3"
1262
+
"\x5c\x99\xdb\x92\x5b\x36",
1263
+
.c_size = 54,
1264
+
.public_key_vec = true,
1265
+
}, {
1266
+
.key = /* secp192r1(sha256) */
1267
+
"\x04\xe2\x51\x24\x9b\xf7\xb6\x32\x82\x39\x66\x3d\x5b\xec\x3b\xae"
1268
+
"\x0c\xd5\xf2\x67\xd1\xc7\xe1\x02\xe4\xbf\x90\x62\xb8\x55\x75\x56"
1269
+
"\x69\x20\x5e\xcb\x4e\xca\x33\xd6\xcb\x62\x6b\x94\xa9\xa2\xe9\x58"
1270
+
"\x91",
1271
+
.key_len = 49,
1272
+
.m =
1273
+
"\x35\xec\xa1\xa0\x9e\x14\xde\x33\x03\xb6\xf6\xbd\x0c\x2f\xb2\xfd"
1274
+
"\x1f\x27\x82\xa5\xd7\x70\x3f\xef\xa0\x82\x69\x8e\x73\x31\x8e\xd7",
1275
+
.m_size = 32,
1276
+
.c =
1277
+
"\x30\x35\x02\x18\x3f\x72\x3f\x1f\x42\xd2\x3f\x1d\x6b\x1a\x58\x56"
1278
+
"\xf1\x8f\xf7\xfd\x01\x48\xfb\x5f\x72\x2a\xd4\x8f\x02\x19\x00\xb3"
1279
+
"\x69\x43\xfd\x48\x19\x86\xcf\x32\xdd\x41\x74\x6a\x51\xc7\xd9\x7d"
1280
+
"\x3a\x97\xd9\xcd\x1a\x6a\x49",
1281
+
.c_size = 55,
1282
+
.public_key_vec = true,
1283
+
}, {
1284
+
.key = /* secp192r1(sha384) */
1285
+
"\x04\x5a\x13\xfe\x68\x86\x4d\xf4\x17\xc7\xa4\xe5\x8c\x65\x57\xb7"
1286
+
"\x03\x73\x26\x57\xfb\xe5\x58\x40\xd8\xfd\x49\x05\xab\xf1\x66\x1f"
1287
+
"\xe2\x9d\x93\x9e\xc2\x22\x5a\x8b\x4f\xf3\x77\x22\x59\x7e\xa6\x4e"
1288
+
"\x8b",
1289
+
.key_len = 49,
1290
+
.m =
1291
+
"\x9d\x2e\x1a\x8f\xed\x6c\x4b\x61\xae\xac\xd5\x19\x79\xce\x67\xf9"
1292
+
"\xa0\x34\xeb\xb0\x81\xf9\xd9\xdc\x6e\xb3\x5c\xa8\x69\xfc\x8a\x61"
1293
+
"\x39\x81\xfb\xfd\x5c\x30\x6b\xa8\xee\xed\x89\xaf\xa3\x05\xe4\x78",
1294
+
.m_size = 48,
1295
+
.c =
1296
+
"\x30\x35\x02\x19\x00\xf0\xa3\x38\xce\x2b\xf8\x9d\x1a\xcf\x7f\x34"
1297
+
"\xb4\xb4\xe5\xc5\x00\xdd\x15\xbb\xd6\x8c\xa7\x03\x78\x02\x18\x64"
1298
+
"\xbc\x5a\x1f\x82\x96\x61\xd7\xd1\x01\x77\x44\x5d\x53\xa4\x7c\x93"
1299
+
"\x12\x3b\x3b\x28\xfb\x6d\xe1",
1300
+
.c_size = 55,
1301
+
.public_key_vec = true,
1302
+
}, {
1303
+
.key = /* secp192r1(sha512) */
1304
+
"\x04\xd5\xf2\x6e\xc3\x94\x5c\x52\xbc\xdf\x86\x6c\x14\xd1\xca\xea"
1305
+
"\xcc\x72\x3a\x8a\xf6\x7a\x3a\x56\x36\x3b\xca\xc6\x94\x0e\x17\x1d"
1306
+
"\x9e\xa0\x58\x28\xf9\x4b\xe6\xd1\xa5\x44\x91\x35\x0d\xe7\xf5\x11"
1307
+
"\x57",
1308
+
.key_len = 49,
1309
+
.m =
1310
+
"\xd5\x4b\xe9\x36\xda\xd8\x6e\xc0\x50\x03\xbe\x00\x43\xff\xf0\x23"
1311
+
"\xac\xa2\x42\xe7\x37\x77\x79\x52\x8f\x3e\xc0\x16\xc1\xfc\x8c\x67"
1312
+
"\x16\xbc\x8a\x5d\x3b\xd3\x13\xbb\xb6\xc0\x26\x1b\xeb\x33\xcc\x70"
1313
+
"\x4a\xf2\x11\x37\xe8\x1b\xba\x55\xac\x69\xe1\x74\x62\x7c\x6e\xb5",
1314
+
.m_size = 64,
1315
+
.c =
1316
+
"\x30\x35\x02\x19\x00\x88\x5b\x8f\x59\x43\xbf\xcf\xc6\xdd\x3f\x07"
1317
+
"\x87\x12\xa0\xd4\xac\x2b\x11\x2d\x1c\xb6\x06\xc9\x6c\x02\x18\x73"
1318
+
"\xb4\x22\x9a\x98\x73\x3c\x83\xa9\x14\x2a\x5e\xf5\xe5\xfb\x72\x28"
1319
+
"\x6a\xdf\x97\xfd\x82\x76\x24",
1320
+
.c_size = 55,
1321
+
.public_key_vec = true,
1322
+
},
1323
+
};
1324
+
1325
+
static const struct sig_testvec x962_ecdsa_nist_p256_tv_template[] = {
1326
+
{
1327
+
.key = /* secp256r1(sha1) */
1328
+
"\x04\xb9\x7b\xbb\xd7\x17\x64\xd2\x7e\xfc\x81\x5d\x87\x06\x83\x41"
1329
+
"\x22\xd6\x9a\xaa\x87\x17\xec\x4f\x63\x55\x2f\x94\xba\xdd\x83\xe9"
1330
+
"\x34\x4b\xf3\xe9\x91\x13\x50\xb6\xcb\xca\x62\x08\xe7\x3b\x09\xdc"
1331
+
"\xc3\x63\x4b\x2d\xb9\x73\x53\xe4\x45\xe6\x7c\xad\xe7\x6b\xb0\xe8"
1332
+
"\xaf",
1333
+
.key_len = 65,
1334
+
.m =
1335
+
"\xc2\x2b\x5f\x91\x78\x34\x26\x09\x42\x8d\x6f\x51\xb2\xc5\xaf\x4c"
1336
+
"\x0b\xde\x6a\x42",
1337
+
.m_size = 20,
1338
+
.c =
1339
+
"\x30\x46\x02\x21\x00\xf9\x25\xce\x9f\x3a\xa6\x35\x81\xcf\xd4\xe7"
1340
+
"\xb7\xf0\x82\x56\x41\xf7\xd4\xad\x8d\x94\x5a\x69\x89\xee\xca\x6a"
1341
+
"\x52\x0e\x48\x4d\xcc\x02\x21\x00\xd7\xe4\xef\x52\x66\xd3\x5b\x9d"
1342
+
"\x8a\xfa\x54\x93\x29\xa7\x70\x86\xf1\x03\x03\xf3\x3b\xe2\x73\xf7"
1343
+
"\xfb\x9d\x8b\xde\xd4\x8d\x6f\xad",
1344
+
.c_size = 72,
1345
+
.public_key_vec = true,
1346
+
}, {
1347
+
.key = /* secp256r1(sha224) */
1348
+
"\x04\x8b\x6d\xc0\x33\x8e\x2d\x8b\x67\xf5\xeb\xc4\x7f\xa0\xf5\xd9"
1349
+
"\x7b\x03\xa5\x78\x9a\xb5\xea\x14\xe4\x23\xd0\xaf\xd7\x0e\x2e\xa0"
1350
+
"\xc9\x8b\xdb\x95\xf8\xb3\xaf\xac\x00\x2c\x2c\x1f\x7a\xfd\x95\x88"
1351
+
"\x43\x13\xbf\xf3\x1c\x05\x1a\x14\x18\x09\x3f\xd6\x28\x3e\xc5\xa0"
1352
+
"\xd4",
1353
+
.key_len = 65,
1354
+
.m =
1355
+
"\x1a\x15\xbc\xa3\xe4\xed\x3a\xb8\x23\x67\xc6\xc4\x34\xf8\x6c\x41"
1356
+
"\x04\x0b\xda\xc5\x77\xfa\x1c\x2d\xe6\x2c\x3b\xe0",
1357
+
.m_size = 28,
1358
+
.c =
1359
+
"\x30\x44\x02\x20\x20\x43\xfa\xc0\x9f\x9d\x7b\xe7\xae\xce\x77\x59"
1360
+
"\x1a\xdb\x59\xd5\x34\x62\x79\xcb\x6a\x91\x67\x2e\x7d\x25\xd8\x25"
1361
+
"\xf5\x81\xd2\x1e\x02\x20\x5f\xf8\x74\xf8\x57\xd0\x5e\x54\x76\x20"
1362
+
"\x4a\x77\x22\xec\xc8\x66\xbf\x50\x05\x58\x39\x0e\x26\x92\xce\xd5"
1363
+
"\x2e\x8b\xde\x5a\x04\x0e",
1364
+
.c_size = 70,
1365
+
.public_key_vec = true,
1366
+
}, {
1367
+
.key = /* secp256r1(sha256) */
1368
+
"\x04\xf1\xea\xc4\x53\xf3\xb9\x0e\x9f\x7e\xad\xe3\xea\xd7\x0e\x0f"
1369
+
"\xd6\x98\x9a\xca\x92\x4d\x0a\x80\xdb\x2d\x45\xc7\xec\x4b\x97\x00"
1370
+
"\x2f\xe9\x42\x6c\x29\xdc\x55\x0e\x0b\x53\x12\x9b\x2b\xad\x2c\xe9"
1371
+
"\x80\xe6\xc5\x43\xc2\x1d\x5e\xbb\x65\x21\x50\xb6\x37\xb0\x03\x8e"
1372
+
"\xb8",
1373
+
.key_len = 65,
1374
+
.m =
1375
+
"\x8f\x43\x43\x46\x64\x8f\x6b\x96\xdf\x89\xdd\xa9\x01\xc5\x17\x6b"
1376
+
"\x10\xa6\xd8\x39\x61\xdd\x3c\x1a\xc8\x8b\x59\xb2\xdc\x32\x7a\xa4",
1377
+
.m_size = 32,
1378
+
.c =
1379
+
"\x30\x45\x02\x20\x08\x31\xfa\x74\x0d\x1d\x21\x5d\x09\xdc\x29\x63"
1380
+
"\xa8\x1a\xad\xfc\xac\x44\xc3\xe8\x24\x11\x2d\xa4\x91\xdc\x02\x67"
1381
+
"\xdc\x0c\xd0\x82\x02\x21\x00\xbd\xff\xce\xee\x42\xc3\x97\xff\xf9"
1382
+
"\xa9\x81\xac\x4a\x50\xd0\x91\x0a\x6e\x1b\xc4\xaf\xe1\x83\xc3\x4f"
1383
+
"\x2a\x65\x35\x23\xe3\x1d\xfa",
1384
+
.c_size = 71,
1385
+
.public_key_vec = true,
1386
+
}, {
1387
+
.key = /* secp256r1(sha384) */
1388
+
"\x04\xc5\xc6\xea\x60\xc9\xce\xad\x02\x8d\xf5\x3e\x24\xe3\x52\x1d"
1389
+
"\x28\x47\x3b\xc3\x6b\xa4\x99\x35\x99\x11\x88\x88\xc8\xf4\xee\x7e"
1390
+
"\x8c\x33\x8f\x41\x03\x24\x46\x2b\x1a\x82\xf9\x9f\xe1\x97\x1b\x00"
1391
+
"\xda\x3b\x24\x41\xf7\x66\x33\x58\x3d\x3a\x81\xad\xcf\x16\xe9\xe2"
1392
+
"\x7c",
1393
+
.key_len = 65,
1394
+
.m =
1395
+
"\x3e\x78\x70\xfb\xcd\x66\xba\x91\xa1\x79\xff\x1e\x1c\x6b\x78\xe6"
1396
+
"\xc0\x81\x3a\x65\x97\x14\x84\x36\x14\x1a\x9a\xb7\xc5\xab\x84\x94"
1397
+
"\x5e\xbb\x1b\x34\x71\xcb\x41\xe1\xf6\xfc\x92\x7b\x34\xbb\x86\xbb",
1398
+
.m_size = 48,
1399
+
.c =
1400
+
"\x30\x46\x02\x21\x00\x8e\xf3\x6f\xdc\xf8\x69\xa6\x2e\xd0\x2e\x95"
1401
+
"\x54\xd1\x95\x64\x93\x08\xb2\x6b\x24\x94\x48\x46\x5e\xf2\xe4\x6c"
1402
+
"\xc7\x94\xb1\xd5\xfe\x02\x21\x00\xeb\xa7\x80\x26\xdc\xf9\x3a\x44"
1403
+
"\x19\xfb\x5f\x92\xf4\xc9\x23\x37\x69\xf4\x3b\x4f\x47\xcf\x9b\x16"
1404
+
"\xc0\x60\x11\x92\xdc\x17\x89\x12",
1405
+
.c_size = 72,
1406
+
.public_key_vec = true,
1407
+
}, {
1408
+
.key = /* secp256r1(sha512) */
1409
+
"\x04\xd7\x27\x46\x49\xf6\x26\x85\x12\x40\x76\x8e\xe2\xe6\x2a\x7a"
1410
+
"\x83\xb1\x4e\x7a\xeb\x3b\x5c\x67\x4a\xb5\xa4\x92\x8c\x69\xff\x38"
1411
+
"\xee\xd9\x4e\x13\x29\x59\xad\xde\x6b\xbb\x45\x31\xee\xfd\xd1\x1b"
1412
+
"\x64\xd3\xb5\xfc\xaf\x9b\x4b\x88\x3b\x0e\xb7\xd6\xdf\xf1\xd5\x92"
1413
+
"\xbf",
1414
+
.key_len = 65,
1415
+
.m =
1416
+
"\x57\xb7\x9e\xe9\x05\x0a\x8c\x1b\xc9\x13\xe5\x4a\x24\xc7\xe2\xe9"
1417
+
"\x43\xc3\xd1\x76\x62\xf4\x98\x1a\x9c\x13\xb0\x20\x1b\xe5\x39\xca"
1418
+
"\x4f\xd9\x85\x34\x95\xa2\x31\xbc\xbb\xde\xdd\x76\xbb\x61\xe3\xcf"
1419
+
"\x9d\xc0\x49\x7a\xf3\x7a\xc4\x7d\xa8\x04\x4b\x8d\xb4\x4d\x5b\xd6",
1420
+
.m_size = 64,
1421
+
.c =
1422
+
"\x30\x45\x02\x21\x00\xb8\x6d\x87\x81\x43\xdf\xfb\x9f\x40\xea\x44"
1423
+
"\x81\x00\x4e\x29\x08\xed\x8c\x73\x30\x6c\x22\xb3\x97\x76\xf6\x04"
1424
+
"\x99\x09\x37\x4d\xfa\x02\x20\x1e\xb9\x75\x31\xf6\x04\xa5\x4d\xf8"
1425
+
"\x00\xdd\xab\xd4\xc0\x2b\xe6\x5c\xad\xc3\x78\x1c\xc2\xc1\x19\x76"
1426
+
"\x31\x79\x4a\xe9\x81\x6a\xee",
1427
+
.c_size = 71,
1428
+
.public_key_vec = true,
1429
+
},
1430
+
};
1431
+
1432
+
static const struct sig_testvec x962_ecdsa_nist_p384_tv_template[] = {
1433
+
{
1434
+
.key = /* secp384r1(sha1) */
1435
+
"\x04\x89\x25\xf3\x97\x88\xcb\xb0\x78\xc5\x72\x9a\x14\x6e\x7a\xb1"
1436
+
"\x5a\xa5\x24\xf1\x95\x06\x9e\x28\xfb\xc4\xb9\xbe\x5a\x0d\xd9\x9f"
1437
+
"\xf3\xd1\x4d\x2d\x07\x99\xbd\xda\xa7\x66\xec\xbb\xea\xba\x79\x42"
1438
+
"\xc9\x34\x89\x6a\xe7\x0b\xc3\xf2\xfe\x32\x30\xbe\xba\xf9\xdf\x7e"
1439
+
"\x4b\x6a\x07\x8e\x26\x66\x3f\x1d\xec\xa2\x57\x91\x51\xdd\x17\x0e"
1440
+
"\x0b\x25\xd6\x80\x5c\x3b\xe6\x1a\x98\x48\x91\x45\x7a\x73\xb0\xc3"
1441
+
"\xf1",
1442
+
.key_len = 97,
1443
+
.m =
1444
+
"\x12\x55\x28\xf0\x77\xd5\xb6\x21\x71\x32\x48\xcd\x28\xa8\x25\x22"
1445
+
"\x3a\x69\xc1\x93",
1446
+
.m_size = 20,
1447
+
.c =
1448
+
"\x30\x66\x02\x31\x00\xf5\x0f\x24\x4c\x07\x93\x6f\x21\x57\x55\x07"
1449
+
"\x20\x43\x30\xde\xa0\x8d\x26\x8e\xae\x63\x3f\xbc\x20\x3a\xc6\xf1"
1450
+
"\x32\x3c\xce\x70\x2b\x78\xf1\x4c\x26\xe6\x5b\x86\xcf\xec\x7c\x7e"
1451
+
"\xd0\x87\xd7\xd7\x6e\x02\x31\x00\xcd\xbb\x7e\x81\x5d\x8f\x63\xc0"
1452
+
"\x5f\x63\xb1\xbe\x5e\x4c\x0e\xa1\xdf\x28\x8c\x1b\xfa\xf9\x95\x88"
1453
+
"\x74\xa0\x0f\xbf\xaf\xc3\x36\x76\x4a\xa1\x59\xf1\x1c\xa4\x58\x26"
1454
+
"\x79\x12\x2a\xb7\xc5\x15\x92\xc5",
1455
+
.c_size = 104,
1456
+
.public_key_vec = true,
1457
+
}, {
1458
+
.key = /* secp384r1(sha224) */
1459
+
"\x04\x69\x6c\xcf\x62\xee\xd0\x0d\xe5\xb5\x2f\x70\x54\xcf\x26\xa0"
1460
+
"\xd9\x98\x8d\x92\x2a\xab\x9b\x11\xcb\x48\x18\xa1\xa9\x0d\xd5\x18"
1461
+
"\x3e\xe8\x29\x6e\xf6\xe4\xb5\x8e\xc7\x4a\xc2\x5f\x37\x13\x99\x05"
1462
+
"\xb6\xa4\x9d\xf9\xfb\x79\x41\xe7\xd7\x96\x9f\x73\x3b\x39\x43\xdc"
1463
+
"\xda\xf4\x06\xb9\xa5\x29\x01\x9d\x3b\xe1\xd8\x68\x77\x2a\xf4\x50"
1464
+
"\x6b\x93\x99\x6c\x66\x4c\x42\x3f\x65\x60\x6c\x1c\x0b\x93\x9b\x9d"
1465
+
"\xe0",
1466
+
.key_len = 97,
1467
+
.m =
1468
+
"\x12\x80\xb6\xeb\x25\xe2\x3d\xf0\x21\x32\x96\x17\x3a\x38\x39\xfd"
1469
+
"\x1f\x05\x34\x7b\xb8\xf9\x71\x66\x03\x4f\xd5\xe5",
1470
+
.m_size = 28,
1471
+
.c =
1472
+
"\x30\x66\x02\x31\x00\x8a\x51\x84\xce\x13\x1e\xd2\xdc\xec\xcb\xe4"
1473
+
"\x89\x47\xb2\xf7\xbc\x97\xf1\xc8\x72\x26\xcf\x5a\x5e\xc5\xda\xb4"
1474
+
"\xe3\x93\x07\xe0\x99\xc9\x9c\x11\xb8\x10\x01\xc5\x41\x3f\xdd\x15"
1475
+
"\x1b\x68\x2b\x9d\x8b\x02\x31\x00\x8b\x03\x2c\xfc\x1f\xd1\xa9\xa4"
1476
+
"\x4b\x00\x08\x31\x6c\xf5\xd5\xf6\xdf\xd8\x68\xa2\x64\x42\x65\xf3"
1477
+
"\x4d\xd0\xc6\x6e\xb0\xe9\xfc\x14\x9f\x19\xd0\x42\x8b\x93\xc2\x11"
1478
+
"\x88\x2b\x82\x26\x5e\x1c\xda\xfb",
1479
+
.c_size = 104,
1480
+
.public_key_vec = true,
1481
+
}, {
1482
+
.key = /* secp384r1(sha256) */
1483
+
"\x04\xee\xd6\xda\x3e\x94\x90\x00\x27\xed\xf8\x64\x55\xd6\x51\x9a"
1484
+
"\x1f\x52\x00\x63\x78\xf1\xa9\xfd\x75\x4c\x9e\xb2\x20\x1a\x91\x5a"
1485
+
"\xba\x7a\xa3\xe5\x6c\xb6\x25\x68\x4b\xe8\x13\xa6\x54\x87\x2c\x0e"
1486
+
"\xd0\x83\x95\xbc\xbf\xc5\x28\x4f\x77\x1c\x46\xa6\xf0\xbc\xd4\xa4"
1487
+
"\x8d\xc2\x8f\xb3\x32\x37\x40\xd6\xca\xf8\xae\x07\x34\x52\x39\x52"
1488
+
"\x17\xc3\x34\x29\xd6\x40\xea\x5c\xb9\x3f\xfb\x32\x2e\x12\x33\xbc"
1489
+
"\xab",
1490
+
.key_len = 97,
1491
+
.m =
1492
+
"\xaa\xe7\xfd\x03\x26\xcb\x94\x71\xe4\xce\x0f\xc5\xff\xa6\x29\xa3"
1493
+
"\xe1\xcc\x4c\x35\x4e\xde\xca\x80\xab\x26\x0c\x25\xe6\x68\x11\xc2",
1494
+
.m_size = 32,
1495
+
.c =
1496
+
"\x30\x64\x02\x30\x08\x09\x12\x9d\x6e\x96\x64\xa6\x8e\x3f\x7e\xce"
1497
+
"\x0a\x9b\xaa\x59\xcc\x47\x53\x87\xbc\xbd\x83\x3f\xaf\x06\x3f\x84"
1498
+
"\x04\xe2\xf9\x67\xb6\xc6\xfc\x70\x2e\x66\x3c\x77\xc8\x8d\x2c\x79"
1499
+
"\x3a\x8e\x32\xc4\x02\x30\x40\x34\xb8\x90\xa9\x80\xab\x47\x26\xa2"
1500
+
"\xb0\x89\x42\x0a\xda\xd9\xdd\xce\xbc\xb2\x97\xf4\x9c\xf3\x15\x68"
1501
+
"\xc0\x75\x3e\x23\x5e\x36\x4f\x8d\xde\x1e\x93\x8d\x95\xbb\x10\x0e"
1502
+
"\xf4\x1f\x39\xca\x4d\x43",
1503
+
.c_size = 102,
1504
+
.public_key_vec = true,
1505
+
}, {
1506
+
.key = /* secp384r1(sha384) */
1507
+
"\x04\x3a\x2f\x62\xe7\x1a\xcf\x24\xd0\x0b\x7c\xe0\xed\x46\x0a\x4f"
1508
+
"\x74\x16\x43\xe9\x1a\x25\x7c\x55\xff\xf0\x29\x68\x66\x20\x91\xf9"
1509
+
"\xdb\x2b\xf6\xb3\x6c\x54\x01\xca\xc7\x6a\x5c\x0d\xeb\x68\xd9\x3c"
1510
+
"\xf1\x01\x74\x1f\xf9\x6c\xe5\x5b\x60\xe9\x7f\x5d\xb3\x12\x80\x2a"
1511
+
"\xd8\x67\x92\xc9\x0e\x4c\x4c\x6b\xa1\xb2\xa8\x1e\xac\x1c\x97\xd9"
1512
+
"\x21\x67\xe5\x1b\x5a\x52\x31\x68\xd6\xee\xf0\x19\xb0\x55\xed\x89"
1513
+
"\x9e",
1514
+
.key_len = 97,
1515
+
.m =
1516
+
"\x8d\xf2\xc0\xe9\xa8\xf3\x8e\x44\xc4\x8c\x1a\xa0\xb8\xd7\x17\xdf"
1517
+
"\xf2\x37\x1b\xc6\xe3\xf5\x62\xcc\x68\xf5\xd5\x0b\xbf\x73\x2b\xb1"
1518
+
"\xb0\x4c\x04\x00\x31\xab\xfe\xc8\xd6\x09\xc8\xf2\xea\xd3\x28\xff",
1519
+
.m_size = 48,
1520
+
.c =
1521
+
"\x30\x66\x02\x31\x00\x9b\x28\x68\xc0\xa1\xea\x8c\x50\xee\x2e\x62"
1522
+
"\x35\x46\xfa\x00\xd8\x2d\x7a\x91\x5f\x49\x2d\x22\x08\x29\xe6\xfb"
1523
+
"\xca\x8c\xd6\xb6\xb4\x3b\x1f\x07\x8f\x15\x02\xfe\x1d\xa2\xa4\xc8"
1524
+
"\xf2\xea\x9d\x11\x1f\x02\x31\x00\xfc\x50\xf6\x43\xbd\x50\x82\x0e"
1525
+
"\xbf\xe3\x75\x24\x49\xac\xfb\xc8\x71\xcd\x8f\x18\x99\xf0\x0f\x13"
1526
+
"\x44\x92\x8c\x86\x99\x65\xb3\x97\x96\x17\x04\xc9\x05\x77\xf1\x8e"
1527
+
"\xab\x8d\x4e\xde\xe6\x6d\x9b\x66",
1528
+
.c_size = 104,
1529
+
.public_key_vec = true,
1530
+
}, {
1531
+
.key = /* secp384r1(sha512) */
1532
+
"\x04\xb4\xe7\xc1\xeb\x64\x25\x22\x46\xc3\x86\x61\x80\xbe\x1e\x46"
1533
+
"\xcb\xf6\x05\xc2\xee\x73\x83\xbc\xea\x30\x61\x4d\x40\x05\x41\xf4"
1534
+
"\x8c\xe3\x0e\x5c\xf0\x50\xf2\x07\x19\xe8\x4f\x25\xbe\xee\x0c\x95"
1535
+
"\x54\x36\x86\xec\xc2\x20\x75\xf3\x89\xb5\x11\xa1\xb7\xf5\xaf\xbe"
1536
+
"\x81\xe4\xc3\x39\x06\xbd\xe4\xfe\x68\x1c\x6d\x99\x2b\x1b\x63\xfa"
1537
+
"\xdf\x42\x5c\xc2\x5a\xc7\x0c\xf4\x15\xf7\x1b\xa3\x2e\xd7\x00\xac"
1538
+
"\xa3",
1539
+
.key_len = 97,
1540
+
.m =
1541
+
"\xe8\xb7\x52\x7d\x1a\x44\x20\x05\x53\x6b\x3a\x68\xf2\xe7\x6c\xa1"
1542
+
"\xae\x9d\x84\xbb\xba\x52\x43\x3e\x2c\x42\x78\x49\xbf\x78\xb2\x71"
1543
+
"\xeb\xe1\xe0\xe8\x42\x7b\x11\xad\x2b\x99\x05\x1d\x36\xe6\xac\xfc"
1544
+
"\x55\x73\xf0\x15\x63\x39\xb8\x6a\x6a\xc5\x91\x5b\xca\x6a\xa8\x0e",
1545
+
.m_size = 64,
1546
+
.c =
1547
+
"\x30\x63\x02\x2f\x1d\x20\x94\x77\xfe\x31\xfa\x4d\xc6\xef\xda\x02"
1548
+
"\xe7\x0f\x52\x9a\x02\xde\x93\xe8\x83\xe4\x84\x4c\xfc\x6f\x80\xe3"
1549
+
"\xaf\xb3\xd9\xdc\x2b\x43\x0e\x6a\xb3\x53\x6f\x3e\xb3\xc7\xa8\xb3"
1550
+
"\x17\x77\xd1\x02\x30\x63\xf6\xf0\x3d\x5f\x5f\x99\x3f\xde\x3a\x3d"
1551
+
"\x16\xaf\xb4\x52\x6a\xec\x63\xe3\x0c\xec\x50\xdc\xcc\xc4\x6a\x03"
1552
+
"\x5f\x8d\x7a\xf9\xfb\x34\xe4\x8b\x80\xa5\xb6\xda\x2c\x4e\x45\xcf"
1553
+
"\x3c\x93\xff\x50\x5d",
1554
+
.c_size = 101,
1555
+
.public_key_vec = true,
1556
+
},
1557
+
};
1558
+
1559
+
static const struct sig_testvec x962_ecdsa_nist_p521_tv_template[] = {
1560
+
{
1561
+
.key = /* secp521r1(sha224) */
1562
+
"\x04\x01\x4f\x43\x18\xb6\xa9\xc9\x5d\x68\xd3\xa9\x42\xf8\x98\xc0"
1563
+
"\xd2\xd1\xa9\x50\x3b\xe8\xc4\x40\xe6\x11\x78\x88\x4b\xbd\x76\xa7"
1564
+
"\x9a\xe0\xdd\x31\xa4\x67\x78\x45\x33\x9e\x8c\xd1\xc7\x44\xac\x61"
1565
+
"\x68\xc8\x04\xe7\x5c\x79\xb1\xf1\x41\x0c\x71\xc0\x53\xa8\xbc\xfb"
1566
+
"\xf5\xca\xd4\x01\x40\xfd\xa3\x45\xda\x08\xe0\xb4\xcb\x28\x3b\x0a"
1567
+
"\x02\x35\x5f\x02\x9f\x3f\xcd\xef\x08\x22\x40\x97\x74\x65\xb7\x76"
1568
+
"\x85\xc7\xc0\x5c\xfb\x81\xe1\xa5\xde\x0c\x4e\x8b\x12\x31\xb6\x47"
1569
+
"\xed\x37\x0f\x99\x3f\x26\xba\xa3\x8e\xff\x79\x34\x7c\x3a\xfe\x1f"
1570
+
"\x3b\x83\x82\x2f\x14",
1571
+
.key_len = 133,
1572
+
.m =
1573
+
"\xa2\x3a\x6a\x8c\x7b\x3c\xf2\x51\xf8\xbe\x5f\x4f\x3b\x15\x05\xc4"
1574
+
"\xb5\xbc\x19\xe7\x21\x85\xe9\x23\x06\x33\x62\xfb",
1575
+
.m_size = 28,
1576
+
.c =
1577
+
"\x30\x81\x86\x02\x41\x01\xd6\x43\xe7\xff\x42\xb2\xba\x74\x35\xf6"
1578
+
"\xdc\x6d\x02\x7b\x22\xac\xe2\xef\x07\x92\xee\x60\x94\x06\xf8\x3f"
1579
+
"\x59\x0f\x74\xf0\x3f\xd8\x18\xc6\x37\x8a\xcb\xa7\xd8\x7d\x98\x85"
1580
+
"\x29\x88\xff\x0b\x94\x94\x6c\xa6\x9b\x89\x8b\x1e\xfd\x09\x46\x6b"
1581
+
"\xc7\xaf\x7a\xb9\x19\x0a\x02\x41\x3a\x26\x0d\x55\xcd\x23\x1e\x7d"
1582
+
"\xa0\x5e\xf9\x88\xf3\xd2\x32\x90\x57\x0f\xf8\x65\x97\x6b\x09\x4d"
1583
+
"\x22\x26\x0b\x5f\x49\x32\x6b\x91\x99\x30\x90\x0f\x1c\x8f\x78\xd3"
1584
+
"\x9f\x0e\x64\xcc\xc4\xe8\x43\xd9\x0e\x1c\xad\x22\xda\x82\x00\x35"
1585
+
"\xa3\x50\xb1\xa5\x98\x92\x2a\xa5\x52",
1586
+
.c_size = 137,
1587
+
.public_key_vec = true,
1588
+
},
1589
+
{
1590
+
.key = /* secp521r1(sha256) */
1591
+
"\x04\x01\x05\x3a\x6b\x3b\x5a\x0f\xa7\xb9\xb7\x32\x53\x4e\xe2\xae"
1592
+
"\x0a\x52\xc5\xda\xdd\x5a\x79\x1c\x30\x2d\x33\x07\x79\xd5\x70\x14"
1593
+
"\x61\x0c\xec\x26\x4d\xd8\x35\x57\x04\x1d\x88\x33\x4d\xce\x05\x36"
1594
+
"\xa5\xaf\x56\x84\xfa\x0b\x9e\xff\x7b\x30\x4b\x92\x1d\x06\xf8\x81"
1595
+
"\x24\x1e\x51\x00\x09\x21\x51\xf7\x46\x0a\x77\xdb\xb5\x0c\xe7\x9c"
1596
+
"\xff\x27\x3c\x02\x71\xd7\x85\x36\xf1\xaa\x11\x59\xd8\xb8\xdc\x09"
1597
+
"\xdc\x6d\x5a\x6f\x63\x07\x6c\xe1\xe5\x4d\x6e\x0f\x6e\xfb\x7c\x05"
1598
+
"\x8a\xe9\x53\xa8\xcf\xce\x43\x0e\x82\x20\x86\xbc\x88\x9c\xb7\xe3"
1599
+
"\xe6\x77\x1e\x1f\x8a",
1600
+
.key_len = 133,
1601
+
.m =
1602
+
"\xcc\x97\x73\x0c\x73\xa2\x53\x2b\xfa\xd7\x83\x1d\x0c\x72\x1b\x39"
1603
+
"\x80\x71\x8d\xdd\xc5\x9b\xff\x55\x32\x98\x25\xa2\x58\x2e\xb7\x73",
1604
+
.m_size = 32,
1605
+
.c =
1606
+
"\x30\x81\x88\x02\x42\x00\xcd\xa5\x5f\x57\x52\x27\x78\x3a\xb5\x06"
1607
+
"\x0f\xfd\x83\xfc\x0e\xd9\xce\x50\x9f\x7d\x1f\xca\x8b\xa8\x2d\x56"
1608
+
"\x3c\xf6\xf0\xd8\xe1\xb7\x5d\x95\x35\x6f\x02\x0e\xaf\xe1\x4c\xae"
1609
+
"\xce\x54\x76\x9a\xc2\x8f\xb8\x38\x1f\x46\x0b\x04\x64\x34\x79\xde"
1610
+
"\x7e\xd7\x59\x10\xe9\xd9\xd5\x02\x42\x01\xcf\x50\x85\x38\xf9\x15"
1611
+
"\x83\x18\x04\x6b\x35\xae\x65\xb5\x99\x12\x0a\xa9\x79\x24\xb9\x37"
1612
+
"\x35\xdd\xa0\xe0\x87\x2c\x44\x4b\x5a\xee\xaf\xfa\x10\xdd\x9b\xfb"
1613
+
"\x36\x1a\x31\x03\x42\x02\x5f\x50\xf0\xa2\x0d\x1c\x57\x56\x8f\x12"
1614
+
"\xb7\x1d\x91\x55\x38\xb6\xf6\x34\x65\xc7\xbd",
1615
+
.c_size = 139,
1616
+
.public_key_vec = true,
1617
+
},
1618
+
{
1619
+
.key = /* secp521r1(sha384) */
1620
+
"\x04\x00\x2e\xd6\x21\x04\x75\xc3\xdc\x7d\xff\x0e\xf3\x70\x25\x2b"
1621
+
"\xad\x72\xfc\x5a\x91\xf1\xd5\x9c\x64\xf3\x1f\x47\x11\x10\x62\x33"
1622
+
"\xfd\x2e\xe8\x32\xca\x9e\x6f\x0a\x4c\x5b\x35\x9a\x46\xc5\xe7\xd4"
1623
+
"\x38\xda\xb2\xf0\xf4\x87\xf3\x86\xf4\xea\x70\xad\x1e\xd4\x78\x8c"
1624
+
"\x36\x18\x17\x00\xa2\xa0\x34\x1b\x2e\x6a\xdf\x06\xd6\x99\x2d\x47"
1625
+
"\x50\x92\x1a\x8a\x72\x9c\x23\x44\xfa\xa7\xa9\xed\xa6\xef\x26\x14"
1626
+
"\xb3\x9d\xfe\x5e\xa3\x8c\xd8\x29\xf8\xdf\xad\xa6\xab\xfc\xdd\x46"
1627
+
"\x22\x6e\xd7\x35\xc7\x23\xb7\x13\xae\xb6\x34\xff\xd7\x80\xe5\x39"
1628
+
"\xb3\x3b\x5b\x1b\x94",
1629
+
.key_len = 133,
1630
+
.m =
1631
+
"\x36\x98\xd6\x82\xfa\xad\xed\x3c\xb9\x40\xb6\x4d\x9e\xb7\x04\x26"
1632
+
"\xad\x72\x34\x44\xd2\x81\xb4\x9b\xbe\x01\x04\x7a\xd8\x50\xf8\x59"
1633
+
"\xba\xad\x23\x85\x6b\x59\xbe\xfb\xf6\x86\xd4\x67\xa8\x43\x28\x76",
1634
+
.m_size = 48,
1635
+
.c =
1636
+
"\x30\x81\x88\x02\x42\x00\x93\x96\x76\x3c\x27\xea\xaa\x9c\x26\xec"
1637
+
"\x51\xdc\xe8\x35\x5e\xae\x16\xf2\x4b\x64\x98\xf7\xec\xda\xc7\x7e"
1638
+
"\x42\x71\x86\x57\x2d\xf1\x7d\xe4\xdf\x9b\x7d\x9e\x47\xca\x33\x32"
1639
+
"\x76\x06\xd0\xf9\xc0\xe4\xe6\x84\x59\xfd\x1a\xc4\x40\xdd\x43\xb8"
1640
+
"\x6a\xdd\xfb\xe6\x63\x4e\x28\x02\x42\x00\xff\xc3\x6a\x87\x6e\xb5"
1641
+
"\x13\x1f\x20\x55\xce\x37\x97\xc9\x05\x51\xe5\xe4\x3c\xbc\x93\x65"
1642
+
"\x57\x1c\x30\xda\xa7\xcd\x26\x28\x76\x3b\x52\xdf\xc4\xc0\xdb\x54"
1643
+
"\xdb\x8a\x0d\x6a\xc3\xf3\x7a\xd1\xfa\xe7\xa7\xe5\x5a\x94\x56\xcf"
1644
+
"\x8f\xb4\x22\xc6\x4f\xab\x2b\x62\xc1\x42\xb1",
1645
+
.c_size = 139,
1646
+
.public_key_vec = true,
1647
+
},
1648
+
{
1649
+
.key = /* secp521r1(sha512) */
1650
+
"\x04\x00\xc7\x65\xee\x0b\x86\x7d\x8f\x02\xf1\x74\x5b\xb0\x4c\x3f"
1651
+
"\xa6\x35\x60\x9f\x55\x23\x11\xcc\xdf\xb8\x42\x99\xee\x6c\x96\x6a"
1652
+
"\x27\xa2\x56\xb2\x2b\x03\xad\x0f\xe7\x97\xde\x09\x5d\xb4\xc5\x5f"
1653
+
"\xbd\x87\x37\xbf\x5a\x16\x35\x56\x08\xfd\x6f\x06\x1a\x1c\x84\xee"
1654
+
"\xc3\x64\xb3\x00\x9e\xbd\x6e\x60\x76\xee\x69\xfd\x3a\xb8\xcd\x7e"
1655
+
"\x91\x68\x53\x57\x44\x13\x2e\x77\x09\x2a\xbe\x48\xbd\x91\xd8\xf6"
1656
+
"\x21\x16\x53\x99\xd5\xf0\x40\xad\xa6\xf8\x58\x26\xb6\x9a\xf8\x77"
1657
+
"\xfe\x3a\x05\x1a\xdb\xa9\x0f\xc0\x6c\x76\x30\x8c\xd8\xde\x44\xae"
1658
+
"\xd0\x17\xdf\x49\x6a",
1659
+
.key_len = 133,
1660
+
.m =
1661
+
"\x5c\xa6\xbc\x79\xb8\xa0\x1e\x11\x83\xf7\xe9\x05\xdf\xba\xf7\x69"
1662
+
"\x97\x22\x32\xe4\x94\x7c\x65\xbd\x74\xc6\x9a\x8b\xbd\x0d\xdc\xed"
1663
+
"\xf5\x9c\xeb\xe1\xc5\x68\x40\xf2\xc7\x04\xde\x9e\x0d\x76\xc5\xa3"
1664
+
"\xf9\x3c\x6c\x98\x08\x31\xbd\x39\xe8\x42\x7f\x80\x39\x6f\xfe\x68",
1665
+
.m_size = 64,
1331
1666
.c =
1332
1667
"\x30\x81\x88\x02\x42\x01\x5c\x71\x86\x96\xac\x21\x33\x7e\x4e\xaa"
1333
1668
"\x86\xec\xa8\x05\x03\x52\x56\x63\x0e\x02\xcc\x94\xa9\x05\xb9\xfb"
···
1802
1213
"\xa6\xe5\x25\x46\x1e\x77\x44\x78\xe0\xd1\x04",
1803
1214
.c_size = 139,
1804
1215
.public_key_vec = true,
1805
-
.siggen_sigver_test = true,
1216
+
},
1217
+
};
1218
+
1219
+
/*
1220
+
* ECDSA P1363 test vectors.
1221
+
*
1222
+
* Identical to ECDSA test vectors, except signature in "c" is P1363 encoded.
1223
+
*/
1224
+
static const struct sig_testvec p1363_ecdsa_nist_p256_tv_template[] = {
1225
+
{
1226
+
.key = /* secp256r1(sha256) */
1227
+
"\x04\xf1\xea\xc4\x53\xf3\xb9\x0e\x9f\x7e\xad\xe3\xea\xd7\x0e\x0f"
1228
+
"\xd6\x98\x9a\xca\x92\x4d\x0a\x80\xdb\x2d\x45\xc7\xec\x4b\x97\x00"
1229
+
"\x2f\xe9\x42\x6c\x29\xdc\x55\x0e\x0b\x53\x12\x9b\x2b\xad\x2c\xe9"
1230
+
"\x80\xe6\xc5\x43\xc2\x1d\x5e\xbb\x65\x21\x50\xb6\x37\xb0\x03\x8e"
1231
+
"\xb8",
1232
+
.key_len = 65,
1233
+
.m =
1234
+
"\x8f\x43\x43\x46\x64\x8f\x6b\x96\xdf\x89\xdd\xa9\x01\xc5\x17\x6b"
1235
+
"\x10\xa6\xd8\x39\x61\xdd\x3c\x1a\xc8\x8b\x59\xb2\xdc\x32\x7a\xa4",
1236
+
.m_size = 32,
1237
+
.c =
1238
+
"\x08\x31\xfa\x74\x0d\x1d\x21\x5d\x09\xdc\x29\x63\xa8\x1a\xad\xfc"
1239
+
"\xac\x44\xc3\xe8\x24\x11\x2d\xa4\x91\xdc\x02\x67\xdc\x0c\xd0\x82"
1240
+
"\xbd\xff\xce\xee\x42\xc3\x97\xff\xf9\xa9\x81\xac\x4a\x50\xd0\x91"
1241
+
"\x0a\x6e\x1b\xc4\xaf\xe1\x83\xc3\x4f\x2a\x65\x35\x23\xe3\x1d\xfa",
1242
+
.c_size = 64,
1243
+
.public_key_vec = true,
1806
1244
},
1807
1245
};
1808
1246
1809
1247
/*
1810
1248
* EC-RDSA test vectors are generated by gost-engine.
1811
1249
*/
1812
-
static const struct akcipher_testvec ecrdsa_tv_template[] = {
1250
+
static const struct sig_testvec ecrdsa_tv_template[] = {
1813
1251
{
1814
1252
.key =
1815
1253
"\x04\x40\xd5\xa7\x77\xf9\x26\x2f\x8c\xbd\xcc\xe3\x1f\x01\x94\x05"
···
1861
1245
"\x79\xd2\x76\x64\xa3\xbd\x66\x10\x79\x05\x5a\x06\x42\xec\xb9\xc9",
1862
1246
.m_size = 32,
1863
1247
.public_key_vec = true,
1864
-
.siggen_sigver_test = true,
1865
1248
},
1866
1249
{
1867
1250
.key =
···
1886
1271
"\x11\x23\x4a\x70\x43\x52\x7a\x68\x11\x65\x45\x37\xbb\x25\xb7\x40",
1887
1272
.m_size = 32,
1888
1273
.public_key_vec = true,
1889
-
.siggen_sigver_test = true,
1890
1274
},
1891
1275
{
1892
1276
.key =
···
1911
1297
"\x9f\x16\xc6\x1c\xb1\x3f\x84\x41\x69\xec\x34\xfd\xf1\xf9\xa3\x39",
1912
1298
.m_size = 32,
1913
1299
.public_key_vec = true,
1914
-
.siggen_sigver_test = true,
1915
1300
},
1916
1301
{
1917
1302
.key =
···
1945
1332
"\xa8\xf6\x80\x01\xb9\x27\xac\xd8\x45\x96\x66\xa1\xee\x48\x08\x3f",
1946
1333
.m_size = 64,
1947
1334
.public_key_vec = true,
1948
-
.siggen_sigver_test = true,
1949
1335
},
1950
1336
{
1951
1337
.key =
···
1979
1367
"\x6d\xf4\xd2\x45\xc2\x83\xa0\x42\x95\x05\x9d\x89\x8e\x0a\xca\xcc",
1980
1368
.m_size = 64,
1981
1369
.public_key_vec = true,
1982
-
.siggen_sigver_test = true,
1370
+
},
1371
+
};
1372
+
1373
+
/*
1374
+
* PKCS#1 RSA test vectors for hash algorithm "none"
1375
+
* (i.e. the hash in "m" is not prepended by a Full Hash Prefix)
1376
+
*
1377
+
* Obtained from:
1378
+
* https://vcsjones.dev/sometimes-valid-rsa-dotnet/
1379
+
* https://gist.github.com/vcsjones/ab4c2327b53ed018eada76b75ef4fd99
1380
+
*/
1381
+
static const struct sig_testvec pkcs1_rsa_none_tv_template[] = {
1382
+
{
1383
+
.key =
1384
+
"\x30\x82\x01\x0a\x02\x82\x01\x01\x00\xa2\x63\x0b\x39\x44\xb8\xbb"
1385
+
"\x23\xa7\x44\x49\xbb\x0e\xff\xa1\xf0\x61\x0a\x53\x93\xb0\x98\xdb"
1386
+
"\xad\x2c\x0f\x4a\xc5\x6e\xff\x86\x3c\x53\x55\x0f\x15\xce\x04\x3f"
1387
+
"\x2b\xfd\xa9\x96\x96\xd9\xbe\x61\x79\x0b\x5b\xc9\x4c\x86\x76\xe5"
1388
+
"\xe0\x43\x4b\x22\x95\xee\xc2\x2b\x43\xc1\x9f\xd8\x68\xb4\x8e\x40"
1389
+
"\x4f\xee\x85\x38\xb9\x11\xc5\x23\xf2\x64\x58\xf0\x15\x32\x6f\x4e"
1390
+
"\x57\xa1\xae\x88\xa4\x02\xd7\x2a\x1e\xcd\x4b\xe1\xdd\x63\xd5\x17"
1391
+
"\x89\x32\x5b\xb0\x5e\x99\x5a\xa8\x9d\x28\x50\x0e\x17\xee\x96\xdb"
1392
+
"\x61\x3b\x45\x51\x1d\xcf\x12\x56\x0b\x92\x47\xfc\xab\xae\xf6\x66"
1393
+
"\x3d\x47\xac\x70\x72\xe7\x92\xe7\x5f\xcd\x10\xb9\xc4\x83\x64\x94"
1394
+
"\x19\xbd\x25\x80\xe1\xe8\xd2\x22\xa5\xd0\xba\x02\x7a\xa1\x77\x93"
1395
+
"\x5b\x65\xc3\xee\x17\x74\xbc\x41\x86\x2a\xdc\x08\x4c\x8c\x92\x8c"
1396
+
"\x91\x2d\x9e\x77\x44\x1f\x68\xd6\xa8\x74\x77\xdb\x0e\x5b\x32\x8b"
1397
+
"\x56\x8b\x33\xbd\xd9\x63\xc8\x49\x9d\x3a\xc5\xc5\xea\x33\x0b\xd2"
1398
+
"\xf1\xa3\x1b\xf4\x8b\xbe\xd9\xb3\x57\x8b\x3b\xde\x04\xa7\x7a\x22"
1399
+
"\xb2\x24\xae\x2e\xc7\x70\xc5\xbe\x4e\x83\x26\x08\xfb\x0b\xbd\xa9"
1400
+
"\x4f\x99\x08\xe1\x10\x28\x72\xaa\xcd\x02\x03\x01\x00\x01",
1401
+
.key_len = 270,
1402
+
.m =
1403
+
"\x68\xb4\xf9\x26\x34\x31\x25\xdd\x26\x50\x13\x68\xc1\x99\x26\x71"
1404
+
"\x19\xa2\xde\x81",
1405
+
.m_size = 20,
1406
+
.c =
1407
+
"\x6a\xdb\x39\xe5\x63\xb3\x25\xde\x58\xca\xc3\xf1\x36\x9c\x0b\x36"
1408
+
"\xb7\xd6\x69\xf9\xba\xa6\x68\x14\x8c\x24\x52\xd3\x25\xa5\xf3\xad"
1409
+
"\xc9\x47\x44\xde\x06\xd8\x0f\x56\xca\x2d\xfb\x0f\xe9\x99\xe2\x9d"
1410
+
"\x8a\xe8\x7f\xfb\x9a\x99\x96\xf1\x2c\x4a\xe4\xc0\xae\x4d\x29\x47"
1411
+
"\x38\x96\x51\x2f\x6d\x8e\xb8\x88\xbd\x1a\x0a\x70\xbc\x23\x38\x67"
1412
+
"\x62\x22\x01\x23\x71\xe5\xbb\x95\xea\x6b\x8d\x31\x62\xbf\xf0\xc4"
1413
+
"\xb9\x46\xd6\x67\xfc\x4c\xe6\x1f\xd6\x5d\xf7\xa9\xad\x3a\xf1\xbf"
1414
+
"\xa2\xf9\x66\xde\xb6\x8e\xec\x8f\x81\x8d\x1e\x3a\x12\x27\x6a\xfc"
1415
+
"\xae\x92\x9f\xc3\x87\xc3\xba\x8d\x04\xb8\x8f\x0f\x61\x68\x9a\x96"
1416
+
"\x2c\x80\x2c\x32\x40\xde\x9d\xb9\x9b\xe2\xe4\x45\x2e\x91\x47\x5c"
1417
+
"\x47\xa4\x9d\x02\x57\x59\xf7\x75\x5d\x5f\x32\x82\x75\x5d\xe5\x78"
1418
+
"\xc9\x19\x61\x46\x06\x9d\xa5\x1d\xd6\x32\x48\x9a\xdb\x09\x29\x81"
1419
+
"\x14\x2e\xf0\x27\xe9\x37\x13\x74\xec\xa5\xcd\x67\x6b\x19\xf6\x88"
1420
+
"\xf0\xc2\x8b\xa8\x7f\x2f\x76\x5a\x3e\x0c\x47\x5d\xe8\x82\x50\x27"
1421
+
"\x40\xce\x27\x41\x45\xa0\xcf\xaa\x2f\xd3\xad\x3c\xbf\x73\xff\x93"
1422
+
"\xe3\x78\x49\xd9\xa9\x78\x22\x81\x9a\xe5\xe2\x94\xe9\x40\xab\xf1",
1423
+
.c_size = 256,
1424
+
.public_key_vec = true,
1983
1425
},
1984
1426
};
1985
1427
1986
1428
/*
1987
1429
* PKCS#1 RSA test vectors. Obtained from CAVS testing.
1988
1430
*/
1989
-
static const struct akcipher_testvec pkcs1pad_rsa_tv_template[] = {
1431
+
static const struct sig_testvec pkcs1_rsa_tv_template[] = {
1990
1432
{
1991
1433
.key =
1992
1434
"\x30\x82\x04\xa5\x02\x01\x00\x02\x82\x01\x01\x00\xd7\x1e\x77\x82"
···
2152
1486
"\xda\x62\x8d\xe1\x2a\x71\x91\x43\x40\x61\x3c\x5a\xbe\x86\xfc\x5b"
2153
1487
"\xe6\xf9\xa9\x16\x31\x1f\xaf\x25\x6d\xc2\x4a\x23\x6e\x63\x02\xa2",
2154
1488
.c_size = 256,
2155
-
.siggen_sigver_test = true,
2156
1489
}
2157
1490
};
2158
1491
+28
-2
drivers/char/hw_random/Kconfig
+28
-2
drivers/char/hw_random/Kconfig
···
50
50
51
51
config HW_RANDOM_AMD
52
52
tristate "AMD HW Random Number Generator support"
53
-
depends on (X86 || PPC_MAPLE || COMPILE_TEST)
53
+
depends on (X86 || COMPILE_TEST)
54
54
depends on PCI && HAS_IOPORT_MAP
55
55
default HW_RANDOM
56
56
help
···
59
59
60
60
To compile this driver as a module, choose M here: the
61
61
module will be called amd-rng.
62
+
63
+
If unsure, say Y.
64
+
65
+
config HW_RANDOM_AIROHA
66
+
tristate "Airoha True HW Random Number Generator support"
67
+
depends on ARCH_AIROHA || COMPILE_TEST
68
+
default HW_RANDOM
69
+
help
70
+
This driver provides kernel-side support for the True Random Number
71
+
Generator hardware found on Airoha SoC.
72
+
73
+
To compile this driver as a module, choose M here: the
74
+
module will be called airoha-rng.
62
75
63
76
If unsure, say Y.
64
77
···
112
99
113
100
If unsure, say Y.
114
101
102
+
config HW_RANDOM_BCM74110
103
+
tristate "Broadcom BCM74110 Random Number Generator support"
104
+
depends on ARCH_BRCMSTB || COMPILE_TEST
105
+
default HW_RANDOM
106
+
help
107
+
This driver provides kernel-side support for the Random Number
108
+
Generator hardware found on the Broadcom BCM74110 SoCs.
109
+
110
+
To compile this driver as a module, choose M here: the
111
+
module will be called bcm74110-rng
112
+
113
+
If unsure, say Y.
114
+
115
115
config HW_RANDOM_IPROC_RNG200
116
116
tristate "Broadcom iProc/STB RNG200 support"
117
-
depends on ARCH_BCM_IPROC || ARCH_BCM2835 || ARCH_BRCMSTB || COMPILE_TEST
117
+
depends on ARCH_BCM_IPROC || ARCH_BCM2835 || ARCH_BCMBCA || ARCH_BRCMSTB || COMPILE_TEST
118
118
default HW_RANDOM
119
119
help
120
120
This driver provides kernel-side support for the RNG200
+2
drivers/char/hw_random/Makefile
+2
drivers/char/hw_random/Makefile
···
8
8
obj-$(CONFIG_HW_RANDOM_TIMERIOMEM) += timeriomem-rng.o
9
9
obj-$(CONFIG_HW_RANDOM_INTEL) += intel-rng.o
10
10
obj-$(CONFIG_HW_RANDOM_AMD) += amd-rng.o
11
+
obj-$(CONFIG_HW_RANDOM_AIROHA) += airoha-trng.o
11
12
obj-$(CONFIG_HW_RANDOM_ATMEL) += atmel-rng.o
12
13
obj-$(CONFIG_HW_RANDOM_BA431) += ba431-rng.o
13
14
obj-$(CONFIG_HW_RANDOM_GEODE) += geode-rng.o
···
32
31
obj-$(CONFIG_HW_RANDOM_HISI) += hisi-rng.o
33
32
obj-$(CONFIG_HW_RANDOM_HISTB) += histb-rng.o
34
33
obj-$(CONFIG_HW_RANDOM_BCM2835) += bcm2835-rng.o
34
+
obj-$(CONFIG_HW_RANDOM_BCM74110) += bcm74110-rng.o
35
35
obj-$(CONFIG_HW_RANDOM_IPROC_RNG200) += iproc-rng200.o
36
36
obj-$(CONFIG_HW_RANDOM_ST) += st-rng.o
37
37
obj-$(CONFIG_HW_RANDOM_XGENE) += xgene-rng.o
+243
drivers/char/hw_random/airoha-trng.c
+243
drivers/char/hw_random/airoha-trng.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (C) 2024 Christian Marangi */
3
+
4
+
#include <linux/kernel.h>
5
+
#include <linux/module.h>
6
+
#include <linux/mod_devicetable.h>
7
+
#include <linux/bitfield.h>
8
+
#include <linux/delay.h>
9
+
#include <linux/hw_random.h>
10
+
#include <linux/interrupt.h>
11
+
#include <linux/io.h>
12
+
#include <linux/iopoll.h>
13
+
#include <linux/platform_device.h>
14
+
15
+
#define TRNG_IP_RDY 0x800
16
+
#define CNT_TRANS GENMASK(15, 8)
17
+
#define SAMPLE_RDY BIT(0)
18
+
#define TRNG_NS_SEK_AND_DAT_EN 0x804
19
+
#define RNG_EN BIT(31) /* referenced as ring_en */
20
+
#define RAW_DATA_EN BIT(16)
21
+
#define TRNG_HEALTH_TEST_SW_RST 0x808
22
+
#define SW_RST BIT(0) /* Active High */
23
+
#define TRNG_INTR_EN 0x818
24
+
#define INTR_MASK BIT(16)
25
+
#define CONTINUOUS_HEALTH_INITR_EN BIT(2)
26
+
#define SW_STARTUP_INITR_EN BIT(1)
27
+
#define RST_STARTUP_INITR_EN BIT(0)
28
+
/* Notice that Health Test are done only out of Reset and with RNG_EN */
29
+
#define TRNG_HEALTH_TEST_STATUS 0x824
30
+
#define CONTINUOUS_HEALTH_AP_TEST_FAIL BIT(23)
31
+
#define CONTINUOUS_HEALTH_RC_TEST_FAIL BIT(22)
32
+
#define SW_STARTUP_TEST_DONE BIT(21)
33
+
#define SW_STARTUP_AP_TEST_FAIL BIT(20)
34
+
#define SW_STARTUP_RC_TEST_FAIL BIT(19)
35
+
#define RST_STARTUP_TEST_DONE BIT(18)
36
+
#define RST_STARTUP_AP_TEST_FAIL BIT(17)
37
+
#define RST_STARTUP_RC_TEST_FAIL BIT(16)
38
+
#define RAW_DATA_VALID BIT(7)
39
+
40
+
#define TRNG_RAW_DATA_OUT 0x828
41
+
42
+
#define TRNG_CNT_TRANS_VALID 0x80
43
+
#define BUSY_LOOP_SLEEP 10
44
+
#define BUSY_LOOP_TIMEOUT (BUSY_LOOP_SLEEP * 10000)
45
+
46
+
struct airoha_trng {
47
+
void __iomem *base;
48
+
struct hwrng rng;
49
+
struct device *dev;
50
+
51
+
struct completion rng_op_done;
52
+
};
53
+
54
+
static int airoha_trng_irq_mask(struct airoha_trng *trng)
55
+
{
56
+
u32 val;
57
+
58
+
val = readl(trng->base + TRNG_INTR_EN);
59
+
val |= INTR_MASK;
60
+
writel(val, trng->base + TRNG_INTR_EN);
61
+
62
+
return 0;
63
+
}
64
+
65
+
static int airoha_trng_irq_unmask(struct airoha_trng *trng)
66
+
{
67
+
u32 val;
68
+
69
+
val = readl(trng->base + TRNG_INTR_EN);
70
+
val &= ~INTR_MASK;
71
+
writel(val, trng->base + TRNG_INTR_EN);
72
+
73
+
return 0;
74
+
}
75
+
76
+
static int airoha_trng_init(struct hwrng *rng)
77
+
{
78
+
struct airoha_trng *trng = container_of(rng, struct airoha_trng, rng);
79
+
int ret;
80
+
u32 val;
81
+
82
+
val = readl(trng->base + TRNG_NS_SEK_AND_DAT_EN);
83
+
val |= RNG_EN;
84
+
writel(val, trng->base + TRNG_NS_SEK_AND_DAT_EN);
85
+
86
+
/* Set out of SW Reset */
87
+
airoha_trng_irq_unmask(trng);
88
+
writel(0, trng->base + TRNG_HEALTH_TEST_SW_RST);
89
+
90
+
ret = wait_for_completion_timeout(&trng->rng_op_done, BUSY_LOOP_TIMEOUT);
91
+
if (ret <= 0) {
92
+
dev_err(trng->dev, "Timeout waiting for Health Check\n");
93
+
airoha_trng_irq_mask(trng);
94
+
return -ENODEV;
95
+
}
96
+
97
+
/* Check if Health Test Failed */
98
+
val = readl(trng->base + TRNG_HEALTH_TEST_STATUS);
99
+
if (val & (RST_STARTUP_AP_TEST_FAIL | RST_STARTUP_RC_TEST_FAIL)) {
100
+
dev_err(trng->dev, "Health Check fail: %s test fail\n",
101
+
val & RST_STARTUP_AP_TEST_FAIL ? "AP" : "RC");
102
+
return -ENODEV;
103
+
}
104
+
105
+
/* Check if IP is ready */
106
+
ret = readl_poll_timeout(trng->base + TRNG_IP_RDY, val,
107
+
val & SAMPLE_RDY, 10, 1000);
108
+
if (ret < 0) {
109
+
dev_err(trng->dev, "Timeout waiting for IP ready");
110
+
return -ENODEV;
111
+
}
112
+
113
+
/* CNT_TRANS must be 0x80 for IP to be considered ready */
114
+
ret = readl_poll_timeout(trng->base + TRNG_IP_RDY, val,
115
+
FIELD_GET(CNT_TRANS, val) == TRNG_CNT_TRANS_VALID,
116
+
10, 1000);
117
+
if (ret < 0) {
118
+
dev_err(trng->dev, "Timeout waiting for IP ready");
119
+
return -ENODEV;
120
+
}
121
+
122
+
return 0;
123
+
}
124
+
125
+
static void airoha_trng_cleanup(struct hwrng *rng)
126
+
{
127
+
struct airoha_trng *trng = container_of(rng, struct airoha_trng, rng);
128
+
u32 val;
129
+
130
+
val = readl(trng->base + TRNG_NS_SEK_AND_DAT_EN);
131
+
val &= ~RNG_EN;
132
+
writel(val, trng->base + TRNG_NS_SEK_AND_DAT_EN);
133
+
134
+
/* Put it in SW Reset */
135
+
writel(SW_RST, trng->base + TRNG_HEALTH_TEST_SW_RST);
136
+
}
137
+
138
+
static int airoha_trng_read(struct hwrng *rng, void *buf, size_t max, bool wait)
139
+
{
140
+
struct airoha_trng *trng = container_of(rng, struct airoha_trng, rng);
141
+
u32 *data = buf;
142
+
u32 status;
143
+
int ret;
144
+
145
+
ret = readl_poll_timeout(trng->base + TRNG_HEALTH_TEST_STATUS, status,
146
+
status & RAW_DATA_VALID, 10, 1000);
147
+
if (ret < 0) {
148
+
dev_err(trng->dev, "Timeout waiting for TRNG RAW Data valid\n");
149
+
return ret;
150
+
}
151
+
152
+
*data = readl(trng->base + TRNG_RAW_DATA_OUT);
153
+
154
+
return 4;
155
+
}
156
+
157
+
static irqreturn_t airoha_trng_irq(int irq, void *priv)
158
+
{
159
+
struct airoha_trng *trng = (struct airoha_trng *)priv;
160
+
161
+
airoha_trng_irq_mask(trng);
162
+
/* Just complete the task, we will read the value later */
163
+
complete(&trng->rng_op_done);
164
+
165
+
return IRQ_HANDLED;
166
+
}
167
+
168
+
static int airoha_trng_probe(struct platform_device *pdev)
169
+
{
170
+
struct device *dev = &pdev->dev;
171
+
struct airoha_trng *trng;
172
+
int irq, ret;
173
+
u32 val;
174
+
175
+
trng = devm_kzalloc(dev, sizeof(*trng), GFP_KERNEL);
176
+
if (!trng)
177
+
return -ENOMEM;
178
+
179
+
trng->base = devm_platform_ioremap_resource(pdev, 0);
180
+
if (IS_ERR(trng->base))
181
+
return PTR_ERR(trng->base);
182
+
183
+
irq = platform_get_irq(pdev, 0);
184
+
if (irq < 0)
185
+
return irq;
186
+
187
+
airoha_trng_irq_mask(trng);
188
+
ret = devm_request_irq(&pdev->dev, irq, airoha_trng_irq, 0,
189
+
pdev->name, (void *)trng);
190
+
if (ret) {
191
+
dev_err(dev, "Can't get interrupt working.\n");
192
+
return ret;
193
+
}
194
+
195
+
init_completion(&trng->rng_op_done);
196
+
197
+
/* Enable interrupt for SW reset Health Check */
198
+
val = readl(trng->base + TRNG_INTR_EN);
199
+
val |= RST_STARTUP_INITR_EN;
200
+
writel(val, trng->base + TRNG_INTR_EN);
201
+
202
+
/* Set output to raw data */
203
+
val = readl(trng->base + TRNG_NS_SEK_AND_DAT_EN);
204
+
val |= RAW_DATA_EN;
205
+
writel(val, trng->base + TRNG_NS_SEK_AND_DAT_EN);
206
+
207
+
/* Put it in SW Reset */
208
+
writel(SW_RST, trng->base + TRNG_HEALTH_TEST_SW_RST);
209
+
210
+
trng->dev = dev;
211
+
trng->rng.name = pdev->name;
212
+
trng->rng.init = airoha_trng_init;
213
+
trng->rng.cleanup = airoha_trng_cleanup;
214
+
trng->rng.read = airoha_trng_read;
215
+
216
+
ret = devm_hwrng_register(dev, &trng->rng);
217
+
if (ret) {
218
+
dev_err(dev, "failed to register rng device: %d\n", ret);
219
+
return ret;
220
+
}
221
+
222
+
return 0;
223
+
}
224
+
225
+
static const struct of_device_id airoha_trng_of_match[] = {
226
+
{ .compatible = "airoha,en7581-trng", },
227
+
{},
228
+
};
229
+
MODULE_DEVICE_TABLE(of, airoha_trng_of_match);
230
+
231
+
static struct platform_driver airoha_trng_driver = {
232
+
.driver = {
233
+
.name = "airoha-trng",
234
+
.of_match_table = airoha_trng_of_match,
235
+
},
236
+
.probe = airoha_trng_probe,
237
+
};
238
+
239
+
module_platform_driver(airoha_trng_driver);
240
+
241
+
MODULE_LICENSE("GPL");
242
+
MODULE_AUTHOR("Christian Marangi <ansuelsmth@gmail.com>");
243
+
MODULE_DESCRIPTION("Airoha True Random Number Generator driver");
+1
-1
drivers/char/hw_random/atmel-rng.c
+1
-1
drivers/char/hw_random/atmel-rng.c
+125
drivers/char/hw_random/bcm74110-rng.c
+125
drivers/char/hw_random/bcm74110-rng.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/*
3
+
* Copyright (c) 2024 Broadcom
4
+
*/
5
+
6
+
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
7
+
8
+
#include <linux/module.h>
9
+
#include <linux/mod_devicetable.h>
10
+
#include <linux/kernel.h>
11
+
#include <linux/io.h>
12
+
#include <linux/delay.h>
13
+
#include <linux/platform_device.h>
14
+
#include <linux/random.h>
15
+
#include <linux/hw_random.h>
16
+
17
+
#define HOST_REV_ID 0x00
18
+
#define HOST_FIFO_DEPTH 0x04
19
+
#define HOST_FIFO_COUNT 0x08
20
+
#define HOST_FIFO_THRESHOLD 0x0c
21
+
#define HOST_FIFO_DATA 0x10
22
+
23
+
#define HOST_FIFO_COUNT_MASK 0xffff
24
+
25
+
/* Delay range in microseconds */
26
+
#define FIFO_DELAY_MIN_US 3
27
+
#define FIFO_DELAY_MAX_US 7
28
+
#define FIFO_DELAY_MAX_COUNT 10
29
+
30
+
struct bcm74110_priv {
31
+
void __iomem *base;
32
+
};
33
+
34
+
static inline int bcm74110_rng_fifo_count(void __iomem *mem)
35
+
{
36
+
return readl_relaxed(mem) & HOST_FIFO_COUNT_MASK;
37
+
}
38
+
39
+
static int bcm74110_rng_read(struct hwrng *rng, void *buf, size_t max,
40
+
bool wait)
41
+
{
42
+
struct bcm74110_priv *priv = (struct bcm74110_priv *)rng->priv;
43
+
void __iomem *fc_addr = priv->base + HOST_FIFO_COUNT;
44
+
void __iomem *fd_addr = priv->base + HOST_FIFO_DATA;
45
+
unsigned underrun_count = 0;
46
+
u32 max_words = max / sizeof(u32);
47
+
u32 num_words;
48
+
unsigned i;
49
+
50
+
/*
51
+
* We need to check how many words are available in the RNG FIFO. If
52
+
* there aren't any, we need to wait for some to become available.
53
+
*/
54
+
while ((num_words = bcm74110_rng_fifo_count(fc_addr)) == 0) {
55
+
if (!wait)
56
+
return 0;
57
+
/*
58
+
* As a precaution, limit how long we wait. If the FIFO doesn't
59
+
* refill within the allotted time, return 0 (=no data) to the
60
+
* caller.
61
+
*/
62
+
if (likely(underrun_count < FIFO_DELAY_MAX_COUNT))
63
+
usleep_range(FIFO_DELAY_MIN_US, FIFO_DELAY_MAX_US);
64
+
else
65
+
return 0;
66
+
underrun_count++;
67
+
}
68
+
if (num_words > max_words)
69
+
num_words = max_words;
70
+
71
+
/* Bail early if we run out of random numbers unexpectedly */
72
+
for (i = 0; i < num_words && bcm74110_rng_fifo_count(fc_addr) > 0; i++)
73
+
((u32 *)buf)[i] = readl_relaxed(fd_addr);
74
+
75
+
return i * sizeof(u32);
76
+
}
77
+
78
+
static struct hwrng bcm74110_hwrng = {
79
+
.read = bcm74110_rng_read,
80
+
};
81
+
82
+
static int bcm74110_rng_probe(struct platform_device *pdev)
83
+
{
84
+
struct device *dev = &pdev->dev;
85
+
struct bcm74110_priv *priv;
86
+
int rc;
87
+
88
+
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
89
+
if (!priv)
90
+
return -ENOMEM;
91
+
92
+
bcm74110_hwrng.name = pdev->name;
93
+
bcm74110_hwrng.priv = (unsigned long)priv;
94
+
95
+
priv->base = devm_platform_ioremap_resource(pdev, 0);
96
+
if (IS_ERR(priv->base))
97
+
return PTR_ERR(priv->base);
98
+
99
+
rc = devm_hwrng_register(dev, &bcm74110_hwrng);
100
+
if (rc)
101
+
dev_err(dev, "hwrng registration failed (%d)\n", rc);
102
+
else
103
+
dev_info(dev, "hwrng registered\n");
104
+
105
+
return rc;
106
+
}
107
+
108
+
static const struct of_device_id bcm74110_rng_match[] = {
109
+
{ .compatible = "brcm,bcm74110-rng", },
110
+
{},
111
+
};
112
+
MODULE_DEVICE_TABLE(of, bcm74110_rng_match);
113
+
114
+
static struct platform_driver bcm74110_rng_driver = {
115
+
.driver = {
116
+
.name = KBUILD_MODNAME,
117
+
.of_match_table = bcm74110_rng_match,
118
+
},
119
+
.probe = bcm74110_rng_probe,
120
+
};
121
+
module_platform_driver(bcm74110_rng_driver);
122
+
123
+
MODULE_AUTHOR("Markus Mayer <mmayer@broadcom.com>");
124
+
MODULE_DESCRIPTION("BCM 74110 Random Number Generator (RNG) driver");
125
+
MODULE_LICENSE("GPL v2");
+1
-1
drivers/char/hw_random/cctrng.c
+1
-1
drivers/char/hw_random/cctrng.c
+9
-2
drivers/char/hw_random/core.c
+9
-2
drivers/char/hw_random/core.c
···
181
181
int present;
182
182
183
183
BUG_ON(!mutex_is_locked(&reading_mutex));
184
-
if (rng->read)
185
-
return rng->read(rng, (void *)buffer, size, wait);
184
+
if (rng->read) {
185
+
int err;
186
+
187
+
err = rng->read(rng, buffer, size, wait);
188
+
if (WARN_ON_ONCE(err > 0 && err > size))
189
+
err = size;
190
+
191
+
return err;
192
+
}
186
193
187
194
if (rng->data_present)
188
195
present = rng->data_present(rng, wait);
+1
-1
drivers/char/hw_random/exynos-trng.c
+1
-1
drivers/char/hw_random/exynos-trng.c
+1
-1
drivers/char/hw_random/histb-rng.c
+1
-1
drivers/char/hw_random/histb-rng.c
+1
-1
drivers/char/hw_random/ingenic-rng.c
+1
-1
drivers/char/hw_random/ingenic-rng.c
+1
-1
drivers/char/hw_random/ks-sa-rng.c
+1
-1
drivers/char/hw_random/ks-sa-rng.c
+1
-1
drivers/char/hw_random/mxc-rnga.c
+1
-1
drivers/char/hw_random/mxc-rnga.c
+1
-1
drivers/char/hw_random/n2-drv.c
+1
-1
drivers/char/hw_random/n2-drv.c
+1
-1
drivers/char/hw_random/npcm-rng.c
+1
-1
drivers/char/hw_random/npcm-rng.c
+1
-1
drivers/char/hw_random/omap-rng.c
+1
-1
drivers/char/hw_random/omap-rng.c
+58
-20
drivers/char/hw_random/stm32-rng.c
+58
-20
drivers/char/hw_random/stm32-rng.c
···
4
4
*/
5
5
6
6
#include <linux/clk.h>
7
+
#include <linux/clk-provider.h>
7
8
#include <linux/delay.h>
8
9
#include <linux/hw_random.h>
9
10
#include <linux/io.h>
···
50
49
51
50
struct stm32_rng_data {
52
51
uint max_clock_rate;
52
+
uint nb_clock;
53
53
u32 cr;
54
54
u32 nscr;
55
55
u32 htcr;
···
74
72
struct hwrng rng;
75
73
struct device *dev;
76
74
void __iomem *base;
77
-
struct clk *clk;
75
+
struct clk_bulk_data *clk_bulk;
78
76
struct reset_control *rst;
79
77
struct stm32_rng_config pm_conf;
80
78
const struct stm32_rng_data *data;
···
268
266
unsigned long clock_rate = 0;
269
267
uint clock_div = 0;
270
268
271
-
clock_rate = clk_get_rate(priv->clk);
269
+
clock_rate = clk_get_rate(priv->clk_bulk[0].clk);
272
270
273
271
/*
274
272
* Get the exponent to apply on the CLKDIV field in RNG_CR register
···
278
276
while ((clock_rate >> clock_div) > priv->data->max_clock_rate)
279
277
clock_div++;
280
278
281
-
pr_debug("RNG clk rate : %lu\n", clk_get_rate(priv->clk) >> clock_div);
279
+
pr_debug("RNG clk rate : %lu\n", clk_get_rate(priv->clk_bulk[0].clk) >> clock_div);
282
280
283
281
return clock_div;
284
282
}
···
290
288
int err;
291
289
u32 reg;
292
290
293
-
err = clk_prepare_enable(priv->clk);
291
+
err = clk_bulk_prepare_enable(priv->data->nb_clock, priv->clk_bulk);
294
292
if (err)
295
293
return err;
296
294
···
330
328
(!(reg & RNG_CR_CONDRST)),
331
329
10, 50000);
332
330
if (err) {
333
-
clk_disable_unprepare(priv->clk);
331
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
334
332
dev_err(priv->dev, "%s: timeout %x!\n", __func__, reg);
335
333
return -EINVAL;
336
334
}
···
358
356
reg & RNG_SR_DRDY,
359
357
10, 100000);
360
358
if (err || (reg & ~RNG_SR_DRDY)) {
361
-
clk_disable_unprepare(priv->clk);
359
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
362
360
dev_err(priv->dev, "%s: timeout:%x SR: %x!\n", __func__, err, reg);
361
+
363
362
return -EINVAL;
364
363
}
365
364
366
-
clk_disable_unprepare(priv->clk);
365
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
367
366
368
367
return 0;
369
368
}
···
382
379
reg = readl_relaxed(priv->base + RNG_CR);
383
380
reg &= ~RNG_CR_RNGEN;
384
381
writel_relaxed(reg, priv->base + RNG_CR);
385
-
clk_disable_unprepare(priv->clk);
382
+
383
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
386
384
387
385
return 0;
388
386
}
···
393
389
struct stm32_rng_private *priv = dev_get_drvdata(dev);
394
390
int err;
395
391
396
-
err = clk_prepare_enable(priv->clk);
392
+
err = clk_bulk_prepare_enable(priv->data->nb_clock, priv->clk_bulk);
397
393
if (err)
398
394
return err;
399
395
···
407
403
408
404
writel_relaxed(priv->pm_conf.cr, priv->base + RNG_CR);
409
405
410
-
clk_disable_unprepare(priv->clk);
406
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
411
407
412
408
return 0;
413
409
}
···
418
414
int err;
419
415
u32 reg;
420
416
421
-
err = clk_prepare_enable(priv->clk);
417
+
err = clk_bulk_prepare_enable(priv->data->nb_clock, priv->clk_bulk);
422
418
if (err)
423
419
return err;
424
420
···
438
434
int err;
439
435
u32 reg;
440
436
441
-
err = clk_prepare_enable(priv->clk);
437
+
err = clk_bulk_prepare_enable(priv->data->nb_clock, priv->clk_bulk);
442
438
if (err)
443
439
return err;
444
440
···
466
462
reg & ~RNG_CR_CONDRST, 10, 100000);
467
463
468
464
if (err) {
469
-
clk_disable_unprepare(priv->clk);
465
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
470
466
dev_err(priv->dev, "%s: timeout:%x CR: %x!\n", __func__, err, reg);
471
467
return -EINVAL;
472
468
}
···
476
472
writel_relaxed(reg, priv->base + RNG_CR);
477
473
}
478
474
479
-
clk_disable_unprepare(priv->clk);
475
+
clk_bulk_disable_unprepare(priv->data->nb_clock, priv->clk_bulk);
480
476
481
477
return 0;
482
478
}
···
488
484
stm32_rng_resume)
489
485
};
490
486
487
+
static const struct stm32_rng_data stm32mp25_rng_data = {
488
+
.has_cond_reset = true,
489
+
.max_clock_rate = 48000000,
490
+
.nb_clock = 2,
491
+
.cr = 0x00F00D00,
492
+
.nscr = 0x2B5BB,
493
+
.htcr = 0x969D,
494
+
};
495
+
491
496
static const struct stm32_rng_data stm32mp13_rng_data = {
492
497
.has_cond_reset = true,
493
498
.max_clock_rate = 48000000,
499
+
.nb_clock = 1,
494
500
.cr = 0x00F00D00,
495
501
.nscr = 0x2B5BB,
496
502
.htcr = 0x969D,
···
508
494
509
495
static const struct stm32_rng_data stm32_rng_data = {
510
496
.has_cond_reset = false,
511
-
.max_clock_rate = 3000000,
497
+
.max_clock_rate = 48000000,
498
+
.nb_clock = 1,
512
499
};
513
500
514
501
static const struct of_device_id stm32_rng_match[] = {
502
+
{
503
+
.compatible = "st,stm32mp25-rng",
504
+
.data = &stm32mp25_rng_data,
505
+
},
515
506
{
516
507
.compatible = "st,stm32mp13-rng",
517
508
.data = &stm32mp13_rng_data,
···
535
516
struct device_node *np = ofdev->dev.of_node;
536
517
struct stm32_rng_private *priv;
537
518
struct resource *res;
519
+
int ret;
538
520
539
521
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
540
522
if (!priv)
···
544
524
priv->base = devm_platform_get_and_ioremap_resource(ofdev, 0, &res);
545
525
if (IS_ERR(priv->base))
546
526
return PTR_ERR(priv->base);
547
-
548
-
priv->clk = devm_clk_get(&ofdev->dev, NULL);
549
-
if (IS_ERR(priv->clk))
550
-
return PTR_ERR(priv->clk);
551
527
552
528
priv->rst = devm_reset_control_get(&ofdev->dev, NULL);
553
529
if (!IS_ERR(priv->rst)) {
···
567
551
priv->rng.read = stm32_rng_read;
568
552
priv->rng.quality = 900;
569
553
554
+
if (!priv->data->nb_clock || priv->data->nb_clock > 2)
555
+
return -EINVAL;
556
+
557
+
ret = devm_clk_bulk_get_all(dev, &priv->clk_bulk);
558
+
if (ret != priv->data->nb_clock)
559
+
return dev_err_probe(dev, -EINVAL, "Failed to get clocks: %d\n", ret);
560
+
561
+
if (priv->data->nb_clock == 2) {
562
+
const char *id = priv->clk_bulk[1].id;
563
+
struct clk *clk = priv->clk_bulk[1].clk;
564
+
565
+
if (!priv->clk_bulk[0].id || !priv->clk_bulk[1].id)
566
+
return dev_err_probe(dev, -EINVAL, "Missing clock name\n");
567
+
568
+
if (strcmp(priv->clk_bulk[0].id, "core")) {
569
+
priv->clk_bulk[1].id = priv->clk_bulk[0].id;
570
+
priv->clk_bulk[1].clk = priv->clk_bulk[0].clk;
571
+
priv->clk_bulk[0].id = id;
572
+
priv->clk_bulk[0].clk = clk;
573
+
}
574
+
}
575
+
570
576
pm_runtime_set_autosuspend_delay(dev, 100);
571
577
pm_runtime_use_autosuspend(dev);
572
578
pm_runtime_enable(dev);
···
603
565
.of_match_table = stm32_rng_match,
604
566
},
605
567
.probe = stm32_rng_probe,
606
-
.remove_new = stm32_rng_remove,
568
+
.remove = stm32_rng_remove,
607
569
};
608
570
609
571
module_platform_driver(stm32_rng_driver);
+1
-1
drivers/char/hw_random/timeriomem-rng.c
+1
-1
drivers/char/hw_random/timeriomem-rng.c
+1
-1
drivers/char/hw_random/xgene-rng.c
+1
-1
drivers/char/hw_random/xgene-rng.c
+1
-1
drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c
+1
-1
drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c
+1
-1
drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c
+1
-1
drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c
+1
-1
drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c
+1
-1
drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c
+20
-38
drivers/crypto/amcc/crypto4xx_core.c
+20
-38
drivers/crypto/amcc/crypto4xx_core.c
···
653
653
crypto4xx_destroy_pdr(core_dev->dev);
654
654
crypto4xx_destroy_gdr(core_dev->dev);
655
655
crypto4xx_destroy_sdr(core_dev->dev);
656
-
iounmap(core_dev->dev->ce_base);
657
-
kfree(core_dev->dev);
658
-
kfree(core_dev);
659
656
}
660
657
661
658
static u32 get_next_gd(u32 current)
···
1330
1333
static int crypto4xx_probe(struct platform_device *ofdev)
1331
1334
{
1332
1335
int rc;
1333
-
struct resource res;
1334
1336
struct device *dev = &ofdev->dev;
1335
1337
struct crypto4xx_core_device *core_dev;
1336
1338
struct device_node *np;
1337
1339
u32 pvr;
1338
1340
bool is_revb = true;
1339
-
1340
-
rc = of_address_to_resource(ofdev->dev.of_node, 0, &res);
1341
-
if (rc)
1342
-
return -ENODEV;
1343
1341
1344
1342
np = of_find_compatible_node(NULL, NULL, "amcc,ppc460ex-crypto");
1345
1343
if (np) {
···
1366
1374
1367
1375
of_node_put(np);
1368
1376
1369
-
core_dev = kzalloc(sizeof(struct crypto4xx_core_device), GFP_KERNEL);
1377
+
core_dev = devm_kzalloc(
1378
+
&ofdev->dev, sizeof(struct crypto4xx_core_device), GFP_KERNEL);
1370
1379
if (!core_dev)
1371
1380
return -ENOMEM;
1372
1381
1373
1382
dev_set_drvdata(dev, core_dev);
1374
1383
core_dev->ofdev = ofdev;
1375
-
core_dev->dev = kzalloc(sizeof(struct crypto4xx_device), GFP_KERNEL);
1376
-
rc = -ENOMEM;
1384
+
core_dev->dev = devm_kzalloc(
1385
+
&ofdev->dev, sizeof(struct crypto4xx_device), GFP_KERNEL);
1377
1386
if (!core_dev->dev)
1378
-
goto err_alloc_dev;
1387
+
return -ENOMEM;
1379
1388
1380
1389
/*
1381
1390
* Older version of 460EX/GT have a hardware bug.
···
1395
1402
core_dev->dev->core_dev = core_dev;
1396
1403
core_dev->dev->is_revb = is_revb;
1397
1404
core_dev->device = dev;
1398
-
mutex_init(&core_dev->rng_lock);
1405
+
rc = devm_mutex_init(&ofdev->dev, &core_dev->rng_lock);
1406
+
if (rc)
1407
+
return rc;
1399
1408
spin_lock_init(&core_dev->lock);
1400
1409
INIT_LIST_HEAD(&core_dev->dev->alg_list);
1401
1410
ratelimit_default_init(&core_dev->dev->aead_ratelimit);
···
1416
1421
tasklet_init(&core_dev->tasklet, crypto4xx_bh_tasklet_cb,
1417
1422
(unsigned long) dev);
1418
1423
1419
-
core_dev->dev->ce_base = of_iomap(ofdev->dev.of_node, 0);
1420
-
if (!core_dev->dev->ce_base) {
1421
-
dev_err(dev, "failed to of_iomap\n");
1422
-
rc = -ENOMEM;
1423
-
goto err_iomap;
1424
+
core_dev->dev->ce_base = devm_platform_ioremap_resource(ofdev, 0);
1425
+
if (IS_ERR(core_dev->dev->ce_base)) {
1426
+
dev_err(&ofdev->dev, "failed to ioremap resource");
1427
+
rc = PTR_ERR(core_dev->dev->ce_base);
1428
+
goto err_build_sdr;
1424
1429
}
1425
1430
1426
1431
/* Register for Crypto isr, Crypto Engine IRQ */
1427
1432
core_dev->irq = irq_of_parse_and_map(ofdev->dev.of_node, 0);
1428
-
rc = request_irq(core_dev->irq, is_revb ?
1429
-
crypto4xx_ce_interrupt_handler_revb :
1430
-
crypto4xx_ce_interrupt_handler, 0,
1431
-
KBUILD_MODNAME, dev);
1433
+
rc = devm_request_irq(&ofdev->dev, core_dev->irq,
1434
+
is_revb ? crypto4xx_ce_interrupt_handler_revb :
1435
+
crypto4xx_ce_interrupt_handler,
1436
+
0, KBUILD_MODNAME, dev);
1432
1437
if (rc)
1433
-
goto err_request_irq;
1438
+
goto err_iomap;
1434
1439
1435
1440
/* need to setup pdr, rdr, gdr and sdr before this */
1436
1441
crypto4xx_hw_init(core_dev->dev);
···
1439
1444
rc = crypto4xx_register_alg(core_dev->dev, crypto4xx_alg,
1440
1445
ARRAY_SIZE(crypto4xx_alg));
1441
1446
if (rc)
1442
-
goto err_start_dev;
1447
+
goto err_iomap;
1443
1448
1444
1449
ppc4xx_trng_probe(core_dev);
1445
1450
return 0;
1446
1451
1447
-
err_start_dev:
1448
-
free_irq(core_dev->irq, dev);
1449
-
err_request_irq:
1450
-
irq_dispose_mapping(core_dev->irq);
1451
-
iounmap(core_dev->dev->ce_base);
1452
1452
err_iomap:
1453
1453
tasklet_kill(&core_dev->tasklet);
1454
1454
err_build_sdr:
1455
1455
crypto4xx_destroy_sdr(core_dev->dev);
1456
1456
crypto4xx_destroy_gdr(core_dev->dev);
1457
1457
crypto4xx_destroy_pdr(core_dev->dev);
1458
-
kfree(core_dev->dev);
1459
-
err_alloc_dev:
1460
-
kfree(core_dev);
1461
-
1462
1458
return rc;
1463
1459
}
1464
1460
···
1460
1474
1461
1475
ppc4xx_trng_remove(core_dev);
1462
1476
1463
-
free_irq(core_dev->irq, dev);
1464
-
irq_dispose_mapping(core_dev->irq);
1465
-
1466
1477
tasklet_kill(&core_dev->tasklet);
1467
1478
/* Un-register with Linux CryptoAPI */
1468
1479
crypto4xx_unregister_alg(core_dev->dev);
1469
-
mutex_destroy(&core_dev->rng_lock);
1470
1480
/* Free all allocated memory */
1471
1481
crypto4xx_stop_all(core_dev);
1472
1482
}
···
1479
1497
.of_match_table = crypto4xx_match,
1480
1498
},
1481
1499
.probe = crypto4xx_probe,
1482
-
.remove_new = crypto4xx_remove,
1500
+
.remove = crypto4xx_remove,
1483
1501
};
1484
1502
1485
1503
module_platform_driver(crypto4xx_driver);
+4
-6
drivers/crypto/amlogic/amlogic-gxl-core.c
+4
-6
drivers/crypto/amlogic/amlogic-gxl-core.c
···
240
240
platform_set_drvdata(pdev, mc);
241
241
242
242
mc->base = devm_platform_ioremap_resource(pdev, 0);
243
-
if (IS_ERR(mc->base)) {
244
-
err = PTR_ERR(mc->base);
245
-
dev_err(&pdev->dev, "Cannot request MMIO err=%d\n", err);
246
-
return err;
247
-
}
243
+
if (IS_ERR(mc->base))
244
+
return PTR_ERR(mc->base);
245
+
248
246
mc->busclk = devm_clk_get(&pdev->dev, "blkmv");
249
247
if (IS_ERR(mc->busclk)) {
250
248
err = PTR_ERR(mc->busclk);
···
320
322
321
323
static struct platform_driver meson_crypto_driver = {
322
324
.probe = meson_crypto_probe,
323
-
.remove_new = meson_crypto_remove,
325
+
.remove = meson_crypto_remove,
324
326
.driver = {
325
327
.name = "gxl-crypto",
326
328
.of_match_table = meson_crypto_of_match_table,
+1
-3
drivers/crypto/aspeed/aspeed-acry.c
+1
-3
drivers/crypto/aspeed/aspeed-acry.c
···
601
601
.akcipher.base = {
602
602
.encrypt = aspeed_acry_rsa_enc,
603
603
.decrypt = aspeed_acry_rsa_dec,
604
-
.sign = aspeed_acry_rsa_dec,
605
-
.verify = aspeed_acry_rsa_enc,
606
604
.set_pub_key = aspeed_acry_rsa_set_pub_key,
607
605
.set_priv_key = aspeed_acry_rsa_set_priv_key,
608
606
.max_size = aspeed_acry_rsa_max_size,
···
806
808
807
809
static struct platform_driver aspeed_acry_driver = {
808
810
.probe = aspeed_acry_probe,
809
-
.remove_new = aspeed_acry_remove,
811
+
.remove = aspeed_acry_remove,
810
812
.driver = {
811
813
.name = KBUILD_MODNAME,
812
814
.of_match_table = aspeed_acry_of_matches,
+1
-1
drivers/crypto/aspeed/aspeed-hace.c
+1
-1
drivers/crypto/aspeed/aspeed-hace.c
+1
-1
drivers/crypto/atmel-aes.c
+1
-1
drivers/crypto/atmel-aes.c
+1
-1
drivers/crypto/atmel-ecc.c
+1
-1
drivers/crypto/atmel-ecc.c
+1
-1
drivers/crypto/atmel-sha.c
+1
-1
drivers/crypto/atmel-sha.c
+2
-2
drivers/crypto/atmel-sha204a.c
+2
-2
drivers/crypto/atmel-sha204a.c
···
202
202
MODULE_DEVICE_TABLE(of, atmel_sha204a_dt_ids);
203
203
204
204
static const struct i2c_device_id atmel_sha204a_id[] = {
205
-
{ "atsha204", 0 },
206
-
{ "atsha204a", 0 },
205
+
{ "atsha204" },
206
+
{ "atsha204a" },
207
207
{ /* sentinel */ }
208
208
};
209
209
MODULE_DEVICE_TABLE(i2c, atmel_sha204a_id);
+2
-2
drivers/crypto/atmel-tdes.c
+2
-2
drivers/crypto/atmel-tdes.c
···
872
872
if (!err)
873
873
err = atmel_tdes_crypt_start(dd);
874
874
if (!err)
875
-
return; /* DMA started. Not fininishing. */
875
+
return; /* DMA started. Not finishing. */
876
876
}
877
877
878
878
atmel_tdes_finish_req(dd, err);
···
1074
1074
1075
1075
static struct platform_driver atmel_tdes_driver = {
1076
1076
.probe = atmel_tdes_probe,
1077
-
.remove_new = atmel_tdes_remove,
1077
+
.remove = atmel_tdes_remove,
1078
1078
.driver = {
1079
1079
.name = "atmel_tdes",
1080
1080
.of_match_table = atmel_tdes_dt_ids,
+1
-1
drivers/crypto/axis/artpec6_crypto.c
+1
-1
drivers/crypto/axis/artpec6_crypto.c
···
2975
2975
2976
2976
static struct platform_driver artpec6_crypto_driver = {
2977
2977
.probe = artpec6_crypto_probe,
2978
-
.remove_new = artpec6_crypto_remove,
2978
+
.remove = artpec6_crypto_remove,
2979
2979
.driver = {
2980
2980
.name = "artpec6-crypto",
2981
2981
.of_match_table = artpec6_crypto_of_match,
+5
-2
drivers/crypto/bcm/cipher.c
+5
-2
drivers/crypto/bcm/cipher.c
···
2415
2415
2416
2416
static int ahash_hmac_init(struct ahash_request *req)
2417
2417
{
2418
+
int ret;
2418
2419
struct iproc_reqctx_s *rctx = ahash_request_ctx(req);
2419
2420
struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
2420
2421
struct iproc_ctx_s *ctx = crypto_ahash_ctx(tfm);
···
2425
2424
flow_log("ahash_hmac_init()\n");
2426
2425
2427
2426
/* init the context as a hash */
2428
-
ahash_init(req);
2427
+
ret = ahash_init(req);
2428
+
if (ret)
2429
+
return ret;
2429
2430
2430
2431
if (!spu_no_incr_hash(ctx)) {
2431
2432
/* SPU-M can do incr hashing but needs sw for outer HMAC */
···
4707
4704
.of_match_table = of_match_ptr(bcm_spu_dt_ids),
4708
4705
},
4709
4706
.probe = bcm_spu_probe,
4710
-
.remove_new = bcm_spu_remove,
4707
+
.remove = bcm_spu_remove,
4711
4708
};
4712
4709
module_platform_driver(bcm_spu_pdriver);
4713
4710
+7
-4
drivers/crypto/caam/caampkc.c
+7
-4
drivers/crypto/caam/caampkc.c
···
984
984
return -ENOMEM;
985
985
}
986
986
987
-
static void caam_rsa_set_priv_key_form(struct caam_rsa_ctx *ctx,
987
+
static int caam_rsa_set_priv_key_form(struct caam_rsa_ctx *ctx,
988
988
struct rsa_key *raw_key)
989
989
{
990
990
struct caam_rsa_key *rsa_key = &ctx->key;
···
994
994
995
995
rsa_key->p = caam_read_raw_data(raw_key->p, &p_sz);
996
996
if (!rsa_key->p)
997
-
return;
997
+
return -ENOMEM;
998
998
rsa_key->p_sz = p_sz;
999
999
1000
1000
rsa_key->q = caam_read_raw_data(raw_key->q, &q_sz);
···
1029
1029
1030
1030
rsa_key->priv_form = FORM3;
1031
1031
1032
-
return;
1032
+
return 0;
1033
1033
1034
1034
free_dq:
1035
1035
kfree_sensitive(rsa_key->dq);
···
1043
1043
kfree_sensitive(rsa_key->q);
1044
1044
free_p:
1045
1045
kfree_sensitive(rsa_key->p);
1046
+
return -ENOMEM;
1046
1047
}
1047
1048
1048
1049
static int caam_rsa_set_priv_key(struct crypto_akcipher *tfm, const void *key,
···
1089
1088
rsa_key->e_sz = raw_key.e_sz;
1090
1089
rsa_key->n_sz = raw_key.n_sz;
1091
1090
1092
-
caam_rsa_set_priv_key_form(ctx, &raw_key);
1091
+
ret = caam_rsa_set_priv_key_form(ctx, &raw_key);
1092
+
if (ret)
1093
+
goto err;
1093
1094
1094
1095
return 0;
1095
1096
+1
-1
drivers/crypto/caam/jr.c
+1
-1
drivers/crypto/caam/jr.c
+3
-4
drivers/crypto/caam/qi.c
+3
-4
drivers/crypto/caam/qi.c
···
733
733
int caam_qi_init(struct platform_device *caam_pdev)
734
734
{
735
735
int err, i;
736
-
struct device *ctrldev = &caam_pdev->dev, *qidev;
736
+
struct device *qidev = &caam_pdev->dev;
737
737
struct caam_drv_private *ctrlpriv;
738
738
const cpumask_t *cpus = qman_affine_cpus();
739
739
cpumask_var_t clean_mask;
···
742
742
if (!zalloc_cpumask_var(&clean_mask, GFP_KERNEL))
743
743
goto fail_cpumask;
744
744
745
-
ctrlpriv = dev_get_drvdata(ctrldev);
746
-
qidev = ctrldev;
745
+
ctrlpriv = dev_get_drvdata(qidev);
747
746
748
747
/* Initialize the congestion detection */
749
748
err = init_cgr(qidev);
···
793
794
794
795
caam_debugfs_qi_init(ctrlpriv);
795
796
796
-
err = devm_add_action_or_reset(qidev, caam_qi_shutdown, ctrlpriv);
797
+
err = devm_add_action_or_reset(qidev, caam_qi_shutdown, qidev);
797
798
if (err)
798
799
goto fail2;
799
800
+4
-2
drivers/crypto/cavium/cpt/cptpf_main.c
+4
-2
drivers/crypto/cavium/cpt/cptpf_main.c
···
44
44
dev_err(dev, "Cores still busy %llx", coremask);
45
45
grp = cpt_read_csr64(cpt->reg_base,
46
46
CPTX_PF_EXEC_BUSY(0));
47
-
if (timeout--)
47
+
if (!timeout--)
48
48
break;
49
49
50
50
udelay(CSR_DELAY);
···
302
302
303
303
ret = do_cpt_init(cpt, mcode);
304
304
if (ret) {
305
+
dma_free_coherent(&cpt->pdev->dev, mcode->code_size,
306
+
mcode->code, mcode->phys_base);
305
307
dev_err(dev, "do_cpt_init failed with ret: %d\n", ret);
306
308
goto fw_release;
307
309
}
···
396
394
dev_err(dev, "Cores still busy");
397
395
grp = cpt_read_csr64(cpt->reg_base,
398
396
CPTX_PF_EXEC_BUSY(0));
399
-
if (timeout--)
397
+
if (!timeout--)
400
398
break;
401
399
402
400
udelay(CSR_DELAY);
+2
-2
drivers/crypto/cavium/cpt/cptvf_reqmanager.c
+2
-2
drivers/crypto/cavium/cpt/cptvf_reqmanager.c
···
238
238
239
239
qinfo = &cptvf->cqinfo;
240
240
queue = &qinfo->queue[qno];
241
-
/* lock commad queue */
241
+
/* lock command queue */
242
242
spin_lock(&queue->lock);
243
243
ent = &queue->qhead->head[queue->idx * qinfo->cmd_size];
244
244
memcpy(ent, (void *)cmd, qinfo->cmd_size);
···
510
510
info->time_in = jiffies;
511
511
info->req = req;
512
512
513
-
/* Create the CPT_INST_S type command for HW intrepretation */
513
+
/* Create the CPT_INST_S type command for HW interpretation */
514
514
cptinst.s.doneint = true;
515
515
cptinst.s.res_addr = (u64)info->comp_baddr;
516
516
cptinst.s.tag = 0;
+1
-1
drivers/crypto/cavium/nitrox/nitrox_lib.c
+1
-1
drivers/crypto/cavium/nitrox/nitrox_lib.c
+1
-1
drivers/crypto/ccp/sp-platform.c
+1
-1
drivers/crypto/ccp/sp-platform.c
+2
-2
drivers/crypto/ccree/cc_aead.c
+2
-2
drivers/crypto/ccree/cc_aead.c
···
2226
2226
2227
2227
memset(areq_ctx, 0, sizeof(*areq_ctx));
2228
2228
2229
-
//plaintext is not encryped with rfc4543
2229
+
//plaintext is not encrypted with rfc4543
2230
2230
areq_ctx->plaintext_authenticate_only = true;
2231
2231
2232
2232
/* No generated IV required */
···
2277
2277
2278
2278
memset(areq_ctx, 0, sizeof(*areq_ctx));
2279
2279
2280
-
//plaintext is not decryped with rfc4543
2280
+
//plaintext is not decrypted with rfc4543
2281
2281
areq_ctx->plaintext_authenticate_only = true;
2282
2282
2283
2283
/* No generated IV required */
+1
-1
drivers/crypto/ccree/cc_cipher.c
+1
-1
drivers/crypto/ccree/cc_cipher.c
···
179
179
}
180
180
max_key_buf_size <<= 1;
181
181
182
-
/* Alloc fallabck tfm or essiv when key size != 256 bit */
182
+
/* Alloc fallback tfm or essiv when key size != 256 bit */
183
183
ctx_p->fallback_tfm =
184
184
crypto_alloc_skcipher(name, 0, CRYPTO_ALG_NEED_FALLBACK | CRYPTO_ALG_ASYNC);
185
185
+1
-1
drivers/crypto/ccree/cc_driver.c
+1
-1
drivers/crypto/ccree/cc_driver.c
+1
-1
drivers/crypto/ccree/cc_hash.c
+1
-1
drivers/crypto/ccree/cc_hash.c
+1
-1
drivers/crypto/chelsio/chcr_algo.c
+1
-1
drivers/crypto/chelsio/chcr_algo.c
+1
-1
drivers/crypto/exynos-rng.c
+1
-1
drivers/crypto/exynos-rng.c
+1
-1
drivers/crypto/gemini/sl3516-ce-core.c
+1
-1
drivers/crypto/gemini/sl3516-ce-core.c
+23
drivers/crypto/hisilicon/hpre/hpre.h
+23
drivers/crypto/hisilicon/hpre/hpre.h
···
100
100
__le32 rsvd1[_HPRE_SQE_ALIGN_EXT];
101
101
};
102
102
103
+
enum hpre_cap_table_type {
104
+
QM_RAS_NFE_TYPE = 0x0,
105
+
QM_RAS_NFE_RESET,
106
+
QM_RAS_CE_TYPE,
107
+
HPRE_RAS_NFE_TYPE,
108
+
HPRE_RAS_NFE_RESET,
109
+
HPRE_RAS_CE_TYPE,
110
+
HPRE_CORE_INFO,
111
+
HPRE_CORE_EN,
112
+
HPRE_DRV_ALG_BITMAP,
113
+
HPRE_ALG_BITMAP,
114
+
HPRE_CORE1_BITMAP_CAP,
115
+
HPRE_CORE2_BITMAP_CAP,
116
+
HPRE_CORE3_BITMAP_CAP,
117
+
HPRE_CORE4_BITMAP_CAP,
118
+
HPRE_CORE5_BITMAP_CAP,
119
+
HPRE_CORE6_BITMAP_CAP,
120
+
HPRE_CORE7_BITMAP_CAP,
121
+
HPRE_CORE8_BITMAP_CAP,
122
+
HPRE_CORE9_BITMAP_CAP,
123
+
HPRE_CORE10_BITMAP_CAP,
124
+
};
125
+
103
126
struct hisi_qp *hpre_create_qp(u8 type);
104
127
int hpre_algs_register(struct hisi_qm *qm);
105
128
void hpre_algs_unregister(struct hisi_qm *qm);
-2
drivers/crypto/hisilicon/hpre/hpre_crypto.c
-2
drivers/crypto/hisilicon/hpre/hpre_crypto.c
+144
-46
drivers/crypto/hisilicon/hpre/hpre_main.c
+144
-46
drivers/crypto/hisilicon/hpre/hpre_main.c
···
13
13
#include <linux/uacce.h>
14
14
#include "hpre.h"
15
15
16
+
#define CAP_FILE_PERMISSION 0444
16
17
#define HPRE_CTRL_CNT_CLR_CE_BIT BIT(0)
17
18
#define HPRE_CTRL_CNT_CLR_CE 0x301000
18
19
#define HPRE_FSM_MAX_CNT 0x301008
···
204
203
{HPRE_RESET_MASK_CAP, 0x3134, 0, GENMASK(31, 0), 0x0, 0x3FFFFE, 0xBFFC3E},
205
204
{HPRE_OOO_SHUTDOWN_MASK_CAP, 0x3134, 0, GENMASK(31, 0), 0x0, 0x22, 0xBFFC3E},
206
205
{HPRE_CE_MASK_CAP, 0x3138, 0, GENMASK(31, 0), 0x0, 0x1, 0x1},
207
-
{HPRE_CLUSTER_NUM_CAP, 0x313c, 20, GENMASK(3, 0), 0x0, 0x4, 0x1},
206
+
{HPRE_CLUSTER_NUM_CAP, 0x313c, 20, GENMASK(3, 0), 0x0, 0x4, 0x1},
208
207
{HPRE_CORE_TYPE_NUM_CAP, 0x313c, 16, GENMASK(3, 0), 0x0, 0x2, 0x2},
209
208
{HPRE_CORE_NUM_CAP, 0x313c, 8, GENMASK(7, 0), 0x0, 0x8, 0xA},
210
209
{HPRE_CLUSTER_CORE_NUM_CAP, 0x313c, 0, GENMASK(7, 0), 0x0, 0x2, 0xA},
···
223
222
{HPRE_CORE10_ALG_BITMAP_CAP, 0x3170, 0, GENMASK(31, 0), 0x0, 0x10, 0x10}
224
223
};
225
224
226
-
enum hpre_pre_store_cap_idx {
227
-
HPRE_CLUSTER_NUM_CAP_IDX = 0x0,
228
-
HPRE_CORE_ENABLE_BITMAP_CAP_IDX,
229
-
HPRE_DRV_ALG_BITMAP_CAP_IDX,
230
-
HPRE_DEV_ALG_BITMAP_CAP_IDX,
231
-
};
232
-
233
-
static const u32 hpre_pre_store_caps[] = {
234
-
HPRE_CLUSTER_NUM_CAP,
235
-
HPRE_CORE_ENABLE_BITMAP_CAP,
236
-
HPRE_DRV_ALG_BITMAP_CAP,
237
-
HPRE_DEV_ALG_BITMAP_CAP,
225
+
static const struct hisi_qm_cap_query_info hpre_cap_query_info[] = {
226
+
{QM_RAS_NFE_TYPE, "QM_RAS_NFE_TYPE ", 0x3124, 0x0, 0x1C37, 0x7C37},
227
+
{QM_RAS_NFE_RESET, "QM_RAS_NFE_RESET ", 0x3128, 0x0, 0xC77, 0x6C77},
228
+
{QM_RAS_CE_TYPE, "QM_RAS_CE_TYPE ", 0x312C, 0x0, 0x8, 0x8},
229
+
{HPRE_RAS_NFE_TYPE, "HPRE_RAS_NFE_TYPE ", 0x3130, 0x0, 0x3FFFFE, 0x1FFFC3E},
230
+
{HPRE_RAS_NFE_RESET, "HPRE_RAS_NFE_RESET ", 0x3134, 0x0, 0x3FFFFE, 0xBFFC3E},
231
+
{HPRE_RAS_CE_TYPE, "HPRE_RAS_CE_TYPE ", 0x3138, 0x0, 0x1, 0x1},
232
+
{HPRE_CORE_INFO, "HPRE_CORE_INFO ", 0x313c, 0x0, 0x420802, 0x120A0A},
233
+
{HPRE_CORE_EN, "HPRE_CORE_EN ", 0x3140, 0x0, 0xF, 0x3FF},
234
+
{HPRE_DRV_ALG_BITMAP, "HPRE_DRV_ALG_BITMAP ", 0x3144, 0x0, 0x03, 0x27},
235
+
{HPRE_ALG_BITMAP, "HPRE_ALG_BITMAP ", 0x3148, 0x0, 0x03, 0x7F},
236
+
{HPRE_CORE1_BITMAP_CAP, "HPRE_CORE1_BITMAP_CAP ", 0x314c, 0x0, 0x7F, 0x7F},
237
+
{HPRE_CORE2_BITMAP_CAP, "HPRE_CORE2_BITMAP_CAP ", 0x3150, 0x0, 0x7F, 0x7F},
238
+
{HPRE_CORE3_BITMAP_CAP, "HPRE_CORE3_BITMAP_CAP ", 0x3154, 0x0, 0x7F, 0x7F},
239
+
{HPRE_CORE4_BITMAP_CAP, "HPRE_CORE4_BITMAP_CAP ", 0x3158, 0x0, 0x7F, 0x7F},
240
+
{HPRE_CORE5_BITMAP_CAP, "HPRE_CORE5_BITMAP_CAP ", 0x315c, 0x0, 0x7F, 0x7F},
241
+
{HPRE_CORE6_BITMAP_CAP, "HPRE_CORE6_BITMAP_CAP ", 0x3160, 0x0, 0x7F, 0x7F},
242
+
{HPRE_CORE7_BITMAP_CAP, "HPRE_CORE7_BITMAP_CAP ", 0x3164, 0x0, 0x7F, 0x7F},
243
+
{HPRE_CORE8_BITMAP_CAP, "HPRE_CORE8_BITMAP_CAP ", 0x3168, 0x0, 0x7F, 0x7F},
244
+
{HPRE_CORE9_BITMAP_CAP, "HPRE_CORE9_BITMAP_CAP ", 0x316c, 0x0, 0x10, 0x10},
245
+
{HPRE_CORE10_BITMAP_CAP, "HPRE_CORE10_BITMAP_CAP ", 0x3170, 0x0, 0x10, 0x10},
238
246
};
239
247
240
248
static const struct hpre_hw_error hpre_hw_errors[] = {
···
370
360
{
371
361
u32 cap_val;
372
362
373
-
cap_val = qm->cap_tables.dev_cap_table[HPRE_DRV_ALG_BITMAP_CAP_IDX].cap_val;
363
+
cap_val = qm->cap_tables.dev_cap_table[HPRE_DRV_ALG_BITMAP].cap_val;
374
364
if (alg & cap_val)
375
365
return true;
376
366
···
425
415
{
426
416
pf_q_num_flag = true;
427
417
428
-
return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
418
+
return hisi_qm_q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_HPRE_PF);
429
419
}
430
420
431
421
static const struct kernel_param_ops hpre_pf_q_num_ops = {
···
513
503
static int hpre_set_cluster(struct hisi_qm *qm)
514
504
{
515
505
struct device *dev = &qm->pdev->dev;
516
-
unsigned long offset;
517
506
u32 cluster_core_mask;
507
+
unsigned long offset;
508
+
u32 hpre_core_info;
518
509
u8 clusters_num;
519
510
u32 val = 0;
520
511
int ret, i;
521
512
522
-
cluster_core_mask = qm->cap_tables.dev_cap_table[HPRE_CORE_ENABLE_BITMAP_CAP_IDX].cap_val;
523
-
clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
513
+
cluster_core_mask = qm->cap_tables.dev_cap_table[HPRE_CORE_EN].cap_val;
514
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
515
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
516
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
524
517
for (i = 0; i < clusters_num; i++) {
525
518
offset = i * HPRE_CLSTR_ADDR_INTRVL;
526
519
···
606
593
607
594
static void hpre_enable_clock_gate(struct hisi_qm *qm)
608
595
{
596
+
unsigned long offset;
597
+
u8 clusters_num, i;
598
+
u32 hpre_core_info;
609
599
u32 val;
610
600
611
601
if (qm->ver < QM_HW_V3)
···
622
606
val |= HPRE_PEH_CFG_AUTO_GATE_EN;
623
607
writel(val, qm->io_base + HPRE_PEH_CFG_AUTO_GATE);
624
608
625
-
val = readl(qm->io_base + HPRE_CLUSTER_DYN_CTL);
626
-
val |= HPRE_CLUSTER_DYN_CTL_EN;
627
-
writel(val, qm->io_base + HPRE_CLUSTER_DYN_CTL);
609
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
610
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
611
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
612
+
for (i = 0; i < clusters_num; i++) {
613
+
offset = (unsigned long)i * HPRE_CLSTR_ADDR_INTRVL;
614
+
val = readl(qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
615
+
val |= HPRE_CLUSTER_DYN_CTL_EN;
616
+
writel(val, qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
628
617
629
-
val = readl_relaxed(qm->io_base + HPRE_CORE_SHB_CFG);
630
-
val |= HPRE_CORE_GATE_EN;
631
-
writel(val, qm->io_base + HPRE_CORE_SHB_CFG);
618
+
val = readl(qm->io_base + offset + HPRE_CORE_SHB_CFG);
619
+
val |= HPRE_CORE_GATE_EN;
620
+
writel(val, qm->io_base + offset + HPRE_CORE_SHB_CFG);
621
+
}
632
622
}
633
623
634
624
static void hpre_disable_clock_gate(struct hisi_qm *qm)
635
625
{
626
+
unsigned long offset;
627
+
u8 clusters_num, i;
628
+
u32 hpre_core_info;
636
629
u32 val;
637
630
638
631
if (qm->ver < QM_HW_V3)
···
655
630
val &= ~HPRE_PEH_CFG_AUTO_GATE_EN;
656
631
writel(val, qm->io_base + HPRE_PEH_CFG_AUTO_GATE);
657
632
658
-
val = readl(qm->io_base + HPRE_CLUSTER_DYN_CTL);
659
-
val &= ~HPRE_CLUSTER_DYN_CTL_EN;
660
-
writel(val, qm->io_base + HPRE_CLUSTER_DYN_CTL);
633
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
634
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
635
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
636
+
for (i = 0; i < clusters_num; i++) {
637
+
offset = (unsigned long)i * HPRE_CLSTR_ADDR_INTRVL;
638
+
val = readl(qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
639
+
val &= ~HPRE_CLUSTER_DYN_CTL_EN;
640
+
writel(val, qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
661
641
662
-
val = readl_relaxed(qm->io_base + HPRE_CORE_SHB_CFG);
663
-
val &= ~HPRE_CORE_GATE_EN;
664
-
writel(val, qm->io_base + HPRE_CORE_SHB_CFG);
642
+
val = readl(qm->io_base + offset + HPRE_CORE_SHB_CFG);
643
+
val &= ~HPRE_CORE_GATE_EN;
644
+
writel(val, qm->io_base + offset + HPRE_CORE_SHB_CFG);
645
+
}
665
646
}
666
647
667
648
static int hpre_set_user_domain_and_cache(struct hisi_qm *qm)
···
730
699
static void hpre_cnt_regs_clear(struct hisi_qm *qm)
731
700
{
732
701
unsigned long offset;
702
+
u32 hpre_core_info;
733
703
u8 clusters_num;
734
704
int i;
735
705
736
706
/* clear clusterX/cluster_ctrl */
737
-
clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
707
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
708
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
709
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
738
710
for (i = 0; i < clusters_num; i++) {
739
711
offset = HPRE_CLSTR_BASE + i * HPRE_CLSTR_ADDR_INTRVL;
740
712
writel(0x0, qm->io_base + offset + HPRE_CLUSTER_INQURY);
···
1029
995
char buf[HPRE_DBGFS_VAL_MAX_LEN];
1030
996
struct debugfs_regset32 *regset;
1031
997
struct dentry *tmp_d;
998
+
u32 hpre_core_info;
1032
999
u8 clusters_num;
1033
1000
int i, ret;
1034
1001
1035
-
clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
1002
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
1003
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
1004
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
1036
1005
for (i = 0; i < clusters_num; i++) {
1037
1006
ret = snprintf(buf, HPRE_DBGFS_VAL_MAX_LEN, "cluster%d", i);
1038
1007
if (ret >= HPRE_DBGFS_VAL_MAX_LEN)
···
1078
1041
return hpre_cluster_debugfs_init(qm);
1079
1042
}
1080
1043
1044
+
static int hpre_cap_regs_show(struct seq_file *s, void *unused)
1045
+
{
1046
+
struct hisi_qm *qm = s->private;
1047
+
u32 i, size;
1048
+
1049
+
size = qm->cap_tables.qm_cap_size;
1050
+
for (i = 0; i < size; i++)
1051
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.qm_cap_table[i].name,
1052
+
qm->cap_tables.qm_cap_table[i].cap_val);
1053
+
1054
+
size = qm->cap_tables.dev_cap_size;
1055
+
for (i = 0; i < size; i++)
1056
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.dev_cap_table[i].name,
1057
+
qm->cap_tables.dev_cap_table[i].cap_val);
1058
+
1059
+
return 0;
1060
+
}
1061
+
1062
+
DEFINE_SHOW_ATTRIBUTE(hpre_cap_regs);
1063
+
1081
1064
static void hpre_dfx_debug_init(struct hisi_qm *qm)
1082
1065
{
1083
1066
struct dfx_diff_registers *hpre_regs = qm->debug.acc_diff_regs;
···
1116
1059
if (qm->fun_type == QM_HW_PF && hpre_regs)
1117
1060
debugfs_create_file("diff_regs", 0444, parent,
1118
1061
qm, &hpre_diff_regs_fops);
1062
+
1063
+
debugfs_create_file("cap_regs", CAP_FILE_PERMISSION,
1064
+
qm->debug.debug_root, qm, &hpre_cap_regs_fops);
1119
1065
}
1120
1066
1121
1067
static int hpre_debugfs_init(struct hisi_qm *qm)
···
1166
1106
{
1167
1107
struct hisi_qm_cap_record *hpre_cap;
1168
1108
struct device *dev = &qm->pdev->dev;
1109
+
u32 hpre_core_info;
1110
+
u8 clusters_num;
1169
1111
size_t i, size;
1170
1112
1171
-
size = ARRAY_SIZE(hpre_pre_store_caps);
1113
+
size = ARRAY_SIZE(hpre_cap_query_info);
1172
1114
hpre_cap = devm_kzalloc(dev, sizeof(*hpre_cap) * size, GFP_KERNEL);
1173
1115
if (!hpre_cap)
1174
1116
return -ENOMEM;
1175
1117
1176
1118
for (i = 0; i < size; i++) {
1177
-
hpre_cap[i].type = hpre_pre_store_caps[i];
1178
-
hpre_cap[i].cap_val = hisi_qm_get_hw_info(qm, hpre_basic_info,
1179
-
hpre_pre_store_caps[i], qm->cap_ver);
1119
+
hpre_cap[i].type = hpre_cap_query_info[i].type;
1120
+
hpre_cap[i].name = hpre_cap_query_info[i].name;
1121
+
hpre_cap[i].cap_val = hisi_qm_get_cap_value(qm, hpre_cap_query_info,
1122
+
i, qm->cap_ver);
1180
1123
}
1181
1124
1182
-
if (hpre_cap[HPRE_CLUSTER_NUM_CAP_IDX].cap_val > HPRE_CLUSTERS_NUM_MAX) {
1125
+
hpre_core_info = hpre_cap[HPRE_CORE_INFO].cap_val;
1126
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
1127
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
1128
+
if (clusters_num > HPRE_CLUSTERS_NUM_MAX) {
1183
1129
dev_err(dev, "Device cluster num %u is out of range for driver supports %d!\n",
1184
-
hpre_cap[HPRE_CLUSTER_NUM_CAP_IDX].cap_val, HPRE_CLUSTERS_NUM_MAX);
1130
+
clusters_num, HPRE_CLUSTERS_NUM_MAX);
1185
1131
return -EINVAL;
1186
1132
}
1187
1133
1188
1134
qm->cap_tables.dev_cap_table = hpre_cap;
1135
+
qm->cap_tables.dev_cap_size = size;
1189
1136
1190
1137
return 0;
1191
1138
}
···
1239
1172
return ret;
1240
1173
}
1241
1174
1242
-
alg_msk = qm->cap_tables.dev_cap_table[HPRE_DEV_ALG_BITMAP_CAP_IDX].cap_val;
1175
+
alg_msk = qm->cap_tables.dev_cap_table[HPRE_ALG_BITMAP].cap_val;
1243
1176
ret = hisi_qm_set_algs(qm, alg_msk, hpre_dev_algs, ARRAY_SIZE(hpre_dev_algs));
1244
1177
if (ret) {
1245
1178
pci_err(pdev, "Failed to set hpre algs!\n");
···
1255
1188
int com_dfx_regs_num = ARRAY_SIZE(hpre_com_dfx_regs);
1256
1189
struct qm_debug *debug = &qm->debug;
1257
1190
void __iomem *io_base;
1191
+
u32 hpre_core_info;
1258
1192
u8 clusters_num;
1259
1193
int i, j, idx;
1260
1194
1261
-
clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
1195
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
1196
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
1197
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
1262
1198
debug->last_words = kcalloc(cluster_dfx_regs_num * clusters_num +
1263
1199
com_dfx_regs_num, sizeof(unsigned int), GFP_KERNEL);
1264
1200
if (!debug->last_words)
···
1301
1231
struct qm_debug *debug = &qm->debug;
1302
1232
struct pci_dev *pdev = qm->pdev;
1303
1233
void __iomem *io_base;
1234
+
u32 hpre_core_info;
1304
1235
u8 clusters_num;
1305
1236
int i, j, idx;
1306
1237
u32 val;
···
1317
1246
hpre_com_dfx_regs[i].name, debug->last_words[i], val);
1318
1247
}
1319
1248
1320
-
clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
1249
+
hpre_core_info = qm->cap_tables.dev_cap_table[HPRE_CORE_INFO].cap_val;
1250
+
clusters_num = (hpre_core_info >> hpre_basic_info[HPRE_CLUSTER_NUM_CAP].shift) &
1251
+
hpre_basic_info[HPRE_CLUSTER_NUM_CAP].mask;
1321
1252
for (i = 0; i < clusters_num; i++) {
1322
1253
io_base = qm->io_base + hpre_cluster_offsets[i];
1323
1254
for (j = 0; j < cluster_dfx_regs_num; j++) {
···
1353
1280
1354
1281
static void hpre_clear_hw_err_status(struct hisi_qm *qm, u32 err_sts)
1355
1282
{
1356
-
u32 nfe;
1357
-
1358
1283
writel(err_sts, qm->io_base + HPRE_HAC_SOURCE_INT);
1359
-
nfe = hisi_qm_get_hw_info(qm, hpre_basic_info, HPRE_NFE_MASK_CAP, qm->cap_ver);
1360
-
writel(nfe, qm->io_base + HPRE_RAS_NFE_ENB);
1284
+
}
1285
+
1286
+
static void hpre_disable_error_report(struct hisi_qm *qm, u32 err_type)
1287
+
{
1288
+
u32 nfe_mask;
1289
+
1290
+
nfe_mask = hisi_qm_get_hw_info(qm, hpre_basic_info, HPRE_NFE_MASK_CAP, qm->cap_ver);
1291
+
writel(nfe_mask & (~err_type), qm->io_base + HPRE_RAS_NFE_ENB);
1361
1292
}
1362
1293
1363
1294
static void hpre_open_axi_master_ooo(struct hisi_qm *qm)
···
1373
1296
qm->io_base + HPRE_AM_OOO_SHUTDOWN_ENB);
1374
1297
writel(value | HPRE_AM_OOO_SHUTDOWN_ENABLE,
1375
1298
qm->io_base + HPRE_AM_OOO_SHUTDOWN_ENB);
1299
+
}
1300
+
1301
+
static enum acc_err_result hpre_get_err_result(struct hisi_qm *qm)
1302
+
{
1303
+
u32 err_status;
1304
+
1305
+
err_status = hpre_get_hw_err_status(qm);
1306
+
if (err_status) {
1307
+
if (err_status & qm->err_info.ecc_2bits_mask)
1308
+
qm->err_status.is_dev_ecc_mbit = true;
1309
+
hpre_log_hw_error(qm, err_status);
1310
+
1311
+
if (err_status & qm->err_info.dev_reset_mask) {
1312
+
/* Disable the same error reporting until device is recovered. */
1313
+
hpre_disable_error_report(qm, err_status);
1314
+
return ACC_ERR_NEED_RESET;
1315
+
}
1316
+
hpre_clear_hw_err_status(qm, err_status);
1317
+
}
1318
+
1319
+
return ACC_ERR_RECOVERED;
1376
1320
}
1377
1321
1378
1322
static void hpre_err_info_init(struct hisi_qm *qm)
···
1422
1324
.hw_err_disable = hpre_hw_error_disable,
1423
1325
.get_dev_hw_err_status = hpre_get_hw_err_status,
1424
1326
.clear_dev_hw_err_status = hpre_clear_hw_err_status,
1425
-
.log_dev_hw_err = hpre_log_hw_error,
1426
1327
.open_axi_master_ooo = hpre_open_axi_master_ooo,
1427
1328
.open_sva_prefetch = hpre_open_sva_prefetch,
1428
1329
.close_sva_prefetch = hpre_close_sva_prefetch,
1429
1330
.show_last_dfx_regs = hpre_show_last_dfx_regs,
1430
1331
.err_info_init = hpre_err_info_init,
1332
+
.get_err_result = hpre_get_err_result,
1431
1333
};
1432
1334
1433
1335
static int hpre_pf_probe_init(struct hpre *hpre)
+105
-61
drivers/crypto/hisilicon/qm.c
+105
-61
drivers/crypto/hisilicon/qm.c
···
271
271
SHAPER_VFT,
272
272
};
273
273
274
-
enum acc_err_result {
275
-
ACC_ERR_NONE,
276
-
ACC_ERR_NEED_RESET,
277
-
ACC_ERR_RECOVERED,
278
-
};
279
-
280
274
enum qm_alg_type {
281
275
ALG_TYPE_0,
282
276
ALG_TYPE_1,
···
301
307
QM_VF_IRQ_NUM_CAP,
302
308
};
303
309
304
-
enum qm_pre_store_cap_idx {
305
-
QM_EQ_IRQ_TYPE_CAP_IDX = 0x0,
306
-
QM_AEQ_IRQ_TYPE_CAP_IDX,
307
-
QM_ABN_IRQ_TYPE_CAP_IDX,
308
-
QM_PF2VF_IRQ_TYPE_CAP_IDX,
310
+
enum qm_cap_table_type {
311
+
QM_CAP_VF = 0x0,
312
+
QM_AEQE_NUM,
313
+
QM_SCQE_NUM,
314
+
QM_EQ_IRQ,
315
+
QM_AEQ_IRQ,
316
+
QM_ABNORMAL_IRQ,
317
+
QM_MB_IRQ,
318
+
MAX_IRQ_NUM,
319
+
EXT_BAR_INDEX,
320
+
};
321
+
322
+
static const struct hisi_qm_cap_query_info qm_cap_query_info[] = {
323
+
{QM_CAP_VF, "QM_CAP_VF ", 0x3100, 0x0, 0x0, 0x6F01},
324
+
{QM_AEQE_NUM, "QM_AEQE_NUM ", 0x3104, 0x800, 0x4000800, 0x4000800},
325
+
{QM_SCQE_NUM, "QM_SCQE_NUM ",
326
+
0x3108, 0x4000400, 0x4000400, 0x4000400},
327
+
{QM_EQ_IRQ, "QM_EQ_IRQ ", 0x310c, 0x10000, 0x10000, 0x10000},
328
+
{QM_AEQ_IRQ, "QM_AEQ_IRQ ", 0x3110, 0x0, 0x10001, 0x10001},
329
+
{QM_ABNORMAL_IRQ, "QM_ABNORMAL_IRQ ", 0x3114, 0x0, 0x10003, 0x10003},
330
+
{QM_MB_IRQ, "QM_MB_IRQ ", 0x3118, 0x0, 0x0, 0x10002},
331
+
{MAX_IRQ_NUM, "MAX_IRQ_NUM ", 0x311c, 0x10001, 0x40002, 0x40003},
332
+
{EXT_BAR_INDEX, "EXT_BAR_INDEX ", 0x3120, 0x0, 0x0, 0x14},
309
333
};
310
334
311
335
static const struct hisi_qm_cap_info qm_cap_info_comm[] = {
···
354
342
{QM_PF2VF_IRQ_TYPE_CAP, 0x3118, 0, GENMASK(31, 0), 0x0, 0x0, 0x10002},
355
343
{QM_PF_IRQ_NUM_CAP, 0x311c, 16, GENMASK(15, 0), 0x1, 0x4, 0x4},
356
344
{QM_VF_IRQ_NUM_CAP, 0x311c, 0, GENMASK(15, 0), 0x1, 0x2, 0x3},
357
-
};
358
-
359
-
static const u32 qm_pre_store_caps[] = {
360
-
QM_EQ_IRQ_TYPE_CAP,
361
-
QM_AEQ_IRQ_TYPE_CAP,
362
-
QM_ABN_IRQ_TYPE_CAP,
363
-
QM_PF2VF_IRQ_TYPE_CAP,
364
345
};
365
346
366
347
struct qm_mailbox {
···
456
451
457
452
static void qm_irqs_unregister(struct hisi_qm *qm);
458
453
static int qm_reset_device(struct hisi_qm *qm);
454
+
int hisi_qm_q_num_set(const char *val, const struct kernel_param *kp,
455
+
unsigned int device)
456
+
{
457
+
struct pci_dev *pdev;
458
+
u32 n, q_num;
459
+
int ret;
460
+
461
+
if (!val)
462
+
return -EINVAL;
463
+
464
+
pdev = pci_get_device(PCI_VENDOR_ID_HUAWEI, device, NULL);
465
+
if (!pdev) {
466
+
q_num = min_t(u32, QM_QNUM_V1, QM_QNUM_V2);
467
+
pr_info("No device found currently, suppose queue number is %u\n",
468
+
q_num);
469
+
} else {
470
+
if (pdev->revision == QM_HW_V1)
471
+
q_num = QM_QNUM_V1;
472
+
else
473
+
q_num = QM_QNUM_V2;
474
+
475
+
pci_dev_put(pdev);
476
+
}
477
+
478
+
ret = kstrtou32(val, 10, &n);
479
+
if (ret || n < QM_MIN_QNUM || n > q_num)
480
+
return -EINVAL;
481
+
482
+
return param_set_int(val, kp);
483
+
}
484
+
EXPORT_SYMBOL_GPL(hisi_qm_q_num_set);
459
485
460
486
static u32 qm_get_hw_error_status(struct hisi_qm *qm)
461
487
{
···
798
762
}
799
763
}
800
764
EXPORT_SYMBOL_GPL(hisi_qm_get_hw_info);
765
+
766
+
u32 hisi_qm_get_cap_value(struct hisi_qm *qm,
767
+
const struct hisi_qm_cap_query_info *info_table,
768
+
u32 index, bool is_read)
769
+
{
770
+
u32 val;
771
+
772
+
switch (qm->ver) {
773
+
case QM_HW_V1:
774
+
return info_table[index].v1_val;
775
+
case QM_HW_V2:
776
+
return info_table[index].v2_val;
777
+
default:
778
+
if (!is_read)
779
+
return info_table[index].v3_val;
780
+
781
+
val = readl(qm->io_base + info_table[index].offset);
782
+
return val;
783
+
}
784
+
}
785
+
EXPORT_SYMBOL_GPL(hisi_qm_get_cap_value);
801
786
802
787
static void qm_get_xqc_depth(struct hisi_qm *qm, u16 *low_bits,
803
788
u16 *high_bits, enum qm_basic_type type)
···
1482
1425
1483
1426
static enum acc_err_result qm_hw_error_handle_v2(struct hisi_qm *qm)
1484
1427
{
1485
-
u32 error_status, tmp;
1428
+
u32 error_status;
1486
1429
1487
-
/* read err sts */
1488
-
tmp = readl(qm->io_base + QM_ABNORMAL_INT_STATUS);
1489
-
error_status = qm->error_mask & tmp;
1490
-
1491
-
if (error_status) {
1430
+
error_status = qm_get_hw_error_status(qm);
1431
+
if (error_status & qm->error_mask) {
1492
1432
if (error_status & QM_ECC_MBIT)
1493
1433
qm->err_status.is_qm_ecc_mbit = true;
1494
1434
1495
1435
qm_log_hw_error(qm, error_status);
1496
-
if (error_status & qm->err_info.qm_reset_mask)
1436
+
if (error_status & qm->err_info.qm_reset_mask) {
1437
+
/* Disable the same error reporting until device is recovered. */
1438
+
writel(qm->err_info.nfe & (~error_status),
1439
+
qm->io_base + QM_RAS_NFE_ENABLE);
1497
1440
return ACC_ERR_NEED_RESET;
1441
+
}
1498
1442
1443
+
/* Clear error source if not need reset. */
1499
1444
writel(error_status, qm->io_base + QM_ABNORMAL_INT_SOURCE);
1500
1445
writel(qm->err_info.nfe, qm->io_base + QM_RAS_NFE_ENABLE);
1446
+
writel(qm->err_info.ce, qm->io_base + QM_RAS_CE_ENABLE);
1501
1447
}
1502
1448
1503
1449
return ACC_ERR_RECOVERED;
···
3921
3861
3922
3862
static enum acc_err_result qm_dev_err_handle(struct hisi_qm *qm)
3923
3863
{
3924
-
u32 err_sts;
3925
-
3926
-
if (!qm->err_ini->get_dev_hw_err_status) {
3927
-
dev_err(&qm->pdev->dev, "Device doesn't support get hw error status!\n");
3864
+
if (!qm->err_ini->get_err_result) {
3865
+
dev_err(&qm->pdev->dev, "Device doesn't support reset!\n");
3928
3866
return ACC_ERR_NONE;
3929
3867
}
3930
3868
3931
-
/* get device hardware error status */
3932
-
err_sts = qm->err_ini->get_dev_hw_err_status(qm);
3933
-
if (err_sts) {
3934
-
if (err_sts & qm->err_info.ecc_2bits_mask)
3935
-
qm->err_status.is_dev_ecc_mbit = true;
3936
-
3937
-
if (qm->err_ini->log_dev_hw_err)
3938
-
qm->err_ini->log_dev_hw_err(qm, err_sts);
3939
-
3940
-
if (err_sts & qm->err_info.dev_reset_mask)
3941
-
return ACC_ERR_NEED_RESET;
3942
-
3943
-
if (qm->err_ini->clear_dev_hw_err_status)
3944
-
qm->err_ini->clear_dev_hw_err_status(qm, err_sts);
3945
-
}
3946
-
3947
-
return ACC_ERR_RECOVERED;
3869
+
return qm->err_ini->get_err_result(qm);
3948
3870
}
3949
3871
3950
3872
static enum acc_err_result qm_process_dev_error(struct hisi_qm *qm)
···
4908
4866
if (qm->fun_type == QM_HW_VF)
4909
4867
return;
4910
4868
4911
-
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
4869
+
val = qm->cap_tables.qm_cap_table[QM_ABNORMAL_IRQ].cap_val;
4912
4870
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_ABN_IRQ_TYPE_MASK))
4913
4871
return;
4914
4872
···
4925
4883
if (qm->fun_type == QM_HW_VF)
4926
4884
return 0;
4927
4885
4928
-
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
4886
+
val = qm->cap_tables.qm_cap_table[QM_ABNORMAL_IRQ].cap_val;
4929
4887
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_ABN_IRQ_TYPE_MASK))
4930
4888
return 0;
4931
4889
···
4942
4900
struct pci_dev *pdev = qm->pdev;
4943
4901
u32 irq_vector, val;
4944
4902
4945
-
val = qm->cap_tables.qm_cap_table[QM_PF2VF_IRQ_TYPE_CAP_IDX].cap_val;
4903
+
val = qm->cap_tables.qm_cap_table[QM_MB_IRQ].cap_val;
4946
4904
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
4947
4905
return;
4948
4906
···
4956
4914
u32 irq_vector, val;
4957
4915
int ret;
4958
4916
4959
-
val = qm->cap_tables.qm_cap_table[QM_PF2VF_IRQ_TYPE_CAP_IDX].cap_val;
4917
+
val = qm->cap_tables.qm_cap_table[QM_MB_IRQ].cap_val;
4960
4918
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
4961
4919
return 0;
4962
4920
···
4973
4931
struct pci_dev *pdev = qm->pdev;
4974
4932
u32 irq_vector, val;
4975
4933
4976
-
val = qm->cap_tables.qm_cap_table[QM_AEQ_IRQ_TYPE_CAP_IDX].cap_val;
4934
+
val = qm->cap_tables.qm_cap_table[QM_AEQ_IRQ].cap_val;
4977
4935
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
4978
4936
return;
4979
4937
···
4987
4945
u32 irq_vector, val;
4988
4946
int ret;
4989
4947
4990
-
val = qm->cap_tables.qm_cap_table[QM_AEQ_IRQ_TYPE_CAP_IDX].cap_val;
4948
+
val = qm->cap_tables.qm_cap_table[QM_AEQ_IRQ].cap_val;
4991
4949
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
4992
4950
return 0;
4993
4951
···
5005
4963
struct pci_dev *pdev = qm->pdev;
5006
4964
u32 irq_vector, val;
5007
4965
5008
-
val = qm->cap_tables.qm_cap_table[QM_EQ_IRQ_TYPE_CAP_IDX].cap_val;
4966
+
val = qm->cap_tables.qm_cap_table[QM_EQ_IRQ].cap_val;
5009
4967
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
5010
4968
return;
5011
4969
···
5019
4977
u32 irq_vector, val;
5020
4978
int ret;
5021
4979
5022
-
val = qm->cap_tables.qm_cap_table[QM_EQ_IRQ_TYPE_CAP_IDX].cap_val;
4980
+
val = qm->cap_tables.qm_cap_table[QM_EQ_IRQ].cap_val;
5023
4981
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_IRQ_TYPE_MASK))
5024
4982
return 0;
5025
4983
···
5107
5065
return 0;
5108
5066
}
5109
5067
5110
-
static int qm_pre_store_irq_type_caps(struct hisi_qm *qm)
5068
+
static int qm_pre_store_caps(struct hisi_qm *qm)
5111
5069
{
5112
5070
struct hisi_qm_cap_record *qm_cap;
5113
5071
struct pci_dev *pdev = qm->pdev;
5114
5072
size_t i, size;
5115
5073
5116
-
size = ARRAY_SIZE(qm_pre_store_caps);
5074
+
size = ARRAY_SIZE(qm_cap_query_info);
5117
5075
qm_cap = devm_kzalloc(&pdev->dev, sizeof(*qm_cap) * size, GFP_KERNEL);
5118
5076
if (!qm_cap)
5119
5077
return -ENOMEM;
5120
5078
5121
5079
for (i = 0; i < size; i++) {
5122
-
qm_cap[i].type = qm_pre_store_caps[i];
5123
-
qm_cap[i].cap_val = hisi_qm_get_hw_info(qm, qm_basic_info,
5124
-
qm_pre_store_caps[i], qm->cap_ver);
5080
+
qm_cap[i].type = qm_cap_query_info[i].type;
5081
+
qm_cap[i].name = qm_cap_query_info[i].name;
5082
+
qm_cap[i].cap_val = hisi_qm_get_cap_value(qm, qm_cap_query_info,
5083
+
i, qm->cap_ver);
5125
5084
}
5126
5085
5127
5086
qm->cap_tables.qm_cap_table = qm_cap;
5087
+
qm->cap_tables.qm_cap_size = size;
5128
5088
5129
5089
return 0;
5130
5090
}
···
5163
5119
set_bit(cap_info[i].type, &qm->caps);
5164
5120
}
5165
5121
5166
-
/* Fetch and save the value of irq type related capability registers */
5167
-
return qm_pre_store_irq_type_caps(qm);
5122
+
/* Fetch and save the value of qm capability registers */
5123
+
return qm_pre_store_caps(qm);
5168
5124
}
5169
5125
5170
5126
static int qm_get_pci_res(struct hisi_qm *qm)
+1
-1
drivers/crypto/hisilicon/sec/sec_drv.c
+1
-1
drivers/crypto/hisilicon/sec/sec_drv.c
+21
-5
drivers/crypto/hisilicon/sec2/sec.h
+21
-5
drivers/crypto/hisilicon/sec2/sec.h
···
220
220
SEC_CORE4_ALG_BITMAP_HIGH,
221
221
};
222
222
223
-
enum sec_cap_reg_record_idx {
224
-
SEC_DRV_ALG_BITMAP_LOW_IDX = 0x0,
225
-
SEC_DRV_ALG_BITMAP_HIGH_IDX,
226
-
SEC_DEV_ALG_BITMAP_LOW_IDX,
227
-
SEC_DEV_ALG_BITMAP_HIGH_IDX,
223
+
enum sec_cap_table_type {
224
+
QM_RAS_NFE_TYPE = 0x0,
225
+
QM_RAS_NFE_RESET,
226
+
QM_RAS_CE_TYPE,
227
+
SEC_RAS_NFE_TYPE,
228
+
SEC_RAS_NFE_RESET,
229
+
SEC_RAS_CE_TYPE,
230
+
SEC_CORE_INFO,
231
+
SEC_CORE_EN,
232
+
SEC_DRV_ALG_BITMAP_LOW_TB,
233
+
SEC_DRV_ALG_BITMAP_HIGH_TB,
234
+
SEC_ALG_BITMAP_LOW,
235
+
SEC_ALG_BITMAP_HIGH,
236
+
SEC_CORE1_BITMAP_LOW,
237
+
SEC_CORE1_BITMAP_HIGH,
238
+
SEC_CORE2_BITMAP_LOW,
239
+
SEC_CORE2_BITMAP_HIGH,
240
+
SEC_CORE3_BITMAP_LOW,
241
+
SEC_CORE3_BITMAP_HIGH,
242
+
SEC_CORE4_BITMAP_LOW,
243
+
SEC_CORE4_BITMAP_HIGH,
228
244
};
229
245
230
246
void sec_destroy_qps(struct hisi_qp **qps, int qp_num);
+4
-4
drivers/crypto/hisilicon/sec2/sec_crypto.c
+4
-4
drivers/crypto/hisilicon/sec2/sec_crypto.c
···
2520
2520
u64 alg_mask;
2521
2521
int ret = 0;
2522
2522
2523
-
alg_mask = sec_get_alg_bitmap(qm, SEC_DRV_ALG_BITMAP_HIGH_IDX,
2524
-
SEC_DRV_ALG_BITMAP_LOW_IDX);
2523
+
alg_mask = sec_get_alg_bitmap(qm, SEC_DRV_ALG_BITMAP_HIGH_TB,
2524
+
SEC_DRV_ALG_BITMAP_LOW_TB);
2525
2525
2526
2526
mutex_lock(&sec_algs_lock);
2527
2527
if (sec_available_devs) {
···
2553
2553
{
2554
2554
u64 alg_mask;
2555
2555
2556
-
alg_mask = sec_get_alg_bitmap(qm, SEC_DRV_ALG_BITMAP_HIGH_IDX,
2557
-
SEC_DRV_ALG_BITMAP_LOW_IDX);
2556
+
alg_mask = sec_get_alg_bitmap(qm, SEC_DRV_ALG_BITMAP_HIGH_TB,
2557
+
SEC_DRV_ALG_BITMAP_LOW_TB);
2558
2558
2559
2559
mutex_lock(&sec_algs_lock);
2560
2560
if (--sec_available_devs)
+90
-18
drivers/crypto/hisilicon/sec2/sec_main.c
+90
-18
drivers/crypto/hisilicon/sec2/sec_main.c
···
14
14
#include <linux/seq_file.h>
15
15
#include <linux/topology.h>
16
16
#include <linux/uacce.h>
17
-
18
17
#include "sec.h"
19
18
19
+
#define CAP_FILE_PERMISSION 0444
20
20
#define SEC_VF_NUM 63
21
21
#define SEC_QUEUE_NUM_V1 4096
22
22
#define PCI_DEVICE_ID_HUAWEI_SEC_PF 0xa255
···
167
167
{SEC_CORE4_ALG_BITMAP_HIGH, 0x3170, 0, GENMASK(31, 0), 0x3FFF, 0x3FFF, 0x3FFF},
168
168
};
169
169
170
-
static const u32 sec_pre_store_caps[] = {
171
-
SEC_DRV_ALG_BITMAP_LOW,
172
-
SEC_DRV_ALG_BITMAP_HIGH,
173
-
SEC_DEV_ALG_BITMAP_LOW,
174
-
SEC_DEV_ALG_BITMAP_HIGH,
170
+
static const struct hisi_qm_cap_query_info sec_cap_query_info[] = {
171
+
{QM_RAS_NFE_TYPE, "QM_RAS_NFE_TYPE ", 0x3124, 0x0, 0x1C77, 0x7C77},
172
+
{QM_RAS_NFE_RESET, "QM_RAS_NFE_RESET ", 0x3128, 0x0, 0xC77, 0x6C77},
173
+
{QM_RAS_CE_TYPE, "QM_RAS_CE_TYPE ", 0x312C, 0x0, 0x8, 0x8},
174
+
{SEC_RAS_NFE_TYPE, "SEC_RAS_NFE_TYPE ", 0x3130, 0x0, 0x177, 0x60177},
175
+
{SEC_RAS_NFE_RESET, "SEC_RAS_NFE_RESET ", 0x3134, 0x0, 0x177, 0x177},
176
+
{SEC_RAS_CE_TYPE, "SEC_RAS_CE_TYPE ", 0x3138, 0x0, 0x88, 0xC088},
177
+
{SEC_CORE_INFO, "SEC_CORE_INFO ", 0x313c, 0x110404, 0x110404, 0x110404},
178
+
{SEC_CORE_EN, "SEC_CORE_EN ", 0x3140, 0x17F, 0x17F, 0xF},
179
+
{SEC_DRV_ALG_BITMAP_LOW_TB, "SEC_DRV_ALG_BITMAP_LOW ",
180
+
0x3144, 0x18050CB, 0x18050CB, 0x18670CF},
181
+
{SEC_DRV_ALG_BITMAP_HIGH_TB, "SEC_DRV_ALG_BITMAP_HIGH ",
182
+
0x3148, 0x395C, 0x395C, 0x395C},
183
+
{SEC_ALG_BITMAP_LOW, "SEC_ALG_BITMAP_LOW ",
184
+
0x314c, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF},
185
+
{SEC_ALG_BITMAP_HIGH, "SEC_ALG_BITMAP_HIGH ", 0x3150, 0x3FFF, 0x3FFF, 0x3FFF},
186
+
{SEC_CORE1_BITMAP_LOW, "SEC_CORE1_BITMAP_LOW ",
187
+
0x3154, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF},
188
+
{SEC_CORE1_BITMAP_HIGH, "SEC_CORE1_BITMAP_HIGH ", 0x3158, 0x3FFF, 0x3FFF, 0x3FFF},
189
+
{SEC_CORE2_BITMAP_LOW, "SEC_CORE2_BITMAP_LOW ",
190
+
0x315c, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF},
191
+
{SEC_CORE2_BITMAP_HIGH, "SEC_CORE2_BITMAP_HIGH ", 0x3160, 0x3FFF, 0x3FFF, 0x3FFF},
192
+
{SEC_CORE3_BITMAP_LOW, "SEC_CORE3_BITMAP_LOW ",
193
+
0x3164, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF},
194
+
{SEC_CORE3_BITMAP_HIGH, "SEC_CORE3_BITMAP_HIGH ", 0x3168, 0x3FFF, 0x3FFF, 0x3FFF},
195
+
{SEC_CORE4_BITMAP_LOW, "SEC_CORE4_BITMAP_LOW ",
196
+
0x316c, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF},
197
+
{SEC_CORE4_BITMAP_HIGH, "SEC_CORE4_BITMAP_HIGH ", 0x3170, 0x3FFF, 0x3FFF, 0x3FFF},
175
198
};
176
199
177
200
static const struct qm_dev_alg sec_dev_algs[] = { {
···
345
322
{
346
323
pf_q_num_flag = true;
347
324
348
-
return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
325
+
return hisi_qm_q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_SEC_PF);
349
326
}
350
327
351
328
static const struct kernel_param_ops sec_pf_q_num_ops = {
···
861
838
862
839
DEFINE_SHOW_ATTRIBUTE(sec_regs);
863
840
841
+
static int sec_cap_regs_show(struct seq_file *s, void *unused)
842
+
{
843
+
struct hisi_qm *qm = s->private;
844
+
u32 i, size;
845
+
846
+
size = qm->cap_tables.qm_cap_size;
847
+
for (i = 0; i < size; i++)
848
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.qm_cap_table[i].name,
849
+
qm->cap_tables.qm_cap_table[i].cap_val);
850
+
851
+
size = qm->cap_tables.dev_cap_size;
852
+
for (i = 0; i < size; i++)
853
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.dev_cap_table[i].name,
854
+
qm->cap_tables.dev_cap_table[i].cap_val);
855
+
856
+
return 0;
857
+
}
858
+
859
+
DEFINE_SHOW_ATTRIBUTE(sec_cap_regs);
860
+
864
861
static int sec_core_debug_init(struct hisi_qm *qm)
865
862
{
866
863
struct dfx_diff_registers *sec_regs = qm->debug.acc_diff_regs;
···
914
871
debugfs_create_file(sec_dfx_labels[i].name, 0644,
915
872
tmp_d, data, &sec_atomic64_ops);
916
873
}
874
+
875
+
debugfs_create_file("cap_regs", CAP_FILE_PERMISSION,
876
+
qm->debug.debug_root, qm, &sec_cap_regs_fops);
917
877
918
878
return 0;
919
879
}
···
1056
1010
1057
1011
static void sec_clear_hw_err_status(struct hisi_qm *qm, u32 err_sts)
1058
1012
{
1059
-
u32 nfe;
1060
-
1061
1013
writel(err_sts, qm->io_base + SEC_CORE_INT_SOURCE);
1062
-
nfe = hisi_qm_get_hw_info(qm, sec_basic_info, SEC_NFE_MASK_CAP, qm->cap_ver);
1063
-
writel(nfe, qm->io_base + SEC_RAS_NFE_REG);
1014
+
}
1015
+
1016
+
static void sec_disable_error_report(struct hisi_qm *qm, u32 err_type)
1017
+
{
1018
+
u32 nfe_mask;
1019
+
1020
+
nfe_mask = hisi_qm_get_hw_info(qm, sec_basic_info, SEC_NFE_MASK_CAP, qm->cap_ver);
1021
+
writel(nfe_mask & (~err_type), qm->io_base + SEC_RAS_NFE_REG);
1064
1022
}
1065
1023
1066
1024
static void sec_open_axi_master_ooo(struct hisi_qm *qm)
···
1074
1024
val = readl(qm->io_base + SEC_CONTROL_REG);
1075
1025
writel(val & SEC_AXI_SHUTDOWN_DISABLE, qm->io_base + SEC_CONTROL_REG);
1076
1026
writel(val | SEC_AXI_SHUTDOWN_ENABLE, qm->io_base + SEC_CONTROL_REG);
1027
+
}
1028
+
1029
+
static enum acc_err_result sec_get_err_result(struct hisi_qm *qm)
1030
+
{
1031
+
u32 err_status;
1032
+
1033
+
err_status = sec_get_hw_err_status(qm);
1034
+
if (err_status) {
1035
+
if (err_status & qm->err_info.ecc_2bits_mask)
1036
+
qm->err_status.is_dev_ecc_mbit = true;
1037
+
sec_log_hw_error(qm, err_status);
1038
+
1039
+
if (err_status & qm->err_info.dev_reset_mask) {
1040
+
/* Disable the same error reporting until device is recovered. */
1041
+
sec_disable_error_report(qm, err_status);
1042
+
return ACC_ERR_NEED_RESET;
1043
+
}
1044
+
sec_clear_hw_err_status(qm, err_status);
1045
+
}
1046
+
1047
+
return ACC_ERR_RECOVERED;
1077
1048
}
1078
1049
1079
1050
static void sec_err_info_init(struct hisi_qm *qm)
···
1123
1052
.hw_err_disable = sec_hw_error_disable,
1124
1053
.get_dev_hw_err_status = sec_get_hw_err_status,
1125
1054
.clear_dev_hw_err_status = sec_clear_hw_err_status,
1126
-
.log_dev_hw_err = sec_log_hw_error,
1127
1055
.open_axi_master_ooo = sec_open_axi_master_ooo,
1128
1056
.open_sva_prefetch = sec_open_sva_prefetch,
1129
1057
.close_sva_prefetch = sec_close_sva_prefetch,
1130
1058
.show_last_dfx_regs = sec_show_last_dfx_regs,
1131
1059
.err_info_init = sec_err_info_init,
1060
+
.get_err_result = sec_get_err_result,
1132
1061
};
1133
1062
1134
1063
static int sec_pf_probe_init(struct sec_dev *sec)
···
1156
1085
struct pci_dev *pdev = qm->pdev;
1157
1086
size_t i, size;
1158
1087
1159
-
size = ARRAY_SIZE(sec_pre_store_caps);
1088
+
size = ARRAY_SIZE(sec_cap_query_info);
1160
1089
sec_cap = devm_kzalloc(&pdev->dev, sizeof(*sec_cap) * size, GFP_KERNEL);
1161
1090
if (!sec_cap)
1162
1091
return -ENOMEM;
1163
1092
1164
1093
for (i = 0; i < size; i++) {
1165
-
sec_cap[i].type = sec_pre_store_caps[i];
1166
-
sec_cap[i].cap_val = hisi_qm_get_hw_info(qm, sec_basic_info,
1167
-
sec_pre_store_caps[i], qm->cap_ver);
1094
+
sec_cap[i].type = sec_cap_query_info[i].type;
1095
+
sec_cap[i].name = sec_cap_query_info[i].name;
1096
+
sec_cap[i].cap_val = hisi_qm_get_cap_value(qm, sec_cap_query_info,
1097
+
i, qm->cap_ver);
1168
1098
}
1169
1099
1170
1100
qm->cap_tables.dev_cap_table = sec_cap;
1101
+
qm->cap_tables.dev_cap_size = size;
1171
1102
1172
1103
return 0;
1173
1104
}
···
1219
1146
hisi_qm_uninit(qm);
1220
1147
return ret;
1221
1148
}
1222
-
1223
-
alg_msk = sec_get_alg_bitmap(qm, SEC_DEV_ALG_BITMAP_HIGH_IDX, SEC_DEV_ALG_BITMAP_LOW_IDX);
1149
+
alg_msk = sec_get_alg_bitmap(qm, SEC_ALG_BITMAP_HIGH, SEC_ALG_BITMAP_LOW);
1224
1150
ret = hisi_qm_set_algs(qm, alg_msk, sec_dev_algs, ARRAY_SIZE(sec_dev_algs));
1225
1151
if (ret) {
1226
1152
pci_err(qm->pdev, "Failed to set sec algs!\n");
+1
-1
drivers/crypto/hisilicon/trng/trng.c
+1
-1
drivers/crypto/hisilicon/trng/trng.c
+18
drivers/crypto/hisilicon/zip/zip.h
+18
drivers/crypto/hisilicon/zip/zip.h
···
81
81
u32 rsvd1[4];
82
82
};
83
83
84
+
enum zip_cap_table_type {
85
+
QM_RAS_NFE_TYPE,
86
+
QM_RAS_NFE_RESET,
87
+
QM_RAS_CE_TYPE,
88
+
ZIP_RAS_NFE_TYPE,
89
+
ZIP_RAS_NFE_RESET,
90
+
ZIP_RAS_CE_TYPE,
91
+
ZIP_CORE_INFO,
92
+
ZIP_CORE_EN,
93
+
ZIP_DRV_ALG_BITMAP_TB,
94
+
ZIP_ALG_BITMAP,
95
+
ZIP_CORE1_BITMAP,
96
+
ZIP_CORE2_BITMAP,
97
+
ZIP_CORE3_BITMAP,
98
+
ZIP_CORE4_BITMAP,
99
+
ZIP_CORE5_BITMAP,
100
+
};
101
+
84
102
int zip_create_qps(struct hisi_qp **qps, int qp_num, int node);
85
103
int hisi_zip_register_to_crypto(struct hisi_qm *qm);
86
104
void hisi_zip_unregister_from_crypto(struct hisi_qm *qm);
+114
-39
drivers/crypto/hisilicon/zip/zip_main.c
+114
-39
drivers/crypto/hisilicon/zip/zip_main.c
···
14
14
#include <linux/uacce.h>
15
15
#include "zip.h"
16
16
17
+
#define CAP_FILE_PERMISSION 0444
17
18
#define PCI_DEVICE_ID_HUAWEI_ZIP_PF 0xa250
18
19
19
20
#define HZIP_QUEUE_NUM_V1 4096
···
251
250
{ZIP_CAP_MAX, 0x317c, 0, GENMASK(0, 0), 0x0, 0x0, 0x0}
252
251
};
253
252
254
-
enum zip_pre_store_cap_idx {
255
-
ZIP_CORE_NUM_CAP_IDX = 0x0,
256
-
ZIP_CLUSTER_COMP_NUM_CAP_IDX,
257
-
ZIP_CLUSTER_DECOMP_NUM_CAP_IDX,
258
-
ZIP_DECOMP_ENABLE_BITMAP_IDX,
259
-
ZIP_COMP_ENABLE_BITMAP_IDX,
260
-
ZIP_DRV_ALG_BITMAP_IDX,
261
-
ZIP_DEV_ALG_BITMAP_IDX,
262
-
};
263
-
264
-
static const u32 zip_pre_store_caps[] = {
265
-
ZIP_CORE_NUM_CAP,
266
-
ZIP_CLUSTER_COMP_NUM_CAP,
267
-
ZIP_CLUSTER_DECOMP_NUM_CAP,
268
-
ZIP_DECOMP_ENABLE_BITMAP,
269
-
ZIP_COMP_ENABLE_BITMAP,
270
-
ZIP_DRV_ALG_BITMAP,
271
-
ZIP_DEV_ALG_BITMAP,
253
+
static const struct hisi_qm_cap_query_info zip_cap_query_info[] = {
254
+
{QM_RAS_NFE_TYPE, "QM_RAS_NFE_TYPE ", 0x3124, 0x0, 0x1C57, 0x7C77},
255
+
{QM_RAS_NFE_RESET, "QM_RAS_NFE_RESET ", 0x3128, 0x0, 0xC57, 0x6C77},
256
+
{QM_RAS_CE_TYPE, "QM_RAS_CE_TYPE ", 0x312C, 0x0, 0x8, 0x8},
257
+
{ZIP_RAS_NFE_TYPE, "ZIP_RAS_NFE_TYPE ", 0x3130, 0x0, 0x7FE, 0x1FFE},
258
+
{ZIP_RAS_NFE_RESET, "ZIP_RAS_NFE_RESET ", 0x3134, 0x0, 0x7FE, 0x7FE},
259
+
{ZIP_RAS_CE_TYPE, "ZIP_RAS_CE_TYPE ", 0x3138, 0x0, 0x1, 0x1},
260
+
{ZIP_CORE_INFO, "ZIP_CORE_INFO ", 0x313C, 0x12080206, 0x12080206, 0x12050203},
261
+
{ZIP_CORE_EN, "ZIP_CORE_EN ", 0x3140, 0xFC0003, 0xFC0003, 0x1C0003},
262
+
{ZIP_DRV_ALG_BITMAP_TB, "ZIP_DRV_ALG_BITMAP ", 0x3144, 0x0, 0x0, 0x30},
263
+
{ZIP_ALG_BITMAP, "ZIP_ALG_BITMAP ", 0x3148, 0xF, 0xF, 0x3F},
264
+
{ZIP_CORE1_BITMAP, "ZIP_CORE1_BITMAP ", 0x314C, 0x5, 0x5, 0xD5},
265
+
{ZIP_CORE2_BITMAP, "ZIP_CORE2_BITMAP ", 0x3150, 0x5, 0x5, 0xD5},
266
+
{ZIP_CORE3_BITMAP, "ZIP_CORE3_BITMAP ", 0x3154, 0xA, 0xA, 0x2A},
267
+
{ZIP_CORE4_BITMAP, "ZIP_CORE4_BITMAP ", 0x3158, 0xA, 0xA, 0x2A},
268
+
{ZIP_CORE5_BITMAP, "ZIP_CORE5_BITMAP ", 0x315C, 0xA, 0xA, 0x2A},
272
269
};
273
270
274
271
static const struct debugfs_reg32 hzip_dfx_regs[] = {
···
401
402
{
402
403
pf_q_num_flag = true;
403
404
404
-
return q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
405
+
return hisi_qm_q_num_set(val, kp, PCI_DEVICE_ID_HUAWEI_ZIP_PF);
405
406
}
406
407
407
408
static const struct kernel_param_ops pf_q_num_ops = {
···
441
442
{
442
443
u32 cap_val;
443
444
444
-
cap_val = qm->cap_tables.dev_cap_table[ZIP_DRV_ALG_BITMAP_IDX].cap_val;
445
+
cap_val = qm->cap_tables.dev_cap_table[ZIP_DRV_ALG_BITMAP_TB].cap_val;
445
446
if ((alg & cap_val) == alg)
446
447
return true;
447
448
···
529
530
{
530
531
void __iomem *base = qm->io_base;
531
532
u32 dcomp_bm, comp_bm;
533
+
u32 zip_core_en;
532
534
533
535
/* qm user domain */
534
536
writel(AXUSER_BASE, base + QM_ARUSER_M_CFG_1);
···
567
567
}
568
568
569
569
/* let's open all compression/decompression cores */
570
-
dcomp_bm = qm->cap_tables.dev_cap_table[ZIP_DECOMP_ENABLE_BITMAP_IDX].cap_val;
571
-
comp_bm = qm->cap_tables.dev_cap_table[ZIP_COMP_ENABLE_BITMAP_IDX].cap_val;
570
+
571
+
zip_core_en = qm->cap_tables.dev_cap_table[ZIP_CORE_EN].cap_val;
572
+
dcomp_bm = (zip_core_en >> zip_basic_cap_info[ZIP_DECOMP_ENABLE_BITMAP].shift) &
573
+
zip_basic_cap_info[ZIP_DECOMP_ENABLE_BITMAP].mask;
574
+
comp_bm = (zip_core_en >> zip_basic_cap_info[ZIP_COMP_ENABLE_BITMAP].shift) &
575
+
zip_basic_cap_info[ZIP_COMP_ENABLE_BITMAP].mask;
572
576
writel(HZIP_DECOMP_CHECK_ENABLE | dcomp_bm | comp_bm, base + HZIP_CLOCK_GATE_CTRL);
573
577
574
578
/* enable sqc,cqc writeback */
···
792
788
793
789
static void __iomem *get_zip_core_addr(struct hisi_qm *qm, int core_num)
794
790
{
795
-
u32 zip_comp_core_num = qm->cap_tables.dev_cap_table[ZIP_CLUSTER_COMP_NUM_CAP_IDX].cap_val;
791
+
u8 zip_comp_core_num;
792
+
u32 zip_core_info;
793
+
794
+
zip_core_info = qm->cap_tables.dev_cap_table[ZIP_CORE_INFO].cap_val;
795
+
zip_comp_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].shift) &
796
+
zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].mask;
796
797
797
798
if (core_num < zip_comp_core_num)
798
799
return qm->io_base + HZIP_CORE_DFX_BASE +
···
812
803
u32 zip_core_num, zip_comp_core_num;
813
804
struct device *dev = &qm->pdev->dev;
814
805
struct debugfs_regset32 *regset;
806
+
u32 zip_core_info;
815
807
struct dentry *tmp_d;
816
808
char buf[HZIP_BUF_SIZE];
817
809
int i;
818
810
819
-
zip_core_num = qm->cap_tables.dev_cap_table[ZIP_CORE_NUM_CAP_IDX].cap_val;
820
-
zip_comp_core_num = qm->cap_tables.dev_cap_table[ZIP_CLUSTER_COMP_NUM_CAP_IDX].cap_val;
811
+
zip_core_info = qm->cap_tables.dev_cap_table[ZIP_CORE_INFO].cap_val;
812
+
zip_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CORE_NUM_CAP].shift) &
813
+
zip_basic_cap_info[ZIP_CORE_NUM_CAP].mask;
814
+
zip_comp_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].shift) &
815
+
zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].mask;
821
816
822
817
for (i = 0; i < zip_core_num; i++) {
823
818
if (i < zip_comp_core_num)
···
847
834
return 0;
848
835
}
849
836
837
+
static int zip_cap_regs_show(struct seq_file *s, void *unused)
838
+
{
839
+
struct hisi_qm *qm = s->private;
840
+
u32 i, size;
841
+
842
+
size = qm->cap_tables.qm_cap_size;
843
+
for (i = 0; i < size; i++)
844
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.qm_cap_table[i].name,
845
+
qm->cap_tables.qm_cap_table[i].cap_val);
846
+
847
+
size = qm->cap_tables.dev_cap_size;
848
+
for (i = 0; i < size; i++)
849
+
seq_printf(s, "%s= 0x%08x\n", qm->cap_tables.dev_cap_table[i].name,
850
+
qm->cap_tables.dev_cap_table[i].cap_val);
851
+
852
+
return 0;
853
+
}
854
+
855
+
DEFINE_SHOW_ATTRIBUTE(zip_cap_regs);
856
+
850
857
static void hisi_zip_dfx_debug_init(struct hisi_qm *qm)
851
858
{
852
859
struct dfx_diff_registers *hzip_regs = qm->debug.acc_diff_regs;
···
887
854
if (qm->fun_type == QM_HW_PF && hzip_regs)
888
855
debugfs_create_file("diff_regs", 0444, tmp_dir,
889
856
qm, &hzip_diff_regs_fops);
857
+
858
+
debugfs_create_file("cap_regs", CAP_FILE_PERMISSION,
859
+
qm->debug.debug_root, qm, &zip_cap_regs_fops);
890
860
}
891
861
892
862
static int hisi_zip_ctrl_debug_init(struct hisi_qm *qm)
···
948
912
/* hisi_zip_debug_regs_clear() - clear the zip debug regs */
949
913
static void hisi_zip_debug_regs_clear(struct hisi_qm *qm)
950
914
{
951
-
u32 zip_core_num = qm->cap_tables.dev_cap_table[ZIP_CORE_NUM_CAP_IDX].cap_val;
915
+
u32 zip_core_info;
916
+
u8 zip_core_num;
952
917
int i, j;
918
+
919
+
zip_core_info = qm->cap_tables.dev_cap_table[ZIP_CORE_INFO].cap_val;
920
+
zip_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CORE_NUM_CAP].shift) &
921
+
zip_basic_cap_info[ZIP_CORE_NUM_CAP].mask;
953
922
954
923
/* enable register read_clear bit */
955
924
writel(HZIP_RD_CNT_CLR_CE_EN, qm->io_base + HZIP_SOFT_CTRL_CNT_CLR_CE);
···
987
946
int com_dfx_regs_num = ARRAY_SIZE(hzip_com_dfx_regs);
988
947
struct qm_debug *debug = &qm->debug;
989
948
void __iomem *io_base;
949
+
u32 zip_core_info;
990
950
u32 zip_core_num;
991
951
int i, j, idx;
992
952
993
-
zip_core_num = qm->cap_tables.dev_cap_table[ZIP_CORE_NUM_CAP_IDX].cap_val;
953
+
zip_core_info = qm->cap_tables.dev_cap_table[ZIP_CORE_INFO].cap_val;
954
+
zip_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CORE_NUM_CAP].shift) &
955
+
zip_basic_cap_info[ZIP_CORE_NUM_CAP].mask;
994
956
995
957
debug->last_words = kcalloc(core_dfx_regs_num * zip_core_num + com_dfx_regs_num,
996
958
sizeof(unsigned int), GFP_KERNEL);
···
1035
991
u32 zip_core_num, zip_comp_core_num;
1036
992
struct qm_debug *debug = &qm->debug;
1037
993
char buf[HZIP_BUF_SIZE];
994
+
u32 zip_core_info;
1038
995
void __iomem *base;
1039
996
int i, j, idx;
1040
997
u32 val;
···
1050
1005
hzip_com_dfx_regs[i].name, debug->last_words[i], val);
1051
1006
}
1052
1007
1053
-
zip_core_num = qm->cap_tables.dev_cap_table[ZIP_CORE_NUM_CAP_IDX].cap_val;
1054
-
zip_comp_core_num = qm->cap_tables.dev_cap_table[ZIP_CLUSTER_COMP_NUM_CAP_IDX].cap_val;
1008
+
zip_core_info = qm->cap_tables.dev_cap_table[ZIP_CORE_INFO].cap_val;
1009
+
zip_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CORE_NUM_CAP].shift) &
1010
+
zip_basic_cap_info[ZIP_CORE_NUM_CAP].mask;
1011
+
zip_comp_core_num = (zip_core_info >> zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].shift) &
1012
+
zip_basic_cap_info[ZIP_CLUSTER_COMP_NUM_CAP].mask;
1055
1013
1056
1014
for (i = 0; i < zip_core_num; i++) {
1057
1015
if (i < zip_comp_core_num)
···
1107
1059
1108
1060
static void hisi_zip_clear_hw_err_status(struct hisi_qm *qm, u32 err_sts)
1109
1061
{
1110
-
u32 nfe;
1111
-
1112
1062
writel(err_sts, qm->io_base + HZIP_CORE_INT_SOURCE);
1113
-
nfe = hisi_qm_get_hw_info(qm, zip_basic_cap_info, ZIP_NFE_MASK_CAP, qm->cap_ver);
1114
-
writel(nfe, qm->io_base + HZIP_CORE_INT_RAS_NFE_ENB);
1063
+
}
1064
+
1065
+
static void hisi_zip_disable_error_report(struct hisi_qm *qm, u32 err_type)
1066
+
{
1067
+
u32 nfe_mask;
1068
+
1069
+
nfe_mask = hisi_qm_get_hw_info(qm, zip_basic_cap_info, ZIP_NFE_MASK_CAP, qm->cap_ver);
1070
+
writel(nfe_mask & (~err_type), qm->io_base + HZIP_CORE_INT_RAS_NFE_ENB);
1115
1071
}
1116
1072
1117
1073
static void hisi_zip_open_axi_master_ooo(struct hisi_qm *qm)
···
1145
1093
qm->io_base + HZIP_CORE_INT_SET);
1146
1094
}
1147
1095
1096
+
static enum acc_err_result hisi_zip_get_err_result(struct hisi_qm *qm)
1097
+
{
1098
+
u32 err_status;
1099
+
1100
+
err_status = hisi_zip_get_hw_err_status(qm);
1101
+
if (err_status) {
1102
+
if (err_status & qm->err_info.ecc_2bits_mask)
1103
+
qm->err_status.is_dev_ecc_mbit = true;
1104
+
hisi_zip_log_hw_error(qm, err_status);
1105
+
1106
+
if (err_status & qm->err_info.dev_reset_mask) {
1107
+
/* Disable the same error reporting until device is recovered. */
1108
+
hisi_zip_disable_error_report(qm, err_status);
1109
+
return ACC_ERR_NEED_RESET;
1110
+
}
1111
+
hisi_zip_clear_hw_err_status(qm, err_status);
1112
+
}
1113
+
1114
+
return ACC_ERR_RECOVERED;
1115
+
}
1116
+
1148
1117
static void hisi_zip_err_info_init(struct hisi_qm *qm)
1149
1118
{
1150
1119
struct hisi_qm_err_info *err_info = &qm->err_info;
···
1193
1120
.hw_err_disable = hisi_zip_hw_error_disable,
1194
1121
.get_dev_hw_err_status = hisi_zip_get_hw_err_status,
1195
1122
.clear_dev_hw_err_status = hisi_zip_clear_hw_err_status,
1196
-
.log_dev_hw_err = hisi_zip_log_hw_error,
1197
1123
.open_axi_master_ooo = hisi_zip_open_axi_master_ooo,
1198
1124
.close_axi_master_ooo = hisi_zip_close_axi_master_ooo,
1199
1125
.open_sva_prefetch = hisi_zip_open_sva_prefetch,
1200
1126
.close_sva_prefetch = hisi_zip_close_sva_prefetch,
1201
1127
.show_last_dfx_regs = hisi_zip_show_last_dfx_regs,
1202
1128
.err_info_init = hisi_zip_err_info_init,
1129
+
.get_err_result = hisi_zip_get_err_result,
1203
1130
};
1204
1131
1205
1132
static int hisi_zip_pf_probe_init(struct hisi_zip *hisi_zip)
···
1240
1167
struct pci_dev *pdev = qm->pdev;
1241
1168
size_t i, size;
1242
1169
1243
-
size = ARRAY_SIZE(zip_pre_store_caps);
1170
+
size = ARRAY_SIZE(zip_cap_query_info);
1244
1171
zip_cap = devm_kzalloc(&pdev->dev, sizeof(*zip_cap) * size, GFP_KERNEL);
1245
1172
if (!zip_cap)
1246
1173
return -ENOMEM;
1247
1174
1248
1175
for (i = 0; i < size; i++) {
1249
-
zip_cap[i].type = zip_pre_store_caps[i];
1250
-
zip_cap[i].cap_val = hisi_qm_get_hw_info(qm, zip_basic_cap_info,
1251
-
zip_pre_store_caps[i], qm->cap_ver);
1176
+
zip_cap[i].type = zip_cap_query_info[i].type;
1177
+
zip_cap[i].name = zip_cap_query_info[i].name;
1178
+
zip_cap[i].cap_val = hisi_qm_get_cap_value(qm, zip_cap_query_info,
1179
+
i, qm->cap_ver);
1252
1180
}
1253
1181
1254
1182
qm->cap_tables.dev_cap_table = zip_cap;
1183
+
qm->cap_tables.dev_cap_size = size;
1255
1184
1256
1185
return 0;
1257
1186
}
···
1305
1230
return ret;
1306
1231
}
1307
1232
1308
-
alg_msk = qm->cap_tables.dev_cap_table[ZIP_DEV_ALG_BITMAP_IDX].cap_val;
1233
+
alg_msk = qm->cap_tables.dev_cap_table[ZIP_ALG_BITMAP].cap_val;
1309
1234
ret = hisi_qm_set_algs(qm, alg_msk, zip_dev_algs, ARRAY_SIZE(zip_dev_algs));
1310
1235
if (ret) {
1311
1236
pci_err(qm->pdev, "Failed to set zip algs!\n");
+1
-1
drivers/crypto/img-hash.c
+1
-1
drivers/crypto/img-hash.c
+1
-1
drivers/crypto/inside-secure/safexcel.c
+1
-1
drivers/crypto/inside-secure/safexcel.c
+1
-1
drivers/crypto/inside-secure/safexcel_hash.c
+1
-1
drivers/crypto/inside-secure/safexcel_hash.c
+10
drivers/crypto/intel/iaa/iaa_crypto_main.c
+10
drivers/crypto/intel/iaa/iaa_crypto_main.c
···
945
945
bool only_once)
946
946
{
947
947
char *op_str = compress ? "compress" : "decompress";
948
+
int status_checks = 0;
948
949
int ret = 0;
949
950
950
951
while (!comp->status) {
951
952
if (only_once)
952
953
return -EAGAIN;
953
954
cpu_relax();
955
+
if (status_checks++ >= IAA_COMPLETION_TIMEOUT) {
956
+
/* Something is wrong with the hw, disable it. */
957
+
dev_err(dev, "%s completion timed out - "
958
+
"assuming broken hw, iaa_crypto now DISABLED\n",
959
+
op_str);
960
+
iaa_crypto_enabled = false;
961
+
ret = -ETIMEDOUT;
962
+
goto out;
963
+
}
954
964
}
955
965
956
966
if (comp->status != IAX_COMP_SUCCESS) {
+1
-1
drivers/crypto/intel/ixp4xx/ixp4xx_crypto.c
+1
-1
drivers/crypto/intel/ixp4xx/ixp4xx_crypto.c
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-aes-core.c
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-aes-core.c
···
1656
1656
/* The OCS driver is a platform device. */
1657
1657
static struct platform_driver kmb_ocs_aes_driver = {
1658
1658
.probe = kmb_ocs_aes_probe,
1659
-
.remove_new = kmb_ocs_aes_remove,
1659
+
.remove = kmb_ocs_aes_remove,
1660
1660
.driver = {
1661
1661
.name = DRV_NAME,
1662
1662
.of_match_table = kmb_ocs_aes_of_match,
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-ecc.c
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-ecc.c
···
991
991
/* The OCS driver is a platform device. */
992
992
static struct platform_driver kmb_ocs_ecc_driver = {
993
993
.probe = kmb_ocs_ecc_probe,
994
-
.remove_new = kmb_ocs_ecc_remove,
994
+
.remove = kmb_ocs_ecc_remove,
995
995
.driver = {
996
996
.name = DRV_NAME,
997
997
.of_match_table = kmb_ocs_ecc_of_match,
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-hcu-core.c
+1
-1
drivers/crypto/intel/keembay/keembay-ocs-hcu-core.c
···
1243
1243
/* The OCS driver is a platform device. */
1244
1244
static struct platform_driver kmb_ocs_hcu_driver = {
1245
1245
.probe = kmb_ocs_hcu_probe,
1246
-
.remove_new = kmb_ocs_hcu_remove,
1246
+
.remove = kmb_ocs_hcu_remove,
1247
1247
.driver = {
1248
1248
.name = DRV_NAME,
1249
1249
.of_match_table = kmb_ocs_hcu_of_match,
+1
-1
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
+1
-1
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
+1
-1
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c
+1
-1
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c
+4
-1
drivers/crypto/intel/qat/qat_common/adf_aer.c
+4
-1
drivers/crypto/intel/qat/qat_common/adf_aer.c
···
281
281
return -EFAULT;
282
282
283
283
device_sriov_wq = alloc_workqueue("qat_device_sriov_wq", 0, 0);
284
-
if (!device_sriov_wq)
284
+
if (!device_sriov_wq) {
285
+
destroy_workqueue(device_reset_wq);
286
+
device_reset_wq = NULL;
285
287
return -EFAULT;
288
+
}
286
289
287
290
return 0;
288
291
}
-1
drivers/crypto/intel/qat/qat_common/adf_common_drv.h
-1
drivers/crypto/intel/qat/qat_common/adf_common_drv.h
···
69
69
struct adf_accel_dev *pf);
70
70
struct list_head *adf_devmgr_get_head(void);
71
71
struct adf_accel_dev *adf_devmgr_get_dev_by_id(u32 id);
72
-
struct adf_accel_dev *adf_devmgr_get_first(void);
73
72
struct adf_accel_dev *adf_devmgr_pci_to_accel_dev(struct pci_dev *pci_dev);
74
73
int adf_devmgr_verify_id(u32 id);
75
74
void adf_devmgr_get_num_dev(u32 *num);
+1
-12
drivers/crypto/intel/qat/qat_common/adf_dbgfs.c
+1
-12
drivers/crypto/intel/qat/qat_common/adf_dbgfs.c
···
22
22
void adf_dbgfs_init(struct adf_accel_dev *accel_dev)
23
23
{
24
24
char name[ADF_DEVICE_NAME_LENGTH];
25
-
void *ret;
26
25
27
26
/* Create dev top level debugfs entry */
28
27
snprintf(name, sizeof(name), "%s%s_%s", ADF_DEVICE_NAME_PREFIX,
29
28
accel_dev->hw_device->dev_class->name,
30
29
pci_name(accel_dev->accel_pci_dev.pci_dev));
31
30
32
-
ret = debugfs_create_dir(name, NULL);
33
-
if (IS_ERR_OR_NULL(ret))
34
-
return;
35
-
36
-
accel_dev->debugfs_dir = ret;
31
+
accel_dev->debugfs_dir = debugfs_create_dir(name, NULL);
37
32
38
33
adf_cfg_dev_dbgfs_add(accel_dev);
39
34
}
···
54
59
*/
55
60
void adf_dbgfs_add(struct adf_accel_dev *accel_dev)
56
61
{
57
-
if (!accel_dev->debugfs_dir)
58
-
return;
59
-
60
62
if (!accel_dev->is_vf) {
61
63
adf_fw_counters_dbgfs_add(accel_dev);
62
64
adf_heartbeat_dbgfs_add(accel_dev);
···
69
77
*/
70
78
void adf_dbgfs_rm(struct adf_accel_dev *accel_dev)
71
79
{
72
-
if (!accel_dev->debugfs_dir)
73
-
return;
74
-
75
80
if (!accel_dev->is_vf) {
76
81
adf_tl_dbgfs_rm(accel_dev);
77
82
adf_cnv_dbgfs_rm(accel_dev);
-10
drivers/crypto/intel/qat/qat_common/adf_dev_mgr.c
-10
drivers/crypto/intel/qat/qat_common/adf_dev_mgr.c
···
276
276
}
277
277
EXPORT_SYMBOL_GPL(adf_devmgr_rm_dev);
278
278
279
-
struct adf_accel_dev *adf_devmgr_get_first(void)
280
-
{
281
-
struct adf_accel_dev *dev = NULL;
282
-
283
-
if (!list_empty(&accel_table))
284
-
dev = list_first_entry(&accel_table, struct adf_accel_dev,
285
-
list);
286
-
return dev;
287
-
}
288
-
289
279
/**
290
280
* adf_devmgr_pci_to_accel_dev() - Get accel_dev associated with the pci_dev.
291
281
* @pci_dev: Pointer to PCI device.
+9
-9
drivers/crypto/intel/qat/qat_common/adf_gen4_pm_debugfs.c
+9
-9
drivers/crypto/intel/qat/qat_common/adf_gen4_pm_debugfs.c
···
42
42
const char *key;
43
43
};
44
44
45
-
static struct pm_status_row pm_fuse_rows[] = {
45
+
static const struct pm_status_row pm_fuse_rows[] = {
46
46
PM_INFO_REGSET_ENTRY(fusectl0, ENABLE_PM),
47
47
PM_INFO_REGSET_ENTRY(fusectl0, ENABLE_PM_IDLE),
48
48
PM_INFO_REGSET_ENTRY(fusectl0, ENABLE_DEEP_PM_IDLE),
49
49
};
50
50
51
-
static struct pm_status_row pm_info_rows[] = {
51
+
static const struct pm_status_row pm_info_rows[] = {
52
52
PM_INFO_REGSET_ENTRY(pm.status, CPM_PM_STATE),
53
53
PM_INFO_REGSET_ENTRY(pm.status, PENDING_WP),
54
54
PM_INFO_REGSET_ENTRY(pm.status, CURRENT_WP),
···
59
59
PM_INFO_REGSET_ENTRY(pm.main, THR_VALUE),
60
60
};
61
61
62
-
static struct pm_status_row pm_ssm_rows[] = {
62
+
static const struct pm_status_row pm_ssm_rows[] = {
63
63
PM_INFO_REGSET_ENTRY(ssm.pm_enable, SSM_PM_ENABLE),
64
64
PM_INFO_REGSET_ENTRY32(ssm.active_constraint, ACTIVE_CONSTRAINT),
65
65
PM_INFO_REGSET_ENTRY(ssm.pm_domain_status, DOMAIN_POWER_GATED),
···
83
83
PM_INFO_REGSET_ENTRY(ssm.pm_managed_status, WCP_MANAGED_COUNT),
84
84
};
85
85
86
-
static struct pm_status_row pm_log_rows[] = {
86
+
static const struct pm_status_row pm_log_rows[] = {
87
87
PM_INFO_REGSET_ENTRY32(event_counters.host_msg, HOST_MSG_EVENT_COUNT),
88
88
PM_INFO_REGSET_ENTRY32(event_counters.sys_pm, SYS_PM_EVENT_COUNT),
89
89
PM_INFO_REGSET_ENTRY32(event_counters.local_ssm, SSM_EVENT_COUNT),
···
91
91
PM_INFO_REGSET_ENTRY32(event_counters.unknown, UNKNOWN_EVENT_COUNT),
92
92
};
93
93
94
-
static struct pm_status_row pm_event_rows[ICP_QAT_NUMBER_OF_PM_EVENTS] = {
94
+
static const struct pm_status_row pm_event_rows[ICP_QAT_NUMBER_OF_PM_EVENTS] = {
95
95
PM_INFO_REGSET_ENTRY32(event_log[0], EVENT0),
96
96
PM_INFO_REGSET_ENTRY32(event_log[1], EVENT1),
97
97
PM_INFO_REGSET_ENTRY32(event_log[2], EVENT2),
···
102
102
PM_INFO_REGSET_ENTRY32(event_log[7], EVENT7),
103
103
};
104
104
105
-
static struct pm_status_row pm_csrs_rows[] = {
105
+
static const struct pm_status_row pm_csrs_rows[] = {
106
106
PM_INFO_REGSET_ENTRY32(pm.fw_init, CPM_PM_FW_INIT),
107
107
PM_INFO_REGSET_ENTRY32(pm.status, CPM_PM_STATUS),
108
108
PM_INFO_REGSET_ENTRY32(pm.main, CPM_PM_MASTER_FW),
109
109
PM_INFO_REGSET_ENTRY32(pm.pwrreq, CPM_PM_PWRREQ),
110
110
};
111
111
112
-
static int pm_scnprint_table(char *buff, struct pm_status_row *table,
112
+
static int pm_scnprint_table(char *buff, const struct pm_status_row *table,
113
113
u32 *pm_info_regs, size_t buff_size, int table_len,
114
114
bool lowercase)
115
115
{
···
131
131
return wr;
132
132
}
133
133
134
-
static int pm_scnprint_table_upper_keys(char *buff, struct pm_status_row *table,
134
+
static int pm_scnprint_table_upper_keys(char *buff, const struct pm_status_row *table,
135
135
u32 *pm_info_regs, size_t buff_size,
136
136
int table_len)
137
137
{
···
139
139
table_len, false);
140
140
}
141
141
142
-
static int pm_scnprint_table_lower_keys(char *buff, struct pm_status_row *table,
142
+
static int pm_scnprint_table_lower_keys(char *buff, const struct pm_status_row *table,
143
143
u32 *pm_info_regs, size_t buff_size,
144
144
int table_len)
145
145
{
-4
drivers/crypto/intel/qat/qat_common/adf_hw_arbiter.c
-4
drivers/crypto/intel/qat/qat_common/adf_hw_arbiter.c
···
90
90
91
91
hw_data->get_arb_info(&info);
92
92
93
-
/* Reset arbiter configuration */
94
-
for (i = 0; i < ADF_ARB_NUM; i++)
95
-
WRITE_CSR_ARB_SARCONFIG(csr, arb_off, i, 0);
96
-
97
93
/* Unmap worker threads to service arbiters */
98
94
for (i = 0; i < hw_data->num_engines; i++)
99
95
WRITE_CSR_ARB_WT2SAM(csr, arb_off, wt_off, i, 0);
+1
-1
drivers/crypto/intel/qat/qat_common/qat_hal.c
+1
-1
drivers/crypto/intel/qat/qat_common/qat_hal.c
···
163
163
return -EINVAL;
164
164
}
165
165
166
-
/* Sets the accelaration engine context mode to either four or eight */
166
+
/* Sets the acceleration engine context mode to either four or eight */
167
167
csr = qat_hal_rd_ae_csr(handle, ae, CTX_ENABLES);
168
168
csr = IGNORE_W1C_MASK & csr;
169
169
new_csr = (mode == 4) ?
+1
-1
drivers/crypto/marvell/Kconfig
+1
-1
drivers/crypto/marvell/Kconfig
+14
-40
drivers/crypto/marvell/cesa/cesa.c
+14
-40
drivers/crypto/marvell/cesa/cesa.c
···
375
375
{
376
376
struct mv_cesa_dev *cesa = platform_get_drvdata(pdev);
377
377
struct mv_cesa_engine *engine = &cesa->engines[idx];
378
-
const char *res_name = "sram";
379
378
struct resource *res;
380
379
381
380
engine->pool = of_gen_pool_get(cesa->dev->of_node,
···
390
391
return -ENOMEM;
391
392
}
392
393
393
-
if (cesa->caps->nengines > 1) {
394
-
if (!idx)
395
-
res_name = "sram0";
396
-
else
397
-
res_name = "sram1";
398
-
}
399
-
400
-
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
401
-
res_name);
402
-
if (!res || resource_size(res) < cesa->sram_size)
403
-
return -EINVAL;
404
-
405
-
engine->sram = devm_ioremap_resource(cesa->dev, res);
394
+
engine->sram = devm_platform_get_and_ioremap_resource(pdev, idx, &res);
406
395
if (IS_ERR(engine->sram))
407
396
return PTR_ERR(engine->sram);
408
397
···
497
510
* if the clock does not exist.
498
511
*/
499
512
snprintf(res_name, sizeof(res_name), "cesa%u", i);
500
-
engine->clk = devm_clk_get(dev, res_name);
513
+
engine->clk = devm_clk_get_optional_enabled(dev, res_name);
501
514
if (IS_ERR(engine->clk)) {
502
-
engine->clk = devm_clk_get(dev, NULL);
503
-
if (IS_ERR(engine->clk))
504
-
engine->clk = NULL;
515
+
engine->clk = devm_clk_get_optional_enabled(dev, NULL);
516
+
if (IS_ERR(engine->clk)) {
517
+
ret = PTR_ERR(engine->clk);
518
+
goto err_cleanup;
519
+
}
505
520
}
506
521
507
522
snprintf(res_name, sizeof(res_name), "cesaz%u", i);
508
-
engine->zclk = devm_clk_get(dev, res_name);
509
-
if (IS_ERR(engine->zclk))
510
-
engine->zclk = NULL;
511
-
512
-
ret = clk_prepare_enable(engine->clk);
513
-
if (ret)
523
+
engine->zclk = devm_clk_get_optional_enabled(dev, res_name);
524
+
if (IS_ERR(engine->zclk)) {
525
+
ret = PTR_ERR(engine->zclk);
514
526
goto err_cleanup;
515
-
516
-
ret = clk_prepare_enable(engine->zclk);
517
-
if (ret)
518
-
goto err_cleanup;
527
+
}
519
528
520
529
engine->regs = cesa->regs + CESA_ENGINE_OFF(i);
521
530
···
553
570
return 0;
554
571
555
572
err_cleanup:
556
-
for (i = 0; i < caps->nengines; i++) {
557
-
clk_disable_unprepare(cesa->engines[i].zclk);
558
-
clk_disable_unprepare(cesa->engines[i].clk);
573
+
for (i = 0; i < caps->nengines; i++)
559
574
mv_cesa_put_sram(pdev, i);
560
-
if (cesa->engines[i].irq > 0)
561
-
irq_set_affinity_hint(cesa->engines[i].irq, NULL);
562
-
}
563
575
564
576
return ret;
565
577
}
···
566
588
567
589
mv_cesa_remove_algs(cesa);
568
590
569
-
for (i = 0; i < cesa->caps->nengines; i++) {
570
-
clk_disable_unprepare(cesa->engines[i].zclk);
571
-
clk_disable_unprepare(cesa->engines[i].clk);
591
+
for (i = 0; i < cesa->caps->nengines; i++)
572
592
mv_cesa_put_sram(pdev, i);
573
-
irq_set_affinity_hint(cesa->engines[i].irq, NULL);
574
-
}
575
593
}
576
594
577
595
static const struct platform_device_id mv_cesa_plat_id_table[] = {
···
578
604
579
605
static struct platform_driver marvell_cesa = {
580
606
.probe = mv_cesa_probe,
581
-
.remove_new = mv_cesa_remove,
607
+
.remove = mv_cesa_remove,
582
608
.id_table = mv_cesa_plat_id_table,
583
609
.driver = {
584
610
.name = "marvell-cesa",
+12
-12
drivers/crypto/marvell/cesa/cipher.c
+12
-12
drivers/crypto/marvell/cesa/cipher.c
···
489
489
490
490
static int mv_cesa_ecb_des_encrypt(struct skcipher_request *req)
491
491
{
492
-
struct mv_cesa_op_ctx tmpl;
492
+
struct mv_cesa_op_ctx tmpl = { };
493
493
494
494
mv_cesa_set_op_cfg(&tmpl,
495
495
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
500
500
501
501
static int mv_cesa_ecb_des_decrypt(struct skcipher_request *req)
502
502
{
503
-
struct mv_cesa_op_ctx tmpl;
503
+
struct mv_cesa_op_ctx tmpl = { };
504
504
505
505
mv_cesa_set_op_cfg(&tmpl,
506
506
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
543
543
544
544
static int mv_cesa_cbc_des_encrypt(struct skcipher_request *req)
545
545
{
546
-
struct mv_cesa_op_ctx tmpl;
546
+
struct mv_cesa_op_ctx tmpl = { };
547
547
548
548
mv_cesa_set_op_cfg(&tmpl, CESA_SA_DESC_CFG_DIR_ENC);
549
549
···
552
552
553
553
static int mv_cesa_cbc_des_decrypt(struct skcipher_request *req)
554
554
{
555
-
struct mv_cesa_op_ctx tmpl;
555
+
struct mv_cesa_op_ctx tmpl = { };
556
556
557
557
mv_cesa_set_op_cfg(&tmpl, CESA_SA_DESC_CFG_DIR_DEC);
558
558
···
596
596
597
597
static int mv_cesa_ecb_des3_ede_encrypt(struct skcipher_request *req)
598
598
{
599
-
struct mv_cesa_op_ctx tmpl;
599
+
struct mv_cesa_op_ctx tmpl = { };
600
600
601
601
mv_cesa_set_op_cfg(&tmpl,
602
602
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
608
608
609
609
static int mv_cesa_ecb_des3_ede_decrypt(struct skcipher_request *req)
610
610
{
611
-
struct mv_cesa_op_ctx tmpl;
611
+
struct mv_cesa_op_ctx tmpl = { };
612
612
613
613
mv_cesa_set_op_cfg(&tmpl,
614
614
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
649
649
650
650
static int mv_cesa_cbc_des3_ede_encrypt(struct skcipher_request *req)
651
651
{
652
-
struct mv_cesa_op_ctx tmpl;
652
+
struct mv_cesa_op_ctx tmpl = { };
653
653
654
654
mv_cesa_set_op_cfg(&tmpl,
655
655
CESA_SA_DESC_CFG_CRYPTCM_CBC |
···
661
661
662
662
static int mv_cesa_cbc_des3_ede_decrypt(struct skcipher_request *req)
663
663
{
664
-
struct mv_cesa_op_ctx tmpl;
664
+
struct mv_cesa_op_ctx tmpl = { };
665
665
666
666
mv_cesa_set_op_cfg(&tmpl,
667
667
CESA_SA_DESC_CFG_CRYPTCM_CBC |
···
725
725
726
726
static int mv_cesa_ecb_aes_encrypt(struct skcipher_request *req)
727
727
{
728
-
struct mv_cesa_op_ctx tmpl;
728
+
struct mv_cesa_op_ctx tmpl = { };
729
729
730
730
mv_cesa_set_op_cfg(&tmpl,
731
731
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
736
736
737
737
static int mv_cesa_ecb_aes_decrypt(struct skcipher_request *req)
738
738
{
739
-
struct mv_cesa_op_ctx tmpl;
739
+
struct mv_cesa_op_ctx tmpl = { };
740
740
741
741
mv_cesa_set_op_cfg(&tmpl,
742
742
CESA_SA_DESC_CFG_CRYPTCM_ECB |
···
778
778
779
779
static int mv_cesa_cbc_aes_encrypt(struct skcipher_request *req)
780
780
{
781
-
struct mv_cesa_op_ctx tmpl;
781
+
struct mv_cesa_op_ctx tmpl = { };
782
782
783
783
mv_cesa_set_op_cfg(&tmpl, CESA_SA_DESC_CFG_DIR_ENC);
784
784
···
787
787
788
788
static int mv_cesa_cbc_aes_decrypt(struct skcipher_request *req)
789
789
{
790
-
struct mv_cesa_op_ctx tmpl;
790
+
struct mv_cesa_op_ctx tmpl = { };
791
791
792
792
mv_cesa_set_op_cfg(&tmpl, CESA_SA_DESC_CFG_DIR_DEC);
793
793
+13
-9
drivers/crypto/mxs-dcp.c
+13
-9
drivers/crypto/mxs-dcp.c
···
225
225
static int mxs_dcp_run_aes(struct dcp_async_ctx *actx,
226
226
struct skcipher_request *req, int init)
227
227
{
228
-
dma_addr_t key_phys = 0;
229
-
dma_addr_t src_phys, dst_phys;
228
+
dma_addr_t key_phys, src_phys, dst_phys;
230
229
struct dcp *sdcp = global_sdcp;
231
230
struct dcp_dma_desc *desc = &sdcp->coh->desc[actx->chan];
232
231
struct dcp_aes_req_ctx *rctx = skcipher_request_ctx(req);
233
232
bool key_referenced = actx->key_referenced;
234
233
int ret;
235
234
236
-
if (!key_referenced) {
235
+
if (key_referenced)
236
+
key_phys = dma_map_single(sdcp->dev, sdcp->coh->aes_key + AES_KEYSIZE_128,
237
+
AES_KEYSIZE_128, DMA_TO_DEVICE);
238
+
else
237
239
key_phys = dma_map_single(sdcp->dev, sdcp->coh->aes_key,
238
240
2 * AES_KEYSIZE_128, DMA_TO_DEVICE);
239
-
ret = dma_mapping_error(sdcp->dev, key_phys);
240
-
if (ret)
241
-
return ret;
242
-
}
241
+
ret = dma_mapping_error(sdcp->dev, key_phys);
242
+
if (ret)
243
+
return ret;
243
244
244
245
src_phys = dma_map_single(sdcp->dev, sdcp->coh->aes_in_buf,
245
246
DCP_BUF_SZ, DMA_TO_DEVICE);
···
301
300
err_dst:
302
301
dma_unmap_single(sdcp->dev, src_phys, DCP_BUF_SZ, DMA_TO_DEVICE);
303
302
err_src:
304
-
if (!key_referenced)
303
+
if (key_referenced)
304
+
dma_unmap_single(sdcp->dev, key_phys, AES_KEYSIZE_128,
305
+
DMA_TO_DEVICE);
306
+
else
305
307
dma_unmap_single(sdcp->dev, key_phys, 2 * AES_KEYSIZE_128,
306
308
DMA_TO_DEVICE);
307
309
return ret;
···
1247
1243
1248
1244
static struct platform_driver mxs_dcp_driver = {
1249
1245
.probe = mxs_dcp_probe,
1250
-
.remove_new = mxs_dcp_remove,
1246
+
.remove = mxs_dcp_remove,
1251
1247
.driver = {
1252
1248
.name = "mxs-dcp",
1253
1249
.of_match_table = mxs_dcp_dt_ids,
+2
-2
drivers/crypto/n2_core.c
+2
-2
drivers/crypto/n2_core.c
···
2119
2119
.of_match_table = n2_crypto_match,
2120
2120
},
2121
2121
.probe = n2_crypto_probe,
2122
-
.remove_new = n2_crypto_remove,
2122
+
.remove = n2_crypto_remove,
2123
2123
};
2124
2124
2125
2125
static const struct of_device_id n2_mau_match[] = {
···
2146
2146
.of_match_table = n2_mau_match,
2147
2147
},
2148
2148
.probe = n2_mau_probe,
2149
-
.remove_new = n2_mau_remove,
2149
+
.remove = n2_mau_remove,
2150
2150
};
2151
2151
2152
2152
static struct platform_driver * const drivers[] = {
+19
-18
drivers/crypto/nx/nx-common-pseries.c
+19
-18
drivers/crypto/nx/nx-common-pseries.c
···
133
133
};
134
134
135
135
static struct nx842_devdata __rcu *devdata;
136
-
static DEFINE_SPINLOCK(devdata_mutex);
136
+
static DEFINE_SPINLOCK(devdata_spinlock);
137
137
138
138
#define NX842_COUNTER_INC(_x) \
139
139
static inline void nx842_inc_##_x( \
···
750
750
if (!new_devdata)
751
751
return -ENOMEM;
752
752
753
-
spin_lock_irqsave(&devdata_mutex, flags);
753
+
spin_lock_irqsave(&devdata_spinlock, flags);
754
754
old_devdata = rcu_dereference_check(devdata,
755
-
lockdep_is_held(&devdata_mutex));
755
+
lockdep_is_held(&devdata_spinlock));
756
756
if (old_devdata)
757
757
of_node = old_devdata->dev->of_node;
758
758
759
759
if (!old_devdata || !of_node) {
760
760
pr_err("%s: device is not available\n", __func__);
761
-
spin_unlock_irqrestore(&devdata_mutex, flags);
761
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
762
762
kfree(new_devdata);
763
763
return -ENODEV;
764
764
}
···
810
810
old_devdata->max_sg_len);
811
811
812
812
rcu_assign_pointer(devdata, new_devdata);
813
-
spin_unlock_irqrestore(&devdata_mutex, flags);
813
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
814
814
synchronize_rcu();
815
815
dev_set_drvdata(new_devdata->dev, new_devdata);
816
816
kfree(old_devdata);
···
821
821
dev_info(old_devdata->dev, "%s: device disabled\n", __func__);
822
822
nx842_OF_set_defaults(new_devdata);
823
823
rcu_assign_pointer(devdata, new_devdata);
824
-
spin_unlock_irqrestore(&devdata_mutex, flags);
824
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
825
825
synchronize_rcu();
826
826
dev_set_drvdata(new_devdata->dev, new_devdata);
827
827
kfree(old_devdata);
828
828
} else {
829
829
dev_err(old_devdata->dev, "%s: could not update driver from hardware\n", __func__);
830
-
spin_unlock_irqrestore(&devdata_mutex, flags);
830
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
831
831
}
832
832
833
833
if (!ret)
···
1045
1045
return -ENOMEM;
1046
1046
}
1047
1047
1048
-
spin_lock_irqsave(&devdata_mutex, flags);
1048
+
spin_lock_irqsave(&devdata_spinlock, flags);
1049
1049
old_devdata = rcu_dereference_check(devdata,
1050
-
lockdep_is_held(&devdata_mutex));
1050
+
lockdep_is_held(&devdata_spinlock));
1051
1051
1052
1052
if (old_devdata && old_devdata->vdev != NULL) {
1053
1053
dev_err(&viodev->dev, "%s: Attempt to register more than one instance of the hardware\n", __func__);
···
1062
1062
nx842_OF_set_defaults(new_devdata);
1063
1063
1064
1064
rcu_assign_pointer(devdata, new_devdata);
1065
-
spin_unlock_irqrestore(&devdata_mutex, flags);
1065
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
1066
1066
synchronize_rcu();
1067
1067
kfree(old_devdata);
1068
1068
···
1101
1101
return 0;
1102
1102
1103
1103
error_unlock:
1104
-
spin_unlock_irqrestore(&devdata_mutex, flags);
1104
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
1105
1105
if (new_devdata)
1106
1106
kfree(new_devdata->counters);
1107
1107
kfree(new_devdata);
···
1122
1122
1123
1123
crypto_unregister_alg(&nx842_pseries_alg);
1124
1124
1125
-
spin_lock_irqsave(&devdata_mutex, flags);
1126
-
old_devdata = rcu_dereference_check(devdata,
1127
-
lockdep_is_held(&devdata_mutex));
1128
1125
of_reconfig_notifier_unregister(&nx842_of_nb);
1126
+
1127
+
spin_lock_irqsave(&devdata_spinlock, flags);
1128
+
old_devdata = rcu_dereference_check(devdata,
1129
+
lockdep_is_held(&devdata_spinlock));
1129
1130
RCU_INIT_POINTER(devdata, NULL);
1130
-
spin_unlock_irqrestore(&devdata_mutex, flags);
1131
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
1131
1132
synchronize_rcu();
1132
1133
dev_set_drvdata(&viodev->dev, NULL);
1133
1134
if (old_devdata)
···
1258
1257
1259
1258
crypto_unregister_alg(&nx842_pseries_alg);
1260
1259
1261
-
spin_lock_irqsave(&devdata_mutex, flags);
1260
+
spin_lock_irqsave(&devdata_spinlock, flags);
1262
1261
old_devdata = rcu_dereference_check(devdata,
1263
-
lockdep_is_held(&devdata_mutex));
1262
+
lockdep_is_held(&devdata_spinlock));
1264
1263
RCU_INIT_POINTER(devdata, NULL);
1265
-
spin_unlock_irqrestore(&devdata_mutex, flags);
1264
+
spin_unlock_irqrestore(&devdata_spinlock, flags);
1266
1265
synchronize_rcu();
1267
1266
if (old_devdata && old_devdata->dev)
1268
1267
dev_set_drvdata(old_devdata->dev, NULL);
+1
-1
drivers/crypto/omap-aes.c
+1
-1
drivers/crypto/omap-aes.c
+1
-1
drivers/crypto/omap-des.c
+1
-1
drivers/crypto/omap-des.c
+1
-1
drivers/crypto/omap-sham.c
+1
-1
drivers/crypto/omap-sham.c
+1
-1
drivers/crypto/qce/core.c
+1
-1
drivers/crypto/qce/core.c
+1
-1
drivers/crypto/qcom-rng.c
+1
-1
drivers/crypto/qcom-rng.c
+1
-1
drivers/crypto/rockchip/rk3288_crypto.c
+1
-1
drivers/crypto/rockchip/rk3288_crypto.c
+1
-1
drivers/crypto/s5p-sss.c
+1
-1
drivers/crypto/s5p-sss.c
+2
-2
drivers/crypto/sa2ul.c
+2
-2
drivers/crypto/sa2ul.c
···
574
574
/* Clear the command label */
575
575
memzero_explicit(cmdl, (SA_MAX_CMDL_WORDS * sizeof(u32)));
576
576
577
-
/* Iniialize the command update structure */
577
+
/* Initialize the command update structure */
578
578
memzero_explicit(upd_info, sizeof(*upd_info));
579
579
580
580
if (cfg->enc_eng_id && cfg->auth_eng_id) {
···
2489
2489
2490
2490
static struct platform_driver sa_ul_driver = {
2491
2491
.probe = sa_ul_probe,
2492
-
.remove_new = sa_ul_remove,
2492
+
.remove = sa_ul_remove,
2493
2493
.driver = {
2494
2494
.name = "saul-crypto",
2495
2495
.of_match_table = of_match,
+1
-1
drivers/crypto/sahara.c
+1
-1
drivers/crypto/sahara.c
+2
-5
drivers/crypto/starfive/jh7110-cryp.c
+2
-5
drivers/crypto/starfive/jh7110-cryp.c
···
151
151
152
152
ret = starfive_aes_register_algs();
153
153
if (ret)
154
-
goto err_algs_aes;
154
+
goto err_engine_start;
155
155
156
156
ret = starfive_hash_register_algs();
157
157
if (ret)
···
167
167
starfive_hash_unregister_algs();
168
168
err_algs_hash:
169
169
starfive_aes_unregister_algs();
170
-
err_algs_aes:
171
-
crypto_engine_stop(cryp->engine);
172
170
err_engine_start:
173
171
crypto_engine_exit(cryp->engine);
174
172
err_engine:
···
191
193
starfive_hash_unregister_algs();
192
194
starfive_rsa_unregister_algs();
193
195
194
-
crypto_engine_stop(cryp->engine);
195
196
crypto_engine_exit(cryp->engine);
196
197
197
198
starfive_dma_cleanup(cryp);
···
212
215
213
216
static struct platform_driver starfive_cryp_driver = {
214
217
.probe = starfive_cryp_probe,
215
-
.remove_new = starfive_cryp_remove,
218
+
.remove = starfive_cryp_remove,
216
219
.driver = {
217
220
.name = DRIVER_NAME,
218
221
.of_match_table = starfive_dt_ids,
-2
drivers/crypto/starfive/jh7110-rsa.c
-2
drivers/crypto/starfive/jh7110-rsa.c
···
565
565
static struct akcipher_alg starfive_rsa = {
566
566
.encrypt = starfive_rsa_enc,
567
567
.decrypt = starfive_rsa_dec,
568
-
.sign = starfive_rsa_dec,
569
-
.verify = starfive_rsa_enc,
570
568
.set_pub_key = starfive_rsa_set_pub_key,
571
569
.set_priv_key = starfive_rsa_set_priv_key,
572
570
.max_size = starfive_rsa_max_size,
+1
-1
drivers/crypto/stm32/stm32-crc32.c
+1
-1
drivers/crypto/stm32/stm32-crc32.c
+1
-1
drivers/crypto/stm32/stm32-cryp.c
+1
-1
drivers/crypto/stm32/stm32-cryp.c
+1
-1
drivers/crypto/stm32/stm32-hash.c
+1
-1
drivers/crypto/stm32/stm32-hash.c
+1
-1
drivers/crypto/talitos.c
+1
-1
drivers/crypto/talitos.c
-2
drivers/crypto/tegra/tegra-se-aes.c
-2
drivers/crypto/tegra/tegra-se-aes.c
+1
-3
drivers/crypto/tegra/tegra-se-main.c
+1
-3
drivers/crypto/tegra/tegra-se-main.c
···
312
312
313
313
ret = tegra_se_host1x_register(se);
314
314
if (ret) {
315
-
crypto_engine_stop(se->engine);
316
315
crypto_engine_exit(se->engine);
317
316
return dev_err_probe(dev, ret, "failed to init host1x params\n");
318
317
}
···
323
324
{
324
325
struct tegra_se *se = platform_get_drvdata(pdev);
325
326
326
-
crypto_engine_stop(se->engine);
327
327
crypto_engine_exit(se->engine);
328
328
host1x_client_unregister(&se->client);
329
329
}
···
385
387
.of_match_table = tegra_se_of_match,
386
388
},
387
389
.probe = tegra_se_probe,
388
-
.remove_new = tegra_se_remove,
390
+
.remove = tegra_se_remove,
389
391
};
390
392
391
393
static int tegra_se_host1x_probe(struct host1x_device *dev)
+21
-44
drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
+21
-44
drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
···
83
83
case VIRTIO_CRYPTO_BADMSG:
84
84
error = -EBADMSG;
85
85
break;
86
-
87
-
case VIRTIO_CRYPTO_KEY_REJECTED:
88
-
error = -EKEYREJECTED;
89
-
break;
90
-
91
86
default:
92
87
error = -EIO;
93
88
break;
94
89
}
95
90
96
91
akcipher_req = vc_akcipher_req->akcipher_req;
97
-
if (vc_akcipher_req->opcode != VIRTIO_CRYPTO_AKCIPHER_VERIFY) {
98
-
/* actuall length maybe less than dst buffer */
99
-
akcipher_req->dst_len = len - sizeof(vc_req->status);
100
-
sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
101
-
vc_akcipher_req->dst_buf, akcipher_req->dst_len);
102
-
}
92
+
/* actual length maybe less than dst buffer */
93
+
akcipher_req->dst_len = len - sizeof(vc_req->status);
94
+
sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
95
+
vc_akcipher_req->dst_buf, akcipher_req->dst_len);
103
96
virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error);
104
97
}
105
98
···
223
230
int node = dev_to_node(&vcrypto->vdev->dev);
224
231
unsigned long flags;
225
232
int ret;
226
-
bool verify = vc_akcipher_req->opcode == VIRTIO_CRYPTO_AKCIPHER_VERIFY;
227
-
unsigned int src_len = verify ? req->src_len + req->dst_len : req->src_len;
228
233
229
234
/* out header */
230
235
sg_init_one(&outhdr_sg, req_data, sizeof(*req_data));
231
236
sgs[num_out++] = &outhdr_sg;
232
237
233
238
/* src data */
234
-
src_buf = kcalloc_node(src_len, 1, GFP_KERNEL, node);
239
+
src_buf = kcalloc_node(req->src_len, 1, GFP_KERNEL, node);
235
240
if (!src_buf)
236
241
return -ENOMEM;
237
242
238
-
if (verify) {
239
-
/* for verify operation, both src and dst data work as OUT direction */
240
-
sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, src_len);
241
-
sg_init_one(&srcdata_sg, src_buf, src_len);
242
-
sgs[num_out++] = &srcdata_sg;
243
-
} else {
244
-
sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, src_len);
245
-
sg_init_one(&srcdata_sg, src_buf, src_len);
246
-
sgs[num_out++] = &srcdata_sg;
243
+
sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, req->src_len);
244
+
sg_init_one(&srcdata_sg, src_buf, req->src_len);
245
+
sgs[num_out++] = &srcdata_sg;
247
246
248
-
/* dst data */
249
-
dst_buf = kcalloc_node(req->dst_len, 1, GFP_KERNEL, node);
250
-
if (!dst_buf)
251
-
goto free_src;
247
+
/* dst data */
248
+
dst_buf = kcalloc_node(req->dst_len, 1, GFP_KERNEL, node);
249
+
if (!dst_buf)
250
+
goto free_src;
252
251
253
-
sg_init_one(&dstdata_sg, dst_buf, req->dst_len);
254
-
sgs[num_out + num_in++] = &dstdata_sg;
255
-
}
252
+
sg_init_one(&dstdata_sg, dst_buf, req->dst_len);
253
+
sgs[num_out + num_in++] = &dstdata_sg;
256
254
257
255
vc_akcipher_req->src_buf = src_buf;
258
256
vc_akcipher_req->dst_buf = dst_buf;
···
334
350
static int virtio_crypto_rsa_decrypt(struct akcipher_request *req)
335
351
{
336
352
return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_DECRYPT);
337
-
}
338
-
339
-
static int virtio_crypto_rsa_sign(struct akcipher_request *req)
340
-
{
341
-
return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_SIGN);
342
-
}
343
-
344
-
static int virtio_crypto_rsa_verify(struct akcipher_request *req)
345
-
{
346
-
return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_VERIFY);
347
353
}
348
354
349
355
static int virtio_crypto_rsa_set_key(struct crypto_akcipher *tfm,
···
498
524
.algo.base = {
499
525
.encrypt = virtio_crypto_rsa_encrypt,
500
526
.decrypt = virtio_crypto_rsa_decrypt,
501
-
.sign = virtio_crypto_rsa_sign,
502
-
.verify = virtio_crypto_rsa_verify,
527
+
/*
528
+
* Must specify an arbitrary hash algorithm upon
529
+
* set_{pub,priv}_key (even though it's not used
530
+
* by encrypt/decrypt) because qemu checks for it.
531
+
*/
503
532
.set_pub_key = virtio_crypto_p1pad_rsa_sha1_set_pub_key,
504
533
.set_priv_key = virtio_crypto_p1pad_rsa_sha1_set_priv_key,
505
534
.max_size = virtio_crypto_rsa_max_size,
506
535
.init = virtio_crypto_rsa_init_tfm,
507
536
.exit = virtio_crypto_rsa_exit_tfm,
508
537
.base = {
509
-
.cra_name = "pkcs1pad(rsa,sha1)",
510
-
.cra_driver_name = "virtio-pkcs1-rsa-with-sha1",
538
+
.cra_name = "pkcs1pad(rsa)",
539
+
.cra_driver_name = "virtio-pkcs1-rsa",
511
540
.cra_priority = 150,
512
541
.cra_module = THIS_MODULE,
513
542
.cra_ctxsize = sizeof(struct virtio_crypto_akcipher_ctx),
+1
-1
drivers/crypto/xilinx/zynqmp-aes-gcm.c
+1
-1
drivers/crypto/xilinx/zynqmp-aes-gcm.c
+1
-1
drivers/crypto/xilinx/zynqmp-sha.c
+1
-1
drivers/crypto/xilinx/zynqmp-sha.c
+8
-61
include/crypto/akcipher.h
+8
-61
include/crypto/akcipher.h
···
12
12
#include <linux/crypto.h>
13
13
14
14
/**
15
-
* struct akcipher_request - public key request
15
+
* struct akcipher_request - public key cipher request
16
16
*
17
17
* @base: Common attributes for async crypto requests
18
18
* @src: Source data
19
-
* For verify op this is signature + digest, in that case
20
-
* total size of @src is @src_len + @dst_len.
21
-
* @dst: Destination data (Should be NULL for verify op)
19
+
* @dst: Destination data
22
20
* @src_len: Size of the input buffer
23
-
* For verify op it's size of signature part of @src, this part
24
-
* is supposed to be operated by cipher.
25
-
* @dst_len: Size of @dst buffer (for all ops except verify).
21
+
* @dst_len: Size of @dst buffer
26
22
* It needs to be at least as big as the expected result
27
23
* depending on the operation.
28
24
* After operation it will be updated with the actual size of the
29
25
* result.
30
26
* In case of error where the dst sgl size was insufficient,
31
27
* it will be updated to the size required for the operation.
32
-
* For verify op this is size of digest part in @src.
33
28
* @__ctx: Start of private context data
34
29
*/
35
30
struct akcipher_request {
···
50
55
};
51
56
52
57
/**
53
-
* struct akcipher_alg - generic public key algorithm
58
+
* struct akcipher_alg - generic public key cipher algorithm
54
59
*
55
-
* @sign: Function performs a sign operation as defined by public key
56
-
* algorithm. In case of error, where the dst_len was insufficient,
57
-
* the req->dst_len will be updated to the size required for the
58
-
* operation
59
-
* @verify: Function performs a complete verify operation as defined by
60
-
* public key algorithm, returning verification status. Requires
61
-
* digest value as input parameter.
62
60
* @encrypt: Function performs an encrypt operation as defined by public key
63
61
* algorithm. In case of error, where the dst_len was insufficient,
64
62
* the req->dst_len will be updated to the size required for the
···
82
94
* @base: Common crypto API algorithm data structure
83
95
*/
84
96
struct akcipher_alg {
85
-
int (*sign)(struct akcipher_request *req);
86
-
int (*verify)(struct akcipher_request *req);
87
97
int (*encrypt)(struct akcipher_request *req);
88
98
int (*decrypt)(struct akcipher_request *req);
89
99
int (*set_pub_key)(struct crypto_akcipher *tfm, const void *key,
···
96
110
};
97
111
98
112
/**
99
-
* DOC: Generic Public Key API
113
+
* DOC: Generic Public Key Cipher API
100
114
*
101
-
* The Public Key API is used with the algorithms of type
115
+
* The Public Key Cipher API is used with the algorithms of type
102
116
* CRYPTO_ALG_TYPE_AKCIPHER (listed as type "akcipher" in /proc/crypto)
103
117
*/
104
118
···
229
243
*
230
244
* @req: public key request
231
245
* @src: ptr to input scatter list
232
-
* @dst: ptr to output scatter list or NULL for verify op
246
+
* @dst: ptr to output scatter list
233
247
* @src_len: size of the src input scatter list to be processed
234
-
* @dst_len: size of the dst output scatter list or size of signature
235
-
* portion in @src for verify op
248
+
* @dst_len: size of the dst output scatter list
236
249
*/
237
250
static inline void akcipher_request_set_crypt(struct akcipher_request *req,
238
251
struct scatterlist *src,
···
331
346
int crypto_akcipher_sync_decrypt(struct crypto_akcipher *tfm,
332
347
const void *src, unsigned int slen,
333
348
void *dst, unsigned int dlen);
334
-
335
-
/**
336
-
* crypto_akcipher_sign() - Invoke public key sign operation
337
-
*
338
-
* Function invokes the specific public key sign operation for a given
339
-
* public key algorithm
340
-
*
341
-
* @req: asymmetric key request
342
-
*
343
-
* Return: zero on success; error code in case of error
344
-
*/
345
-
static inline int crypto_akcipher_sign(struct akcipher_request *req)
346
-
{
347
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
348
-
349
-
return crypto_akcipher_alg(tfm)->sign(req);
350
-
}
351
-
352
-
/**
353
-
* crypto_akcipher_verify() - Invoke public key signature verification
354
-
*
355
-
* Function invokes the specific public key signature verification operation
356
-
* for a given public key algorithm.
357
-
*
358
-
* @req: asymmetric key request
359
-
*
360
-
* Note: req->dst should be NULL, req->src should point to SG of size
361
-
* (req->src_size + req->dst_size), containing signature (of req->src_size
362
-
* length) with appended digest (of req->dst_size length).
363
-
*
364
-
* Return: zero on verification success; error code in case of error.
365
-
*/
366
-
static inline int crypto_akcipher_verify(struct akcipher_request *req)
367
-
{
368
-
struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
369
-
370
-
return crypto_akcipher_alg(tfm)->verify(req);
371
-
}
372
349
373
350
/**
374
351
* crypto_akcipher_set_pub_key() - Invoke set public key operation
+2
-2
include/crypto/internal/akcipher.h
+2
-2
include/crypto/internal/akcipher.h
···
124
124
/**
125
125
* crypto_register_akcipher() -- Register public key algorithm
126
126
*
127
-
* Function registers an implementation of a public key verify algorithm
127
+
* Function registers an implementation of a public key cipher algorithm
128
128
*
129
129
* @alg: algorithm definition
130
130
*
···
135
135
/**
136
136
* crypto_unregister_akcipher() -- Unregister public key algorithm
137
137
*
138
-
* Function unregisters an implementation of a public key verify algorithm
138
+
* Function unregisters an implementation of a public key cipher algorithm
139
139
*
140
140
* @alg: algorithm definition
141
141
*/
+14
include/crypto/internal/ecc.h
+14
include/crypto/internal/ecc.h
···
42
42
43
43
#define ECC_POINT_INIT(x, y, ndigits) (struct ecc_point) { x, y, ndigits }
44
44
45
+
/*
46
+
* The integers r and s making up the signature are expected to be
47
+
* formatted as two consecutive u64 arrays of size ECC_MAX_BYTES.
48
+
* The bytes within each u64 digit are in native endianness,
49
+
* but the order of the u64 digits themselves is little endian.
50
+
* This format allows direct use by internal vli_*() functions.
51
+
*/
52
+
struct ecdsa_raw_sig {
53
+
u64 r[ECC_MAX_DIGITS];
54
+
u64 s[ECC_MAX_DIGITS];
55
+
};
56
+
45
57
/**
46
58
* ecc_swap_digits() - Copy ndigits from big endian array to native array
47
59
* @in: Input array
···
305
293
const u64 *y, const struct ecc_point *q,
306
294
const struct ecc_curve *curve);
307
295
296
+
extern struct crypto_template ecdsa_x962_tmpl;
297
+
extern struct crypto_template ecdsa_p1363_tmpl;
308
298
#endif
+29
include/crypto/internal/rsa.h
+29
include/crypto/internal/rsa.h
···
8
8
#ifndef _RSA_HELPER_
9
9
#define _RSA_HELPER_
10
10
#include <linux/types.h>
11
+
#include <crypto/akcipher.h>
11
12
12
13
/**
13
14
* rsa_key - RSA key structure
···
54
53
int rsa_parse_priv_key(struct rsa_key *rsa_key, const void *key,
55
54
unsigned int key_len);
56
55
56
+
#define RSA_PUB (true)
57
+
#define RSA_PRIV (false)
58
+
59
+
static inline int rsa_set_key(struct crypto_akcipher *child,
60
+
unsigned int *key_size, bool is_pubkey,
61
+
const void *key, unsigned int keylen)
62
+
{
63
+
int err;
64
+
65
+
*key_size = 0;
66
+
67
+
if (is_pubkey)
68
+
err = crypto_akcipher_set_pub_key(child, key, keylen);
69
+
else
70
+
err = crypto_akcipher_set_priv_key(child, key, keylen);
71
+
if (err)
72
+
return err;
73
+
74
+
/* Find out new modulus size from rsa implementation */
75
+
err = crypto_akcipher_maxsize(child);
76
+
if (err > PAGE_SIZE)
77
+
return -ENOTSUPP;
78
+
79
+
*key_size = err;
80
+
return 0;
81
+
}
82
+
57
83
extern struct crypto_template rsa_pkcs1pad_tmpl;
84
+
extern struct crypto_template rsassa_pkcs1_tmpl;
58
85
#endif
+80
include/crypto/internal/sig.h
+80
include/crypto/internal/sig.h
···
10
10
#include <crypto/algapi.h>
11
11
#include <crypto/sig.h>
12
12
13
+
struct sig_instance {
14
+
void (*free)(struct sig_instance *inst);
15
+
union {
16
+
struct {
17
+
char head[offsetof(struct sig_alg, base)];
18
+
struct crypto_instance base;
19
+
};
20
+
struct sig_alg alg;
21
+
};
22
+
};
23
+
24
+
struct crypto_sig_spawn {
25
+
struct crypto_spawn base;
26
+
};
27
+
13
28
static inline void *crypto_sig_ctx(struct crypto_sig *tfm)
14
29
{
15
30
return crypto_tfm_ctx(&tfm->base);
31
+
}
32
+
33
+
/**
34
+
* crypto_register_sig() -- Register public key signature algorithm
35
+
*
36
+
* Function registers an implementation of a public key signature algorithm
37
+
*
38
+
* @alg: algorithm definition
39
+
*
40
+
* Return: zero on success; error code in case of error
41
+
*/
42
+
int crypto_register_sig(struct sig_alg *alg);
43
+
44
+
/**
45
+
* crypto_unregister_sig() -- Unregister public key signature algorithm
46
+
*
47
+
* Function unregisters an implementation of a public key signature algorithm
48
+
*
49
+
* @alg: algorithm definition
50
+
*/
51
+
void crypto_unregister_sig(struct sig_alg *alg);
52
+
53
+
int sig_register_instance(struct crypto_template *tmpl,
54
+
struct sig_instance *inst);
55
+
56
+
static inline struct sig_instance *sig_instance(struct crypto_instance *inst)
57
+
{
58
+
return container_of(&inst->alg, struct sig_instance, alg.base);
59
+
}
60
+
61
+
static inline struct sig_instance *sig_alg_instance(struct crypto_sig *tfm)
62
+
{
63
+
return sig_instance(crypto_tfm_alg_instance(&tfm->base));
64
+
}
65
+
66
+
static inline struct crypto_instance *sig_crypto_instance(struct sig_instance
67
+
*inst)
68
+
{
69
+
return container_of(&inst->alg.base, struct crypto_instance, alg);
70
+
}
71
+
72
+
static inline void *sig_instance_ctx(struct sig_instance *inst)
73
+
{
74
+
return crypto_instance_ctx(sig_crypto_instance(inst));
75
+
}
76
+
77
+
int crypto_grab_sig(struct crypto_sig_spawn *spawn,
78
+
struct crypto_instance *inst,
79
+
const char *name, u32 type, u32 mask);
80
+
81
+
static inline struct crypto_sig *crypto_spawn_sig(struct crypto_sig_spawn
82
+
*spawn)
83
+
{
84
+
return crypto_spawn_tfm2(&spawn->base);
85
+
}
86
+
87
+
static inline void crypto_drop_sig(struct crypto_sig_spawn *spawn)
88
+
{
89
+
crypto_drop_spawn(&spawn->base);
90
+
}
91
+
92
+
static inline struct sig_alg *crypto_spawn_sig_alg(struct crypto_sig_spawn
93
+
*spawn)
94
+
{
95
+
return container_of(spawn->base.alg, struct sig_alg, base);
16
96
}
17
97
#endif
-3
include/crypto/public_key.h
-3
include/crypto/public_key.h
···
104
104
extern int query_asymmetric_key(const struct kernel_pkey_params *,
105
105
struct kernel_pkey_query *);
106
106
107
-
extern int encrypt_blob(struct kernel_pkey_params *, const void *, void *);
108
-
extern int decrypt_blob(struct kernel_pkey_params *, const void *, void *);
109
-
extern int create_signature(struct kernel_pkey_params *, const void *, void *);
110
107
extern int verify_signature(const struct key *,
111
108
const struct public_key_signature *);
112
109
+138
-14
include/crypto/sig.h
+138
-14
include/crypto/sig.h
···
20
20
};
21
21
22
22
/**
23
+
* struct sig_alg - generic public key signature algorithm
24
+
*
25
+
* @sign: Function performs a sign operation as defined by public key
26
+
* algorithm. Optional.
27
+
* @verify: Function performs a complete verify operation as defined by
28
+
* public key algorithm, returning verification status. Optional.
29
+
* @set_pub_key: Function invokes the algorithm specific set public key
30
+
* function, which knows how to decode and interpret
31
+
* the BER encoded public key and parameters. Mandatory.
32
+
* @set_priv_key: Function invokes the algorithm specific set private key
33
+
* function, which knows how to decode and interpret
34
+
* the BER encoded private key and parameters. Optional.
35
+
* @key_size: Function returns key size. Mandatory.
36
+
* @digest_size: Function returns maximum digest size. Optional.
37
+
* @max_size: Function returns maximum signature size. Optional.
38
+
* @init: Initialize the cryptographic transformation object.
39
+
* This function is used to initialize the cryptographic
40
+
* transformation object. This function is called only once at
41
+
* the instantiation time, right after the transformation context
42
+
* was allocated. In case the cryptographic hardware has some
43
+
* special requirements which need to be handled by software, this
44
+
* function shall check for the precise requirement of the
45
+
* transformation and put any software fallbacks in place.
46
+
* @exit: Deinitialize the cryptographic transformation object. This is a
47
+
* counterpart to @init, used to remove various changes set in
48
+
* @init.
49
+
*
50
+
* @base: Common crypto API algorithm data structure
51
+
*/
52
+
struct sig_alg {
53
+
int (*sign)(struct crypto_sig *tfm,
54
+
const void *src, unsigned int slen,
55
+
void *dst, unsigned int dlen);
56
+
int (*verify)(struct crypto_sig *tfm,
57
+
const void *src, unsigned int slen,
58
+
const void *digest, unsigned int dlen);
59
+
int (*set_pub_key)(struct crypto_sig *tfm,
60
+
const void *key, unsigned int keylen);
61
+
int (*set_priv_key)(struct crypto_sig *tfm,
62
+
const void *key, unsigned int keylen);
63
+
unsigned int (*key_size)(struct crypto_sig *tfm);
64
+
unsigned int (*digest_size)(struct crypto_sig *tfm);
65
+
unsigned int (*max_size)(struct crypto_sig *tfm);
66
+
int (*init)(struct crypto_sig *tfm);
67
+
void (*exit)(struct crypto_sig *tfm);
68
+
69
+
struct crypto_alg base;
70
+
};
71
+
72
+
/**
23
73
* DOC: Generic Public Key Signature API
24
74
*
25
75
* The Public Key Signature API is used with the algorithms of type
···
97
47
return &tfm->base;
98
48
}
99
49
50
+
static inline struct crypto_sig *__crypto_sig_tfm(struct crypto_tfm *tfm)
51
+
{
52
+
return container_of(tfm, struct crypto_sig, base);
53
+
}
54
+
55
+
static inline struct sig_alg *__crypto_sig_alg(struct crypto_alg *alg)
56
+
{
57
+
return container_of(alg, struct sig_alg, base);
58
+
}
59
+
60
+
static inline struct sig_alg *crypto_sig_alg(struct crypto_sig *tfm)
61
+
{
62
+
return __crypto_sig_alg(crypto_sig_tfm(tfm)->__crt_alg);
63
+
}
64
+
100
65
/**
101
66
* crypto_free_sig() - free signature tfm handle
102
67
*
···
125
60
}
126
61
127
62
/**
128
-
* crypto_sig_maxsize() - Get len for output buffer
63
+
* crypto_sig_keysize() - Get key size
129
64
*
130
-
* Function returns the dest buffer size required for a given key.
65
+
* Function returns the key size in bytes.
131
66
* Function assumes that the key is already set in the transformation. If this
132
-
* function is called without a setkey or with a failed setkey, you will end up
67
+
* function is called without a setkey or with a failed setkey, you may end up
133
68
* in a NULL dereference.
134
69
*
135
70
* @tfm: signature tfm handle allocated with crypto_alloc_sig()
136
71
*/
137
-
int crypto_sig_maxsize(struct crypto_sig *tfm);
72
+
static inline unsigned int crypto_sig_keysize(struct crypto_sig *tfm)
73
+
{
74
+
struct sig_alg *alg = crypto_sig_alg(tfm);
75
+
76
+
return alg->key_size(tfm);
77
+
}
78
+
79
+
/**
80
+
* crypto_sig_digestsize() - Get maximum digest size
81
+
*
82
+
* Function returns the maximum digest size in bytes.
83
+
* Function assumes that the key is already set in the transformation. If this
84
+
* function is called without a setkey or with a failed setkey, you may end up
85
+
* in a NULL dereference.
86
+
*
87
+
* @tfm: signature tfm handle allocated with crypto_alloc_sig()
88
+
*/
89
+
static inline unsigned int crypto_sig_digestsize(struct crypto_sig *tfm)
90
+
{
91
+
struct sig_alg *alg = crypto_sig_alg(tfm);
92
+
93
+
return alg->digest_size(tfm);
94
+
}
95
+
96
+
/**
97
+
* crypto_sig_maxsize() - Get maximum signature size
98
+
*
99
+
* Function returns the maximum signature size in bytes.
100
+
* Function assumes that the key is already set in the transformation. If this
101
+
* function is called without a setkey or with a failed setkey, you may end up
102
+
* in a NULL dereference.
103
+
*
104
+
* @tfm: signature tfm handle allocated with crypto_alloc_sig()
105
+
*/
106
+
static inline unsigned int crypto_sig_maxsize(struct crypto_sig *tfm)
107
+
{
108
+
struct sig_alg *alg = crypto_sig_alg(tfm);
109
+
110
+
return alg->max_size(tfm);
111
+
}
138
112
139
113
/**
140
114
* crypto_sig_sign() - Invoke signing operation
···
188
84
*
189
85
* Return: zero on success; error code in case of error
190
86
*/
191
-
int crypto_sig_sign(struct crypto_sig *tfm,
192
-
const void *src, unsigned int slen,
193
-
void *dst, unsigned int dlen);
87
+
static inline int crypto_sig_sign(struct crypto_sig *tfm,
88
+
const void *src, unsigned int slen,
89
+
void *dst, unsigned int dlen)
90
+
{
91
+
struct sig_alg *alg = crypto_sig_alg(tfm);
92
+
93
+
return alg->sign(tfm, src, slen, dst, dlen);
94
+
}
194
95
195
96
/**
196
97
* crypto_sig_verify() - Invoke signature verification
···
211
102
*
212
103
* Return: zero on verification success; error code in case of error.
213
104
*/
214
-
int crypto_sig_verify(struct crypto_sig *tfm,
215
-
const void *src, unsigned int slen,
216
-
const void *digest, unsigned int dlen);
105
+
static inline int crypto_sig_verify(struct crypto_sig *tfm,
106
+
const void *src, unsigned int slen,
107
+
const void *digest, unsigned int dlen)
108
+
{
109
+
struct sig_alg *alg = crypto_sig_alg(tfm);
110
+
111
+
return alg->verify(tfm, src, slen, digest, dlen);
112
+
}
217
113
218
114
/**
219
115
* crypto_sig_set_pubkey() - Invoke set public key operation
···
233
119
*
234
120
* Return: zero on success; error code in case of error
235
121
*/
236
-
int crypto_sig_set_pubkey(struct crypto_sig *tfm,
237
-
const void *key, unsigned int keylen);
122
+
static inline int crypto_sig_set_pubkey(struct crypto_sig *tfm,
123
+
const void *key, unsigned int keylen)
124
+
{
125
+
struct sig_alg *alg = crypto_sig_alg(tfm);
126
+
127
+
return alg->set_pub_key(tfm, key, keylen);
128
+
}
238
129
239
130
/**
240
131
* crypto_sig_set_privkey() - Invoke set private key operation
···
254
135
*
255
136
* Return: zero on success; error code in case of error
256
137
*/
257
-
int crypto_sig_set_privkey(struct crypto_sig *tfm,
258
-
const void *key, unsigned int keylen);
138
+
static inline int crypto_sig_set_privkey(struct crypto_sig *tfm,
139
+
const void *key, unsigned int keylen)
140
+
{
141
+
struct sig_alg *alg = crypto_sig_alg(tfm);
142
+
143
+
return alg->set_priv_key(tfm, key, keylen);
144
+
}
259
145
#endif
+1
include/linux/asn1_decoder.h
+1
include/linux/asn1_decoder.h
-1
include/linux/asn1_encoder.h
-1
include/linux/asn1_encoder.h
+24
-32
include/linux/hisi_acc_qm.h
+24
-32
include/linux/hisi_acc_qm.h
···
229
229
230
230
struct hisi_qm;
231
231
232
+
enum acc_err_result {
233
+
ACC_ERR_NONE,
234
+
ACC_ERR_NEED_RESET,
235
+
ACC_ERR_RECOVERED,
236
+
};
237
+
232
238
struct hisi_qm_err_info {
233
239
char *acpi_rst;
234
240
u32 msi_wr_port;
···
263
257
void (*close_axi_master_ooo)(struct hisi_qm *qm);
264
258
void (*open_sva_prefetch)(struct hisi_qm *qm);
265
259
void (*close_sva_prefetch)(struct hisi_qm *qm);
266
-
void (*log_dev_hw_err)(struct hisi_qm *qm, u32 err_sts);
267
260
void (*show_last_dfx_regs)(struct hisi_qm *qm);
268
261
void (*err_info_init)(struct hisi_qm *qm);
262
+
enum acc_err_result (*get_err_result)(struct hisi_qm *qm);
269
263
};
270
264
271
265
struct hisi_qm_cap_info {
···
280
274
u32 v3_val;
281
275
};
282
276
277
+
struct hisi_qm_cap_query_info {
278
+
u32 type;
279
+
const char *name;
280
+
u32 offset;
281
+
u32 v1_val;
282
+
u32 v2_val;
283
+
u32 v3_val;
284
+
};
285
+
283
286
struct hisi_qm_cap_record {
284
287
u32 type;
288
+
const char *name;
285
289
u32 cap_val;
286
290
};
287
291
288
292
struct hisi_qm_cap_tables {
293
+
u32 qm_cap_size;
289
294
struct hisi_qm_cap_record *qm_cap_table;
295
+
u32 dev_cap_size;
290
296
struct hisi_qm_cap_record *dev_cap_table;
291
297
};
292
298
···
454
436
struct uacce_queue *uacce_q;
455
437
};
456
438
457
-
static inline int q_num_set(const char *val, const struct kernel_param *kp,
458
-
unsigned int device)
459
-
{
460
-
struct pci_dev *pdev;
461
-
u32 n, q_num;
462
-
int ret;
463
-
464
-
if (!val)
465
-
return -EINVAL;
466
-
467
-
pdev = pci_get_device(PCI_VENDOR_ID_HUAWEI, device, NULL);
468
-
if (!pdev) {
469
-
q_num = min_t(u32, QM_QNUM_V1, QM_QNUM_V2);
470
-
pr_info("No device found currently, suppose queue number is %u\n",
471
-
q_num);
472
-
} else {
473
-
if (pdev->revision == QM_HW_V1)
474
-
q_num = QM_QNUM_V1;
475
-
else
476
-
q_num = QM_QNUM_V2;
477
-
478
-
pci_dev_put(pdev);
479
-
}
480
-
481
-
ret = kstrtou32(val, 10, &n);
482
-
if (ret || n < QM_MIN_QNUM || n > q_num)
483
-
return -EINVAL;
484
-
485
-
return param_set_int(val, kp);
486
-
}
487
-
488
439
static inline int vfs_num_set(const char *val, const struct kernel_param *kp)
489
440
{
490
441
u32 n;
···
513
526
mutex_unlock(&qm_list->lock);
514
527
}
515
528
529
+
int hisi_qm_q_num_set(const char *val, const struct kernel_param *kp,
530
+
unsigned int device);
516
531
int hisi_qm_init(struct hisi_qm *qm);
517
532
void hisi_qm_uninit(struct hisi_qm *qm);
518
533
int hisi_qm_start(struct hisi_qm *qm);
···
571
582
void hisi_qm_regs_dump(struct seq_file *s, struct debugfs_regset32 *regset);
572
583
u32 hisi_qm_get_hw_info(struct hisi_qm *qm,
573
584
const struct hisi_qm_cap_info *info_table,
585
+
u32 index, bool is_read);
586
+
u32 hisi_qm_get_cap_value(struct hisi_qm *qm,
587
+
const struct hisi_qm_cap_query_info *info_table,
574
588
u32 index, bool is_read);
575
589
int hisi_qm_set_algs(struct hisi_qm *qm, u64 alg_msk, const struct qm_dev_alg *dev_algs,
576
590
u32 dev_algs_size);
+1
include/linux/slab.h
+1
include/linux/slab.h
+5
include/uapi/linux/cryptouser.h
+5
include/uapi/linux/cryptouser.h
···
64
64
CRYPTOCFGA_STAT_AKCIPHER, /* No longer supported, do not use. */
65
65
CRYPTOCFGA_STAT_KPP, /* No longer supported, do not use. */
66
66
CRYPTOCFGA_STAT_ACOMP, /* No longer supported, do not use. */
67
+
CRYPTOCFGA_REPORT_SIG, /* struct crypto_report_sig */
67
68
__CRYPTOCFGA_MAX
68
69
69
70
#define CRYPTOCFGA_MAX (__CRYPTOCFGA_MAX - 1)
···
205
204
};
206
205
207
206
struct crypto_report_acomp {
207
+
char type[CRYPTO_MAX_NAME];
208
+
};
209
+
210
+
struct crypto_report_sig {
208
211
char type[CRYPTO_MAX_NAME];
209
212
};
210
213
+1
include/uapi/linux/virtio_crypto.h
+1
include/uapi/linux/virtio_crypto.h
···
329
329
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x00)
330
330
#define VIRTIO_CRYPTO_AKCIPHER_DECRYPT \
331
331
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x01)
332
+
/* akcipher sign/verify opcodes are deprecated */
332
333
#define VIRTIO_CRYPTO_AKCIPHER_SIGN \
333
334
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x02)
334
335
#define VIRTIO_CRYPTO_AKCIPHER_VERIFY \
-7
kernel/padata.c
-7
kernel/padata.c
···
521
521
ps.chunk_size = max(ps.chunk_size, 1ul);
522
522
ps.chunk_size = roundup(ps.chunk_size, job->align);
523
523
524
-
/*
525
-
* chunk_size can be 0 if the caller sets min_chunk to 0. So force it
526
-
* to at least 1 to prevent divide-by-0 panic in padata_mt_helper().`
527
-
*/
528
-
if (!ps.chunk_size)
529
-
ps.chunk_size = 1U;
530
-
531
524
list_for_each_entry(pw, &works, pw_list)
532
525
if (job->numa_aware) {
533
526
int old_node = atomic_read(&last_used_nid);
+4
lib/crc32.c
+4
lib/crc32.c
···
205
205
EXPORT_SYMBOL(__crc32c_le);
206
206
207
207
u32 __pure crc32_le_base(u32, unsigned char const *, size_t) __alias(crc32_le);
208
+
EXPORT_SYMBOL(crc32_le_base);
209
+
208
210
u32 __pure __crc32c_le_base(u32, unsigned char const *, size_t) __alias(__crc32c_le);
211
+
EXPORT_SYMBOL(__crc32c_le_base);
212
+
209
213
u32 __pure crc32_be_base(u32, unsigned char const *, size_t) __alias(crc32_be);
210
214
211
215
/*
+2
lib/crypto/Makefile
+2
lib/crypto/Makefile
+1
lib/crypto/mpi/mpi-bit.c
+1
lib/crypto/mpi/mpi-bit.c
+11
lib/crypto/simd.c
+11
lib/crypto/simd.c
···
1
+
// SPDX-License-Identifier: GPL-2.0-or-later
2
+
/*
3
+
* SIMD testing utility functions
4
+
*
5
+
* Copyright 2024 Google LLC
6
+
*/
7
+
8
+
#include <crypto/internal/simd.h>
9
+
10
+
DEFINE_PER_CPU(bool, crypto_simd_disabled_for_test);
11
+
EXPORT_PER_CPU_SYMBOL_GPL(crypto_simd_disabled_for_test);
+3
-3
security/integrity/ima/ima_main.c
+3
-3
security/integrity/ima/ima_main.c
···
1111
1111
#ifdef CONFIG_INTEGRITY_ASYMMETRIC_KEYS
1112
1112
1113
1113
/**
1114
-
* ima_kernel_module_request - Prevent crypto-pkcs1pad(rsa,*) requests
1114
+
* ima_kernel_module_request - Prevent crypto-pkcs1(rsa,*) requests
1115
1115
* @kmod_name: kernel module name
1116
1116
*
1117
1117
* Avoid a verification loop where verifying the signature of the modprobe
···
1125
1125
* algorithm on the fly, but crypto_larval_lookup() will try to use alg_name
1126
1126
* in order to load a kernel module with same name.
1127
1127
*
1128
-
* Since we don't have any real "crypto-pkcs1pad(rsa,*)" kernel modules,
1128
+
* Since we don't have any real "crypto-pkcs1(rsa,*)" kernel modules,
1129
1129
* we are safe to fail such module request from crypto_larval_lookup(), and
1130
1130
* avoid the verification loop.
1131
1131
*
···
1133
1133
*/
1134
1134
static int ima_kernel_module_request(char *kmod_name)
1135
1135
{
1136
-
if (strncmp(kmod_name, "crypto-pkcs1pad(rsa,", 20) == 0)
1136
+
if (strncmp(kmod_name, "crypto-pkcs1(rsa,", 17) == 0)
1137
1137
return -EINVAL;
1138
1138
1139
1139
return 0;