+14

Documentation/ABI/testing/debugfs-hisi-hpre

reviewed

··· 104 104 Four states: initiated, started, stopped and closed. 105 105 Available for both PF and VF, and take no other effect on HPRE. 106 106 107 107 + What: /sys/kernel/debug/hisi_hpre/<bdf>/qm/diff_regs 108 108 + Date: Mar 2022 109 109 + Contact: linux-crypto@vger.kernel.org 110 110 + Description: QM debug registers(regs) read hardware register value. This 111 111 + node is used to show the change of the qm register values. This 112 112 + node can be help users to check the change of register values. 113 113 + 114 114 + What: /sys/kernel/debug/hisi_hpre/<bdf>/hpre_dfx/diff_regs 115 115 + Date: Mar 2022 116 116 + Contact: linux-crypto@vger.kernel.org 117 117 + Description: HPRE debug registers(regs) read hardware register value. This 118 118 + node is used to show the change of the register values. This 119 119 + node can be help users to check the change of register values. 120 120 + 107 121 What: /sys/kernel/debug/hisi_hpre/<bdf>/hpre_dfx/send_cnt 108 122 Date: Apr 2020 109 123 Contact: linux-crypto@vger.kernel.org

+14

Documentation/ABI/testing/debugfs-hisi-sec

reviewed

··· 84 84 Four states: initiated, started, stopped and closed. 85 85 Available for both PF and VF, and take no other effect on SEC. 86 86 87 87 + What: /sys/kernel/debug/hisi_sec2/<bdf>/qm/diff_regs 88 88 + Date: Mar 2022 89 89 + Contact: linux-crypto@vger.kernel.org 90 90 + Description: QM debug registers(regs) read hardware register value. This 91 91 + node is used to show the change of the qm register values. This 92 92 + node can be help users to check the change of register values. 93 93 + 94 94 + What: /sys/kernel/debug/hisi_sec2/<bdf>/sec_dfx/diff_regs 95 95 + Date: Mar 2022 96 96 + Contact: linux-crypto@vger.kernel.org 97 97 + Description: SEC debug registers(regs) read hardware register value. This 98 98 + node is used to show the change of the register values. This 99 99 + node can be help users to check the change of register values. 100 100 + 87 101 What: /sys/kernel/debug/hisi_sec2/<bdf>/sec_dfx/send_cnt 88 102 Date: Apr 2020 89 103 Contact: linux-crypto@vger.kernel.org

+14

Documentation/ABI/testing/debugfs-hisi-zip

reviewed

··· 97 97 Four states: initiated, started, stopped and closed. 98 98 Available for both PF and VF, and take no other effect on ZIP. 99 99 100 100 + What: /sys/kernel/debug/hisi_zip/<bdf>/qm/diff_regs 101 101 + Date: Mar 2022 102 102 + Contact: linux-crypto@vger.kernel.org 103 103 + Description: QM debug registers(regs) read hardware register value. This 104 104 + node is used to show the change of the qm registers value. This 105 105 + node can be help users to check the change of register values. 106 106 + 107 107 + What: /sys/kernel/debug/hisi_zip/<bdf>/zip_dfx/diff_regs 108 108 + Date: Mar 2022 109 109 + Contact: linux-crypto@vger.kernel.org 110 110 + Description: ZIP debug registers(regs) read hardware register value. This 111 111 + node is used to show the change of the registers value. this 112 112 + node can be help users to check the change of register values. 113 113 + 100 114 What: /sys/kernel/debug/hisi_zip/<bdf>/zip_dfx/send_cnt 101 115 Date: Apr 2020 102 116 Contact: linux-crypto@vger.kernel.org

+87

Documentation/ABI/testing/sysfs-driver-ccp

reviewed

··· 1 1 + What: /sys/bus/pci/devices/<BDF>/fused_part 2 2 + Date: June 2022 3 3 + KernelVersion: 5.19 4 4 + Contact: mario.limonciello@amd.com 5 5 + Description: 6 6 + The /sys/bus/pci/devices/<BDF>/fused_part file reports 7 7 + whether the CPU or APU has been fused to prevent tampering. 8 8 + 0: Not fused 9 9 + 1: Fused 10 10 + 11 11 + What: /sys/bus/pci/devices/<BDF>/debug_lock_on 12 12 + Date: June 2022 13 13 + KernelVersion: 5.19 14 14 + Contact: mario.limonciello@amd.com 15 15 + Description: 16 16 + The /sys/bus/pci/devices/<BDF>/debug_lock_on reports 17 17 + whether the AMD CPU or APU has been unlocked for debugging. 18 18 + Possible values: 19 19 + 0: Not locked 20 20 + 1: Locked 21 21 + 22 22 + What: /sys/bus/pci/devices/<BDF>/tsme_status 23 23 + Date: June 2022 24 24 + KernelVersion: 5.19 25 25 + Contact: mario.limonciello@amd.com 26 26 + Description: 27 27 + The /sys/bus/pci/devices/<BDF>/tsme_status file reports 28 28 + the status of transparent secure memory encryption on AMD systems. 29 29 + Possible values: 30 30 + 0: Not active 31 31 + 1: Active 32 32 + 33 33 + What: /sys/bus/pci/devices/<BDF>/anti_rollback_status 34 34 + Date: June 2022 35 35 + KernelVersion: 5.19 36 36 + Contact: mario.limonciello@amd.com 37 37 + Description: 38 38 + The /sys/bus/pci/devices/<BDF>/anti_rollback_status file reports 39 39 + whether the PSP is enforcing rollback protection. 40 40 + Possible values: 41 41 + 0: Not enforcing 42 42 + 1: Enforcing 43 43 + 44 44 + What: /sys/bus/pci/devices/<BDF>/rpmc_production_enabled 45 45 + Date: June 2022 46 46 + KernelVersion: 5.19 47 47 + Contact: mario.limonciello@amd.com 48 48 + Description: 49 49 + The /sys/bus/pci/devices/<BDF>/rpmc_production_enabled file reports 50 50 + whether Replay Protected Monotonic Counter support has been enabled. 51 51 + Possible values: 52 52 + 0: Not enabled 53 53 + 1: Enabled 54 54 + 55 55 + What: /sys/bus/pci/devices/<BDF>/rpmc_spirom_available 56 56 + Date: June 2022 57 57 + KernelVersion: 5.19 58 58 + Contact: mario.limonciello@amd.com 59 59 + Description: 60 60 + The /sys/bus/pci/devices/<BDF>/rpmc_spirom_available file reports 61 61 + whether an Replay Protected Monotonic Counter supported SPI is installed 62 62 + on the system. 63 63 + Possible values: 64 64 + 0: Not present 65 65 + 1: Present 66 66 + 67 67 + What: /sys/bus/pci/devices/<BDF>/hsp_tpm_available 68 68 + Date: June 2022 69 69 + KernelVersion: 5.19 70 70 + Contact: mario.limonciello@amd.com 71 71 + Description: 72 72 + The /sys/bus/pci/devices/<BDF>/hsp_tpm_available file reports 73 73 + whether the HSP TPM has been activated. 74 74 + Possible values: 75 75 + 0: Not activated or present 76 76 + 1: Activated 77 77 + 78 78 + What: /sys/bus/pci/devices/<BDF>/rom_armor_enforced 79 79 + Date: June 2022 80 80 + KernelVersion: 5.19 81 81 + Contact: mario.limonciello@amd.com 82 82 + Description: 83 83 + The /sys/bus/pci/devices/<BDF>/rom_armor_enforced file reports 84 84 + whether RomArmor SPI protection is enforced. 85 85 + Possible values: 86 86 + 0: Not enforced 87 87 + 1: Enforced

+1

Documentation/devicetree/bindings/crypto/ti,sa2ul.yaml

reviewed

··· 15 15 - ti,j721e-sa2ul 16 16 - ti,am654-sa2ul 17 17 - ti,am64-sa2ul 18 18 + - ti,am62-sa3ul 18 19 19 20 reg: 20 21 maxItems: 1

+3 -1

Documentation/devicetree/bindings/trivial-devices.yaml

reviewed

··· 47 47 - at,24c08 48 48 # i2c trusted platform module (TPM) 49 49 - atmel,at97sc3204t 50 50 - # i2c h/w symmetric crypto module 50 50 + # ATSHA204 - i2c h/w symmetric crypto module 51 51 + - atmel,atsha204 52 52 + # ATSHA204A - i2c h/w symmetric crypto module 51 53 - atmel,atsha204a 52 54 # i2c h/w elliptic curve crypto module 53 55 - atmel,atecc508a

+14 -2

arch/arm64/crypto/Kconfig

reviewed

··· 45 45 tristate "SM3 digest algorithm (ARMv8.2 Crypto Extensions)" 46 46 depends on KERNEL_MODE_NEON 47 47 select CRYPTO_HASH 48 48 - select CRYPTO_LIB_SM3 48 48 + select CRYPTO_SM3 49 49 50 50 config CRYPTO_SM4_ARM64_CE 51 51 tristate "SM4 symmetric cipher (ARMv8.2 Crypto Extensions)" 52 52 depends on KERNEL_MODE_NEON 53 53 select CRYPTO_ALGAPI 54 54 - select CRYPTO_LIB_SM4 54 54 + select CRYPTO_SM4 55 55 + 56 56 + config CRYPTO_SM4_ARM64_CE_BLK 57 57 + tristate "SM4 in ECB/CBC/CFB/CTR modes using ARMv8 Crypto Extensions" 58 58 + depends on KERNEL_MODE_NEON 59 59 + select CRYPTO_SKCIPHER 60 60 + select CRYPTO_SM4 61 61 + 62 62 + config CRYPTO_SM4_ARM64_NEON_BLK 63 63 + tristate "SM4 in ECB/CBC/CFB/CTR modes using NEON instructions" 64 64 + depends on KERNEL_MODE_NEON 65 65 + select CRYPTO_SKCIPHER 66 66 + select CRYPTO_SM4 55 67 56 68 config CRYPTO_GHASH_ARM64_CE 57 69 tristate "GHASH/AES-GCM using ARMv8 Crypto Extensions"

+7 -1

arch/arm64/crypto/Makefile

reviewed

··· 20 20 obj-$(CONFIG_CRYPTO_SM3_ARM64_CE) += sm3-ce.o 21 21 sm3-ce-y := sm3-ce-glue.o sm3-ce-core.o 22 22 23 23 - obj-$(CONFIG_CRYPTO_SM4_ARM64_CE) += sm4-ce.o 23 23 + obj-$(CONFIG_CRYPTO_SM4_ARM64_CE) += sm4-ce-cipher.o 24 24 + sm4-ce-cipher-y := sm4-ce-cipher-glue.o sm4-ce-cipher-core.o 25 25 + 26 26 + obj-$(CONFIG_CRYPTO_SM4_ARM64_CE_BLK) += sm4-ce.o 24 27 sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o 28 28 + 29 29 + obj-$(CONFIG_CRYPTO_SM4_ARM64_NEON_BLK) += sm4-neon.o 30 30 + sm4-neon-y := sm4-neon-glue.o sm4-neon-core.o 25 31 26 32 obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o 27 33 ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o

+36

arch/arm64/crypto/sm4-ce-cipher-core.S

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + 3 3 + #include <linux/linkage.h> 4 4 + #include <asm/assembler.h> 5 5 + 6 6 + .irp b, 0, 1, 2, 3, 4, 5, 6, 7, 8 7 7 + .set .Lv\b\().4s, \b 8 8 + .endr 9 9 + 10 10 + .macro sm4e, rd, rn 11 11 + .inst 0xcec08400 | .L\rd | (.L\rn << 5) 12 12 + .endm 13 13 + 14 14 + /* 15 15 + * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in); 16 16 + */ 17 17 + .text 18 18 + SYM_FUNC_START(sm4_ce_do_crypt) 19 19 + ld1 {v8.4s}, [x2] 20 20 + ld1 {v0.4s-v3.4s}, [x0], #64 21 21 + CPU_LE( rev32 v8.16b, v8.16b ) 22 22 + ld1 {v4.4s-v7.4s}, [x0] 23 23 + sm4e v8.4s, v0.4s 24 24 + sm4e v8.4s, v1.4s 25 25 + sm4e v8.4s, v2.4s 26 26 + sm4e v8.4s, v3.4s 27 27 + sm4e v8.4s, v4.4s 28 28 + sm4e v8.4s, v5.4s 29 29 + sm4e v8.4s, v6.4s 30 30 + sm4e v8.4s, v7.4s 31 31 + rev64 v8.4s, v8.4s 32 32 + ext v8.16b, v8.16b, v8.16b, #8 33 33 + CPU_LE( rev32 v8.16b, v8.16b ) 34 34 + st1 {v8.4s}, [x1] 35 35 + ret 36 36 + SYM_FUNC_END(sm4_ce_do_crypt)

+82

arch/arm64/crypto/sm4-ce-cipher-glue.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + 3 3 + #include <asm/neon.h> 4 4 + #include <asm/simd.h> 5 5 + #include <crypto/sm4.h> 6 6 + #include <crypto/internal/simd.h> 7 7 + #include <linux/module.h> 8 8 + #include <linux/cpufeature.h> 9 9 + #include <linux/crypto.h> 10 10 + #include <linux/types.h> 11 11 + 12 12 + MODULE_ALIAS_CRYPTO("sm4"); 13 13 + MODULE_ALIAS_CRYPTO("sm4-ce"); 14 14 + MODULE_DESCRIPTION("SM4 symmetric cipher using ARMv8 Crypto Extensions"); 15 15 + MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>"); 16 16 + MODULE_LICENSE("GPL v2"); 17 17 + 18 18 + asmlinkage void sm4_ce_do_crypt(const u32 *rk, void *out, const void *in); 19 19 + 20 20 + static int sm4_ce_setkey(struct crypto_tfm *tfm, const u8 *key, 21 21 + unsigned int key_len) 22 22 + { 23 23 + struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 24 24 + 25 25 + return sm4_expandkey(ctx, key, key_len); 26 26 + } 27 27 + 28 28 + static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) 29 29 + { 30 30 + const struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 31 31 + 32 32 + if (!crypto_simd_usable()) { 33 33 + sm4_crypt_block(ctx->rkey_enc, out, in); 34 34 + } else { 35 35 + kernel_neon_begin(); 36 36 + sm4_ce_do_crypt(ctx->rkey_enc, out, in); 37 37 + kernel_neon_end(); 38 38 + } 39 39 + } 40 40 + 41 41 + static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) 42 42 + { 43 43 + const struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 44 44 + 45 45 + if (!crypto_simd_usable()) { 46 46 + sm4_crypt_block(ctx->rkey_dec, out, in); 47 47 + } else { 48 48 + kernel_neon_begin(); 49 49 + sm4_ce_do_crypt(ctx->rkey_dec, out, in); 50 50 + kernel_neon_end(); 51 51 + } 52 52 + } 53 53 + 54 54 + static struct crypto_alg sm4_ce_alg = { 55 55 + .cra_name = "sm4", 56 56 + .cra_driver_name = "sm4-ce", 57 57 + .cra_priority = 300, 58 58 + .cra_flags = CRYPTO_ALG_TYPE_CIPHER, 59 59 + .cra_blocksize = SM4_BLOCK_SIZE, 60 60 + .cra_ctxsize = sizeof(struct sm4_ctx), 61 61 + .cra_module = THIS_MODULE, 62 62 + .cra_u.cipher = { 63 63 + .cia_min_keysize = SM4_KEY_SIZE, 64 64 + .cia_max_keysize = SM4_KEY_SIZE, 65 65 + .cia_setkey = sm4_ce_setkey, 66 66 + .cia_encrypt = sm4_ce_encrypt, 67 67 + .cia_decrypt = sm4_ce_decrypt 68 68 + } 69 69 + }; 70 70 + 71 71 + static int __init sm4_ce_mod_init(void) 72 72 + { 73 73 + return crypto_register_alg(&sm4_ce_alg); 74 74 + } 75 75 + 76 76 + static void __exit sm4_ce_mod_fini(void) 77 77 + { 78 78 + crypto_unregister_alg(&sm4_ce_alg); 79 79 + } 80 80 + 81 81 + module_cpu_feature_match(SM4, sm4_ce_mod_init); 82 82 + module_exit(sm4_ce_mod_fini);

+653 -29

arch/arm64/crypto/sm4-ce-core.S

reviewed

··· 1 1 - // SPDX-License-Identifier: GPL-2.0 1 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 2 + /* 3 3 + * SM4 Cipher Algorithm for ARMv8 with Crypto Extensions 4 4 + * as specified in 5 5 + * https://tools.ietf.org/id/draft-ribose-cfrg-sm4-10.html 6 6 + * 7 7 + * Copyright (C) 2022, Alibaba Group. 8 8 + * Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com> 9 9 + */ 2 10 3 11 #include <linux/linkage.h> 4 12 #include <asm/assembler.h> 5 13 6 6 - .irp b, 0, 1, 2, 3, 4, 5, 6, 7, 8 7 7 - .set .Lv\b\().4s, \b 8 8 - .endr 14 14 + .arch armv8-a+crypto 9 15 10 10 - .macro sm4e, rd, rn 11 11 - .inst 0xcec08400 | .L\rd | (.L\rn << 5) 12 12 - .endm 16 16 + .irp b, 0, 1, 2, 3, 4, 5, 6, 7, 16, 20, 24, 25, 26, 27, 28, 29, 30, 31 17 17 + .set .Lv\b\().4s, \b 18 18 + .endr 13 19 14 14 - /* 15 15 - * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in); 20 20 + .macro sm4e, vd, vn 21 21 + .inst 0xcec08400 | (.L\vn << 5) | .L\vd 22 22 + .endm 23 23 + 24 24 + .macro sm4ekey, vd, vn, vm 25 25 + .inst 0xce60c800 | (.L\vm << 16) | (.L\vn << 5) | .L\vd 26 26 + .endm 27 27 + 28 28 + /* Register macros */ 29 29 + 30 30 + #define RTMP0 v16 31 31 + #define RTMP1 v17 32 32 + #define RTMP2 v18 33 33 + #define RTMP3 v19 34 34 + 35 35 + #define RIV v20 36 36 + 37 37 + /* Helper macros. */ 38 38 + 39 39 + #define PREPARE \ 40 40 + ld1 {v24.16b-v27.16b}, [x0], #64; \ 41 41 + ld1 {v28.16b-v31.16b}, [x0]; 42 42 + 43 43 + #define SM4_CRYPT_BLK(b0) \ 44 44 + rev32 b0.16b, b0.16b; \ 45 45 + sm4e b0.4s, v24.4s; \ 46 46 + sm4e b0.4s, v25.4s; \ 47 47 + sm4e b0.4s, v26.4s; \ 48 48 + sm4e b0.4s, v27.4s; \ 49 49 + sm4e b0.4s, v28.4s; \ 50 50 + sm4e b0.4s, v29.4s; \ 51 51 + sm4e b0.4s, v30.4s; \ 52 52 + sm4e b0.4s, v31.4s; \ 53 53 + rev64 b0.4s, b0.4s; \ 54 54 + ext b0.16b, b0.16b, b0.16b, #8; \ 55 55 + rev32 b0.16b, b0.16b; 56 56 + 57 57 + #define SM4_CRYPT_BLK4(b0, b1, b2, b3) \ 58 58 + rev32 b0.16b, b0.16b; \ 59 59 + rev32 b1.16b, b1.16b; \ 60 60 + rev32 b2.16b, b2.16b; \ 61 61 + rev32 b3.16b, b3.16b; \ 62 62 + sm4e b0.4s, v24.4s; \ 63 63 + sm4e b1.4s, v24.4s; \ 64 64 + sm4e b2.4s, v24.4s; \ 65 65 + sm4e b3.4s, v24.4s; \ 66 66 + sm4e b0.4s, v25.4s; \ 67 67 + sm4e b1.4s, v25.4s; \ 68 68 + sm4e b2.4s, v25.4s; \ 69 69 + sm4e b3.4s, v25.4s; \ 70 70 + sm4e b0.4s, v26.4s; \ 71 71 + sm4e b1.4s, v26.4s; \ 72 72 + sm4e b2.4s, v26.4s; \ 73 73 + sm4e b3.4s, v26.4s; \ 74 74 + sm4e b0.4s, v27.4s; \ 75 75 + sm4e b1.4s, v27.4s; \ 76 76 + sm4e b2.4s, v27.4s; \ 77 77 + sm4e b3.4s, v27.4s; \ 78 78 + sm4e b0.4s, v28.4s; \ 79 79 + sm4e b1.4s, v28.4s; \ 80 80 + sm4e b2.4s, v28.4s; \ 81 81 + sm4e b3.4s, v28.4s; \ 82 82 + sm4e b0.4s, v29.4s; \ 83 83 + sm4e b1.4s, v29.4s; \ 84 84 + sm4e b2.4s, v29.4s; \ 85 85 + sm4e b3.4s, v29.4s; \ 86 86 + sm4e b0.4s, v30.4s; \ 87 87 + sm4e b1.4s, v30.4s; \ 88 88 + sm4e b2.4s, v30.4s; \ 89 89 + sm4e b3.4s, v30.4s; \ 90 90 + sm4e b0.4s, v31.4s; \ 91 91 + sm4e b1.4s, v31.4s; \ 92 92 + sm4e b2.4s, v31.4s; \ 93 93 + sm4e b3.4s, v31.4s; \ 94 94 + rev64 b0.4s, b0.4s; \ 95 95 + rev64 b1.4s, b1.4s; \ 96 96 + rev64 b2.4s, b2.4s; \ 97 97 + rev64 b3.4s, b3.4s; \ 98 98 + ext b0.16b, b0.16b, b0.16b, #8; \ 99 99 + ext b1.16b, b1.16b, b1.16b, #8; \ 100 100 + ext b2.16b, b2.16b, b2.16b, #8; \ 101 101 + ext b3.16b, b3.16b, b3.16b, #8; \ 102 102 + rev32 b0.16b, b0.16b; \ 103 103 + rev32 b1.16b, b1.16b; \ 104 104 + rev32 b2.16b, b2.16b; \ 105 105 + rev32 b3.16b, b3.16b; 106 106 + 107 107 + #define SM4_CRYPT_BLK8(b0, b1, b2, b3, b4, b5, b6, b7) \ 108 108 + rev32 b0.16b, b0.16b; \ 109 109 + rev32 b1.16b, b1.16b; \ 110 110 + rev32 b2.16b, b2.16b; \ 111 111 + rev32 b3.16b, b3.16b; \ 112 112 + rev32 b4.16b, b4.16b; \ 113 113 + rev32 b5.16b, b5.16b; \ 114 114 + rev32 b6.16b, b6.16b; \ 115 115 + rev32 b7.16b, b7.16b; \ 116 116 + sm4e b0.4s, v24.4s; \ 117 117 + sm4e b1.4s, v24.4s; \ 118 118 + sm4e b2.4s, v24.4s; \ 119 119 + sm4e b3.4s, v24.4s; \ 120 120 + sm4e b4.4s, v24.4s; \ 121 121 + sm4e b5.4s, v24.4s; \ 122 122 + sm4e b6.4s, v24.4s; \ 123 123 + sm4e b7.4s, v24.4s; \ 124 124 + sm4e b0.4s, v25.4s; \ 125 125 + sm4e b1.4s, v25.4s; \ 126 126 + sm4e b2.4s, v25.4s; \ 127 127 + sm4e b3.4s, v25.4s; \ 128 128 + sm4e b4.4s, v25.4s; \ 129 129 + sm4e b5.4s, v25.4s; \ 130 130 + sm4e b6.4s, v25.4s; \ 131 131 + sm4e b7.4s, v25.4s; \ 132 132 + sm4e b0.4s, v26.4s; \ 133 133 + sm4e b1.4s, v26.4s; \ 134 134 + sm4e b2.4s, v26.4s; \ 135 135 + sm4e b3.4s, v26.4s; \ 136 136 + sm4e b4.4s, v26.4s; \ 137 137 + sm4e b5.4s, v26.4s; \ 138 138 + sm4e b6.4s, v26.4s; \ 139 139 + sm4e b7.4s, v26.4s; \ 140 140 + sm4e b0.4s, v27.4s; \ 141 141 + sm4e b1.4s, v27.4s; \ 142 142 + sm4e b2.4s, v27.4s; \ 143 143 + sm4e b3.4s, v27.4s; \ 144 144 + sm4e b4.4s, v27.4s; \ 145 145 + sm4e b5.4s, v27.4s; \ 146 146 + sm4e b6.4s, v27.4s; \ 147 147 + sm4e b7.4s, v27.4s; \ 148 148 + sm4e b0.4s, v28.4s; \ 149 149 + sm4e b1.4s, v28.4s; \ 150 150 + sm4e b2.4s, v28.4s; \ 151 151 + sm4e b3.4s, v28.4s; \ 152 152 + sm4e b4.4s, v28.4s; \ 153 153 + sm4e b5.4s, v28.4s; \ 154 154 + sm4e b6.4s, v28.4s; \ 155 155 + sm4e b7.4s, v28.4s; \ 156 156 + sm4e b0.4s, v29.4s; \ 157 157 + sm4e b1.4s, v29.4s; \ 158 158 + sm4e b2.4s, v29.4s; \ 159 159 + sm4e b3.4s, v29.4s; \ 160 160 + sm4e b4.4s, v29.4s; \ 161 161 + sm4e b5.4s, v29.4s; \ 162 162 + sm4e b6.4s, v29.4s; \ 163 163 + sm4e b7.4s, v29.4s; \ 164 164 + sm4e b0.4s, v30.4s; \ 165 165 + sm4e b1.4s, v30.4s; \ 166 166 + sm4e b2.4s, v30.4s; \ 167 167 + sm4e b3.4s, v30.4s; \ 168 168 + sm4e b4.4s, v30.4s; \ 169 169 + sm4e b5.4s, v30.4s; \ 170 170 + sm4e b6.4s, v30.4s; \ 171 171 + sm4e b7.4s, v30.4s; \ 172 172 + sm4e b0.4s, v31.4s; \ 173 173 + sm4e b1.4s, v31.4s; \ 174 174 + sm4e b2.4s, v31.4s; \ 175 175 + sm4e b3.4s, v31.4s; \ 176 176 + sm4e b4.4s, v31.4s; \ 177 177 + sm4e b5.4s, v31.4s; \ 178 178 + sm4e b6.4s, v31.4s; \ 179 179 + sm4e b7.4s, v31.4s; \ 180 180 + rev64 b0.4s, b0.4s; \ 181 181 + rev64 b1.4s, b1.4s; \ 182 182 + rev64 b2.4s, b2.4s; \ 183 183 + rev64 b3.4s, b3.4s; \ 184 184 + rev64 b4.4s, b4.4s; \ 185 185 + rev64 b5.4s, b5.4s; \ 186 186 + rev64 b6.4s, b6.4s; \ 187 187 + rev64 b7.4s, b7.4s; \ 188 188 + ext b0.16b, b0.16b, b0.16b, #8; \ 189 189 + ext b1.16b, b1.16b, b1.16b, #8; \ 190 190 + ext b2.16b, b2.16b, b2.16b, #8; \ 191 191 + ext b3.16b, b3.16b, b3.16b, #8; \ 192 192 + ext b4.16b, b4.16b, b4.16b, #8; \ 193 193 + ext b5.16b, b5.16b, b5.16b, #8; \ 194 194 + ext b6.16b, b6.16b, b6.16b, #8; \ 195 195 + ext b7.16b, b7.16b, b7.16b, #8; \ 196 196 + rev32 b0.16b, b0.16b; \ 197 197 + rev32 b1.16b, b1.16b; \ 198 198 + rev32 b2.16b, b2.16b; \ 199 199 + rev32 b3.16b, b3.16b; \ 200 200 + rev32 b4.16b, b4.16b; \ 201 201 + rev32 b5.16b, b5.16b; \ 202 202 + rev32 b6.16b, b6.16b; \ 203 203 + rev32 b7.16b, b7.16b; 204 204 + 205 205 + 206 206 + .align 3 207 207 + SYM_FUNC_START(sm4_ce_expand_key) 208 208 + /* input: 209 209 + * x0: 128-bit key 210 210 + * x1: rkey_enc 211 211 + * x2: rkey_dec 212 212 + * x3: fk array 213 213 + * x4: ck array 16 214 */ 17 17 - .text 18 18 - SYM_FUNC_START(sm4_ce_do_crypt) 19 19 - ld1 {v8.4s}, [x2] 20 20 - ld1 {v0.4s-v3.4s}, [x0], #64 21 21 - CPU_LE( rev32 v8.16b, v8.16b ) 22 22 - ld1 {v4.4s-v7.4s}, [x0] 23 23 - sm4e v8.4s, v0.4s 24 24 - sm4e v8.4s, v1.4s 25 25 - sm4e v8.4s, v2.4s 26 26 - sm4e v8.4s, v3.4s 27 27 - sm4e v8.4s, v4.4s 28 28 - sm4e v8.4s, v5.4s 29 29 - sm4e v8.4s, v6.4s 30 30 - sm4e v8.4s, v7.4s 31 31 - rev64 v8.4s, v8.4s 32 32 - ext v8.16b, v8.16b, v8.16b, #8 33 33 - CPU_LE( rev32 v8.16b, v8.16b ) 34 34 - st1 {v8.4s}, [x1] 35 35 - ret 36 36 - SYM_FUNC_END(sm4_ce_do_crypt) 215 215 + ld1 {v0.16b}, [x0]; 216 216 + rev32 v0.16b, v0.16b; 217 217 + ld1 {v1.16b}, [x3]; 218 218 + /* load ck */ 219 219 + ld1 {v24.16b-v27.16b}, [x4], #64; 220 220 + ld1 {v28.16b-v31.16b}, [x4]; 221 221 + 222 222 + /* input ^ fk */ 223 223 + eor v0.16b, v0.16b, v1.16b; 224 224 + 225 225 + sm4ekey v0.4s, v0.4s, v24.4s; 226 226 + sm4ekey v1.4s, v0.4s, v25.4s; 227 227 + sm4ekey v2.4s, v1.4s, v26.4s; 228 228 + sm4ekey v3.4s, v2.4s, v27.4s; 229 229 + sm4ekey v4.4s, v3.4s, v28.4s; 230 230 + sm4ekey v5.4s, v4.4s, v29.4s; 231 231 + sm4ekey v6.4s, v5.4s, v30.4s; 232 232 + sm4ekey v7.4s, v6.4s, v31.4s; 233 233 + 234 234 + st1 {v0.16b-v3.16b}, [x1], #64; 235 235 + st1 {v4.16b-v7.16b}, [x1]; 236 236 + rev64 v7.4s, v7.4s; 237 237 + rev64 v6.4s, v6.4s; 238 238 + rev64 v5.4s, v5.4s; 239 239 + rev64 v4.4s, v4.4s; 240 240 + rev64 v3.4s, v3.4s; 241 241 + rev64 v2.4s, v2.4s; 242 242 + rev64 v1.4s, v1.4s; 243 243 + rev64 v0.4s, v0.4s; 244 244 + ext v7.16b, v7.16b, v7.16b, #8; 245 245 + ext v6.16b, v6.16b, v6.16b, #8; 246 246 + ext v5.16b, v5.16b, v5.16b, #8; 247 247 + ext v4.16b, v4.16b, v4.16b, #8; 248 248 + ext v3.16b, v3.16b, v3.16b, #8; 249 249 + ext v2.16b, v2.16b, v2.16b, #8; 250 250 + ext v1.16b, v1.16b, v1.16b, #8; 251 251 + ext v0.16b, v0.16b, v0.16b, #8; 252 252 + st1 {v7.16b}, [x2], #16; 253 253 + st1 {v6.16b}, [x2], #16; 254 254 + st1 {v5.16b}, [x2], #16; 255 255 + st1 {v4.16b}, [x2], #16; 256 256 + st1 {v3.16b}, [x2], #16; 257 257 + st1 {v2.16b}, [x2], #16; 258 258 + st1 {v1.16b}, [x2], #16; 259 259 + st1 {v0.16b}, [x2]; 260 260 + 261 261 + ret; 262 262 + SYM_FUNC_END(sm4_ce_expand_key) 263 263 + 264 264 + .align 3 265 265 + SYM_FUNC_START(sm4_ce_crypt_block) 266 266 + /* input: 267 267 + * x0: round key array, CTX 268 268 + * x1: dst 269 269 + * x2: src 270 270 + */ 271 271 + PREPARE; 272 272 + 273 273 + ld1 {v0.16b}, [x2]; 274 274 + SM4_CRYPT_BLK(v0); 275 275 + st1 {v0.16b}, [x1]; 276 276 + 277 277 + ret; 278 278 + SYM_FUNC_END(sm4_ce_crypt_block) 279 279 + 280 280 + .align 3 281 281 + SYM_FUNC_START(sm4_ce_crypt) 282 282 + /* input: 283 283 + * x0: round key array, CTX 284 284 + * x1: dst 285 285 + * x2: src 286 286 + * w3: nblocks 287 287 + */ 288 288 + PREPARE; 289 289 + 290 290 + .Lcrypt_loop_blk: 291 291 + sub w3, w3, #8; 292 292 + tbnz w3, #31, .Lcrypt_tail8; 293 293 + 294 294 + ld1 {v0.16b-v3.16b}, [x2], #64; 295 295 + ld1 {v4.16b-v7.16b}, [x2], #64; 296 296 + 297 297 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 298 298 + 299 299 + st1 {v0.16b-v3.16b}, [x1], #64; 300 300 + st1 {v4.16b-v7.16b}, [x1], #64; 301 301 + 302 302 + cbz w3, .Lcrypt_end; 303 303 + b .Lcrypt_loop_blk; 304 304 + 305 305 + .Lcrypt_tail8: 306 306 + add w3, w3, #8; 307 307 + cmp w3, #4; 308 308 + blt .Lcrypt_tail4; 309 309 + 310 310 + sub w3, w3, #4; 311 311 + 312 312 + ld1 {v0.16b-v3.16b}, [x2], #64; 313 313 + SM4_CRYPT_BLK4(v0, v1, v2, v3); 314 314 + st1 {v0.16b-v3.16b}, [x1], #64; 315 315 + 316 316 + cbz w3, .Lcrypt_end; 317 317 + 318 318 + .Lcrypt_tail4: 319 319 + sub w3, w3, #1; 320 320 + 321 321 + ld1 {v0.16b}, [x2], #16; 322 322 + SM4_CRYPT_BLK(v0); 323 323 + st1 {v0.16b}, [x1], #16; 324 324 + 325 325 + cbnz w3, .Lcrypt_tail4; 326 326 + 327 327 + .Lcrypt_end: 328 328 + ret; 329 329 + SYM_FUNC_END(sm4_ce_crypt) 330 330 + 331 331 + .align 3 332 332 + SYM_FUNC_START(sm4_ce_cbc_enc) 333 333 + /* input: 334 334 + * x0: round key array, CTX 335 335 + * x1: dst 336 336 + * x2: src 337 337 + * x3: iv (big endian, 128 bit) 338 338 + * w4: nblocks 339 339 + */ 340 340 + PREPARE; 341 341 + 342 342 + ld1 {RIV.16b}, [x3]; 343 343 + 344 344 + .Lcbc_enc_loop: 345 345 + sub w4, w4, #1; 346 346 + 347 347 + ld1 {RTMP0.16b}, [x2], #16; 348 348 + eor RIV.16b, RIV.16b, RTMP0.16b; 349 349 + 350 350 + SM4_CRYPT_BLK(RIV); 351 351 + 352 352 + st1 {RIV.16b}, [x1], #16; 353 353 + 354 354 + cbnz w4, .Lcbc_enc_loop; 355 355 + 356 356 + /* store new IV */ 357 357 + st1 {RIV.16b}, [x3]; 358 358 + 359 359 + ret; 360 360 + SYM_FUNC_END(sm4_ce_cbc_enc) 361 361 + 362 362 + .align 3 363 363 + SYM_FUNC_START(sm4_ce_cbc_dec) 364 364 + /* input: 365 365 + * x0: round key array, CTX 366 366 + * x1: dst 367 367 + * x2: src 368 368 + * x3: iv (big endian, 128 bit) 369 369 + * w4: nblocks 370 370 + */ 371 371 + PREPARE; 372 372 + 373 373 + ld1 {RIV.16b}, [x3]; 374 374 + 375 375 + .Lcbc_loop_blk: 376 376 + sub w4, w4, #8; 377 377 + tbnz w4, #31, .Lcbc_tail8; 378 378 + 379 379 + ld1 {v0.16b-v3.16b}, [x2], #64; 380 380 + ld1 {v4.16b-v7.16b}, [x2]; 381 381 + 382 382 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 383 383 + 384 384 + sub x2, x2, #64; 385 385 + eor v0.16b, v0.16b, RIV.16b; 386 386 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 387 387 + eor v1.16b, v1.16b, RTMP0.16b; 388 388 + eor v2.16b, v2.16b, RTMP1.16b; 389 389 + eor v3.16b, v3.16b, RTMP2.16b; 390 390 + st1 {v0.16b-v3.16b}, [x1], #64; 391 391 + 392 392 + eor v4.16b, v4.16b, RTMP3.16b; 393 393 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 394 394 + eor v5.16b, v5.16b, RTMP0.16b; 395 395 + eor v6.16b, v6.16b, RTMP1.16b; 396 396 + eor v7.16b, v7.16b, RTMP2.16b; 397 397 + 398 398 + mov RIV.16b, RTMP3.16b; 399 399 + st1 {v4.16b-v7.16b}, [x1], #64; 400 400 + 401 401 + cbz w4, .Lcbc_end; 402 402 + b .Lcbc_loop_blk; 403 403 + 404 404 + .Lcbc_tail8: 405 405 + add w4, w4, #8; 406 406 + cmp w4, #4; 407 407 + blt .Lcbc_tail4; 408 408 + 409 409 + sub w4, w4, #4; 410 410 + 411 411 + ld1 {v0.16b-v3.16b}, [x2]; 412 412 + 413 413 + SM4_CRYPT_BLK4(v0, v1, v2, v3); 414 414 + 415 415 + eor v0.16b, v0.16b, RIV.16b; 416 416 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 417 417 + eor v1.16b, v1.16b, RTMP0.16b; 418 418 + eor v2.16b, v2.16b, RTMP1.16b; 419 419 + eor v3.16b, v3.16b, RTMP2.16b; 420 420 + 421 421 + mov RIV.16b, RTMP3.16b; 422 422 + st1 {v0.16b-v3.16b}, [x1], #64; 423 423 + 424 424 + cbz w4, .Lcbc_end; 425 425 + 426 426 + .Lcbc_tail4: 427 427 + sub w4, w4, #1; 428 428 + 429 429 + ld1 {v0.16b}, [x2]; 430 430 + 431 431 + SM4_CRYPT_BLK(v0); 432 432 + 433 433 + eor v0.16b, v0.16b, RIV.16b; 434 434 + ld1 {RIV.16b}, [x2], #16; 435 435 + st1 {v0.16b}, [x1], #16; 436 436 + 437 437 + cbnz w4, .Lcbc_tail4; 438 438 + 439 439 + .Lcbc_end: 440 440 + /* store new IV */ 441 441 + st1 {RIV.16b}, [x3]; 442 442 + 443 443 + ret; 444 444 + SYM_FUNC_END(sm4_ce_cbc_dec) 445 445 + 446 446 + .align 3 447 447 + SYM_FUNC_START(sm4_ce_cfb_enc) 448 448 + /* input: 449 449 + * x0: round key array, CTX 450 450 + * x1: dst 451 451 + * x2: src 452 452 + * x3: iv (big endian, 128 bit) 453 453 + * w4: nblocks 454 454 + */ 455 455 + PREPARE; 456 456 + 457 457 + ld1 {RIV.16b}, [x3]; 458 458 + 459 459 + .Lcfb_enc_loop: 460 460 + sub w4, w4, #1; 461 461 + 462 462 + SM4_CRYPT_BLK(RIV); 463 463 + 464 464 + ld1 {RTMP0.16b}, [x2], #16; 465 465 + eor RIV.16b, RIV.16b, RTMP0.16b; 466 466 + st1 {RIV.16b}, [x1], #16; 467 467 + 468 468 + cbnz w4, .Lcfb_enc_loop; 469 469 + 470 470 + /* store new IV */ 471 471 + st1 {RIV.16b}, [x3]; 472 472 + 473 473 + ret; 474 474 + SYM_FUNC_END(sm4_ce_cfb_enc) 475 475 + 476 476 + .align 3 477 477 + SYM_FUNC_START(sm4_ce_cfb_dec) 478 478 + /* input: 479 479 + * x0: round key array, CTX 480 480 + * x1: dst 481 481 + * x2: src 482 482 + * x3: iv (big endian, 128 bit) 483 483 + * w4: nblocks 484 484 + */ 485 485 + PREPARE; 486 486 + 487 487 + ld1 {v0.16b}, [x3]; 488 488 + 489 489 + .Lcfb_loop_blk: 490 490 + sub w4, w4, #8; 491 491 + tbnz w4, #31, .Lcfb_tail8; 492 492 + 493 493 + ld1 {v1.16b, v2.16b, v3.16b}, [x2], #48; 494 494 + ld1 {v4.16b-v7.16b}, [x2]; 495 495 + 496 496 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 497 497 + 498 498 + sub x2, x2, #48; 499 499 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 500 500 + eor v0.16b, v0.16b, RTMP0.16b; 501 501 + eor v1.16b, v1.16b, RTMP1.16b; 502 502 + eor v2.16b, v2.16b, RTMP2.16b; 503 503 + eor v3.16b, v3.16b, RTMP3.16b; 504 504 + st1 {v0.16b-v3.16b}, [x1], #64; 505 505 + 506 506 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 507 507 + eor v4.16b, v4.16b, RTMP0.16b; 508 508 + eor v5.16b, v5.16b, RTMP1.16b; 509 509 + eor v6.16b, v6.16b, RTMP2.16b; 510 510 + eor v7.16b, v7.16b, RTMP3.16b; 511 511 + st1 {v4.16b-v7.16b}, [x1], #64; 512 512 + 513 513 + mov v0.16b, RTMP3.16b; 514 514 + 515 515 + cbz w4, .Lcfb_end; 516 516 + b .Lcfb_loop_blk; 517 517 + 518 518 + .Lcfb_tail8: 519 519 + add w4, w4, #8; 520 520 + cmp w4, #4; 521 521 + blt .Lcfb_tail4; 522 522 + 523 523 + sub w4, w4, #4; 524 524 + 525 525 + ld1 {v1.16b, v2.16b, v3.16b}, [x2]; 526 526 + 527 527 + SM4_CRYPT_BLK4(v0, v1, v2, v3); 528 528 + 529 529 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 530 530 + eor v0.16b, v0.16b, RTMP0.16b; 531 531 + eor v1.16b, v1.16b, RTMP1.16b; 532 532 + eor v2.16b, v2.16b, RTMP2.16b; 533 533 + eor v3.16b, v3.16b, RTMP3.16b; 534 534 + st1 {v0.16b-v3.16b}, [x1], #64; 535 535 + 536 536 + mov v0.16b, RTMP3.16b; 537 537 + 538 538 + cbz w4, .Lcfb_end; 539 539 + 540 540 + .Lcfb_tail4: 541 541 + sub w4, w4, #1; 542 542 + 543 543 + SM4_CRYPT_BLK(v0); 544 544 + 545 545 + ld1 {RTMP0.16b}, [x2], #16; 546 546 + eor v0.16b, v0.16b, RTMP0.16b; 547 547 + st1 {v0.16b}, [x1], #16; 548 548 + 549 549 + mov v0.16b, RTMP0.16b; 550 550 + 551 551 + cbnz w4, .Lcfb_tail4; 552 552 + 553 553 + .Lcfb_end: 554 554 + /* store new IV */ 555 555 + st1 {v0.16b}, [x3]; 556 556 + 557 557 + ret; 558 558 + SYM_FUNC_END(sm4_ce_cfb_dec) 559 559 + 560 560 + .align 3 561 561 + SYM_FUNC_START(sm4_ce_ctr_enc) 562 562 + /* input: 563 563 + * x0: round key array, CTX 564 564 + * x1: dst 565 565 + * x2: src 566 566 + * x3: ctr (big endian, 128 bit) 567 567 + * w4: nblocks 568 568 + */ 569 569 + PREPARE; 570 570 + 571 571 + ldp x7, x8, [x3]; 572 572 + rev x7, x7; 573 573 + rev x8, x8; 574 574 + 575 575 + .Lctr_loop_blk: 576 576 + sub w4, w4, #8; 577 577 + tbnz w4, #31, .Lctr_tail8; 578 578 + 579 579 + #define inc_le128(vctr) \ 580 580 + mov vctr.d[1], x8; \ 581 581 + mov vctr.d[0], x7; \ 582 582 + adds x8, x8, #1; \ 583 583 + adc x7, x7, xzr; \ 584 584 + rev64 vctr.16b, vctr.16b; 585 585 + 586 586 + /* construct CTRs */ 587 587 + inc_le128(v0); /* +0 */ 588 588 + inc_le128(v1); /* +1 */ 589 589 + inc_le128(v2); /* +2 */ 590 590 + inc_le128(v3); /* +3 */ 591 591 + inc_le128(v4); /* +4 */ 592 592 + inc_le128(v5); /* +5 */ 593 593 + inc_le128(v6); /* +6 */ 594 594 + inc_le128(v7); /* +7 */ 595 595 + 596 596 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 597 597 + 598 598 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 599 599 + eor v0.16b, v0.16b, RTMP0.16b; 600 600 + eor v1.16b, v1.16b, RTMP1.16b; 601 601 + eor v2.16b, v2.16b, RTMP2.16b; 602 602 + eor v3.16b, v3.16b, RTMP3.16b; 603 603 + st1 {v0.16b-v3.16b}, [x1], #64; 604 604 + 605 605 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 606 606 + eor v4.16b, v4.16b, RTMP0.16b; 607 607 + eor v5.16b, v5.16b, RTMP1.16b; 608 608 + eor v6.16b, v6.16b, RTMP2.16b; 609 609 + eor v7.16b, v7.16b, RTMP3.16b; 610 610 + st1 {v4.16b-v7.16b}, [x1], #64; 611 611 + 612 612 + cbz w4, .Lctr_end; 613 613 + b .Lctr_loop_blk; 614 614 + 615 615 + .Lctr_tail8: 616 616 + add w4, w4, #8; 617 617 + cmp w4, #4; 618 618 + blt .Lctr_tail4; 619 619 + 620 620 + sub w4, w4, #4; 621 621 + 622 622 + /* construct CTRs */ 623 623 + inc_le128(v0); /* +0 */ 624 624 + inc_le128(v1); /* +1 */ 625 625 + inc_le128(v2); /* +2 */ 626 626 + inc_le128(v3); /* +3 */ 627 627 + 628 628 + SM4_CRYPT_BLK4(v0, v1, v2, v3); 629 629 + 630 630 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 631 631 + eor v0.16b, v0.16b, RTMP0.16b; 632 632 + eor v1.16b, v1.16b, RTMP1.16b; 633 633 + eor v2.16b, v2.16b, RTMP2.16b; 634 634 + eor v3.16b, v3.16b, RTMP3.16b; 635 635 + st1 {v0.16b-v3.16b}, [x1], #64; 636 636 + 637 637 + cbz w4, .Lctr_end; 638 638 + 639 639 + .Lctr_tail4: 640 640 + sub w4, w4, #1; 641 641 + 642 642 + /* construct CTRs */ 643 643 + inc_le128(v0); 644 644 + 645 645 + SM4_CRYPT_BLK(v0); 646 646 + 647 647 + ld1 {RTMP0.16b}, [x2], #16; 648 648 + eor v0.16b, v0.16b, RTMP0.16b; 649 649 + st1 {v0.16b}, [x1], #16; 650 650 + 651 651 + cbnz w4, .Lctr_tail4; 652 652 + 653 653 + .Lctr_end: 654 654 + /* store new CTR */ 655 655 + rev x7, x7; 656 656 + rev x8, x8; 657 657 + stp x7, x8, [x3]; 658 658 + 659 659 + ret; 660 660 + SYM_FUNC_END(sm4_ce_ctr_enc)

+341 -51

arch/arm64/crypto/sm4-ce-glue.c

reviewed

··· 1 1 - // SPDX-License-Identifier: GPL-2.0 1 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 2 + /* 3 3 + * SM4 Cipher Algorithm, using ARMv8 Crypto Extensions 4 4 + * as specified in 5 5 + * https://tools.ietf.org/id/draft-ribose-cfrg-sm4-10.html 6 6 + * 7 7 + * Copyright (C) 2022, Alibaba Group. 8 8 + * Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com> 9 9 + */ 2 10 11 11 + #include <linux/module.h> 12 12 + #include <linux/crypto.h> 13 13 + #include <linux/kernel.h> 14 14 + #include <linux/cpufeature.h> 3 15 #include <asm/neon.h> 4 16 #include <asm/simd.h> 5 5 - #include <crypto/sm4.h> 6 17 #include <crypto/internal/simd.h> 7 7 - #include <linux/module.h> 8 8 - #include <linux/cpufeature.h> 9 9 - #include <linux/crypto.h> 10 10 - #include <linux/types.h> 18 18 + #include <crypto/internal/skcipher.h> 19 19 + #include <crypto/sm4.h> 11 20 12 12 - MODULE_ALIAS_CRYPTO("sm4"); 13 13 - MODULE_ALIAS_CRYPTO("sm4-ce"); 14 14 - MODULE_DESCRIPTION("SM4 symmetric cipher using ARMv8 Crypto Extensions"); 15 15 - MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>"); 16 16 - MODULE_LICENSE("GPL v2"); 21 21 + #define BYTES2BLKS(nbytes) ((nbytes) >> 4) 17 22 18 18 - asmlinkage void sm4_ce_do_crypt(const u32 *rk, void *out, const void *in); 23 23 + asmlinkage void sm4_ce_expand_key(const u8 *key, u32 *rkey_enc, u32 *rkey_dec, 24 24 + const u32 *fk, const u32 *ck); 25 25 + asmlinkage void sm4_ce_crypt_block(const u32 *rkey, u8 *dst, const u8 *src); 26 26 + asmlinkage void sm4_ce_crypt(const u32 *rkey, u8 *dst, const u8 *src, 27 27 + unsigned int nblks); 28 28 + asmlinkage void sm4_ce_cbc_enc(const u32 *rkey, u8 *dst, const u8 *src, 29 29 + u8 *iv, unsigned int nblks); 30 30 + asmlinkage void sm4_ce_cbc_dec(const u32 *rkey, u8 *dst, const u8 *src, 31 31 + u8 *iv, unsigned int nblks); 32 32 + asmlinkage void sm4_ce_cfb_enc(const u32 *rkey, u8 *dst, const u8 *src, 33 33 + u8 *iv, unsigned int nblks); 34 34 + asmlinkage void sm4_ce_cfb_dec(const u32 *rkey, u8 *dst, const u8 *src, 35 35 + u8 *iv, unsigned int nblks); 36 36 + asmlinkage void sm4_ce_ctr_enc(const u32 *rkey, u8 *dst, const u8 *src, 37 37 + u8 *iv, unsigned int nblks); 19 38 20 20 - static int sm4_ce_setkey(struct crypto_tfm *tfm, const u8 *key, 21 21 - unsigned int key_len) 39 39 + static int sm4_setkey(struct crypto_skcipher *tfm, const u8 *key, 40 40 + unsigned int key_len) 22 41 { 23 23 - struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 42 42 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 24 43 25 25 - return sm4_expandkey(ctx, key, key_len); 44 44 + if (key_len != SM4_KEY_SIZE) 45 45 + return -EINVAL; 46 46 + 47 47 + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, 48 48 + crypto_sm4_fk, crypto_sm4_ck); 49 49 + return 0; 26 50 } 27 51 28 28 - static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) 52 52 + static int sm4_ecb_do_crypt(struct skcipher_request *req, const u32 *rkey) 29 53 { 30 30 - const struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 54 54 + struct skcipher_walk walk; 55 55 + unsigned int nbytes; 56 56 + int err; 31 57 32 32 - if (!crypto_simd_usable()) { 33 33 - sm4_crypt_block(ctx->rkey_enc, out, in); 34 34 - } else { 58 58 + err = skcipher_walk_virt(&walk, req, false); 59 59 + 60 60 + while ((nbytes = walk.nbytes) > 0) { 61 61 + const u8 *src = walk.src.virt.addr; 62 62 + u8 *dst = walk.dst.virt.addr; 63 63 + unsigned int nblks; 64 64 + 35 65 kernel_neon_begin(); 36 36 - sm4_ce_do_crypt(ctx->rkey_enc, out, in); 66 66 + 67 67 + nblks = BYTES2BLKS(nbytes); 68 68 + if (nblks) { 69 69 + sm4_ce_crypt(rkey, dst, src, nblks); 70 70 + nbytes -= nblks * SM4_BLOCK_SIZE; 71 71 + } 72 72 + 37 73 kernel_neon_end(); 74 74 + 75 75 + err = skcipher_walk_done(&walk, nbytes); 38 76 } 77 77 + 78 78 + return err; 39 79 } 40 80 41 41 - static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) 81 81 + static int sm4_ecb_encrypt(struct skcipher_request *req) 42 82 { 43 43 - const struct sm4_ctx *ctx = crypto_tfm_ctx(tfm); 83 83 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 84 84 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 44 85 45 45 - if (!crypto_simd_usable()) { 46 46 - sm4_crypt_block(ctx->rkey_dec, out, in); 47 47 - } else { 48 48 - kernel_neon_begin(); 49 49 - sm4_ce_do_crypt(ctx->rkey_dec, out, in); 50 50 - kernel_neon_end(); 51 51 - } 86 86 + return sm4_ecb_do_crypt(req, ctx->rkey_enc); 52 87 } 53 88 54 54 - static struct crypto_alg sm4_ce_alg = { 55 55 - .cra_name = "sm4", 56 56 - .cra_driver_name = "sm4-ce", 57 57 - .cra_priority = 200, 58 58 - .cra_flags = CRYPTO_ALG_TYPE_CIPHER, 59 59 - .cra_blocksize = SM4_BLOCK_SIZE, 60 60 - .cra_ctxsize = sizeof(struct sm4_ctx), 61 61 - .cra_module = THIS_MODULE, 62 62 - .cra_u.cipher = { 63 63 - .cia_min_keysize = SM4_KEY_SIZE, 64 64 - .cia_max_keysize = SM4_KEY_SIZE, 65 65 - .cia_setkey = sm4_ce_setkey, 66 66 - .cia_encrypt = sm4_ce_encrypt, 67 67 - .cia_decrypt = sm4_ce_decrypt 89 89 + static int sm4_ecb_decrypt(struct skcipher_request *req) 90 90 + { 91 91 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 92 92 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 93 93 + 94 94 + return sm4_ecb_do_crypt(req, ctx->rkey_dec); 95 95 + } 96 96 + 97 97 + static int sm4_cbc_encrypt(struct skcipher_request *req) 98 98 + { 99 99 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 100 100 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 101 101 + struct skcipher_walk walk; 102 102 + unsigned int nbytes; 103 103 + int err; 104 104 + 105 105 + err = skcipher_walk_virt(&walk, req, false); 106 106 + 107 107 + while ((nbytes = walk.nbytes) > 0) { 108 108 + const u8 *src = walk.src.virt.addr; 109 109 + u8 *dst = walk.dst.virt.addr; 110 110 + unsigned int nblks; 111 111 + 112 112 + kernel_neon_begin(); 113 113 + 114 114 + nblks = BYTES2BLKS(nbytes); 115 115 + if (nblks) { 116 116 + sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); 117 117 + nbytes -= nblks * SM4_BLOCK_SIZE; 118 118 + } 119 119 + 120 120 + kernel_neon_end(); 121 121 + 122 122 + err = skcipher_walk_done(&walk, nbytes); 123 123 + } 124 124 + 125 125 + return err; 126 126 + } 127 127 + 128 128 + static int sm4_cbc_decrypt(struct skcipher_request *req) 129 129 + { 130 130 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 131 131 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 132 132 + struct skcipher_walk walk; 133 133 + unsigned int nbytes; 134 134 + int err; 135 135 + 136 136 + err = skcipher_walk_virt(&walk, req, false); 137 137 + 138 138 + while ((nbytes = walk.nbytes) > 0) { 139 139 + const u8 *src = walk.src.virt.addr; 140 140 + u8 *dst = walk.dst.virt.addr; 141 141 + unsigned int nblks; 142 142 + 143 143 + kernel_neon_begin(); 144 144 + 145 145 + nblks = BYTES2BLKS(nbytes); 146 146 + if (nblks) { 147 147 + sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, walk.iv, nblks); 148 148 + nbytes -= nblks * SM4_BLOCK_SIZE; 149 149 + } 150 150 + 151 151 + kernel_neon_end(); 152 152 + 153 153 + err = skcipher_walk_done(&walk, nbytes); 154 154 + } 155 155 + 156 156 + return err; 157 157 + } 158 158 + 159 159 + static int sm4_cfb_encrypt(struct skcipher_request *req) 160 160 + { 161 161 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 162 162 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 163 163 + struct skcipher_walk walk; 164 164 + unsigned int nbytes; 165 165 + int err; 166 166 + 167 167 + err = skcipher_walk_virt(&walk, req, false); 168 168 + 169 169 + while ((nbytes = walk.nbytes) > 0) { 170 170 + const u8 *src = walk.src.virt.addr; 171 171 + u8 *dst = walk.dst.virt.addr; 172 172 + unsigned int nblks; 173 173 + 174 174 + kernel_neon_begin(); 175 175 + 176 176 + nblks = BYTES2BLKS(nbytes); 177 177 + if (nblks) { 178 178 + sm4_ce_cfb_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); 179 179 + dst += nblks * SM4_BLOCK_SIZE; 180 180 + src += nblks * SM4_BLOCK_SIZE; 181 181 + nbytes -= nblks * SM4_BLOCK_SIZE; 182 182 + } 183 183 + 184 184 + /* tail */ 185 185 + if (walk.nbytes == walk.total && nbytes > 0) { 186 186 + u8 keystream[SM4_BLOCK_SIZE]; 187 187 + 188 188 + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); 189 189 + crypto_xor_cpy(dst, src, keystream, nbytes); 190 190 + nbytes = 0; 191 191 + } 192 192 + 193 193 + kernel_neon_end(); 194 194 + 195 195 + err = skcipher_walk_done(&walk, nbytes); 196 196 + } 197 197 + 198 198 + return err; 199 199 + } 200 200 + 201 201 + static int sm4_cfb_decrypt(struct skcipher_request *req) 202 202 + { 203 203 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 204 204 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 205 205 + struct skcipher_walk walk; 206 206 + unsigned int nbytes; 207 207 + int err; 208 208 + 209 209 + err = skcipher_walk_virt(&walk, req, false); 210 210 + 211 211 + while ((nbytes = walk.nbytes) > 0) { 212 212 + const u8 *src = walk.src.virt.addr; 213 213 + u8 *dst = walk.dst.virt.addr; 214 214 + unsigned int nblks; 215 215 + 216 216 + kernel_neon_begin(); 217 217 + 218 218 + nblks = BYTES2BLKS(nbytes); 219 219 + if (nblks) { 220 220 + sm4_ce_cfb_dec(ctx->rkey_enc, dst, src, walk.iv, nblks); 221 221 + dst += nblks * SM4_BLOCK_SIZE; 222 222 + src += nblks * SM4_BLOCK_SIZE; 223 223 + nbytes -= nblks * SM4_BLOCK_SIZE; 224 224 + } 225 225 + 226 226 + /* tail */ 227 227 + if (walk.nbytes == walk.total && nbytes > 0) { 228 228 + u8 keystream[SM4_BLOCK_SIZE]; 229 229 + 230 230 + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); 231 231 + crypto_xor_cpy(dst, src, keystream, nbytes); 232 232 + nbytes = 0; 233 233 + } 234 234 + 235 235 + kernel_neon_end(); 236 236 + 237 237 + err = skcipher_walk_done(&walk, nbytes); 238 238 + } 239 239 + 240 240 + return err; 241 241 + } 242 242 + 243 243 + static int sm4_ctr_crypt(struct skcipher_request *req) 244 244 + { 245 245 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 246 246 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 247 247 + struct skcipher_walk walk; 248 248 + unsigned int nbytes; 249 249 + int err; 250 250 + 251 251 + err = skcipher_walk_virt(&walk, req, false); 252 252 + 253 253 + while ((nbytes = walk.nbytes) > 0) { 254 254 + const u8 *src = walk.src.virt.addr; 255 255 + u8 *dst = walk.dst.virt.addr; 256 256 + unsigned int nblks; 257 257 + 258 258 + kernel_neon_begin(); 259 259 + 260 260 + nblks = BYTES2BLKS(nbytes); 261 261 + if (nblks) { 262 262 + sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); 263 263 + dst += nblks * SM4_BLOCK_SIZE; 264 264 + src += nblks * SM4_BLOCK_SIZE; 265 265 + nbytes -= nblks * SM4_BLOCK_SIZE; 266 266 + } 267 267 + 268 268 + /* tail */ 269 269 + if (walk.nbytes == walk.total && nbytes > 0) { 270 270 + u8 keystream[SM4_BLOCK_SIZE]; 271 271 + 272 272 + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); 273 273 + crypto_inc(walk.iv, SM4_BLOCK_SIZE); 274 274 + crypto_xor_cpy(dst, src, keystream, nbytes); 275 275 + nbytes = 0; 276 276 + } 277 277 + 278 278 + kernel_neon_end(); 279 279 + 280 280 + err = skcipher_walk_done(&walk, nbytes); 281 281 + } 282 282 + 283 283 + return err; 284 284 + } 285 285 + 286 286 + static struct skcipher_alg sm4_algs[] = { 287 287 + { 288 288 + .base = { 289 289 + .cra_name = "ecb(sm4)", 290 290 + .cra_driver_name = "ecb-sm4-ce", 291 291 + .cra_priority = 400, 292 292 + .cra_blocksize = SM4_BLOCK_SIZE, 293 293 + .cra_ctxsize = sizeof(struct sm4_ctx), 294 294 + .cra_module = THIS_MODULE, 295 295 + }, 296 296 + .min_keysize = SM4_KEY_SIZE, 297 297 + .max_keysize = SM4_KEY_SIZE, 298 298 + .setkey = sm4_setkey, 299 299 + .encrypt = sm4_ecb_encrypt, 300 300 + .decrypt = sm4_ecb_decrypt, 301 301 + }, { 302 302 + .base = { 303 303 + .cra_name = "cbc(sm4)", 304 304 + .cra_driver_name = "cbc-sm4-ce", 305 305 + .cra_priority = 400, 306 306 + .cra_blocksize = SM4_BLOCK_SIZE, 307 307 + .cra_ctxsize = sizeof(struct sm4_ctx), 308 308 + .cra_module = THIS_MODULE, 309 309 + }, 310 310 + .min_keysize = SM4_KEY_SIZE, 311 311 + .max_keysize = SM4_KEY_SIZE, 312 312 + .ivsize = SM4_BLOCK_SIZE, 313 313 + .setkey = sm4_setkey, 314 314 + .encrypt = sm4_cbc_encrypt, 315 315 + .decrypt = sm4_cbc_decrypt, 316 316 + }, { 317 317 + .base = { 318 318 + .cra_name = "cfb(sm4)", 319 319 + .cra_driver_name = "cfb-sm4-ce", 320 320 + .cra_priority = 400, 321 321 + .cra_blocksize = 1, 322 322 + .cra_ctxsize = sizeof(struct sm4_ctx), 323 323 + .cra_module = THIS_MODULE, 324 324 + }, 325 325 + .min_keysize = SM4_KEY_SIZE, 326 326 + .max_keysize = SM4_KEY_SIZE, 327 327 + .ivsize = SM4_BLOCK_SIZE, 328 328 + .chunksize = SM4_BLOCK_SIZE, 329 329 + .setkey = sm4_setkey, 330 330 + .encrypt = sm4_cfb_encrypt, 331 331 + .decrypt = sm4_cfb_decrypt, 332 332 + }, { 333 333 + .base = { 334 334 + .cra_name = "ctr(sm4)", 335 335 + .cra_driver_name = "ctr-sm4-ce", 336 336 + .cra_priority = 400, 337 337 + .cra_blocksize = 1, 338 338 + .cra_ctxsize = sizeof(struct sm4_ctx), 339 339 + .cra_module = THIS_MODULE, 340 340 + }, 341 341 + .min_keysize = SM4_KEY_SIZE, 342 342 + .max_keysize = SM4_KEY_SIZE, 343 343 + .ivsize = SM4_BLOCK_SIZE, 344 344 + .chunksize = SM4_BLOCK_SIZE, 345 345 + .setkey = sm4_setkey, 346 346 + .encrypt = sm4_ctr_crypt, 347 347 + .decrypt = sm4_ctr_crypt, 68 348 } 69 349 }; 70 350 71 71 - static int __init sm4_ce_mod_init(void) 351 351 + static int __init sm4_init(void) 72 352 { 73 73 - return crypto_register_alg(&sm4_ce_alg); 353 353 + return crypto_register_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs)); 74 354 } 75 355 76 76 - static void __exit sm4_ce_mod_fini(void) 356 356 + static void __exit sm4_exit(void) 77 357 { 78 78 - crypto_unregister_alg(&sm4_ce_alg); 358 358 + crypto_unregister_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs)); 79 359 } 80 360 81 81 - module_cpu_feature_match(SM4, sm4_ce_mod_init); 82 82 - module_exit(sm4_ce_mod_fini); 361 361 + module_cpu_feature_match(SM4, sm4_init); 362 362 + module_exit(sm4_exit); 363 363 + 364 364 + MODULE_DESCRIPTION("SM4 ECB/CBC/CFB/CTR using ARMv8 Crypto Extensions"); 365 365 + MODULE_ALIAS_CRYPTO("sm4-ce"); 366 366 + MODULE_ALIAS_CRYPTO("sm4"); 367 367 + MODULE_ALIAS_CRYPTO("ecb(sm4)"); 368 368 + MODULE_ALIAS_CRYPTO("cbc(sm4)"); 369 369 + MODULE_ALIAS_CRYPTO("cfb(sm4)"); 370 370 + MODULE_ALIAS_CRYPTO("ctr(sm4)"); 371 371 + MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>"); 372 372 + MODULE_LICENSE("GPL v2");

+487

arch/arm64/crypto/sm4-neon-core.S

reviewed

··· 1 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 2 + /* 3 3 + * SM4 Cipher Algorithm for ARMv8 NEON 4 4 + * as specified in 5 5 + * https://tools.ietf.org/id/draft-ribose-cfrg-sm4-10.html 6 6 + * 7 7 + * Copyright (C) 2022, Alibaba Group. 8 8 + * Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com> 9 9 + */ 10 10 + 11 11 + #include <linux/linkage.h> 12 12 + #include <asm/assembler.h> 13 13 + 14 14 + /* Register macros */ 15 15 + 16 16 + #define RTMP0 v8 17 17 + #define RTMP1 v9 18 18 + #define RTMP2 v10 19 19 + #define RTMP3 v11 20 20 + 21 21 + #define RX0 v12 22 22 + #define RX1 v13 23 23 + #define RKEY v14 24 24 + #define RIV v15 25 25 + 26 26 + /* Helper macros. */ 27 27 + 28 28 + #define PREPARE \ 29 29 + adr_l x5, crypto_sm4_sbox; \ 30 30 + ld1 {v16.16b-v19.16b}, [x5], #64; \ 31 31 + ld1 {v20.16b-v23.16b}, [x5], #64; \ 32 32 + ld1 {v24.16b-v27.16b}, [x5], #64; \ 33 33 + ld1 {v28.16b-v31.16b}, [x5]; 34 34 + 35 35 + #define transpose_4x4(s0, s1, s2, s3) \ 36 36 + zip1 RTMP0.4s, s0.4s, s1.4s; \ 37 37 + zip1 RTMP1.4s, s2.4s, s3.4s; \ 38 38 + zip2 RTMP2.4s, s0.4s, s1.4s; \ 39 39 + zip2 RTMP3.4s, s2.4s, s3.4s; \ 40 40 + zip1 s0.2d, RTMP0.2d, RTMP1.2d; \ 41 41 + zip2 s1.2d, RTMP0.2d, RTMP1.2d; \ 42 42 + zip1 s2.2d, RTMP2.2d, RTMP3.2d; \ 43 43 + zip2 s3.2d, RTMP2.2d, RTMP3.2d; 44 44 + 45 45 + #define rotate_clockwise_90(s0, s1, s2, s3) \ 46 46 + zip1 RTMP0.4s, s1.4s, s0.4s; \ 47 47 + zip2 RTMP1.4s, s1.4s, s0.4s; \ 48 48 + zip1 RTMP2.4s, s3.4s, s2.4s; \ 49 49 + zip2 RTMP3.4s, s3.4s, s2.4s; \ 50 50 + zip1 s0.2d, RTMP2.2d, RTMP0.2d; \ 51 51 + zip2 s1.2d, RTMP2.2d, RTMP0.2d; \ 52 52 + zip1 s2.2d, RTMP3.2d, RTMP1.2d; \ 53 53 + zip2 s3.2d, RTMP3.2d, RTMP1.2d; 54 54 + 55 55 + #define ROUND4(round, s0, s1, s2, s3) \ 56 56 + dup RX0.4s, RKEY.s[round]; \ 57 57 + /* rk ^ s1 ^ s2 ^ s3 */ \ 58 58 + eor RTMP1.16b, s2.16b, s3.16b; \ 59 59 + eor RX0.16b, RX0.16b, s1.16b; \ 60 60 + eor RX0.16b, RX0.16b, RTMP1.16b; \ 61 61 + \ 62 62 + /* sbox, non-linear part */ \ 63 63 + movi RTMP3.16b, #64; /* sizeof(sbox) / 4 */ \ 64 64 + tbl RTMP0.16b, {v16.16b-v19.16b}, RX0.16b; \ 65 65 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 66 66 + tbx RTMP0.16b, {v20.16b-v23.16b}, RX0.16b; \ 67 67 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 68 68 + tbx RTMP0.16b, {v24.16b-v27.16b}, RX0.16b; \ 69 69 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 70 70 + tbx RTMP0.16b, {v28.16b-v31.16b}, RX0.16b; \ 71 71 + \ 72 72 + /* linear part */ \ 73 73 + shl RTMP1.4s, RTMP0.4s, #8; \ 74 74 + shl RTMP2.4s, RTMP0.4s, #16; \ 75 75 + shl RTMP3.4s, RTMP0.4s, #24; \ 76 76 + sri RTMP1.4s, RTMP0.4s, #(32-8); \ 77 77 + sri RTMP2.4s, RTMP0.4s, #(32-16); \ 78 78 + sri RTMP3.4s, RTMP0.4s, #(32-24); \ 79 79 + /* RTMP1 = x ^ rol32(x, 8) ^ rol32(x, 16) */ \ 80 80 + eor RTMP1.16b, RTMP1.16b, RTMP0.16b; \ 81 81 + eor RTMP1.16b, RTMP1.16b, RTMP2.16b; \ 82 82 + /* RTMP3 = x ^ rol32(x, 24) ^ rol32(RTMP1, 2) */ \ 83 83 + eor RTMP3.16b, RTMP3.16b, RTMP0.16b; \ 84 84 + shl RTMP2.4s, RTMP1.4s, 2; \ 85 85 + sri RTMP2.4s, RTMP1.4s, #(32-2); \ 86 86 + eor RTMP3.16b, RTMP3.16b, RTMP2.16b; \ 87 87 + /* s0 ^= RTMP3 */ \ 88 88 + eor s0.16b, s0.16b, RTMP3.16b; 89 89 + 90 90 + #define SM4_CRYPT_BLK4(b0, b1, b2, b3) \ 91 91 + rev32 b0.16b, b0.16b; \ 92 92 + rev32 b1.16b, b1.16b; \ 93 93 + rev32 b2.16b, b2.16b; \ 94 94 + rev32 b3.16b, b3.16b; \ 95 95 + \ 96 96 + transpose_4x4(b0, b1, b2, b3); \ 97 97 + \ 98 98 + mov x6, 8; \ 99 99 + 4: \ 100 100 + ld1 {RKEY.4s}, [x0], #16; \ 101 101 + subs x6, x6, #1; \ 102 102 + \ 103 103 + ROUND4(0, b0, b1, b2, b3); \ 104 104 + ROUND4(1, b1, b2, b3, b0); \ 105 105 + ROUND4(2, b2, b3, b0, b1); \ 106 106 + ROUND4(3, b3, b0, b1, b2); \ 107 107 + \ 108 108 + bne 4b; \ 109 109 + \ 110 110 + rotate_clockwise_90(b0, b1, b2, b3); \ 111 111 + rev32 b0.16b, b0.16b; \ 112 112 + rev32 b1.16b, b1.16b; \ 113 113 + rev32 b2.16b, b2.16b; \ 114 114 + rev32 b3.16b, b3.16b; \ 115 115 + \ 116 116 + /* repoint to rkey */ \ 117 117 + sub x0, x0, #128; 118 118 + 119 119 + #define ROUND8(round, s0, s1, s2, s3, t0, t1, t2, t3) \ 120 120 + /* rk ^ s1 ^ s2 ^ s3 */ \ 121 121 + dup RX0.4s, RKEY.s[round]; \ 122 122 + eor RTMP0.16b, s2.16b, s3.16b; \ 123 123 + mov RX1.16b, RX0.16b; \ 124 124 + eor RTMP1.16b, t2.16b, t3.16b; \ 125 125 + eor RX0.16b, RX0.16b, s1.16b; \ 126 126 + eor RX1.16b, RX1.16b, t1.16b; \ 127 127 + eor RX0.16b, RX0.16b, RTMP0.16b; \ 128 128 + eor RX1.16b, RX1.16b, RTMP1.16b; \ 129 129 + \ 130 130 + /* sbox, non-linear part */ \ 131 131 + movi RTMP3.16b, #64; /* sizeof(sbox) / 4 */ \ 132 132 + tbl RTMP0.16b, {v16.16b-v19.16b}, RX0.16b; \ 133 133 + tbl RTMP1.16b, {v16.16b-v19.16b}, RX1.16b; \ 134 134 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 135 135 + sub RX1.16b, RX1.16b, RTMP3.16b; \ 136 136 + tbx RTMP0.16b, {v20.16b-v23.16b}, RX0.16b; \ 137 137 + tbx RTMP1.16b, {v20.16b-v23.16b}, RX1.16b; \ 138 138 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 139 139 + sub RX1.16b, RX1.16b, RTMP3.16b; \ 140 140 + tbx RTMP0.16b, {v24.16b-v27.16b}, RX0.16b; \ 141 141 + tbx RTMP1.16b, {v24.16b-v27.16b}, RX1.16b; \ 142 142 + sub RX0.16b, RX0.16b, RTMP3.16b; \ 143 143 + sub RX1.16b, RX1.16b, RTMP3.16b; \ 144 144 + tbx RTMP0.16b, {v28.16b-v31.16b}, RX0.16b; \ 145 145 + tbx RTMP1.16b, {v28.16b-v31.16b}, RX1.16b; \ 146 146 + \ 147 147 + /* linear part */ \ 148 148 + shl RX0.4s, RTMP0.4s, #8; \ 149 149 + shl RX1.4s, RTMP1.4s, #8; \ 150 150 + shl RTMP2.4s, RTMP0.4s, #16; \ 151 151 + shl RTMP3.4s, RTMP1.4s, #16; \ 152 152 + sri RX0.4s, RTMP0.4s, #(32 - 8); \ 153 153 + sri RX1.4s, RTMP1.4s, #(32 - 8); \ 154 154 + sri RTMP2.4s, RTMP0.4s, #(32 - 16); \ 155 155 + sri RTMP3.4s, RTMP1.4s, #(32 - 16); \ 156 156 + /* RX = x ^ rol32(x, 8) ^ rol32(x, 16) */ \ 157 157 + eor RX0.16b, RX0.16b, RTMP0.16b; \ 158 158 + eor RX1.16b, RX1.16b, RTMP1.16b; \ 159 159 + eor RX0.16b, RX0.16b, RTMP2.16b; \ 160 160 + eor RX1.16b, RX1.16b, RTMP3.16b; \ 161 161 + /* RTMP0/1 ^= x ^ rol32(x, 24) ^ rol32(RX, 2) */ \ 162 162 + shl RTMP2.4s, RTMP0.4s, #24; \ 163 163 + shl RTMP3.4s, RTMP1.4s, #24; \ 164 164 + sri RTMP2.4s, RTMP0.4s, #(32 - 24); \ 165 165 + sri RTMP3.4s, RTMP1.4s, #(32 - 24); \ 166 166 + eor RTMP0.16b, RTMP0.16b, RTMP2.16b; \ 167 167 + eor RTMP1.16b, RTMP1.16b, RTMP3.16b; \ 168 168 + shl RTMP2.4s, RX0.4s, #2; \ 169 169 + shl RTMP3.4s, RX1.4s, #2; \ 170 170 + sri RTMP2.4s, RX0.4s, #(32 - 2); \ 171 171 + sri RTMP3.4s, RX1.4s, #(32 - 2); \ 172 172 + eor RTMP0.16b, RTMP0.16b, RTMP2.16b; \ 173 173 + eor RTMP1.16b, RTMP1.16b, RTMP3.16b; \ 174 174 + /* s0/t0 ^= RTMP0/1 */ \ 175 175 + eor s0.16b, s0.16b, RTMP0.16b; \ 176 176 + eor t0.16b, t0.16b, RTMP1.16b; 177 177 + 178 178 + #define SM4_CRYPT_BLK8(b0, b1, b2, b3, b4, b5, b6, b7) \ 179 179 + rev32 b0.16b, b0.16b; \ 180 180 + rev32 b1.16b, b1.16b; \ 181 181 + rev32 b2.16b, b2.16b; \ 182 182 + rev32 b3.16b, b3.16b; \ 183 183 + rev32 b4.16b, b4.16b; \ 184 184 + rev32 b5.16b, b5.16b; \ 185 185 + rev32 b6.16b, b6.16b; \ 186 186 + rev32 b7.16b, b7.16b; \ 187 187 + \ 188 188 + transpose_4x4(b0, b1, b2, b3); \ 189 189 + transpose_4x4(b4, b5, b6, b7); \ 190 190 + \ 191 191 + mov x6, 8; \ 192 192 + 8: \ 193 193 + ld1 {RKEY.4s}, [x0], #16; \ 194 194 + subs x6, x6, #1; \ 195 195 + \ 196 196 + ROUND8(0, b0, b1, b2, b3, b4, b5, b6, b7); \ 197 197 + ROUND8(1, b1, b2, b3, b0, b5, b6, b7, b4); \ 198 198 + ROUND8(2, b2, b3, b0, b1, b6, b7, b4, b5); \ 199 199 + ROUND8(3, b3, b0, b1, b2, b7, b4, b5, b6); \ 200 200 + \ 201 201 + bne 8b; \ 202 202 + \ 203 203 + rotate_clockwise_90(b0, b1, b2, b3); \ 204 204 + rotate_clockwise_90(b4, b5, b6, b7); \ 205 205 + rev32 b0.16b, b0.16b; \ 206 206 + rev32 b1.16b, b1.16b; \ 207 207 + rev32 b2.16b, b2.16b; \ 208 208 + rev32 b3.16b, b3.16b; \ 209 209 + rev32 b4.16b, b4.16b; \ 210 210 + rev32 b5.16b, b5.16b; \ 211 211 + rev32 b6.16b, b6.16b; \ 212 212 + rev32 b7.16b, b7.16b; \ 213 213 + \ 214 214 + /* repoint to rkey */ \ 215 215 + sub x0, x0, #128; 216 216 + 217 217 + 218 218 + .align 3 219 219 + SYM_FUNC_START_LOCAL(__sm4_neon_crypt_blk1_4) 220 220 + /* input: 221 221 + * x0: round key array, CTX 222 222 + * x1: dst 223 223 + * x2: src 224 224 + * w3: num blocks (1..4) 225 225 + */ 226 226 + PREPARE; 227 227 + 228 228 + ld1 {v0.16b}, [x2], #16; 229 229 + mov v1.16b, v0.16b; 230 230 + mov v2.16b, v0.16b; 231 231 + mov v3.16b, v0.16b; 232 232 + cmp w3, #2; 233 233 + blt .Lblk4_load_input_done; 234 234 + ld1 {v1.16b}, [x2], #16; 235 235 + beq .Lblk4_load_input_done; 236 236 + ld1 {v2.16b}, [x2], #16; 237 237 + cmp w3, #3; 238 238 + beq .Lblk4_load_input_done; 239 239 + ld1 {v3.16b}, [x2]; 240 240 + 241 241 + .Lblk4_load_input_done: 242 242 + SM4_CRYPT_BLK4(v0, v1, v2, v3); 243 243 + 244 244 + st1 {v0.16b}, [x1], #16; 245 245 + cmp w3, #2; 246 246 + blt .Lblk4_store_output_done; 247 247 + st1 {v1.16b}, [x1], #16; 248 248 + beq .Lblk4_store_output_done; 249 249 + st1 {v2.16b}, [x1], #16; 250 250 + cmp w3, #3; 251 251 + beq .Lblk4_store_output_done; 252 252 + st1 {v3.16b}, [x1]; 253 253 + 254 254 + .Lblk4_store_output_done: 255 255 + ret; 256 256 + SYM_FUNC_END(__sm4_neon_crypt_blk1_4) 257 257 + 258 258 + .align 3 259 259 + SYM_FUNC_START(sm4_neon_crypt_blk1_8) 260 260 + /* input: 261 261 + * x0: round key array, CTX 262 262 + * x1: dst 263 263 + * x2: src 264 264 + * w3: num blocks (1..8) 265 265 + */ 266 266 + cmp w3, #5; 267 267 + blt __sm4_neon_crypt_blk1_4; 268 268 + 269 269 + PREPARE; 270 270 + 271 271 + ld1 {v0.16b-v3.16b}, [x2], #64; 272 272 + ld1 {v4.16b}, [x2], #16; 273 273 + mov v5.16b, v4.16b; 274 274 + mov v6.16b, v4.16b; 275 275 + mov v7.16b, v4.16b; 276 276 + beq .Lblk8_load_input_done; 277 277 + ld1 {v5.16b}, [x2], #16; 278 278 + cmp w3, #7; 279 279 + blt .Lblk8_load_input_done; 280 280 + ld1 {v6.16b}, [x2], #16; 281 281 + beq .Lblk8_load_input_done; 282 282 + ld1 {v7.16b}, [x2]; 283 283 + 284 284 + .Lblk8_load_input_done: 285 285 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 286 286 + 287 287 + cmp w3, #6; 288 288 + st1 {v0.16b-v3.16b}, [x1], #64; 289 289 + st1 {v4.16b}, [x1], #16; 290 290 + blt .Lblk8_store_output_done; 291 291 + st1 {v5.16b}, [x1], #16; 292 292 + beq .Lblk8_store_output_done; 293 293 + st1 {v6.16b}, [x1], #16; 294 294 + cmp w3, #7; 295 295 + beq .Lblk8_store_output_done; 296 296 + st1 {v7.16b}, [x1]; 297 297 + 298 298 + .Lblk8_store_output_done: 299 299 + ret; 300 300 + SYM_FUNC_END(sm4_neon_crypt_blk1_8) 301 301 + 302 302 + .align 3 303 303 + SYM_FUNC_START(sm4_neon_crypt_blk8) 304 304 + /* input: 305 305 + * x0: round key array, CTX 306 306 + * x1: dst 307 307 + * x2: src 308 308 + * w3: nblocks (multiples of 8) 309 309 + */ 310 310 + PREPARE; 311 311 + 312 312 + .Lcrypt_loop_blk: 313 313 + subs w3, w3, #8; 314 314 + bmi .Lcrypt_end; 315 315 + 316 316 + ld1 {v0.16b-v3.16b}, [x2], #64; 317 317 + ld1 {v4.16b-v7.16b}, [x2], #64; 318 318 + 319 319 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 320 320 + 321 321 + st1 {v0.16b-v3.16b}, [x1], #64; 322 322 + st1 {v4.16b-v7.16b}, [x1], #64; 323 323 + 324 324 + b .Lcrypt_loop_blk; 325 325 + 326 326 + .Lcrypt_end: 327 327 + ret; 328 328 + SYM_FUNC_END(sm4_neon_crypt_blk8) 329 329 + 330 330 + .align 3 331 331 + SYM_FUNC_START(sm4_neon_cbc_dec_blk8) 332 332 + /* input: 333 333 + * x0: round key array, CTX 334 334 + * x1: dst 335 335 + * x2: src 336 336 + * x3: iv (big endian, 128 bit) 337 337 + * w4: nblocks (multiples of 8) 338 338 + */ 339 339 + PREPARE; 340 340 + 341 341 + ld1 {RIV.16b}, [x3]; 342 342 + 343 343 + .Lcbc_loop_blk: 344 344 + subs w4, w4, #8; 345 345 + bmi .Lcbc_end; 346 346 + 347 347 + ld1 {v0.16b-v3.16b}, [x2], #64; 348 348 + ld1 {v4.16b-v7.16b}, [x2]; 349 349 + 350 350 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 351 351 + 352 352 + sub x2, x2, #64; 353 353 + eor v0.16b, v0.16b, RIV.16b; 354 354 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 355 355 + eor v1.16b, v1.16b, RTMP0.16b; 356 356 + eor v2.16b, v2.16b, RTMP1.16b; 357 357 + eor v3.16b, v3.16b, RTMP2.16b; 358 358 + st1 {v0.16b-v3.16b}, [x1], #64; 359 359 + 360 360 + eor v4.16b, v4.16b, RTMP3.16b; 361 361 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 362 362 + eor v5.16b, v5.16b, RTMP0.16b; 363 363 + eor v6.16b, v6.16b, RTMP1.16b; 364 364 + eor v7.16b, v7.16b, RTMP2.16b; 365 365 + 366 366 + mov RIV.16b, RTMP3.16b; 367 367 + st1 {v4.16b-v7.16b}, [x1], #64; 368 368 + 369 369 + b .Lcbc_loop_blk; 370 370 + 371 371 + .Lcbc_end: 372 372 + /* store new IV */ 373 373 + st1 {RIV.16b}, [x3]; 374 374 + 375 375 + ret; 376 376 + SYM_FUNC_END(sm4_neon_cbc_dec_blk8) 377 377 + 378 378 + .align 3 379 379 + SYM_FUNC_START(sm4_neon_cfb_dec_blk8) 380 380 + /* input: 381 381 + * x0: round key array, CTX 382 382 + * x1: dst 383 383 + * x2: src 384 384 + * x3: iv (big endian, 128 bit) 385 385 + * w4: nblocks (multiples of 8) 386 386 + */ 387 387 + PREPARE; 388 388 + 389 389 + ld1 {v0.16b}, [x3]; 390 390 + 391 391 + .Lcfb_loop_blk: 392 392 + subs w4, w4, #8; 393 393 + bmi .Lcfb_end; 394 394 + 395 395 + ld1 {v1.16b, v2.16b, v3.16b}, [x2], #48; 396 396 + ld1 {v4.16b-v7.16b}, [x2]; 397 397 + 398 398 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 399 399 + 400 400 + sub x2, x2, #48; 401 401 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 402 402 + eor v0.16b, v0.16b, RTMP0.16b; 403 403 + eor v1.16b, v1.16b, RTMP1.16b; 404 404 + eor v2.16b, v2.16b, RTMP2.16b; 405 405 + eor v3.16b, v3.16b, RTMP3.16b; 406 406 + st1 {v0.16b-v3.16b}, [x1], #64; 407 407 + 408 408 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 409 409 + eor v4.16b, v4.16b, RTMP0.16b; 410 410 + eor v5.16b, v5.16b, RTMP1.16b; 411 411 + eor v6.16b, v6.16b, RTMP2.16b; 412 412 + eor v7.16b, v7.16b, RTMP3.16b; 413 413 + st1 {v4.16b-v7.16b}, [x1], #64; 414 414 + 415 415 + mov v0.16b, RTMP3.16b; 416 416 + 417 417 + b .Lcfb_loop_blk; 418 418 + 419 419 + .Lcfb_end: 420 420 + /* store new IV */ 421 421 + st1 {v0.16b}, [x3]; 422 422 + 423 423 + ret; 424 424 + SYM_FUNC_END(sm4_neon_cfb_dec_blk8) 425 425 + 426 426 + .align 3 427 427 + SYM_FUNC_START(sm4_neon_ctr_enc_blk8) 428 428 + /* input: 429 429 + * x0: round key array, CTX 430 430 + * x1: dst 431 431 + * x2: src 432 432 + * x3: ctr (big endian, 128 bit) 433 433 + * w4: nblocks (multiples of 8) 434 434 + */ 435 435 + PREPARE; 436 436 + 437 437 + ldp x7, x8, [x3]; 438 438 + rev x7, x7; 439 439 + rev x8, x8; 440 440 + 441 441 + .Lctr_loop_blk: 442 442 + subs w4, w4, #8; 443 443 + bmi .Lctr_end; 444 444 + 445 445 + #define inc_le128(vctr) \ 446 446 + mov vctr.d[1], x8; \ 447 447 + mov vctr.d[0], x7; \ 448 448 + adds x8, x8, #1; \ 449 449 + adc x7, x7, xzr; \ 450 450 + rev64 vctr.16b, vctr.16b; 451 451 + 452 452 + /* construct CTRs */ 453 453 + inc_le128(v0); /* +0 */ 454 454 + inc_le128(v1); /* +1 */ 455 455 + inc_le128(v2); /* +2 */ 456 456 + inc_le128(v3); /* +3 */ 457 457 + inc_le128(v4); /* +4 */ 458 458 + inc_le128(v5); /* +5 */ 459 459 + inc_le128(v6); /* +6 */ 460 460 + inc_le128(v7); /* +7 */ 461 461 + 462 462 + SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7); 463 463 + 464 464 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 465 465 + eor v0.16b, v0.16b, RTMP0.16b; 466 466 + eor v1.16b, v1.16b, RTMP1.16b; 467 467 + eor v2.16b, v2.16b, RTMP2.16b; 468 468 + eor v3.16b, v3.16b, RTMP3.16b; 469 469 + st1 {v0.16b-v3.16b}, [x1], #64; 470 470 + 471 471 + ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64; 472 472 + eor v4.16b, v4.16b, RTMP0.16b; 473 473 + eor v5.16b, v5.16b, RTMP1.16b; 474 474 + eor v6.16b, v6.16b, RTMP2.16b; 475 475 + eor v7.16b, v7.16b, RTMP3.16b; 476 476 + st1 {v4.16b-v7.16b}, [x1], #64; 477 477 + 478 478 + b .Lctr_loop_blk; 479 479 + 480 480 + .Lctr_end: 481 481 + /* store new CTR */ 482 482 + rev x7, x7; 483 483 + rev x8, x8; 484 484 + stp x7, x8, [x3]; 485 485 + 486 486 + ret; 487 487 + SYM_FUNC_END(sm4_neon_ctr_enc_blk8)

+442

arch/arm64/crypto/sm4-neon-glue.c

reviewed

··· 1 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 2 + /* 3 3 + * SM4 Cipher Algorithm, using ARMv8 NEON 4 4 + * as specified in 5 5 + * https://tools.ietf.org/id/draft-ribose-cfrg-sm4-10.html 6 6 + * 7 7 + * Copyright (C) 2022, Alibaba Group. 8 8 + * Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com> 9 9 + */ 10 10 + 11 11 + #include <linux/module.h> 12 12 + #include <linux/crypto.h> 13 13 + #include <linux/kernel.h> 14 14 + #include <linux/cpufeature.h> 15 15 + #include <asm/neon.h> 16 16 + #include <asm/simd.h> 17 17 + #include <crypto/internal/simd.h> 18 18 + #include <crypto/internal/skcipher.h> 19 19 + #include <crypto/sm4.h> 20 20 + 21 21 + #define BYTES2BLKS(nbytes) ((nbytes) >> 4) 22 22 + #define BYTES2BLK8(nbytes) (((nbytes) >> 4) & ~(8 - 1)) 23 23 + 24 24 + asmlinkage void sm4_neon_crypt_blk1_8(const u32 *rkey, u8 *dst, const u8 *src, 25 25 + unsigned int nblks); 26 26 + asmlinkage void sm4_neon_crypt_blk8(const u32 *rkey, u8 *dst, const u8 *src, 27 27 + unsigned int nblks); 28 28 + asmlinkage void sm4_neon_cbc_dec_blk8(const u32 *rkey, u8 *dst, const u8 *src, 29 29 + u8 *iv, unsigned int nblks); 30 30 + asmlinkage void sm4_neon_cfb_dec_blk8(const u32 *rkey, u8 *dst, const u8 *src, 31 31 + u8 *iv, unsigned int nblks); 32 32 + asmlinkage void sm4_neon_ctr_enc_blk8(const u32 *rkey, u8 *dst, const u8 *src, 33 33 + u8 *iv, unsigned int nblks); 34 34 + 35 35 + static int sm4_setkey(struct crypto_skcipher *tfm, const u8 *key, 36 36 + unsigned int key_len) 37 37 + { 38 38 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 39 39 + 40 40 + return sm4_expandkey(ctx, key, key_len); 41 41 + } 42 42 + 43 43 + static int sm4_ecb_do_crypt(struct skcipher_request *req, const u32 *rkey) 44 44 + { 45 45 + struct skcipher_walk walk; 46 46 + unsigned int nbytes; 47 47 + int err; 48 48 + 49 49 + err = skcipher_walk_virt(&walk, req, false); 50 50 + 51 51 + while ((nbytes = walk.nbytes) > 0) { 52 52 + const u8 *src = walk.src.virt.addr; 53 53 + u8 *dst = walk.dst.virt.addr; 54 54 + unsigned int nblks; 55 55 + 56 56 + kernel_neon_begin(); 57 57 + 58 58 + nblks = BYTES2BLK8(nbytes); 59 59 + if (nblks) { 60 60 + sm4_neon_crypt_blk8(rkey, dst, src, nblks); 61 61 + dst += nblks * SM4_BLOCK_SIZE; 62 62 + src += nblks * SM4_BLOCK_SIZE; 63 63 + nbytes -= nblks * SM4_BLOCK_SIZE; 64 64 + } 65 65 + 66 66 + nblks = BYTES2BLKS(nbytes); 67 67 + if (nblks) { 68 68 + sm4_neon_crypt_blk1_8(rkey, dst, src, nblks); 69 69 + nbytes -= nblks * SM4_BLOCK_SIZE; 70 70 + } 71 71 + 72 72 + kernel_neon_end(); 73 73 + 74 74 + err = skcipher_walk_done(&walk, nbytes); 75 75 + } 76 76 + 77 77 + return err; 78 78 + } 79 79 + 80 80 + static int sm4_ecb_encrypt(struct skcipher_request *req) 81 81 + { 82 82 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 83 83 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 84 84 + 85 85 + return sm4_ecb_do_crypt(req, ctx->rkey_enc); 86 86 + } 87 87 + 88 88 + static int sm4_ecb_decrypt(struct skcipher_request *req) 89 89 + { 90 90 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 91 91 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 92 92 + 93 93 + return sm4_ecb_do_crypt(req, ctx->rkey_dec); 94 94 + } 95 95 + 96 96 + static int sm4_cbc_encrypt(struct skcipher_request *req) 97 97 + { 98 98 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 99 99 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 100 100 + struct skcipher_walk walk; 101 101 + unsigned int nbytes; 102 102 + int err; 103 103 + 104 104 + err = skcipher_walk_virt(&walk, req, false); 105 105 + 106 106 + while ((nbytes = walk.nbytes) > 0) { 107 107 + const u8 *iv = walk.iv; 108 108 + const u8 *src = walk.src.virt.addr; 109 109 + u8 *dst = walk.dst.virt.addr; 110 110 + 111 111 + while (nbytes >= SM4_BLOCK_SIZE) { 112 112 + crypto_xor_cpy(dst, src, iv, SM4_BLOCK_SIZE); 113 113 + sm4_crypt_block(ctx->rkey_enc, dst, dst); 114 114 + iv = dst; 115 115 + src += SM4_BLOCK_SIZE; 116 116 + dst += SM4_BLOCK_SIZE; 117 117 + nbytes -= SM4_BLOCK_SIZE; 118 118 + } 119 119 + if (iv != walk.iv) 120 120 + memcpy(walk.iv, iv, SM4_BLOCK_SIZE); 121 121 + 122 122 + err = skcipher_walk_done(&walk, nbytes); 123 123 + } 124 124 + 125 125 + return err; 126 126 + } 127 127 + 128 128 + static int sm4_cbc_decrypt(struct skcipher_request *req) 129 129 + { 130 130 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 131 131 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 132 132 + struct skcipher_walk walk; 133 133 + unsigned int nbytes; 134 134 + int err; 135 135 + 136 136 + err = skcipher_walk_virt(&walk, req, false); 137 137 + 138 138 + while ((nbytes = walk.nbytes) > 0) { 139 139 + const u8 *src = walk.src.virt.addr; 140 140 + u8 *dst = walk.dst.virt.addr; 141 141 + unsigned int nblks; 142 142 + 143 143 + kernel_neon_begin(); 144 144 + 145 145 + nblks = BYTES2BLK8(nbytes); 146 146 + if (nblks) { 147 147 + sm4_neon_cbc_dec_blk8(ctx->rkey_dec, dst, src, 148 148 + walk.iv, nblks); 149 149 + dst += nblks * SM4_BLOCK_SIZE; 150 150 + src += nblks * SM4_BLOCK_SIZE; 151 151 + nbytes -= nblks * SM4_BLOCK_SIZE; 152 152 + } 153 153 + 154 154 + nblks = BYTES2BLKS(nbytes); 155 155 + if (nblks) { 156 156 + u8 keystream[SM4_BLOCK_SIZE * 8]; 157 157 + u8 iv[SM4_BLOCK_SIZE]; 158 158 + int i; 159 159 + 160 160 + sm4_neon_crypt_blk1_8(ctx->rkey_dec, keystream, 161 161 + src, nblks); 162 162 + 163 163 + src += ((int)nblks - 2) * SM4_BLOCK_SIZE; 164 164 + dst += (nblks - 1) * SM4_BLOCK_SIZE; 165 165 + memcpy(iv, src + SM4_BLOCK_SIZE, SM4_BLOCK_SIZE); 166 166 + 167 167 + for (i = nblks - 1; i > 0; i--) { 168 168 + crypto_xor_cpy(dst, src, 169 169 + &keystream[i * SM4_BLOCK_SIZE], 170 170 + SM4_BLOCK_SIZE); 171 171 + src -= SM4_BLOCK_SIZE; 172 172 + dst -= SM4_BLOCK_SIZE; 173 173 + } 174 174 + crypto_xor_cpy(dst, walk.iv, 175 175 + keystream, SM4_BLOCK_SIZE); 176 176 + memcpy(walk.iv, iv, SM4_BLOCK_SIZE); 177 177 + nbytes -= nblks * SM4_BLOCK_SIZE; 178 178 + } 179 179 + 180 180 + kernel_neon_end(); 181 181 + 182 182 + err = skcipher_walk_done(&walk, nbytes); 183 183 + } 184 184 + 185 185 + return err; 186 186 + } 187 187 + 188 188 + static int sm4_cfb_encrypt(struct skcipher_request *req) 189 189 + { 190 190 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 191 191 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 192 192 + struct skcipher_walk walk; 193 193 + unsigned int nbytes; 194 194 + int err; 195 195 + 196 196 + err = skcipher_walk_virt(&walk, req, false); 197 197 + 198 198 + while ((nbytes = walk.nbytes) > 0) { 199 199 + u8 keystream[SM4_BLOCK_SIZE]; 200 200 + const u8 *iv = walk.iv; 201 201 + const u8 *src = walk.src.virt.addr; 202 202 + u8 *dst = walk.dst.virt.addr; 203 203 + 204 204 + while (nbytes >= SM4_BLOCK_SIZE) { 205 205 + sm4_crypt_block(ctx->rkey_enc, keystream, iv); 206 206 + crypto_xor_cpy(dst, src, keystream, SM4_BLOCK_SIZE); 207 207 + iv = dst; 208 208 + src += SM4_BLOCK_SIZE; 209 209 + dst += SM4_BLOCK_SIZE; 210 210 + nbytes -= SM4_BLOCK_SIZE; 211 211 + } 212 212 + if (iv != walk.iv) 213 213 + memcpy(walk.iv, iv, SM4_BLOCK_SIZE); 214 214 + 215 215 + /* tail */ 216 216 + if (walk.nbytes == walk.total && nbytes > 0) { 217 217 + sm4_crypt_block(ctx->rkey_enc, keystream, walk.iv); 218 218 + crypto_xor_cpy(dst, src, keystream, nbytes); 219 219 + nbytes = 0; 220 220 + } 221 221 + 222 222 + err = skcipher_walk_done(&walk, nbytes); 223 223 + } 224 224 + 225 225 + return err; 226 226 + } 227 227 + 228 228 + static int sm4_cfb_decrypt(struct skcipher_request *req) 229 229 + { 230 230 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 231 231 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 232 232 + struct skcipher_walk walk; 233 233 + unsigned int nbytes; 234 234 + int err; 235 235 + 236 236 + err = skcipher_walk_virt(&walk, req, false); 237 237 + 238 238 + while ((nbytes = walk.nbytes) > 0) { 239 239 + const u8 *src = walk.src.virt.addr; 240 240 + u8 *dst = walk.dst.virt.addr; 241 241 + unsigned int nblks; 242 242 + 243 243 + kernel_neon_begin(); 244 244 + 245 245 + nblks = BYTES2BLK8(nbytes); 246 246 + if (nblks) { 247 247 + sm4_neon_cfb_dec_blk8(ctx->rkey_enc, dst, src, 248 248 + walk.iv, nblks); 249 249 + dst += nblks * SM4_BLOCK_SIZE; 250 250 + src += nblks * SM4_BLOCK_SIZE; 251 251 + nbytes -= nblks * SM4_BLOCK_SIZE; 252 252 + } 253 253 + 254 254 + nblks = BYTES2BLKS(nbytes); 255 255 + if (nblks) { 256 256 + u8 keystream[SM4_BLOCK_SIZE * 8]; 257 257 + 258 258 + memcpy(keystream, walk.iv, SM4_BLOCK_SIZE); 259 259 + if (nblks > 1) 260 260 + memcpy(&keystream[SM4_BLOCK_SIZE], src, 261 261 + (nblks - 1) * SM4_BLOCK_SIZE); 262 262 + memcpy(walk.iv, src + (nblks - 1) * SM4_BLOCK_SIZE, 263 263 + SM4_BLOCK_SIZE); 264 264 + 265 265 + sm4_neon_crypt_blk1_8(ctx->rkey_enc, keystream, 266 266 + keystream, nblks); 267 267 + 268 268 + crypto_xor_cpy(dst, src, keystream, 269 269 + nblks * SM4_BLOCK_SIZE); 270 270 + dst += nblks * SM4_BLOCK_SIZE; 271 271 + src += nblks * SM4_BLOCK_SIZE; 272 272 + nbytes -= nblks * SM4_BLOCK_SIZE; 273 273 + } 274 274 + 275 275 + kernel_neon_end(); 276 276 + 277 277 + /* tail */ 278 278 + if (walk.nbytes == walk.total && nbytes > 0) { 279 279 + u8 keystream[SM4_BLOCK_SIZE]; 280 280 + 281 281 + sm4_crypt_block(ctx->rkey_enc, keystream, walk.iv); 282 282 + crypto_xor_cpy(dst, src, keystream, nbytes); 283 283 + nbytes = 0; 284 284 + } 285 285 + 286 286 + err = skcipher_walk_done(&walk, nbytes); 287 287 + } 288 288 + 289 289 + return err; 290 290 + } 291 291 + 292 292 + static int sm4_ctr_crypt(struct skcipher_request *req) 293 293 + { 294 294 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); 295 295 + struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm); 296 296 + struct skcipher_walk walk; 297 297 + unsigned int nbytes; 298 298 + int err; 299 299 + 300 300 + err = skcipher_walk_virt(&walk, req, false); 301 301 + 302 302 + while ((nbytes = walk.nbytes) > 0) { 303 303 + const u8 *src = walk.src.virt.addr; 304 304 + u8 *dst = walk.dst.virt.addr; 305 305 + unsigned int nblks; 306 306 + 307 307 + kernel_neon_begin(); 308 308 + 309 309 + nblks = BYTES2BLK8(nbytes); 310 310 + if (nblks) { 311 311 + sm4_neon_ctr_enc_blk8(ctx->rkey_enc, dst, src, 312 312 + walk.iv, nblks); 313 313 + dst += nblks * SM4_BLOCK_SIZE; 314 314 + src += nblks * SM4_BLOCK_SIZE; 315 315 + nbytes -= nblks * SM4_BLOCK_SIZE; 316 316 + } 317 317 + 318 318 + nblks = BYTES2BLKS(nbytes); 319 319 + if (nblks) { 320 320 + u8 keystream[SM4_BLOCK_SIZE * 8]; 321 321 + int i; 322 322 + 323 323 + for (i = 0; i < nblks; i++) { 324 324 + memcpy(&keystream[i * SM4_BLOCK_SIZE], 325 325 + walk.iv, SM4_BLOCK_SIZE); 326 326 + crypto_inc(walk.iv, SM4_BLOCK_SIZE); 327 327 + } 328 328 + sm4_neon_crypt_blk1_8(ctx->rkey_enc, keystream, 329 329 + keystream, nblks); 330 330 + 331 331 + crypto_xor_cpy(dst, src, keystream, 332 332 + nblks * SM4_BLOCK_SIZE); 333 333 + dst += nblks * SM4_BLOCK_SIZE; 334 334 + src += nblks * SM4_BLOCK_SIZE; 335 335 + nbytes -= nblks * SM4_BLOCK_SIZE; 336 336 + } 337 337 + 338 338 + kernel_neon_end(); 339 339 + 340 340 + /* tail */ 341 341 + if (walk.nbytes == walk.total && nbytes > 0) { 342 342 + u8 keystream[SM4_BLOCK_SIZE]; 343 343 + 344 344 + sm4_crypt_block(ctx->rkey_enc, keystream, walk.iv); 345 345 + crypto_inc(walk.iv, SM4_BLOCK_SIZE); 346 346 + crypto_xor_cpy(dst, src, keystream, nbytes); 347 347 + nbytes = 0; 348 348 + } 349 349 + 350 350 + err = skcipher_walk_done(&walk, nbytes); 351 351 + } 352 352 + 353 353 + return err; 354 354 + } 355 355 + 356 356 + static struct skcipher_alg sm4_algs[] = { 357 357 + { 358 358 + .base = { 359 359 + .cra_name = "ecb(sm4)", 360 360 + .cra_driver_name = "ecb-sm4-neon", 361 361 + .cra_priority = 200, 362 362 + .cra_blocksize = SM4_BLOCK_SIZE, 363 363 + .cra_ctxsize = sizeof(struct sm4_ctx), 364 364 + .cra_module = THIS_MODULE, 365 365 + }, 366 366 + .min_keysize = SM4_KEY_SIZE, 367 367 + .max_keysize = SM4_KEY_SIZE, 368 368 + .setkey = sm4_setkey, 369 369 + .encrypt = sm4_ecb_encrypt, 370 370 + .decrypt = sm4_ecb_decrypt, 371 371 + }, { 372 372 + .base = { 373 373 + .cra_name = "cbc(sm4)", 374 374 + .cra_driver_name = "cbc-sm4-neon", 375 375 + .cra_priority = 200, 376 376 + .cra_blocksize = SM4_BLOCK_SIZE, 377 377 + .cra_ctxsize = sizeof(struct sm4_ctx), 378 378 + .cra_module = THIS_MODULE, 379 379 + }, 380 380 + .min_keysize = SM4_KEY_SIZE, 381 381 + .max_keysize = SM4_KEY_SIZE, 382 382 + .ivsize = SM4_BLOCK_SIZE, 383 383 + .setkey = sm4_setkey, 384 384 + .encrypt = sm4_cbc_encrypt, 385 385 + .decrypt = sm4_cbc_decrypt, 386 386 + }, { 387 387 + .base = { 388 388 + .cra_name = "cfb(sm4)", 389 389 + .cra_driver_name = "cfb-sm4-neon", 390 390 + .cra_priority = 200, 391 391 + .cra_blocksize = 1, 392 392 + .cra_ctxsize = sizeof(struct sm4_ctx), 393 393 + .cra_module = THIS_MODULE, 394 394 + }, 395 395 + .min_keysize = SM4_KEY_SIZE, 396 396 + .max_keysize = SM4_KEY_SIZE, 397 397 + .ivsize = SM4_BLOCK_SIZE, 398 398 + .chunksize = SM4_BLOCK_SIZE, 399 399 + .setkey = sm4_setkey, 400 400 + .encrypt = sm4_cfb_encrypt, 401 401 + .decrypt = sm4_cfb_decrypt, 402 402 + }, { 403 403 + .base = { 404 404 + .cra_name = "ctr(sm4)", 405 405 + .cra_driver_name = "ctr-sm4-neon", 406 406 + .cra_priority = 200, 407 407 + .cra_blocksize = 1, 408 408 + .cra_ctxsize = sizeof(struct sm4_ctx), 409 409 + .cra_module = THIS_MODULE, 410 410 + }, 411 411 + .min_keysize = SM4_KEY_SIZE, 412 412 + .max_keysize = SM4_KEY_SIZE, 413 413 + .ivsize = SM4_BLOCK_SIZE, 414 414 + .chunksize = SM4_BLOCK_SIZE, 415 415 + .setkey = sm4_setkey, 416 416 + .encrypt = sm4_ctr_crypt, 417 417 + .decrypt = sm4_ctr_crypt, 418 418 + } 419 419 + }; 420 420 + 421 421 + static int __init sm4_init(void) 422 422 + { 423 423 + return crypto_register_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs)); 424 424 + } 425 425 + 426 426 + static void __exit sm4_exit(void) 427 427 + { 428 428 + crypto_unregister_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs)); 429 429 + } 430 430 + 431 431 + module_init(sm4_init); 432 432 + module_exit(sm4_exit); 433 433 + 434 434 + MODULE_DESCRIPTION("SM4 ECB/CBC/CFB/CTR using ARMv8 NEON"); 435 435 + MODULE_ALIAS_CRYPTO("sm4-neon"); 436 436 + MODULE_ALIAS_CRYPTO("sm4"); 437 437 + MODULE_ALIAS_CRYPTO("ecb(sm4)"); 438 438 + MODULE_ALIAS_CRYPTO("cbc(sm4)"); 439 439 + MODULE_ALIAS_CRYPTO("cfb(sm4)"); 440 440 + MODULE_ALIAS_CRYPTO("ctr(sm4)"); 441 441 + MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>"); 442 442 + MODULE_LICENSE("GPL v2");

+32 -2

arch/s390/crypto/chacha-glue.c

reviewed

··· 62 62 return rc; 63 63 } 64 64 65 65 + void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds) 66 66 + { 67 67 + /* TODO: implement hchacha_block_arch() in assembly */ 68 68 + hchacha_block_generic(state, stream, nrounds); 69 69 + } 70 70 + EXPORT_SYMBOL(hchacha_block_arch); 71 71 + 72 72 + void chacha_init_arch(u32 *state, const u32 *key, const u8 *iv) 73 73 + { 74 74 + chacha_init_generic(state, key, iv); 75 75 + } 76 76 + EXPORT_SYMBOL(chacha_init_arch); 77 77 + 78 78 + void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, 79 79 + unsigned int bytes, int nrounds) 80 80 + { 81 81 + /* s390 chacha20 implementation has 20 rounds hard-coded, 82 82 + * it cannot handle a block of data or less, but otherwise 83 83 + * it can handle data of arbitrary size 84 84 + */ 85 85 + if (bytes <= CHACHA_BLOCK_SIZE || nrounds != 20) 86 86 + chacha_crypt_generic(state, dst, src, bytes, nrounds); 87 87 + else 88 88 + chacha20_crypt_s390(state, dst, src, bytes, 89 89 + &state[4], &state[12]); 90 90 + } 91 91 + EXPORT_SYMBOL(chacha_crypt_arch); 92 92 + 65 93 static struct skcipher_alg chacha_algs[] = { 66 94 { 67 95 .base.cra_name = "chacha20", ··· 111 83 112 84 static int __init chacha_mod_init(void) 113 85 { 114 114 - return crypto_register_skciphers(chacha_algs, ARRAY_SIZE(chacha_algs)); 86 86 + return IS_REACHABLE(CONFIG_CRYPTO_SKCIPHER) ? 87 87 + crypto_register_skciphers(chacha_algs, ARRAY_SIZE(chacha_algs)) : 0; 115 88 } 116 89 117 90 static void __exit chacha_mod_fini(void) 118 91 { 119 119 - crypto_unregister_skciphers(chacha_algs, ARRAY_SIZE(chacha_algs)); 92 92 + if (IS_REACHABLE(CONFIG_CRYPTO_SKCIPHER)) 93 93 + crypto_unregister_skciphers(chacha_algs, ARRAY_SIZE(chacha_algs)); 120 94 } 121 95 122 96 module_cpu_feature_match(VXRS, chacha_mod_init);

+4 -4

arch/x86/crypto/blowfish_glue.c

reviewed

··· 303 303 module_param(force, int, 0); 304 304 MODULE_PARM_DESC(force, "Force module load, ignore CPU blacklist"); 305 305 306 306 - static int __init init(void) 306 306 + static int __init blowfish_init(void) 307 307 { 308 308 int err; 309 309 ··· 327 327 return err; 328 328 } 329 329 330 330 - static void __exit fini(void) 330 330 + static void __exit blowfish_fini(void) 331 331 { 332 332 crypto_unregister_alg(&bf_cipher_alg); 333 333 crypto_unregister_skciphers(bf_skcipher_algs, 334 334 ARRAY_SIZE(bf_skcipher_algs)); 335 335 } 336 336 337 337 - module_init(init); 338 338 - module_exit(fini); 337 337 + module_init(blowfish_init); 338 338 + module_exit(blowfish_fini); 339 339 340 340 MODULE_LICENSE("GPL"); 341 341 MODULE_DESCRIPTION("Blowfish Cipher Algorithm, asm optimized");

+4 -4

arch/x86/crypto/camellia_glue.c

reviewed

··· 1377 1377 module_param(force, int, 0); 1378 1378 MODULE_PARM_DESC(force, "Force module load, ignore CPU blacklist"); 1379 1379 1380 1380 - static int __init init(void) 1380 1380 + static int __init camellia_init(void) 1381 1381 { 1382 1382 int err; 1383 1383 ··· 1401 1401 return err; 1402 1402 } 1403 1403 1404 1404 - static void __exit fini(void) 1404 1404 + static void __exit camellia_fini(void) 1405 1405 { 1406 1406 crypto_unregister_alg(&camellia_cipher_alg); 1407 1407 crypto_unregister_skciphers(camellia_skcipher_algs, 1408 1408 ARRAY_SIZE(camellia_skcipher_algs)); 1409 1409 } 1410 1410 1411 1411 - module_init(init); 1412 1412 - module_exit(fini); 1411 1411 + module_init(camellia_init); 1412 1412 + module_exit(camellia_fini); 1413 1413 1414 1414 MODULE_LICENSE("GPL"); 1415 1415 MODULE_DESCRIPTION("Camellia Cipher Algorithm, asm optimized");

+4 -4

arch/x86/crypto/serpent_avx2_glue.c

reviewed

··· 96 96 97 97 static struct simd_skcipher_alg *serpent_simd_algs[ARRAY_SIZE(serpent_algs)]; 98 98 99 99 - static int __init init(void) 99 99 + static int __init serpent_avx2_init(void) 100 100 { 101 101 const char *feature_name; 102 102 ··· 115 115 serpent_simd_algs); 116 116 } 117 117 118 118 - static void __exit fini(void) 118 118 + static void __exit serpent_avx2_fini(void) 119 119 { 120 120 simd_unregister_skciphers(serpent_algs, ARRAY_SIZE(serpent_algs), 121 121 serpent_simd_algs); 122 122 } 123 123 124 124 - module_init(init); 125 125 - module_exit(fini); 124 124 + module_init(serpent_avx2_init); 125 125 + module_exit(serpent_avx2_fini); 126 126 127 127 MODULE_LICENSE("GPL"); 128 128 MODULE_DESCRIPTION("Serpent Cipher Algorithm, AVX2 optimized");

+4 -4

arch/x86/crypto/twofish_glue.c

reviewed

··· 81 81 } 82 82 }; 83 83 84 84 - static int __init init(void) 84 84 + static int __init twofish_glue_init(void) 85 85 { 86 86 return crypto_register_alg(&alg); 87 87 } 88 88 89 89 - static void __exit fini(void) 89 89 + static void __exit twofish_glue_fini(void) 90 90 { 91 91 crypto_unregister_alg(&alg); 92 92 } 93 93 94 94 - module_init(init); 95 95 - module_exit(fini); 94 94 + module_init(twofish_glue_init); 95 95 + module_exit(twofish_glue_fini); 96 96 97 97 MODULE_LICENSE("GPL"); 98 98 MODULE_DESCRIPTION ("Twofish Cipher Algorithm, asm optimized");

+4 -4

arch/x86/crypto/twofish_glue_3way.c

reviewed

··· 140 140 module_param(force, int, 0); 141 141 MODULE_PARM_DESC(force, "Force module load, ignore CPU blacklist"); 142 142 143 143 - static int __init init(void) 143 143 + static int __init twofish_3way_init(void) 144 144 { 145 145 if (!force && is_blacklisted_cpu()) { 146 146 printk(KERN_INFO ··· 154 154 ARRAY_SIZE(tf_skciphers)); 155 155 } 156 156 157 157 - static void __exit fini(void) 157 157 + static void __exit twofish_3way_fini(void) 158 158 { 159 159 crypto_unregister_skciphers(tf_skciphers, ARRAY_SIZE(tf_skciphers)); 160 160 } 161 161 162 162 - module_init(init); 163 163 - module_exit(fini); 162 162 + module_init(twofish_3way_init); 163 163 + module_exit(twofish_3way_fini); 164 164 165 165 MODULE_LICENSE("GPL"); 166 166 MODULE_DESCRIPTION("Twofish Cipher Algorithm, 3-way parallel asm optimized");

+12 -6

crypto/Kconfig

reviewed

··· 274 274 275 275 config CRYPTO_SM2 276 276 tristate "SM2 algorithm" 277 277 - select CRYPTO_LIB_SM3 277 277 + select CRYPTO_SM3 278 278 select CRYPTO_AKCIPHER 279 279 select CRYPTO_MANAGER 280 280 select MPILIB ··· 1010 1010 http://keccak.noekeon.org/ 1011 1011 1012 1012 config CRYPTO_SM3 1013 1013 + tristate 1014 1014 + 1015 1015 + config CRYPTO_SM3_GENERIC 1013 1016 tristate "SM3 digest algorithm" 1014 1017 select CRYPTO_HASH 1015 1015 - select CRYPTO_LIB_SM3 1018 1018 + select CRYPTO_SM3 1016 1019 help 1017 1020 SM3 secure hash function as defined by OSCCA GM/T 0004-2012 SM3). 1018 1021 It is part of the Chinese Commercial Cryptography suite. ··· 1028 1025 tristate "SM3 digest algorithm (x86_64/AVX)" 1029 1026 depends on X86 && 64BIT 1030 1027 select CRYPTO_HASH 1031 1031 - select CRYPTO_LIB_SM3 1028 1028 + select CRYPTO_SM3 1032 1029 help 1033 1030 SM3 secure hash function as defined by OSCCA GM/T 0004-2012 SM3). 1034 1031 It is part of the Chinese Commercial Cryptography suite. This is ··· 1575 1572 <https://www.cl.cam.ac.uk/~rja14/serpent.html> 1576 1573 1577 1574 config CRYPTO_SM4 1575 1575 + tristate 1576 1576 + 1577 1577 + config CRYPTO_SM4_GENERIC 1578 1578 tristate "SM4 cipher algorithm" 1579 1579 select CRYPTO_ALGAPI 1580 1580 - select CRYPTO_LIB_SM4 1580 1580 + select CRYPTO_SM4 1581 1581 help 1582 1582 SM4 cipher algorithms (OSCCA GB/T 32907-2016). 1583 1583 ··· 1609 1603 select CRYPTO_SKCIPHER 1610 1604 select CRYPTO_SIMD 1611 1605 select CRYPTO_ALGAPI 1612 1612 - select CRYPTO_LIB_SM4 1606 1606 + select CRYPTO_SM4 1613 1607 help 1614 1608 SM4 cipher algorithms (OSCCA GB/T 32907-2016) (x86_64/AES-NI/AVX). 1615 1609 ··· 1630 1624 select CRYPTO_SKCIPHER 1631 1625 select CRYPTO_SIMD 1632 1626 select CRYPTO_ALGAPI 1633 1633 - select CRYPTO_LIB_SM4 1627 1627 + select CRYPTO_SM4 1634 1628 select CRYPTO_SM4_AESNI_AVX_X86_64 1635 1629 help 1636 1630 SM4 cipher algorithms (OSCCA GB/T 32907-2016) (x86_64/AES-NI/AVX2).

+4 -2

crypto/Makefile

reviewed

··· 78 78 obj-$(CONFIG_CRYPTO_SHA256) += sha256_generic.o 79 79 obj-$(CONFIG_CRYPTO_SHA512) += sha512_generic.o 80 80 obj-$(CONFIG_CRYPTO_SHA3) += sha3_generic.o 81 81 - obj-$(CONFIG_CRYPTO_SM3) += sm3_generic.o 81 81 + obj-$(CONFIG_CRYPTO_SM3) += sm3.o 82 82 + obj-$(CONFIG_CRYPTO_SM3_GENERIC) += sm3_generic.o 82 83 obj-$(CONFIG_CRYPTO_STREEBOG) += streebog_generic.o 83 84 obj-$(CONFIG_CRYPTO_WP512) += wp512.o 84 85 CFLAGS_wp512.o := $(call cc-option,-fno-schedule-insns) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149 ··· 135 134 CFLAGS_serpent_generic.o := $(call cc-option,-fsched-pressure) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149 136 135 obj-$(CONFIG_CRYPTO_AES) += aes_generic.o 137 136 CFLAGS_aes_generic.o := $(call cc-option,-fno-code-hoisting) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83356 138 138 - obj-$(CONFIG_CRYPTO_SM4) += sm4_generic.o 137 137 + obj-$(CONFIG_CRYPTO_SM4) += sm4.o 138 138 + obj-$(CONFIG_CRYPTO_SM4_GENERIC) += sm4_generic.o 139 139 obj-$(CONFIG_CRYPTO_AES_TI) += aes_ti.o 140 140 obj-$(CONFIG_CRYPTO_CAMELLIA) += camellia_generic.o 141 141 obj-$(CONFIG_CRYPTO_CAST_COMMON) += cast_common.o

+11 -12

crypto/cryptd.c

reviewed

··· 39 39 }; 40 40 41 41 struct cryptd_queue { 42 42 + /* 43 43 + * Protected by disabling BH to allow enqueueing from softinterrupt and 44 44 + * dequeuing from kworker (cryptd_queue_worker()). 45 45 + */ 42 46 struct cryptd_cpu_queue __percpu *cpu_queue; 43 47 }; 44 48 ··· 129 125 static int cryptd_enqueue_request(struct cryptd_queue *queue, 130 126 struct crypto_async_request *request) 131 127 { 132 132 - int cpu, err; 128 128 + int err; 133 129 struct cryptd_cpu_queue *cpu_queue; 134 130 refcount_t *refcnt; 135 131 136 136 - cpu = get_cpu(); 132 132 + local_bh_disable(); 137 133 cpu_queue = this_cpu_ptr(queue->cpu_queue); 138 134 err = crypto_enqueue_request(&cpu_queue->queue, request); 139 135 140 136 refcnt = crypto_tfm_ctx(request->tfm); 141 137 142 138 if (err == -ENOSPC) 143 143 - goto out_put_cpu; 139 139 + goto out; 144 140 145 145 - queue_work_on(cpu, cryptd_wq, &cpu_queue->work); 141 141 + queue_work_on(smp_processor_id(), cryptd_wq, &cpu_queue->work); 146 142 147 143 if (!refcount_read(refcnt)) 148 148 - goto out_put_cpu; 144 144 + goto out; 149 145 150 146 refcount_inc(refcnt); 151 147 152 152 - out_put_cpu: 153 153 - put_cpu(); 148 148 + out: 149 149 + local_bh_enable(); 154 150 155 151 return err; 156 152 } ··· 166 162 cpu_queue = container_of(work, struct cryptd_cpu_queue, work); 167 163 /* 168 164 * Only handle one request at a time to avoid hogging crypto workqueue. 169 169 - * preempt_disable/enable is used to prevent being preempted by 170 170 - * cryptd_enqueue_request(). local_bh_disable/enable is used to prevent 171 171 - * cryptd_enqueue_request() being accessed from software interrupts. 172 165 */ 173 166 local_bh_disable(); 174 174 - preempt_disable(); 175 167 backlog = crypto_get_backlog(&cpu_queue->queue); 176 168 req = crypto_dequeue_request(&cpu_queue->queue); 177 177 - preempt_enable(); 178 169 local_bh_enable(); 179 170 180 171 if (!req)

+1

crypto/crypto_engine.c

reviewed

··· 253 253 * crypto_transfer_request - transfer the new request into the engine queue 254 254 * @engine: the hardware engine 255 255 * @req: the request need to be listed into the engine queue 256 256 + * @need_pump: indicates whether queue the pump of request to kthread_work 256 257 */ 257 258 static int crypto_transfer_request(struct crypto_engine *engine, 258 259 struct crypto_async_request *req,

+4 -4

crypto/ecrdsa.c

reviewed

··· 113 113 114 114 /* Step 1: verify that 0 < r < q, 0 < s < q */ 115 115 if (vli_is_zero(r, ndigits) || 116 116 - vli_cmp(r, ctx->curve->n, ndigits) == 1 || 116 116 + vli_cmp(r, ctx->curve->n, ndigits) >= 0 || 117 117 vli_is_zero(s, ndigits) || 118 118 - vli_cmp(s, ctx->curve->n, ndigits) == 1) 118 118 + vli_cmp(s, ctx->curve->n, ndigits) >= 0) 119 119 return -EKEYREJECTED; 120 120 121 121 /* Step 2: calculate hash (h) of the message (passed as input) */ 122 122 /* Step 3: calculate e = h \mod q */ 123 123 vli_from_le64(e, digest, ndigits); 124 124 - if (vli_cmp(e, ctx->curve->n, ndigits) == 1) 124 124 + if (vli_cmp(e, ctx->curve->n, ndigits) >= 0) 125 125 vli_sub(e, e, ctx->curve->n, ndigits); 126 126 if (vli_is_zero(e, ndigits)) 127 127 e[0] = 1; ··· 137 137 /* Step 6: calculate point C = z_1P + z_2Q, and R = x_c \mod q */ 138 138 ecc_point_mult_shamir(&cc, z1, &ctx->curve->g, z2, &ctx->pub_key, 139 139 ctx->curve); 140 140 - if (vli_cmp(cc.x, ctx->curve->n, ndigits) == 1) 140 140 + if (vli_cmp(cc.x, ctx->curve->n, ndigits) >= 0) 141 141 vli_sub(cc.x, cc.x, ctx->curve->n, ndigits); 142 142 143 143 /* Step 7: if R == r signature is valid */

+63 -12

crypto/testmgr.c

reviewed

··· 232 232 FINALIZATION_TYPE_DIGEST, /* use digest() */ 233 233 }; 234 234 235 235 + /* 236 236 + * Whether the crypto operation will occur in-place, and if so whether the 237 237 + * source and destination scatterlist pointers will coincide (req->src == 238 238 + * req->dst), or whether they'll merely point to two separate scatterlists 239 239 + * (req->src != req->dst) that reference the same underlying memory. 240 240 + * 241 241 + * This is only relevant for algorithm types that support in-place operation. 242 242 + */ 243 243 + enum inplace_mode { 244 244 + OUT_OF_PLACE, 245 245 + INPLACE_ONE_SGLIST, 246 246 + INPLACE_TWO_SGLISTS, 247 247 + }; 248 248 + 235 249 #define TEST_SG_TOTAL 10000 236 250 237 251 /** ··· 279 265 * crypto test vector can be tested. 280 266 * 281 267 * @name: name of this config, logged for debugging purposes if a test fails 282 282 - * @inplace: operate on the data in-place, if applicable for the algorithm type? 268 268 + * @inplace_mode: whether and how to operate on the data in-place, if applicable 283 269 * @req_flags: extra request_flags, e.g. CRYPTO_TFM_REQ_MAY_SLEEP 284 270 * @src_divs: description of how to arrange the source scatterlist 285 271 * @dst_divs: description of how to arrange the dst scatterlist, if applicable ··· 296 282 */ 297 283 struct testvec_config { 298 284 const char *name; 299 299 - bool inplace; 285 285 + enum inplace_mode inplace_mode; 300 286 u32 req_flags; 301 287 struct test_sg_division src_divs[XBUFSIZE]; 302 288 struct test_sg_division dst_divs[XBUFSIZE]; ··· 321 307 /* Configs for skciphers and aeads */ 322 308 static const struct testvec_config default_cipher_testvec_configs[] = { 323 309 { 324 324 - .name = "in-place", 325 325 - .inplace = true, 310 310 + .name = "in-place (one sglist)", 311 311 + .inplace_mode = INPLACE_ONE_SGLIST, 312 312 + .src_divs = { { .proportion_of_total = 10000 } }, 313 313 + }, { 314 314 + .name = "in-place (two sglists)", 315 315 + .inplace_mode = INPLACE_TWO_SGLISTS, 326 316 .src_divs = { { .proportion_of_total = 10000 } }, 327 317 }, { 328 318 .name = "out-of-place", 319 319 + .inplace_mode = OUT_OF_PLACE, 329 320 .src_divs = { { .proportion_of_total = 10000 } }, 330 321 }, { 331 322 .name = "unaligned buffer, offset=1", ··· 368 349 .key_offset = 3, 369 350 }, { 370 351 .name = "misaligned splits crossing pages, inplace", 371 371 - .inplace = true, 352 352 + .inplace_mode = INPLACE_ONE_SGLIST, 372 353 .src_divs = { 373 354 { 374 355 .proportion_of_total = 7500, ··· 768 749 769 750 iov_iter_kvec(&input, WRITE, inputs, nr_inputs, src_total_len); 770 751 err = build_test_sglist(&tsgls->src, cfg->src_divs, alignmask, 771 771 - cfg->inplace ? 752 752 + cfg->inplace_mode != OUT_OF_PLACE ? 772 753 max(dst_total_len, src_total_len) : 773 754 src_total_len, 774 755 &input, NULL); 775 756 if (err) 776 757 return err; 777 758 778 778 - if (cfg->inplace) { 759 759 + /* 760 760 + * In-place crypto operations can use the same scatterlist for both the 761 761 + * source and destination (req->src == req->dst), or can use separate 762 762 + * scatterlists (req->src != req->dst) which point to the same 763 763 + * underlying memory. Make sure to test both cases. 764 764 + */ 765 765 + if (cfg->inplace_mode == INPLACE_ONE_SGLIST) { 779 766 tsgls->dst.sgl_ptr = tsgls->src.sgl; 780 767 tsgls->dst.nents = tsgls->src.nents; 781 768 return 0; 782 769 } 770 770 + if (cfg->inplace_mode == INPLACE_TWO_SGLISTS) { 771 771 + /* 772 772 + * For now we keep it simple and only test the case where the 773 773 + * two scatterlists have identical entries, rather than 774 774 + * different entries that split up the same memory differently. 775 775 + */ 776 776 + memcpy(tsgls->dst.sgl, tsgls->src.sgl, 777 777 + tsgls->src.nents * sizeof(tsgls->src.sgl[0])); 778 778 + memcpy(tsgls->dst.sgl_saved, tsgls->src.sgl, 779 779 + tsgls->src.nents * sizeof(tsgls->src.sgl[0])); 780 780 + tsgls->dst.sgl_ptr = tsgls->dst.sgl; 781 781 + tsgls->dst.nents = tsgls->src.nents; 782 782 + return 0; 783 783 + } 784 784 + /* Out of place */ 783 785 return build_test_sglist(&tsgls->dst, 784 786 cfg->dst_divs[0].proportion_of_total ? 785 787 cfg->dst_divs : cfg->src_divs, ··· 1035 995 1036 996 p += scnprintf(p, end - p, "random:"); 1037 997 1038 1038 - if (prandom_u32() % 2 == 0) { 1039 1039 - cfg->inplace = true; 1040 1040 - p += scnprintf(p, end - p, " inplace"); 998 998 + switch (prandom_u32() % 4) { 999 999 + case 0: 1000 1000 + case 1: 1001 1001 + cfg->inplace_mode = OUT_OF_PLACE; 1002 1002 + break; 1003 1003 + case 2: 1004 1004 + cfg->inplace_mode = INPLACE_ONE_SGLIST; 1005 1005 + p += scnprintf(p, end - p, " inplace_one_sglist"); 1006 1006 + break; 1007 1007 + default: 1008 1008 + cfg->inplace_mode = INPLACE_TWO_SGLISTS; 1009 1009 + p += scnprintf(p, end - p, " inplace_two_sglists"); 1010 1010 + break; 1041 1011 } 1042 1012 1043 1013 if (prandom_u32() % 2 == 0) { ··· 1084 1034 cfg->req_flags); 1085 1035 p += scnprintf(p, end - p, "]"); 1086 1036 1087 1087 - if (!cfg->inplace && prandom_u32() % 2 == 0) { 1037 1037 + if (cfg->inplace_mode == OUT_OF_PLACE && prandom_u32() % 2 == 0) { 1088 1038 p += scnprintf(p, end - p, " dst_divs=["); 1089 1039 p = generate_random_sgl_divisions(cfg->dst_divs, 1090 1040 ARRAY_SIZE(cfg->dst_divs), ··· 2135 2085 /* Check for the correct output (ciphertext or plaintext) */ 2136 2086 err = verify_correct_output(&tsgls->dst, enc ? vec->ctext : vec->ptext, 2137 2087 enc ? vec->clen : vec->plen, 2138 2138 - vec->alen, enc || !cfg->inplace); 2088 2088 + vec->alen, 2089 2089 + enc || cfg->inplace_mode == OUT_OF_PLACE); 2139 2090 if (err == -EOVERFLOW) { 2140 2091 pr_err("alg: aead: %s %s overran dst buffer on test vector %s, cfg=\"%s\"\n", 2141 2092 driver, op, vec_name, cfg->name);

+14 -1

drivers/char/hw_random/Kconfig

reviewed

··· 385 385 386 386 If unsure, say Y. 387 387 388 388 + config HW_RANDOM_POLARFIRE_SOC 389 389 + tristate "Microchip PolarFire SoC Random Number Generator support" 390 390 + depends on HW_RANDOM && POLARFIRE_SOC_SYS_CTRL 391 391 + help 392 392 + This driver provides kernel-side support for the Random Number 393 393 + Generator hardware found on PolarFire SoC (MPFS). 394 394 + 395 395 + To compile this driver as a module, choose M here. The 396 396 + module will be called mfps_rng. 397 397 + 398 398 + If unsure, say N. 399 399 + 400 400 + 388 401 config HW_RANDOM_MESON 389 402 tristate "Amlogic Meson Random Number Generator support" 390 403 depends on HW_RANDOM ··· 540 527 541 528 config HW_RANDOM_CN10K 542 529 tristate "Marvell CN10K Random Number Generator support" 543 543 - depends on HW_RANDOM && PCI && ARM64 530 530 + depends on HW_RANDOM && PCI && (ARM64 || (64BIT && COMPILE_TEST)) 544 531 default HW_RANDOM 545 532 help 546 533 This driver provides support for the True Random Number

+1

drivers/char/hw_random/Makefile

reviewed

··· 46 46 obj-$(CONFIG_HW_RANDOM_XIPHERA) += xiphera-trng.o 47 47 obj-$(CONFIG_HW_RANDOM_ARM_SMCCC_TRNG) += arm_smccc_trng.o 48 48 obj-$(CONFIG_HW_RANDOM_CN10K) += cn10k-rng.o 49 49 + obj-$(CONFIG_HW_RANDOM_POLARFIRE_SOC) += mpfs-rng.o

+17 -14

drivers/char/hw_random/cn10k-rng.c

reviewed

··· 31 31 32 32 #define PLAT_OCTEONTX_RESET_RNG_EBG_HEALTH_STATE 0xc2000b0f 33 33 34 34 - static int reset_rng_health_state(struct cn10k_rng *rng) 34 34 + static unsigned long reset_rng_health_state(struct cn10k_rng *rng) 35 35 { 36 36 struct arm_smccc_res res; 37 37 38 38 /* Send SMC service call to reset EBG health state */ 39 39 arm_smccc_smc(PLAT_OCTEONTX_RESET_RNG_EBG_HEALTH_STATE, 0, 0, 0, 0, 0, 0, 0, &res); 40 40 - if (res.a0 != 0UL) 41 41 - return -EIO; 42 42 - 43 43 - return 0; 40 40 + return res.a0; 44 41 } 45 42 46 43 static int check_rng_health(struct cn10k_rng *rng) 47 44 { 48 45 u64 status; 49 49 - int err; 46 46 + unsigned long err; 50 47 51 48 /* Skip checking health */ 52 49 if (!rng->reg_base) 53 53 - return 0; 50 50 + return -ENODEV; 54 51 55 52 status = readq(rng->reg_base + RNM_PF_EBG_HEALTH); 56 53 if (status & BIT_ULL(20)) { ··· 55 58 if (err) { 56 59 dev_err(&rng->pdev->dev, "HWRNG: Health test failed (status=%llx)\n", 57 60 status); 58 58 - dev_err(&rng->pdev->dev, "HWRNG: error during reset\n"); 61 61 + dev_err(&rng->pdev->dev, "HWRNG: error during reset (error=%lx)\n", 62 62 + err); 63 63 + return -EIO; 59 64 } 60 65 } 61 66 return 0; ··· 89 90 { 90 91 struct cn10k_rng *rng = (struct cn10k_rng *)hwrng->priv; 91 92 unsigned int size; 93 93 + u8 *pos = data; 92 94 int err = 0; 93 95 u64 value; 94 96 ··· 102 102 while (size >= 8) { 103 103 cn10k_read_trng(rng, &value); 104 104 105 105 - *((u64 *)data) = (u64)value; 105 105 + *((u64 *)pos) = value; 106 106 size -= 8; 107 107 - data += 8; 107 107 + pos += 8; 108 108 } 109 109 110 110 - while (size > 0) { 110 110 + if (size > 0) { 111 111 cn10k_read_trng(rng, &value); 112 112 113 113 - *((u8 *)data) = (u8)value; 114 114 - size--; 115 115 - data++; 113 113 + while (size > 0) { 114 114 + *pos = (u8)value; 115 115 + value >>= 8; 116 116 + size--; 117 117 + pos++; 118 118 + } 116 119 } 117 120 118 121 return max - size;

+104

drivers/char/hw_random/mpfs-rng.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* 3 3 + * Microchip PolarFire SoC (MPFS) hardware random driver 4 4 + * 5 5 + * Copyright (c) 2020-2022 Microchip Corporation. All rights reserved. 6 6 + * 7 7 + * Author: Conor Dooley <conor.dooley@microchip.com> 8 8 + */ 9 9 + 10 10 + #include <linux/module.h> 11 11 + #include <linux/hw_random.h> 12 12 + #include <linux/platform_device.h> 13 13 + #include <soc/microchip/mpfs.h> 14 14 + 15 15 + #define CMD_OPCODE 0x21 16 16 + #define CMD_DATA_SIZE 0U 17 17 + #define CMD_DATA NULL 18 18 + #define MBOX_OFFSET 0U 19 19 + #define RESP_OFFSET 0U 20 20 + #define RNG_RESP_BYTES 32U 21 21 + 22 22 + struct mpfs_rng { 23 23 + struct mpfs_sys_controller *sys_controller; 24 24 + struct hwrng rng; 25 25 + }; 26 26 + 27 27 + static int mpfs_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait) 28 28 + { 29 29 + struct mpfs_rng *rng_priv = container_of(rng, struct mpfs_rng, rng); 30 30 + u32 response_msg[RNG_RESP_BYTES / sizeof(u32)]; 31 31 + unsigned int count = 0, copy_size_bytes; 32 32 + int ret; 33 33 + 34 34 + struct mpfs_mss_response response = { 35 35 + .resp_status = 0U, 36 36 + .resp_msg = (u32 *)response_msg, 37 37 + .resp_size = RNG_RESP_BYTES 38 38 + }; 39 39 + struct mpfs_mss_msg msg = { 40 40 + .cmd_opcode = CMD_OPCODE, 41 41 + .cmd_data_size = CMD_DATA_SIZE, 42 42 + .response = &response, 43 43 + .cmd_data = CMD_DATA, 44 44 + .mbox_offset = MBOX_OFFSET, 45 45 + .resp_offset = RESP_OFFSET 46 46 + }; 47 47 + 48 48 + while (count < max) { 49 49 + ret = mpfs_blocking_transaction(rng_priv->sys_controller, &msg); 50 50 + if (ret) 51 51 + return ret; 52 52 + 53 53 + copy_size_bytes = max - count > RNG_RESP_BYTES ? RNG_RESP_BYTES : max - count; 54 54 + memcpy(buf + count, response_msg, copy_size_bytes); 55 55 + 56 56 + count += copy_size_bytes; 57 57 + if (!wait) 58 58 + break; 59 59 + } 60 60 + 61 61 + return count; 62 62 + } 63 63 + 64 64 + static int mpfs_rng_probe(struct platform_device *pdev) 65 65 + { 66 66 + struct device *dev = &pdev->dev; 67 67 + struct mpfs_rng *rng_priv; 68 68 + int ret; 69 69 + 70 70 + rng_priv = devm_kzalloc(dev, sizeof(*rng_priv), GFP_KERNEL); 71 71 + if (!rng_priv) 72 72 + return -ENOMEM; 73 73 + 74 74 + rng_priv->sys_controller = mpfs_sys_controller_get(&pdev->dev); 75 75 + if (IS_ERR(rng_priv->sys_controller)) 76 76 + return dev_err_probe(dev, PTR_ERR(rng_priv->sys_controller), 77 77 + "Failed to register system controller hwrng sub device\n"); 78 78 + 79 79 + rng_priv->rng.read = mpfs_rng_read; 80 80 + rng_priv->rng.name = pdev->name; 81 81 + rng_priv->rng.quality = 1024; 82 82 + 83 83 + platform_set_drvdata(pdev, rng_priv); 84 84 + 85 85 + ret = devm_hwrng_register(&pdev->dev, &rng_priv->rng); 86 86 + if (ret) 87 87 + return dev_err_probe(&pdev->dev, ret, "Failed to register MPFS hwrng\n"); 88 88 + 89 89 + dev_info(&pdev->dev, "Registered MPFS hwrng\n"); 90 90 + 91 91 + return 0; 92 92 + } 93 93 + 94 94 + static struct platform_driver mpfs_rng_driver = { 95 95 + .driver = { 96 96 + .name = "mpfs-rng", 97 97 + }, 98 98 + .probe = mpfs_rng_probe, 99 99 + }; 100 100 + module_platform_driver(mpfs_rng_driver); 101 101 + 102 102 + MODULE_LICENSE("GPL"); 103 103 + MODULE_AUTHOR("Conor Dooley <conor.dooley@microchip.com>"); 104 104 + MODULE_DESCRIPTION("PolarFire SoC (MPFS) hardware random driver");

+1 -1

drivers/char/hw_random/omap3-rom-rng.c

reviewed

··· 92 92 93 93 r = ddata->rom_rng_call(0, 0, RNG_GEN_PRNG_HW_INIT); 94 94 if (r != 0) { 95 95 - clk_disable(ddata->clk); 95 95 + clk_disable_unprepare(ddata->clk); 96 96 dev_err(dev, "HW init failed: %d\n", r); 97 97 98 98 return -EIO;

+1 -1

drivers/char/hw_random/optee-rng.c

reviewed

··· 115 115 static int optee_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait) 116 116 { 117 117 struct optee_rng_private *pvt_data = to_optee_rng_private(rng); 118 118 - size_t read = 0, rng_size = 0; 118 118 + size_t read = 0, rng_size; 119 119 int timeout = 1; 120 120 u8 *data = buf; 121 121

+2 -2

drivers/crypto/Kconfig

reviewed

··· 216 216 config CRYPTO_CHACHA_S390 217 217 tristate "ChaCha20 stream cipher" 218 218 depends on S390 219 219 - select CRYPTO_ALGAPI 220 219 select CRYPTO_SKCIPHER 221 221 - select CRYPTO_CHACHA20 220 220 + select CRYPTO_LIB_CHACHA_GENERIC 221 221 + select CRYPTO_ARCH_HAVE_LIB_CHACHA 222 222 help 223 223 This is the s390 SIMD implementation of the ChaCha20 stream 224 224 cipher (RFC 7539).

+1

drivers/crypto/Makefile

reviewed

··· 3 3 obj-$(CONFIG_CRYPTO_DEV_ATMEL_AES) += atmel-aes.o 4 4 obj-$(CONFIG_CRYPTO_DEV_ATMEL_SHA) += atmel-sha.o 5 5 obj-$(CONFIG_CRYPTO_DEV_ATMEL_TDES) += atmel-tdes.o 6 6 + # __init ordering requires atmel-i2c being before atmel-ecc and atmel-sha204a. 6 7 obj-$(CONFIG_CRYPTO_DEV_ATMEL_I2C) += atmel-i2c.o 7 8 obj-$(CONFIG_CRYPTO_DEV_ATMEL_ECC) += atmel-ecc.o 8 9 obj-$(CONFIG_CRYPTO_DEV_ATMEL_SHA204A) += atmel-sha204a.o

+8 -14

drivers/crypto/allwinner/sun4i-ss/sun4i-ss-cipher.c

reviewed

··· 20 20 unsigned int ivsize = crypto_skcipher_ivsize(tfm); 21 21 struct sun4i_cipher_req_ctx *ctx = skcipher_request_ctx(areq); 22 22 u32 mode = ctx->mode; 23 23 - void *backup_iv = NULL; 24 23 /* when activating SS, the default FIFO space is SS_RX_DEFAULT(32) */ 25 24 u32 rx_cnt = SS_RX_DEFAULT; 26 25 u32 tx_cnt = 0; ··· 47 48 } 48 49 49 50 if (areq->iv && ivsize > 0 && mode & SS_DECRYPTION) { 50 50 - backup_iv = kzalloc(ivsize, GFP_KERNEL); 51 51 - if (!backup_iv) 52 52 - return -ENOMEM; 53 53 - scatterwalk_map_and_copy(backup_iv, areq->src, areq->cryptlen - ivsize, ivsize, 0); 51 51 + scatterwalk_map_and_copy(ctx->backup_iv, areq->src, 52 52 + areq->cryptlen - ivsize, ivsize, 0); 54 53 } 55 54 56 55 if (IS_ENABLED(CONFIG_CRYPTO_DEV_SUN4I_SS_DEBUG)) { ··· 131 134 132 135 if (areq->iv) { 133 136 if (mode & SS_DECRYPTION) { 134 134 - memcpy(areq->iv, backup_iv, ivsize); 135 135 - kfree_sensitive(backup_iv); 137 137 + memcpy(areq->iv, ctx->backup_iv, ivsize); 138 138 + memzero_explicit(ctx->backup_iv, ivsize); 136 139 } else { 137 140 scatterwalk_map_and_copy(areq->iv, areq->dst, areq->cryptlen - ivsize, 138 141 ivsize, 0); ··· 196 199 unsigned int ileft = areq->cryptlen; 197 200 unsigned int oleft = areq->cryptlen; 198 201 unsigned int todo; 199 199 - void *backup_iv = NULL; 200 202 struct sg_mapping_iter mi, mo; 201 203 unsigned long pi = 0, po = 0; /* progress for in and out */ 202 204 bool miter_err; ··· 240 244 return sun4i_ss_cipher_poll_fallback(areq); 241 245 242 246 if (areq->iv && ivsize > 0 && mode & SS_DECRYPTION) { 243 243 - backup_iv = kzalloc(ivsize, GFP_KERNEL); 244 244 - if (!backup_iv) 245 245 - return -ENOMEM; 246 246 - scatterwalk_map_and_copy(backup_iv, areq->src, areq->cryptlen - ivsize, ivsize, 0); 247 247 + scatterwalk_map_and_copy(ctx->backup_iv, areq->src, 248 248 + areq->cryptlen - ivsize, ivsize, 0); 247 249 } 248 250 249 251 if (IS_ENABLED(CONFIG_CRYPTO_DEV_SUN4I_SS_DEBUG)) { ··· 378 384 } 379 385 if (areq->iv) { 380 386 if (mode & SS_DECRYPTION) { 381 381 - memcpy(areq->iv, backup_iv, ivsize); 382 382 - kfree_sensitive(backup_iv); 387 387 + memcpy(areq->iv, ctx->backup_iv, ivsize); 388 388 + memzero_explicit(ctx->backup_iv, ivsize); 383 389 } else { 384 390 scatterwalk_map_and_copy(areq->iv, areq->dst, areq->cryptlen - ivsize, 385 391 ivsize, 0);

+1

drivers/crypto/allwinner/sun4i-ss/sun4i-ss.h

reviewed

··· 183 183 184 184 struct sun4i_cipher_req_ctx { 185 185 u32 mode; 186 186 + u8 backup_iv[AES_BLOCK_SIZE]; 186 187 struct skcipher_request fallback_req; // keep at the end 187 188 }; 188 189

+61 -41

drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c

reviewed

··· 25 25 { 26 26 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); 27 27 struct scatterlist *sg; 28 28 + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); 29 29 + struct sun8i_ce_alg_template *algt; 30 30 + unsigned int todo, len; 28 31 29 29 - if (sg_nents(areq->src) > MAX_SG || sg_nents(areq->dst) > MAX_SG) 32 32 + algt = container_of(alg, struct sun8i_ce_alg_template, alg.skcipher); 33 33 + 34 34 + if (sg_nents_for_len(areq->src, areq->cryptlen) > MAX_SG || 35 35 + sg_nents_for_len(areq->dst, areq->cryptlen) > MAX_SG) { 36 36 + algt->stat_fb_maxsg++; 30 37 return true; 38 38 + } 31 39 32 32 - if (areq->cryptlen < crypto_skcipher_ivsize(tfm)) 40 40 + if (areq->cryptlen < crypto_skcipher_ivsize(tfm)) { 41 41 + algt->stat_fb_leniv++; 33 42 return true; 43 43 + } 34 44 35 35 - if (areq->cryptlen == 0 || areq->cryptlen % 16) 45 45 + if (areq->cryptlen == 0) { 46 46 + algt->stat_fb_len0++; 36 47 return true; 48 48 + } 37 49 50 50 + if (areq->cryptlen % 16) { 51 51 + algt->stat_fb_mod16++; 52 52 + return true; 53 53 + } 54 54 + 55 55 + len = areq->cryptlen; 38 56 sg = areq->src; 39 57 while (sg) { 40 40 - if (sg->length % 4 || !IS_ALIGNED(sg->offset, sizeof(u32))) 58 58 + if (!IS_ALIGNED(sg->offset, sizeof(u32))) { 59 59 + algt->stat_fb_srcali++; 41 60 return true; 61 61 + } 62 62 + todo = min(len, sg->length); 63 63 + if (todo % 4) { 64 64 + algt->stat_fb_srclen++; 65 65 + return true; 66 66 + } 67 67 + len -= todo; 42 68 sg = sg_next(sg); 43 69 } 70 70 + 71 71 + len = areq->cryptlen; 44 72 sg = areq->dst; 45 73 while (sg) { 46 46 - if (sg->length % 4 || !IS_ALIGNED(sg->offset, sizeof(u32))) 74 74 + if (!IS_ALIGNED(sg->offset, sizeof(u32))) { 75 75 + algt->stat_fb_dstali++; 47 76 return true; 77 77 + } 78 78 + todo = min(len, sg->length); 79 79 + if (todo % 4) { 80 80 + algt->stat_fb_dstlen++; 81 81 + return true; 82 82 + } 83 83 + len -= todo; 48 84 sg = sg_next(sg); 49 85 } 50 86 return false; ··· 130 94 int nr_sgs = 0; 131 95 int nr_sgd = 0; 132 96 int err = 0; 97 97 + int ns = sg_nents_for_len(areq->src, areq->cryptlen); 98 98 + int nd = sg_nents_for_len(areq->dst, areq->cryptlen); 133 99 134 100 algt = container_of(alg, struct sun8i_ce_alg_template, alg.skcipher); 135 101 ··· 190 152 ivsize = crypto_skcipher_ivsize(tfm); 191 153 if (areq->iv && crypto_skcipher_ivsize(tfm) > 0) { 192 154 rctx->ivlen = ivsize; 193 193 - rctx->bounce_iv = kzalloc(ivsize, GFP_KERNEL | GFP_DMA); 194 194 - if (!rctx->bounce_iv) { 195 195 - err = -ENOMEM; 196 196 - goto theend_key; 197 197 - } 198 155 if (rctx->op_dir & CE_DECRYPTION) { 199 199 - rctx->backup_iv = kzalloc(ivsize, GFP_KERNEL); 200 200 - if (!rctx->backup_iv) { 201 201 - err = -ENOMEM; 202 202 - goto theend_key; 203 203 - } 204 156 offset = areq->cryptlen - ivsize; 205 205 - scatterwalk_map_and_copy(rctx->backup_iv, areq->src, 157 157 + scatterwalk_map_and_copy(chan->backup_iv, areq->src, 206 158 offset, ivsize, 0); 207 159 } 208 208 - memcpy(rctx->bounce_iv, areq->iv, ivsize); 209 209 - rctx->addr_iv = dma_map_single(ce->dev, rctx->bounce_iv, rctx->ivlen, 160 160 + memcpy(chan->bounce_iv, areq->iv, ivsize); 161 161 + rctx->addr_iv = dma_map_single(ce->dev, chan->bounce_iv, rctx->ivlen, 210 162 DMA_TO_DEVICE); 211 163 if (dma_mapping_error(ce->dev, rctx->addr_iv)) { 212 164 dev_err(ce->dev, "Cannot DMA MAP IV\n"); ··· 207 179 } 208 180 209 181 if (areq->src == areq->dst) { 210 210 - nr_sgs = dma_map_sg(ce->dev, areq->src, sg_nents(areq->src), 211 211 - DMA_BIDIRECTIONAL); 182 182 + nr_sgs = dma_map_sg(ce->dev, areq->src, ns, DMA_BIDIRECTIONAL); 212 183 if (nr_sgs <= 0 || nr_sgs > MAX_SG) { 213 184 dev_err(ce->dev, "Invalid sg number %d\n", nr_sgs); 214 185 err = -EINVAL; ··· 215 188 } 216 189 nr_sgd = nr_sgs; 217 190 } else { 218 218 - nr_sgs = dma_map_sg(ce->dev, areq->src, sg_nents(areq->src), 219 219 - DMA_TO_DEVICE); 191 191 + nr_sgs = dma_map_sg(ce->dev, areq->src, ns, DMA_TO_DEVICE); 220 192 if (nr_sgs <= 0 || nr_sgs > MAX_SG) { 221 193 dev_err(ce->dev, "Invalid sg number %d\n", nr_sgs); 222 194 err = -EINVAL; 223 195 goto theend_iv; 224 196 } 225 225 - nr_sgd = dma_map_sg(ce->dev, areq->dst, sg_nents(areq->dst), 226 226 - DMA_FROM_DEVICE); 197 197 + nr_sgd = dma_map_sg(ce->dev, areq->dst, nd, DMA_FROM_DEVICE); 227 198 if (nr_sgd <= 0 || nr_sgd > MAX_SG) { 228 199 dev_err(ce->dev, "Invalid sg number %d\n", nr_sgd); 229 200 err = -EINVAL; ··· 266 241 267 242 theend_sgs: 268 243 if (areq->src == areq->dst) { 269 269 - dma_unmap_sg(ce->dev, areq->src, sg_nents(areq->src), 270 270 - DMA_BIDIRECTIONAL); 244 244 + dma_unmap_sg(ce->dev, areq->src, ns, DMA_BIDIRECTIONAL); 271 245 } else { 272 246 if (nr_sgs > 0) 273 273 - dma_unmap_sg(ce->dev, areq->src, sg_nents(areq->src), 274 274 - DMA_TO_DEVICE); 275 275 - dma_unmap_sg(ce->dev, areq->dst, sg_nents(areq->dst), 276 276 - DMA_FROM_DEVICE); 247 247 + dma_unmap_sg(ce->dev, areq->src, ns, DMA_TO_DEVICE); 248 248 + dma_unmap_sg(ce->dev, areq->dst, nd, DMA_FROM_DEVICE); 277 249 } 278 250 279 251 theend_iv: ··· 279 257 dma_unmap_single(ce->dev, rctx->addr_iv, rctx->ivlen, DMA_TO_DEVICE); 280 258 offset = areq->cryptlen - ivsize; 281 259 if (rctx->op_dir & CE_DECRYPTION) { 282 282 - memcpy(areq->iv, rctx->backup_iv, ivsize); 283 283 - kfree_sensitive(rctx->backup_iv); 260 260 + memcpy(areq->iv, chan->backup_iv, ivsize); 261 261 + memzero_explicit(chan->backup_iv, ivsize); 284 262 } else { 285 263 scatterwalk_map_and_copy(areq->iv, areq->dst, offset, 286 264 ivsize, 0); 287 265 } 288 288 - kfree(rctx->bounce_iv); 266 266 + memzero_explicit(chan->bounce_iv, ivsize); 289 267 } 290 268 291 291 - theend_key: 292 269 dma_unmap_single(ce->dev, rctx->addr_key, op->keylen, DMA_TO_DEVICE); 293 270 294 271 theend: ··· 343 322 dma_unmap_single(ce->dev, rctx->addr_iv, rctx->ivlen, DMA_TO_DEVICE); 344 323 offset = areq->cryptlen - ivsize; 345 324 if (rctx->op_dir & CE_DECRYPTION) { 346 346 - memcpy(areq->iv, rctx->backup_iv, ivsize); 347 347 - kfree_sensitive(rctx->backup_iv); 325 325 + memcpy(areq->iv, chan->backup_iv, ivsize); 326 326 + memzero_explicit(chan->backup_iv, ivsize); 348 327 } else { 349 328 scatterwalk_map_and_copy(areq->iv, areq->dst, offset, 350 329 ivsize, 0); 351 330 } 352 352 - kfree(rctx->bounce_iv); 331 331 + memzero_explicit(chan->bounce_iv, ivsize); 353 332 } 354 333 355 334 dma_unmap_single(ce->dev, rctx->addr_key, op->keylen, DMA_TO_DEVICE); ··· 419 398 sktfm->reqsize = sizeof(struct sun8i_cipher_req_ctx) + 420 399 crypto_skcipher_reqsize(op->fallback_tfm); 421 400 422 422 - 423 423 - dev_info(op->ce->dev, "Fallback for %s is %s\n", 424 424 - crypto_tfm_alg_driver_name(&sktfm->base), 425 425 - crypto_tfm_alg_driver_name(crypto_skcipher_tfm(op->fallback_tfm))); 401 401 + memcpy(algt->fbname, 402 402 + crypto_tfm_alg_driver_name(crypto_skcipher_tfm(op->fallback_tfm)), 403 403 + CRYPTO_MAX_ALG_NAME); 426 404 427 405 op->enginectx.op.do_one_request = sun8i_ce_cipher_run; 428 406 op->enginectx.op.prepare_request = sun8i_ce_cipher_prepare;

+47 -7

drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c

reviewed

··· 283 283 .cra_priority = 400, 284 284 .cra_blocksize = AES_BLOCK_SIZE, 285 285 .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | 286 286 - CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY | 286 286 + CRYPTO_ALG_ASYNC | 287 287 CRYPTO_ALG_NEED_FALLBACK, 288 288 .cra_ctxsize = sizeof(struct sun8i_cipher_tfm_ctx), 289 289 .cra_module = THIS_MODULE, ··· 310 310 .cra_priority = 400, 311 311 .cra_blocksize = AES_BLOCK_SIZE, 312 312 .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | 313 313 - CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY | 313 313 + CRYPTO_ALG_ASYNC | 314 314 CRYPTO_ALG_NEED_FALLBACK, 315 315 .cra_ctxsize = sizeof(struct sun8i_cipher_tfm_ctx), 316 316 .cra_module = THIS_MODULE, ··· 336 336 .cra_priority = 400, 337 337 .cra_blocksize = DES3_EDE_BLOCK_SIZE, 338 338 .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | 339 339 - CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY | 339 339 + CRYPTO_ALG_ASYNC | 340 340 CRYPTO_ALG_NEED_FALLBACK, 341 341 .cra_ctxsize = sizeof(struct sun8i_cipher_tfm_ctx), 342 342 .cra_module = THIS_MODULE, ··· 363 363 .cra_priority = 400, 364 364 .cra_blocksize = DES3_EDE_BLOCK_SIZE, 365 365 .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | 366 366 - CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY | 366 366 + CRYPTO_ALG_ASYNC | 367 367 CRYPTO_ALG_NEED_FALLBACK, 368 368 .cra_ctxsize = sizeof(struct sun8i_cipher_tfm_ctx), 369 369 .cra_module = THIS_MODULE, ··· 595 595 continue; 596 596 switch (ce_algs[i].type) { 597 597 case CRYPTO_ALG_TYPE_SKCIPHER: 598 598 - seq_printf(seq, "%s %s %lu %lu\n", 598 598 + seq_printf(seq, "%s %s reqs=%lu fallback=%lu\n", 599 599 ce_algs[i].alg.skcipher.base.cra_driver_name, 600 600 ce_algs[i].alg.skcipher.base.cra_name, 601 601 ce_algs[i].stat_req, ce_algs[i].stat_fb); 602 602 + seq_printf(seq, "\tLast fallback is: %s\n", 603 603 + ce_algs[i].fbname); 604 604 + seq_printf(seq, "\tFallback due to 0 length: %lu\n", 605 605 + ce_algs[i].stat_fb_len0); 606 606 + seq_printf(seq, "\tFallback due to length !mod16: %lu\n", 607 607 + ce_algs[i].stat_fb_mod16); 608 608 + seq_printf(seq, "\tFallback due to length < IV: %lu\n", 609 609 + ce_algs[i].stat_fb_leniv); 610 610 + seq_printf(seq, "\tFallback due to source alignment: %lu\n", 611 611 + ce_algs[i].stat_fb_srcali); 612 612 + seq_printf(seq, "\tFallback due to dest alignment: %lu\n", 613 613 + ce_algs[i].stat_fb_dstali); 614 614 + seq_printf(seq, "\tFallback due to source length: %lu\n", 615 615 + ce_algs[i].stat_fb_srclen); 616 616 + seq_printf(seq, "\tFallback due to dest length: %lu\n", 617 617 + ce_algs[i].stat_fb_dstlen); 618 618 + seq_printf(seq, "\tFallback due to SG numbers: %lu\n", 619 619 + ce_algs[i].stat_fb_maxsg); 602 620 break; 603 621 case CRYPTO_ALG_TYPE_AHASH: 604 604 - seq_printf(seq, "%s %s %lu %lu\n", 622 622 + seq_printf(seq, "%s %s reqs=%lu fallback=%lu\n", 605 623 ce_algs[i].alg.hash.halg.base.cra_driver_name, 606 624 ce_algs[i].alg.hash.halg.base.cra_name, 607 625 ce_algs[i].stat_req, ce_algs[i].stat_fb); 626 626 + seq_printf(seq, "\tLast fallback is: %s\n", 627 627 + ce_algs[i].fbname); 628 628 + seq_printf(seq, "\tFallback due to 0 length: %lu\n", 629 629 + ce_algs[i].stat_fb_len0); 630 630 + seq_printf(seq, "\tFallback due to length: %lu\n", 631 631 + ce_algs[i].stat_fb_srclen); 632 632 + seq_printf(seq, "\tFallback due to alignment: %lu\n", 633 633 + ce_algs[i].stat_fb_srcali); 634 634 + seq_printf(seq, "\tFallback due to SG numbers: %lu\n", 635 635 + ce_algs[i].stat_fb_maxsg); 608 636 break; 609 637 case CRYPTO_ALG_TYPE_RNG: 610 610 - seq_printf(seq, "%s %s %lu %lu\n", 638 638 + seq_printf(seq, "%s %s reqs=%lu bytes=%lu\n", 611 639 ce_algs[i].alg.rng.base.cra_driver_name, 612 640 ce_algs[i].alg.rng.base.cra_name, 613 641 ce_algs[i].stat_req, ce_algs[i].stat_bytes); ··· 698 670 if (!ce->chanlist[i].tl) { 699 671 dev_err(ce->dev, "Cannot get DMA memory for task %d\n", 700 672 i); 673 673 + err = -ENOMEM; 674 674 + goto error_engine; 675 675 + } 676 676 + ce->chanlist[i].bounce_iv = devm_kmalloc(ce->dev, AES_BLOCK_SIZE, 677 677 + GFP_KERNEL | GFP_DMA); 678 678 + if (!ce->chanlist[i].bounce_iv) { 679 679 + err = -ENOMEM; 680 680 + goto error_engine; 681 681 + } 682 682 + ce->chanlist[i].backup_iv = devm_kmalloc(ce->dev, AES_BLOCK_SIZE, 683 683 + GFP_KERNEL); 684 684 + if (!ce->chanlist[i].backup_iv) { 701 685 err = -ENOMEM; 702 686 goto error_engine; 703 687 }

+90 -40

drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c

reviewed

··· 50 50 sizeof(struct sun8i_ce_hash_reqctx) + 51 51 crypto_ahash_reqsize(op->fallback_tfm)); 52 52 53 53 - dev_info(op->ce->dev, "Fallback for %s is %s\n", 54 54 - crypto_tfm_alg_driver_name(tfm), 55 55 - crypto_tfm_alg_driver_name(&op->fallback_tfm->base)); 53 53 + memcpy(algt->fbname, crypto_tfm_alg_driver_name(&op->fallback_tfm->base), 54 54 + CRYPTO_MAX_ALG_NAME); 55 55 + 56 56 err = pm_runtime_get_sync(op->ce->dev); 57 57 if (err < 0) 58 58 goto error_pm; ··· 199 199 200 200 static bool sun8i_ce_hash_need_fallback(struct ahash_request *areq) 201 201 { 202 202 + struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); 203 203 + struct ahash_alg *alg = __crypto_ahash_alg(tfm->base.__crt_alg); 204 204 + struct sun8i_ce_alg_template *algt; 202 205 struct scatterlist *sg; 203 206 204 204 - if (areq->nbytes == 0) 207 207 + algt = container_of(alg, struct sun8i_ce_alg_template, alg.hash); 208 208 + 209 209 + if (areq->nbytes == 0) { 210 210 + algt->stat_fb_len0++; 205 211 return true; 212 212 + } 206 213 /* we need to reserve one SG for padding one */ 207 207 - if (sg_nents(areq->src) > MAX_SG - 1) 214 214 + if (sg_nents_for_len(areq->src, areq->nbytes) > MAX_SG - 1) { 215 215 + algt->stat_fb_maxsg++; 208 216 return true; 217 217 + } 209 218 sg = areq->src; 210 219 while (sg) { 211 211 - if (sg->length % 4 || !IS_ALIGNED(sg->offset, sizeof(u32))) 220 220 + if (sg->length % 4) { 221 221 + algt->stat_fb_srclen++; 212 222 return true; 223 223 + } 224 224 + if (!IS_ALIGNED(sg->offset, sizeof(u32))) { 225 225 + algt->stat_fb_srcali++; 226 226 + return true; 227 227 + } 213 228 sg = sg_next(sg); 214 229 } 215 230 return false; ··· 244 229 if (sun8i_ce_hash_need_fallback(areq)) 245 230 return sun8i_ce_hash_digest_fb(areq); 246 231 247 247 - nr_sgs = sg_nents(areq->src); 232 232 + nr_sgs = sg_nents_for_len(areq->src, areq->nbytes); 248 233 if (nr_sgs > MAX_SG - 1) 249 234 return sun8i_ce_hash_digest_fb(areq); 250 235 ··· 261 246 engine = ce->chanlist[e].engine; 262 247 263 248 return crypto_transfer_hash_request_to_engine(engine, areq); 249 249 + } 250 250 + 251 251 + static u64 hash_pad(__le32 *buf, unsigned int bufsize, u64 padi, u64 byte_count, bool le, int bs) 252 252 + { 253 253 + u64 fill, min_fill, j, k; 254 254 + __be64 *bebits; 255 255 + __le64 *lebits; 256 256 + 257 257 + j = padi; 258 258 + buf[j++] = cpu_to_le32(0x80); 259 259 + 260 260 + if (bs == 64) { 261 261 + fill = 64 - (byte_count % 64); 262 262 + min_fill = 2 * sizeof(u32) + sizeof(u32); 263 263 + } else { 264 264 + fill = 128 - (byte_count % 128); 265 265 + min_fill = 4 * sizeof(u32) + sizeof(u32); 266 266 + } 267 267 + 268 268 + if (fill < min_fill) 269 269 + fill += bs; 270 270 + 271 271 + k = j; 272 272 + j += (fill - min_fill) / sizeof(u32); 273 273 + if (j * 4 > bufsize) { 274 274 + pr_err("%s OVERFLOW %llu\n", __func__, j); 275 275 + return 0; 276 276 + } 277 277 + for (; k < j; k++) 278 278 + buf[k] = 0; 279 279 + 280 280 + if (le) { 281 281 + /* MD5 */ 282 282 + lebits = (__le64 *)&buf[j]; 283 283 + *lebits = cpu_to_le64(byte_count << 3); 284 284 + j += 2; 285 285 + } else { 286 286 + if (bs == 64) { 287 287 + /* sha1 sha224 sha256 */ 288 288 + bebits = (__be64 *)&buf[j]; 289 289 + *bebits = cpu_to_be64(byte_count << 3); 290 290 + j += 2; 291 291 + } else { 292 292 + /* sha384 sha512*/ 293 293 + bebits = (__be64 *)&buf[j]; 294 294 + *bebits = cpu_to_be64(byte_count >> 61); 295 295 + j += 2; 296 296 + bebits = (__be64 *)&buf[j]; 297 297 + *bebits = cpu_to_be64(byte_count << 3); 298 298 + j += 2; 299 299 + } 300 300 + } 301 301 + if (j * 4 > bufsize) { 302 302 + pr_err("%s OVERFLOW %llu\n", __func__, j); 303 303 + return 0; 304 304 + } 305 305 + 306 306 + return j; 264 307 } 265 308 266 309 int sun8i_ce_hash_run(struct crypto_engine *engine, void *breq) ··· 339 266 __le32 *bf; 340 267 void *buf = NULL; 341 268 int j, i, todo; 342 342 - int nbw = 0; 343 343 - u64 fill, min_fill; 344 344 - __be64 *bebits; 345 345 - __le64 *lebits; 346 269 void *result = NULL; 347 270 u64 bs; 348 271 int digestsize; 349 272 dma_addr_t addr_res, addr_pad; 273 273 + int ns = sg_nents_for_len(areq->src, areq->nbytes); 350 274 351 275 algt = container_of(alg, struct sun8i_ce_alg_template, alg.hash); 352 276 ce = algt->ce; ··· 388 318 cet->t_sym_ctl = 0; 389 319 cet->t_asym_ctl = 0; 390 320 391 391 - nr_sgs = dma_map_sg(ce->dev, areq->src, sg_nents(areq->src), DMA_TO_DEVICE); 321 321 + nr_sgs = dma_map_sg(ce->dev, areq->src, ns, DMA_TO_DEVICE); 392 322 if (nr_sgs <= 0 || nr_sgs > MAX_SG) { 393 323 dev_err(ce->dev, "Invalid sg number %d\n", nr_sgs); 394 324 err = -EINVAL; ··· 418 348 419 349 byte_count = areq->nbytes; 420 350 j = 0; 421 421 - bf[j++] = cpu_to_le32(0x80); 422 422 - 423 423 - if (bs == 64) { 424 424 - fill = 64 - (byte_count % 64); 425 425 - min_fill = 2 * sizeof(u32) + (nbw ? 0 : sizeof(u32)); 426 426 - } else { 427 427 - fill = 128 - (byte_count % 128); 428 428 - min_fill = 4 * sizeof(u32) + (nbw ? 0 : sizeof(u32)); 429 429 - } 430 430 - 431 431 - if (fill < min_fill) 432 432 - fill += bs; 433 433 - 434 434 - j += (fill - min_fill) / sizeof(u32); 435 351 436 352 switch (algt->ce_algo_id) { 437 353 case CE_ID_HASH_MD5: 438 438 - lebits = (__le64 *)&bf[j]; 439 439 - *lebits = cpu_to_le64(byte_count << 3); 440 440 - j += 2; 354 354 + j = hash_pad(bf, 2 * bs, j, byte_count, true, bs); 441 355 break; 442 356 case CE_ID_HASH_SHA1: 443 357 case CE_ID_HASH_SHA224: 444 358 case CE_ID_HASH_SHA256: 445 445 - bebits = (__be64 *)&bf[j]; 446 446 - *bebits = cpu_to_be64(byte_count << 3); 447 447 - j += 2; 359 359 + j = hash_pad(bf, 2 * bs, j, byte_count, false, bs); 448 360 break; 449 361 case CE_ID_HASH_SHA384: 450 362 case CE_ID_HASH_SHA512: 451 451 - bebits = (__be64 *)&bf[j]; 452 452 - *bebits = cpu_to_be64(byte_count >> 61); 453 453 - j += 2; 454 454 - bebits = (__be64 *)&bf[j]; 455 455 - *bebits = cpu_to_be64(byte_count << 3); 456 456 - j += 2; 363 363 + j = hash_pad(bf, 2 * bs, j, byte_count, false, bs); 457 364 break; 365 365 + } 366 366 + if (!j) { 367 367 + err = -EINVAL; 368 368 + goto theend; 458 369 } 459 370 460 371 addr_pad = dma_map_single(ce->dev, buf, j * 4, DMA_TO_DEVICE); ··· 457 406 err = sun8i_ce_run_task(ce, flow, crypto_tfm_alg_name(areq->base.tfm)); 458 407 459 408 dma_unmap_single(ce->dev, addr_pad, j * 4, DMA_TO_DEVICE); 460 460 - dma_unmap_sg(ce->dev, areq->src, sg_nents(areq->src), 461 461 - DMA_TO_DEVICE); 409 409 + dma_unmap_sg(ce->dev, areq->src, ns, DMA_TO_DEVICE); 462 410 dma_unmap_single(ce->dev, addr_res, digestsize, DMA_FROM_DEVICE); 463 411 464 412

+2 -4

drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c

reviewed

··· 108 108 goto err_dst; 109 109 } 110 110 111 111 - err = pm_runtime_get_sync(ce->dev); 112 112 - if (err < 0) { 113 113 - pm_runtime_put_noidle(ce->dev); 111 111 + err = pm_runtime_resume_and_get(ce->dev); 112 112 + if (err < 0) 114 113 goto err_pm; 115 115 - } 116 114 117 115 mutex_lock(&ce->rnglock); 118 116 chan = &ce->chanlist[flow];

+13 -6

drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h

reviewed

··· 186 186 * @status: set to 1 by interrupt if task is done 187 187 * @t_phy: Physical address of task 188 188 * @tl: pointer to the current ce_task for this flow 189 189 + * @backup_iv: buffer which contain the next IV to store 190 190 + * @bounce_iv: buffer which contain the IV 189 191 * @stat_req: number of request done by this flow 190 192 */ 191 193 struct sun8i_ce_flow { ··· 197 195 dma_addr_t t_phy; 198 196 int timeout; 199 197 struct ce_task *tl; 198 198 + void *backup_iv; 199 199 + void *bounce_iv; 200 200 #ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_DEBUG 201 201 unsigned long stat_req; 202 202 #endif ··· 245 241 * struct sun8i_cipher_req_ctx - context for a skcipher request 246 242 * @op_dir: direction (encrypt vs decrypt) for this request 247 243 * @flow: the flow to use for this request 248 248 - * @backup_iv: buffer which contain the next IV to store 249 249 - * @bounce_iv: buffer which contain the IV 250 244 * @ivlen: size of bounce_iv 251 245 * @nr_sgs: The number of source SG (as given by dma_map_sg()) 252 246 * @nr_sgd: The number of destination SG (as given by dma_map_sg()) ··· 255 253 struct sun8i_cipher_req_ctx { 256 254 u32 op_dir; 257 255 int flow; 258 258 - void *backup_iv; 259 259 - void *bounce_iv; 260 256 unsigned int ivlen; 261 257 int nr_sgs; 262 258 int nr_sgd; ··· 333 333 struct ahash_alg hash; 334 334 struct rng_alg rng; 335 335 } alg; 336 336 - #ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_DEBUG 337 336 unsigned long stat_req; 338 337 unsigned long stat_fb; 339 338 unsigned long stat_bytes; 340 340 - #endif 339 339 + unsigned long stat_fb_maxsg; 340 340 + unsigned long stat_fb_leniv; 341 341 + unsigned long stat_fb_len0; 342 342 + unsigned long stat_fb_mod16; 343 343 + unsigned long stat_fb_srcali; 344 344 + unsigned long stat_fb_srclen; 345 345 + unsigned long stat_fb_dstali; 346 346 + unsigned long stat_fb_dstlen; 347 347 + char fbname[CRYPTO_MAX_ALG_NAME]; 341 348 }; 342 349 343 350 int sun8i_ce_enqueue(struct crypto_async_request *areq, u32 type);

+117 -63

drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c

reviewed

··· 22 22 23 23 static bool sun8i_ss_need_fallback(struct skcipher_request *areq) 24 24 { 25 25 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); 26 26 + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); 27 27 + struct sun8i_ss_alg_template *algt = container_of(alg, struct sun8i_ss_alg_template, alg.skcipher); 25 28 struct scatterlist *in_sg = areq->src; 26 29 struct scatterlist *out_sg = areq->dst; 27 30 struct scatterlist *sg; 31 31 + unsigned int todo, len; 28 32 29 29 - if (areq->cryptlen == 0 || areq->cryptlen % 16) 33 33 + if (areq->cryptlen == 0 || areq->cryptlen % 16) { 34 34 + algt->stat_fb_len++; 30 35 return true; 36 36 + } 31 37 32 32 - if (sg_nents(areq->src) > 8 || sg_nents(areq->dst) > 8) 38 38 + if (sg_nents_for_len(areq->src, areq->cryptlen) > 8 || 39 39 + sg_nents_for_len(areq->dst, areq->cryptlen) > 8) { 40 40 + algt->stat_fb_sgnum++; 33 41 return true; 42 42 + } 34 43 44 44 + len = areq->cryptlen; 35 45 sg = areq->src; 36 46 while (sg) { 37 37 - if ((sg->length % 16) != 0) 47 47 + todo = min(len, sg->length); 48 48 + if ((todo % 16) != 0) { 49 49 + algt->stat_fb_sglen++; 38 50 return true; 39 39 - if ((sg_dma_len(sg) % 16) != 0) 51 51 + } 52 52 + if (!IS_ALIGNED(sg->offset, 16)) { 53 53 + algt->stat_fb_align++; 40 54 return true; 41 41 - if (!IS_ALIGNED(sg->offset, 16)) 42 42 - return true; 55 55 + } 56 56 + len -= todo; 43 57 sg = sg_next(sg); 44 58 } 59 59 + len = areq->cryptlen; 45 60 sg = areq->dst; 46 61 while (sg) { 47 47 - if ((sg->length % 16) != 0) 62 62 + todo = min(len, sg->length); 63 63 + if ((todo % 16) != 0) { 64 64 + algt->stat_fb_sglen++; 48 65 return true; 49 49 - if ((sg_dma_len(sg) % 16) != 0) 66 66 + } 67 67 + if (!IS_ALIGNED(sg->offset, 16)) { 68 68 + algt->stat_fb_align++; 50 69 return true; 51 51 - if (!IS_ALIGNED(sg->offset, 16)) 52 52 - return true; 70 70 + } 71 71 + len -= todo; 53 72 sg = sg_next(sg); 54 73 } 55 74 ··· 112 93 return err; 113 94 } 114 95 96 96 + static int sun8i_ss_setup_ivs(struct skcipher_request *areq) 97 97 + { 98 98 + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); 99 99 + struct sun8i_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); 100 100 + struct sun8i_ss_dev *ss = op->ss; 101 101 + struct sun8i_cipher_req_ctx *rctx = skcipher_request_ctx(areq); 102 102 + struct scatterlist *sg = areq->src; 103 103 + unsigned int todo, offset; 104 104 + unsigned int len = areq->cryptlen; 105 105 + unsigned int ivsize = crypto_skcipher_ivsize(tfm); 106 106 + struct sun8i_ss_flow *sf = &ss->flows[rctx->flow]; 107 107 + int i = 0; 108 108 + u32 a; 109 109 + int err; 110 110 + 111 111 + rctx->ivlen = ivsize; 112 112 + if (rctx->op_dir & SS_DECRYPTION) { 113 113 + offset = areq->cryptlen - ivsize; 114 114 + scatterwalk_map_and_copy(sf->biv, areq->src, offset, 115 115 + ivsize, 0); 116 116 + } 117 117 + 118 118 + /* we need to copy all IVs from source in case DMA is bi-directionnal */ 119 119 + while (sg && len) { 120 120 + if (sg_dma_len(sg) == 0) { 121 121 + sg = sg_next(sg); 122 122 + continue; 123 123 + } 124 124 + if (i == 0) 125 125 + memcpy(sf->iv[0], areq->iv, ivsize); 126 126 + a = dma_map_single(ss->dev, sf->iv[i], ivsize, DMA_TO_DEVICE); 127 127 + if (dma_mapping_error(ss->dev, a)) { 128 128 + memzero_explicit(sf->iv[i], ivsize); 129 129 + dev_err(ss->dev, "Cannot DMA MAP IV\n"); 130 130 + err = -EFAULT; 131 131 + goto dma_iv_error; 132 132 + } 133 133 + rctx->p_iv[i] = a; 134 134 + /* we need to setup all others IVs only in the decrypt way */ 135 135 + if (rctx->op_dir & SS_ENCRYPTION) 136 136 + return 0; 137 137 + todo = min(len, sg_dma_len(sg)); 138 138 + len -= todo; 139 139 + i++; 140 140 + if (i < MAX_SG) { 141 141 + offset = sg->length - ivsize; 142 142 + scatterwalk_map_and_copy(sf->iv[i], sg, offset, ivsize, 0); 143 143 + } 144 144 + rctx->niv = i; 145 145 + sg = sg_next(sg); 146 146 + } 147 147 + 148 148 + return 0; 149 149 + dma_iv_error: 150 150 + i--; 151 151 + while (i >= 0) { 152 152 + dma_unmap_single(ss->dev, rctx->p_iv[i], ivsize, DMA_TO_DEVICE); 153 153 + memzero_explicit(sf->iv[i], ivsize); 154 154 + } 155 155 + return err; 156 156 + } 157 157 + 115 158 static int sun8i_ss_cipher(struct skcipher_request *areq) 116 159 { 117 160 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); ··· 182 101 struct sun8i_cipher_req_ctx *rctx = skcipher_request_ctx(areq); 183 102 struct skcipher_alg *alg = crypto_skcipher_alg(tfm); 184 103 struct sun8i_ss_alg_template *algt; 104 104 + struct sun8i_ss_flow *sf = &ss->flows[rctx->flow]; 185 105 struct scatterlist *sg; 186 106 unsigned int todo, len, offset, ivsize; 187 187 - void *backup_iv = NULL; 188 107 int nr_sgs = 0; 189 108 int nr_sgd = 0; 190 109 int err = 0; 110 110 + int nsgs = sg_nents_for_len(areq->src, areq->cryptlen); 111 111 + int nsgd = sg_nents_for_len(areq->dst, areq->cryptlen); 191 112 int i; 192 113 193 114 algt = container_of(alg, struct sun8i_ss_alg_template, alg.skcipher); ··· 217 134 218 135 ivsize = crypto_skcipher_ivsize(tfm); 219 136 if (areq->iv && crypto_skcipher_ivsize(tfm) > 0) { 220 220 - rctx->ivlen = ivsize; 221 221 - rctx->biv = kzalloc(ivsize, GFP_KERNEL | GFP_DMA); 222 222 - if (!rctx->biv) { 223 223 - err = -ENOMEM; 137 137 + err = sun8i_ss_setup_ivs(areq); 138 138 + if (err) 224 139 goto theend_key; 225 225 - } 226 226 - if (rctx->op_dir & SS_DECRYPTION) { 227 227 - backup_iv = kzalloc(ivsize, GFP_KERNEL); 228 228 - if (!backup_iv) { 229 229 - err = -ENOMEM; 230 230 - goto theend_key; 231 231 - } 232 232 - offset = areq->cryptlen - ivsize; 233 233 - scatterwalk_map_and_copy(backup_iv, areq->src, offset, 234 234 - ivsize, 0); 235 235 - } 236 236 - memcpy(rctx->biv, areq->iv, ivsize); 237 237 - rctx->p_iv = dma_map_single(ss->dev, rctx->biv, rctx->ivlen, 238 238 - DMA_TO_DEVICE); 239 239 - if (dma_mapping_error(ss->dev, rctx->p_iv)) { 240 240 - dev_err(ss->dev, "Cannot DMA MAP IV\n"); 241 241 - err = -ENOMEM; 242 242 - goto theend_iv; 243 243 - } 244 140 } 245 141 if (areq->src == areq->dst) { 246 246 - nr_sgs = dma_map_sg(ss->dev, areq->src, sg_nents(areq->src), 247 247 - DMA_BIDIRECTIONAL); 142 142 + nr_sgs = dma_map_sg(ss->dev, areq->src, nsgs, DMA_BIDIRECTIONAL); 248 143 if (nr_sgs <= 0 || nr_sgs > 8) { 249 144 dev_err(ss->dev, "Invalid sg number %d\n", nr_sgs); 250 145 err = -EINVAL; ··· 230 169 } 231 170 nr_sgd = nr_sgs; 232 171 } else { 233 233 - nr_sgs = dma_map_sg(ss->dev, areq->src, sg_nents(areq->src), 234 234 - DMA_TO_DEVICE); 172 172 + nr_sgs = dma_map_sg(ss->dev, areq->src, nsgs, DMA_TO_DEVICE); 235 173 if (nr_sgs <= 0 || nr_sgs > 8) { 236 174 dev_err(ss->dev, "Invalid sg number %d\n", nr_sgs); 237 175 err = -EINVAL; 238 176 goto theend_iv; 239 177 } 240 240 - nr_sgd = dma_map_sg(ss->dev, areq->dst, sg_nents(areq->dst), 241 241 - DMA_FROM_DEVICE); 178 178 + nr_sgd = dma_map_sg(ss->dev, areq->dst, nsgd, DMA_FROM_DEVICE); 242 179 if (nr_sgd <= 0 || nr_sgd > 8) { 243 180 dev_err(ss->dev, "Invalid sg number %d\n", nr_sgd); 244 181 err = -EINVAL; ··· 292 233 293 234 theend_sgs: 294 235 if (areq->src == areq->dst) { 295 295 - dma_unmap_sg(ss->dev, areq->src, sg_nents(areq->src), 296 296 - DMA_BIDIRECTIONAL); 236 236 + dma_unmap_sg(ss->dev, areq->src, nsgs, DMA_BIDIRECTIONAL); 297 237 } else { 298 298 - dma_unmap_sg(ss->dev, areq->src, sg_nents(areq->src), 299 299 - DMA_TO_DEVICE); 300 300 - dma_unmap_sg(ss->dev, areq->dst, sg_nents(areq->dst), 301 301 - DMA_FROM_DEVICE); 238 238 + dma_unmap_sg(ss->dev, areq->src, nsgs, DMA_TO_DEVICE); 239 239 + dma_unmap_sg(ss->dev, areq->dst, nsgd, DMA_FROM_DEVICE); 302 240 } 303 241 304 242 theend_iv: 305 305 - if (rctx->p_iv) 306 306 - dma_unmap_single(ss->dev, rctx->p_iv, rctx->ivlen, 307 307 - DMA_TO_DEVICE); 308 308 - 309 243 if (areq->iv && ivsize > 0) { 310 310 - if (rctx->biv) { 311 311 - offset = areq->cryptlen - ivsize; 312 312 - if (rctx->op_dir & SS_DECRYPTION) { 313 313 - memcpy(areq->iv, backup_iv, ivsize); 314 314 - kfree_sensitive(backup_iv); 315 315 - } else { 316 316 - scatterwalk_map_and_copy(areq->iv, areq->dst, offset, 317 317 - ivsize, 0); 318 318 - } 319 319 - kfree(rctx->biv); 244 244 + for (i = 0; i < rctx->niv; i++) { 245 245 + dma_unmap_single(ss->dev, rctx->p_iv[i], ivsize, DMA_TO_DEVICE); 246 246 + memzero_explicit(sf->iv[i], ivsize); 247 247 + } 248 248 + 249 249 + offset = areq->cryptlen - ivsize; 250 250 + if (rctx->op_dir & SS_DECRYPTION) { 251 251 + memcpy(areq->iv, sf->biv, ivsize); 252 252 + memzero_explicit(sf->biv, ivsize); 253 253 + } else { 254 254 + scatterwalk_map_and_copy(areq->iv, areq->dst, offset, 255 255 + ivsize, 0); 320 256 } 321 257 } 322 258 ··· 403 349 crypto_skcipher_reqsize(op->fallback_tfm); 404 350 405 351 406 406 - dev_info(op->ss->dev, "Fallback for %s is %s\n", 407 407 - crypto_tfm_alg_driver_name(&sktfm->base), 408 408 - crypto_tfm_alg_driver_name(crypto_skcipher_tfm(op->fallback_tfm))); 352 352 + memcpy(algt->fbname, 353 353 + crypto_tfm_alg_driver_name(crypto_skcipher_tfm(op->fallback_tfm)), 354 354 + CRYPTO_MAX_ALG_NAME); 409 355 410 356 op->enginectx.op.do_one_request = sun8i_ss_handle_cipher_request; 411 357 op->enginectx.op.prepare_request = NULL;

+83 -9

drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c

reviewed

··· 66 66 const char *name) 67 67 { 68 68 int flow = rctx->flow; 69 69 + unsigned int ivlen = rctx->ivlen; 69 70 u32 v = SS_START; 70 71 int i; 71 72 ··· 105 104 mutex_lock(&ss->mlock); 106 105 writel(rctx->p_key, ss->base + SS_KEY_ADR_REG); 107 106 108 108 - if (i == 0) { 109 109 - if (rctx->p_iv) 110 110 - writel(rctx->p_iv, ss->base + SS_IV_ADR_REG); 111 111 - } else { 112 112 - if (rctx->biv) { 113 113 - if (rctx->op_dir == SS_ENCRYPTION) 114 114 - writel(rctx->t_dst[i - 1].addr + rctx->t_dst[i - 1].len * 4 - rctx->ivlen, ss->base + SS_IV_ADR_REG); 107 107 + if (ivlen) { 108 108 + if (rctx->op_dir == SS_ENCRYPTION) { 109 109 + if (i == 0) 110 110 + writel(rctx->p_iv[0], ss->base + SS_IV_ADR_REG); 115 111 else 116 116 - writel(rctx->t_src[i - 1].addr + rctx->t_src[i - 1].len * 4 - rctx->ivlen, ss->base + SS_IV_ADR_REG); 112 112 + writel(rctx->t_dst[i - 1].addr + rctx->t_dst[i - 1].len * 4 - ivlen, ss->base + SS_IV_ADR_REG); 113 113 + } else { 114 114 + writel(rctx->p_iv[i], ss->base + SS_IV_ADR_REG); 117 115 } 118 116 } 119 117 ··· 409 409 } 410 410 } 411 411 }, 412 412 + { .type = CRYPTO_ALG_TYPE_AHASH, 413 413 + .ss_algo_id = SS_ID_HASH_SHA1, 414 414 + .alg.hash = { 415 415 + .init = sun8i_ss_hash_init, 416 416 + .update = sun8i_ss_hash_update, 417 417 + .final = sun8i_ss_hash_final, 418 418 + .finup = sun8i_ss_hash_finup, 419 419 + .digest = sun8i_ss_hash_digest, 420 420 + .export = sun8i_ss_hash_export, 421 421 + .import = sun8i_ss_hash_import, 422 422 + .setkey = sun8i_ss_hmac_setkey, 423 423 + .halg = { 424 424 + .digestsize = SHA1_DIGEST_SIZE, 425 425 + .statesize = sizeof(struct sha1_state), 426 426 + .base = { 427 427 + .cra_name = "hmac(sha1)", 428 428 + .cra_driver_name = "hmac-sha1-sun8i-ss", 429 429 + .cra_priority = 300, 430 430 + .cra_alignmask = 3, 431 431 + .cra_flags = CRYPTO_ALG_TYPE_AHASH | 432 432 + CRYPTO_ALG_ASYNC | 433 433 + CRYPTO_ALG_NEED_FALLBACK, 434 434 + .cra_blocksize = SHA1_BLOCK_SIZE, 435 435 + .cra_ctxsize = sizeof(struct sun8i_ss_hash_tfm_ctx), 436 436 + .cra_module = THIS_MODULE, 437 437 + .cra_init = sun8i_ss_hash_crainit, 438 438 + .cra_exit = sun8i_ss_hash_craexit, 439 439 + } 440 440 + } 441 441 + } 442 442 + }, 412 443 #endif 413 444 }; 414 445 ··· 461 430 ss_algs[i].alg.skcipher.base.cra_driver_name, 462 431 ss_algs[i].alg.skcipher.base.cra_name, 463 432 ss_algs[i].stat_req, ss_algs[i].stat_fb); 433 433 + 434 434 + seq_printf(seq, "\tLast fallback is: %s\n", 435 435 + ss_algs[i].fbname); 436 436 + seq_printf(seq, "\tFallback due to length: %lu\n", 437 437 + ss_algs[i].stat_fb_len); 438 438 + seq_printf(seq, "\tFallback due to SG length: %lu\n", 439 439 + ss_algs[i].stat_fb_sglen); 440 440 + seq_printf(seq, "\tFallback due to alignment: %lu\n", 441 441 + ss_algs[i].stat_fb_align); 442 442 + seq_printf(seq, "\tFallback due to SG numbers: %lu\n", 443 443 + ss_algs[i].stat_fb_sgnum); 464 444 break; 465 445 case CRYPTO_ALG_TYPE_RNG: 466 446 seq_printf(seq, "%s %s reqs=%lu tsize=%lu\n", ··· 484 442 ss_algs[i].alg.hash.halg.base.cra_driver_name, 485 443 ss_algs[i].alg.hash.halg.base.cra_name, 486 444 ss_algs[i].stat_req, ss_algs[i].stat_fb); 445 445 + seq_printf(seq, "\tLast fallback is: %s\n", 446 446 + ss_algs[i].fbname); 447 447 + seq_printf(seq, "\tFallback due to length: %lu\n", 448 448 + ss_algs[i].stat_fb_len); 449 449 + seq_printf(seq, "\tFallback due to SG length: %lu\n", 450 450 + ss_algs[i].stat_fb_sglen); 451 451 + seq_printf(seq, "\tFallback due to alignment: %lu\n", 452 452 + ss_algs[i].stat_fb_align); 453 453 + seq_printf(seq, "\tFallback due to SG numbers: %lu\n", 454 454 + ss_algs[i].stat_fb_sgnum); 487 455 break; 488 456 } 489 457 } ··· 516 464 */ 517 465 static int allocate_flows(struct sun8i_ss_dev *ss) 518 466 { 519 519 - int i, err; 467 467 + int i, j, err; 520 468 521 469 ss->flows = devm_kcalloc(ss->dev, MAXFLOW, sizeof(struct sun8i_ss_flow), 522 470 GFP_KERNEL); ··· 525 473 526 474 for (i = 0; i < MAXFLOW; i++) { 527 475 init_completion(&ss->flows[i].complete); 476 476 + 477 477 + ss->flows[i].biv = devm_kmalloc(ss->dev, AES_BLOCK_SIZE, 478 478 + GFP_KERNEL | GFP_DMA); 479 479 + if (!ss->flows[i].biv) 480 480 + goto error_engine; 481 481 + 482 482 + for (j = 0; j < MAX_SG; j++) { 483 483 + ss->flows[i].iv[j] = devm_kmalloc(ss->dev, AES_BLOCK_SIZE, 484 484 + GFP_KERNEL | GFP_DMA); 485 485 + if (!ss->flows[i].iv[j]) 486 486 + goto error_engine; 487 487 + } 488 488 + 489 489 + /* the padding could be up to two block. */ 490 490 + ss->flows[i].pad = devm_kmalloc(ss->dev, MAX_PAD_SIZE, 491 491 + GFP_KERNEL | GFP_DMA); 492 492 + if (!ss->flows[i].pad) 493 493 + goto error_engine; 494 494 + ss->flows[i].result = devm_kmalloc(ss->dev, SHA256_DIGEST_SIZE, 495 495 + GFP_KERNEL | GFP_DMA); 496 496 + if (!ss->flows[i].result) 497 497 + goto error_engine; 528 498 529 499 ss->flows[i].engine = crypto_engine_alloc_init(ss->dev, true); 530 500 if (!ss->flows[i].engine) {

+326 -69

drivers/crypto/allwinner/sun8i-ss/sun8i-ss-hash.c

reviewed

··· 14 14 #include <linux/pm_runtime.h> 15 15 #include <linux/scatterlist.h> 16 16 #include <crypto/internal/hash.h> 17 17 + #include <crypto/hmac.h> 18 18 + #include <crypto/scatterwalk.h> 17 19 #include <crypto/sha1.h> 18 20 #include <crypto/sha2.h> 19 21 #include <crypto/md5.h> 20 22 #include "sun8i-ss.h" 23 23 + 24 24 + static int sun8i_ss_hashkey(struct sun8i_ss_hash_tfm_ctx *tfmctx, const u8 *key, 25 25 + unsigned int keylen) 26 26 + { 27 27 + struct crypto_shash *xtfm; 28 28 + struct shash_desc *sdesc; 29 29 + size_t len; 30 30 + int ret = 0; 31 31 + 32 32 + xtfm = crypto_alloc_shash("sha1", 0, CRYPTO_ALG_NEED_FALLBACK); 33 33 + if (!xtfm) 34 34 + return -ENOMEM; 35 35 + 36 36 + len = sizeof(*sdesc) + crypto_shash_descsize(xtfm); 37 37 + sdesc = kmalloc(len, GFP_KERNEL); 38 38 + if (!sdesc) { 39 39 + ret = -ENOMEM; 40 40 + goto err_hashkey_sdesc; 41 41 + } 42 42 + sdesc->tfm = xtfm; 43 43 + 44 44 + ret = crypto_shash_init(sdesc); 45 45 + if (ret) { 46 46 + dev_err(tfmctx->ss->dev, "shash init error ret=%d\n", ret); 47 47 + goto err_hashkey; 48 48 + } 49 49 + ret = crypto_shash_finup(sdesc, key, keylen, tfmctx->key); 50 50 + if (ret) 51 51 + dev_err(tfmctx->ss->dev, "shash finup error\n"); 52 52 + err_hashkey: 53 53 + kfree(sdesc); 54 54 + err_hashkey_sdesc: 55 55 + crypto_free_shash(xtfm); 56 56 + return ret; 57 57 + } 58 58 + 59 59 + int sun8i_ss_hmac_setkey(struct crypto_ahash *ahash, const u8 *key, 60 60 + unsigned int keylen) 61 61 + { 62 62 + struct sun8i_ss_hash_tfm_ctx *tfmctx = crypto_ahash_ctx(ahash); 63 63 + struct ahash_alg *alg = __crypto_ahash_alg(ahash->base.__crt_alg); 64 64 + struct sun8i_ss_alg_template *algt; 65 65 + int digestsize, i; 66 66 + int bs = crypto_ahash_blocksize(ahash); 67 67 + int ret; 68 68 + 69 69 + algt = container_of(alg, struct sun8i_ss_alg_template, alg.hash); 70 70 + digestsize = algt->alg.hash.halg.digestsize; 71 71 + 72 72 + if (keylen > bs) { 73 73 + ret = sun8i_ss_hashkey(tfmctx, key, keylen); 74 74 + if (ret) 75 75 + return ret; 76 76 + tfmctx->keylen = digestsize; 77 77 + } else { 78 78 + tfmctx->keylen = keylen; 79 79 + memcpy(tfmctx->key, key, keylen); 80 80 + } 81 81 + 82 82 + tfmctx->ipad = kzalloc(bs, GFP_KERNEL | GFP_DMA); 83 83 + if (!tfmctx->ipad) 84 84 + return -ENOMEM; 85 85 + tfmctx->opad = kzalloc(bs, GFP_KERNEL | GFP_DMA); 86 86 + if (!tfmctx->opad) { 87 87 + ret = -ENOMEM; 88 88 + goto err_opad; 89 89 + } 90 90 + 91 91 + memset(tfmctx->key + tfmctx->keylen, 0, bs - tfmctx->keylen); 92 92 + memcpy(tfmctx->ipad, tfmctx->key, tfmctx->keylen); 93 93 + memcpy(tfmctx->opad, tfmctx->key, tfmctx->keylen); 94 94 + for (i = 0; i < bs; i++) { 95 95 + tfmctx->ipad[i] ^= HMAC_IPAD_VALUE; 96 96 + tfmctx->opad[i] ^= HMAC_OPAD_VALUE; 97 97 + } 98 98 + 99 99 + ret = crypto_ahash_setkey(tfmctx->fallback_tfm, key, keylen); 100 100 + if (!ret) 101 101 + return 0; 102 102 + 103 103 + memzero_explicit(tfmctx->key, keylen); 104 104 + kfree_sensitive(tfmctx->opad); 105 105 + err_opad: 106 106 + kfree_sensitive(tfmctx->ipad); 107 107 + return ret; 108 108 + } 21 109 22 110 int sun8i_ss_hash_crainit(struct crypto_tfm *tfm) 23 111 { ··· 138 50 sizeof(struct sun8i_ss_hash_reqctx) + 139 51 crypto_ahash_reqsize(op->fallback_tfm)); 140 52 141 141 - dev_info(op->ss->dev, "Fallback for %s is %s\n", 142 142 - crypto_tfm_alg_driver_name(tfm), 143 143 - crypto_tfm_alg_driver_name(&op->fallback_tfm->base)); 53 53 + memcpy(algt->fbname, crypto_tfm_alg_driver_name(&op->fallback_tfm->base), CRYPTO_MAX_ALG_NAME); 54 54 + 144 55 err = pm_runtime_get_sync(op->ss->dev); 145 56 if (err < 0) 146 57 goto error_pm; ··· 153 66 void sun8i_ss_hash_craexit(struct crypto_tfm *tfm) 154 67 { 155 68 struct sun8i_ss_hash_tfm_ctx *tfmctx = crypto_tfm_ctx(tfm); 69 69 + 70 70 + kfree_sensitive(tfmctx->ipad); 71 71 + kfree_sensitive(tfmctx->opad); 156 72 157 73 crypto_free_ahash(tfmctx->fallback_tfm); 158 74 pm_runtime_put_sync_suspend(tfmctx->ss->dev); ··· 348 258 349 259 static bool sun8i_ss_hash_need_fallback(struct ahash_request *areq) 350 260 { 261 261 + struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); 262 262 + struct ahash_alg *alg = __crypto_ahash_alg(tfm->base.__crt_alg); 263 263 + struct sun8i_ss_alg_template *algt; 351 264 struct scatterlist *sg; 352 265 353 353 - if (areq->nbytes == 0) 266 266 + algt = container_of(alg, struct sun8i_ss_alg_template, alg.hash); 267 267 + 268 268 + if (areq->nbytes == 0) { 269 269 + algt->stat_fb_len++; 354 270 return true; 271 271 + } 272 272 + 273 273 + if (areq->nbytes >= MAX_PAD_SIZE - 64) { 274 274 + algt->stat_fb_len++; 275 275 + return true; 276 276 + } 277 277 + 355 278 /* we need to reserve one SG for the padding one */ 356 356 - if (sg_nents(areq->src) > MAX_SG - 1) 279 279 + if (sg_nents(areq->src) > MAX_SG - 1) { 280 280 + algt->stat_fb_sgnum++; 357 281 return true; 282 282 + } 283 283 + 358 284 sg = areq->src; 359 285 while (sg) { 360 286 /* SS can operate hash only on full block size 361 287 * since SS support only MD5,sha1,sha224 and sha256, blocksize 362 288 * is always 64 363 363 - * TODO: handle request if last SG is not len%64 364 364 - * but this will need to copy data on a new SG of size=64 365 289 */ 366 366 - if (sg->length % 64 || !IS_ALIGNED(sg->offset, sizeof(u32))) 290 290 + /* Only the last block could be bounced to the pad buffer */ 291 291 + if (sg->length % 64 && sg_next(sg)) { 292 292 + algt->stat_fb_sglen++; 367 293 return true; 294 294 + } 295 295 + if (!IS_ALIGNED(sg->offset, sizeof(u32))) { 296 296 + algt->stat_fb_align++; 297 297 + return true; 298 298 + } 299 299 + if (sg->length % 4) { 300 300 + algt->stat_fb_sglen++; 301 301 + return true; 302 302 + } 368 303 sg = sg_next(sg); 369 304 } 370 305 return false; ··· 403 288 struct sun8i_ss_alg_template *algt; 404 289 struct sun8i_ss_dev *ss; 405 290 struct crypto_engine *engine; 406 406 - struct scatterlist *sg; 407 407 - int nr_sgs, e, i; 291 291 + int e; 408 292 409 293 if (sun8i_ss_hash_need_fallback(areq)) 410 294 return sun8i_ss_hash_digest_fb(areq); 411 411 - 412 412 - nr_sgs = sg_nents(areq->src); 413 413 - if (nr_sgs > MAX_SG - 1) 414 414 - return sun8i_ss_hash_digest_fb(areq); 415 415 - 416 416 - for_each_sg(areq->src, sg, nr_sgs, i) { 417 417 - if (sg->length % 4 || !IS_ALIGNED(sg->offset, sizeof(u32))) 418 418 - return sun8i_ss_hash_digest_fb(areq); 419 419 - } 420 295 421 296 algt = container_of(alg, struct sun8i_ss_alg_template, alg.hash); 422 297 ss = algt->ss; ··· 418 313 return crypto_transfer_hash_request_to_engine(engine, areq); 419 314 } 420 315 316 316 + static u64 hash_pad(__le32 *buf, unsigned int bufsize, u64 padi, u64 byte_count, bool le, int bs) 317 317 + { 318 318 + u64 fill, min_fill, j, k; 319 319 + __be64 *bebits; 320 320 + __le64 *lebits; 321 321 + 322 322 + j = padi; 323 323 + buf[j++] = cpu_to_le32(0x80); 324 324 + 325 325 + if (bs == 64) { 326 326 + fill = 64 - (byte_count % 64); 327 327 + min_fill = 2 * sizeof(u32) + sizeof(u32); 328 328 + } else { 329 329 + fill = 128 - (byte_count % 128); 330 330 + min_fill = 4 * sizeof(u32) + sizeof(u32); 331 331 + } 332 332 + 333 333 + if (fill < min_fill) 334 334 + fill += bs; 335 335 + 336 336 + k = j; 337 337 + j += (fill - min_fill) / sizeof(u32); 338 338 + if (j * 4 > bufsize) { 339 339 + pr_err("%s OVERFLOW %llu\n", __func__, j); 340 340 + return 0; 341 341 + } 342 342 + for (; k < j; k++) 343 343 + buf[k] = 0; 344 344 + 345 345 + if (le) { 346 346 + /* MD5 */ 347 347 + lebits = (__le64 *)&buf[j]; 348 348 + *lebits = cpu_to_le64(byte_count << 3); 349 349 + j += 2; 350 350 + } else { 351 351 + if (bs == 64) { 352 352 + /* sha1 sha224 sha256 */ 353 353 + bebits = (__be64 *)&buf[j]; 354 354 + *bebits = cpu_to_be64(byte_count << 3); 355 355 + j += 2; 356 356 + } else { 357 357 + /* sha384 sha512*/ 358 358 + bebits = (__be64 *)&buf[j]; 359 359 + *bebits = cpu_to_be64(byte_count >> 61); 360 360 + j += 2; 361 361 + bebits = (__be64 *)&buf[j]; 362 362 + *bebits = cpu_to_be64(byte_count << 3); 363 363 + j += 2; 364 364 + } 365 365 + } 366 366 + if (j * 4 > bufsize) { 367 367 + pr_err("%s OVERFLOW %llu\n", __func__, j); 368 368 + return 0; 369 369 + } 370 370 + 371 371 + return j; 372 372 + } 373 373 + 421 374 /* sun8i_ss_hash_run - run an ahash request 422 375 * Send the data of the request to the SS along with an extra SG with padding 423 376 */ ··· 483 320 { 484 321 struct ahash_request *areq = container_of(breq, struct ahash_request, base); 485 322 struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); 323 323 + struct sun8i_ss_hash_tfm_ctx *tfmctx = crypto_ahash_ctx(tfm); 486 324 struct ahash_alg *alg = __crypto_ahash_alg(tfm->base.__crt_alg); 487 325 struct sun8i_ss_hash_reqctx *rctx = ahash_request_ctx(areq); 488 326 struct sun8i_ss_alg_template *algt; 489 327 struct sun8i_ss_dev *ss; 490 328 struct scatterlist *sg; 329 329 + int bs = crypto_ahash_blocksize(tfm); 491 330 int nr_sgs, err, digestsize; 492 331 unsigned int len; 493 493 - u64 fill, min_fill, byte_count; 332 332 + u64 byte_count; 494 333 void *pad, *result; 495 495 - int j, i, todo; 496 496 - __be64 *bebits; 497 497 - __le64 *lebits; 498 498 - dma_addr_t addr_res, addr_pad; 334 334 + int j, i, k, todo; 335 335 + dma_addr_t addr_res, addr_pad, addr_xpad; 499 336 __le32 *bf; 337 337 + /* HMAC step: 338 338 + * 0: normal hashing 339 339 + * 1: IPAD 340 340 + * 2: OPAD 341 341 + */ 342 342 + int hmac = 0; 500 343 501 344 algt = container_of(alg, struct sun8i_ss_alg_template, alg.hash); 502 345 ss = algt->ss; ··· 511 342 if (digestsize == SHA224_DIGEST_SIZE) 512 343 digestsize = SHA256_DIGEST_SIZE; 513 344 514 514 - /* the padding could be up to two block. */ 515 515 - pad = kzalloc(algt->alg.hash.halg.base.cra_blocksize * 2, GFP_KERNEL | GFP_DMA); 516 516 - if (!pad) 517 517 - return -ENOMEM; 345 345 + result = ss->flows[rctx->flow].result; 346 346 + pad = ss->flows[rctx->flow].pad; 518 347 bf = (__le32 *)pad; 519 519 - 520 520 - result = kzalloc(digestsize, GFP_KERNEL | GFP_DMA); 521 521 - if (!result) { 522 522 - kfree(pad); 523 523 - return -ENOMEM; 524 524 - } 525 348 526 349 for (i = 0; i < MAX_SG; i++) { 527 350 rctx->t_dst[i].addr = 0; ··· 537 376 if (dma_mapping_error(ss->dev, addr_res)) { 538 377 dev_err(ss->dev, "DMA map dest\n"); 539 378 err = -EINVAL; 540 540 - goto theend; 379 379 + goto err_dma_result; 541 380 } 542 381 382 382 + j = 0; 543 383 len = areq->nbytes; 544 544 - for_each_sg(areq->src, sg, nr_sgs, i) { 545 545 - rctx->t_src[i].addr = sg_dma_address(sg); 384 384 + sg = areq->src; 385 385 + i = 0; 386 386 + while (len > 0 && sg) { 387 387 + if (sg_dma_len(sg) == 0) { 388 388 + sg = sg_next(sg); 389 389 + continue; 390 390 + } 546 391 todo = min(len, sg_dma_len(sg)); 547 547 - rctx->t_src[i].len = todo / 4; 548 548 - len -= todo; 549 549 - rctx->t_dst[i].addr = addr_res; 550 550 - rctx->t_dst[i].len = digestsize / 4; 392 392 + /* only the last SG could be with a size not modulo64 */ 393 393 + if (todo % 64 == 0) { 394 394 + rctx->t_src[i].addr = sg_dma_address(sg); 395 395 + rctx->t_src[i].len = todo / 4; 396 396 + rctx->t_dst[i].addr = addr_res; 397 397 + rctx->t_dst[i].len = digestsize / 4; 398 398 + len -= todo; 399 399 + } else { 400 400 + scatterwalk_map_and_copy(bf, sg, 0, todo, 0); 401 401 + j += todo / 4; 402 402 + len -= todo; 403 403 + } 404 404 + sg = sg_next(sg); 405 405 + i++; 551 406 } 552 407 if (len > 0) { 553 408 dev_err(ss->dev, "remaining len %d\n", len); ··· 571 394 goto theend; 572 395 } 573 396 397 397 + if (j > 0) 398 398 + i--; 399 399 + 400 400 + retry: 574 401 byte_count = areq->nbytes; 575 575 - j = 0; 576 576 - bf[j++] = cpu_to_le32(0x80); 402 402 + if (tfmctx->keylen && hmac == 0) { 403 403 + hmac = 1; 404 404 + /* shift all SG one slot up, to free slot 0 for IPAD */ 405 405 + for (k = 6; k >= 0; k--) { 406 406 + rctx->t_src[k + 1].addr = rctx->t_src[k].addr; 407 407 + rctx->t_src[k + 1].len = rctx->t_src[k].len; 408 408 + rctx->t_dst[k + 1].addr = rctx->t_dst[k].addr; 409 409 + rctx->t_dst[k + 1].len = rctx->t_dst[k].len; 410 410 + } 411 411 + addr_xpad = dma_map_single(ss->dev, tfmctx->ipad, bs, DMA_TO_DEVICE); 412 412 + if (dma_mapping_error(ss->dev, addr_xpad)) { 413 413 + dev_err(ss->dev, "Fail to create DMA mapping of ipad\n"); 414 414 + goto err_dma_xpad; 415 415 + } 416 416 + rctx->t_src[0].addr = addr_xpad; 417 417 + rctx->t_src[0].len = bs / 4; 418 418 + rctx->t_dst[0].addr = addr_res; 419 419 + rctx->t_dst[0].len = digestsize / 4; 420 420 + i++; 421 421 + byte_count = areq->nbytes + bs; 422 422 + } 423 423 + if (tfmctx->keylen && hmac == 2) { 424 424 + for (i = 0; i < MAX_SG; i++) { 425 425 + rctx->t_src[i].addr = 0; 426 426 + rctx->t_src[i].len = 0; 427 427 + rctx->t_dst[i].addr = 0; 428 428 + rctx->t_dst[i].len = 0; 429 429 + } 577 430 578 578 - fill = 64 - (byte_count % 64); 579 579 - min_fill = 3 * sizeof(u32); 431 431 + addr_res = dma_map_single(ss->dev, result, digestsize, DMA_FROM_DEVICE); 432 432 + if (dma_mapping_error(ss->dev, addr_res)) { 433 433 + dev_err(ss->dev, "Fail to create DMA mapping of result\n"); 434 434 + err = -EINVAL; 435 435 + goto err_dma_result; 436 436 + } 437 437 + addr_xpad = dma_map_single(ss->dev, tfmctx->opad, bs, DMA_TO_DEVICE); 438 438 + if (dma_mapping_error(ss->dev, addr_xpad)) { 439 439 + dev_err(ss->dev, "Fail to create DMA mapping of opad\n"); 440 440 + goto err_dma_xpad; 441 441 + } 442 442 + rctx->t_src[0].addr = addr_xpad; 443 443 + rctx->t_src[0].len = bs / 4; 580 444 581 581 - if (fill < min_fill) 582 582 - fill += 64; 445 445 + memcpy(bf, result, digestsize); 446 446 + j = digestsize / 4; 447 447 + i = 1; 448 448 + byte_count = digestsize + bs; 583 449 584 584 - j += (fill - min_fill) / sizeof(u32); 450 450 + rctx->t_dst[0].addr = addr_res; 451 451 + rctx->t_dst[0].len = digestsize / 4; 452 452 + } 585 453 586 454 switch (algt->ss_algo_id) { 587 455 case SS_ID_HASH_MD5: 588 588 - lebits = (__le64 *)&bf[j]; 589 589 - *lebits = cpu_to_le64(byte_count << 3); 590 590 - j += 2; 456 456 + j = hash_pad(bf, 4096, j, byte_count, true, bs); 591 457 break; 592 458 case SS_ID_HASH_SHA1: 593 459 case SS_ID_HASH_SHA224: 594 460 case SS_ID_HASH_SHA256: 595 595 - bebits = (__be64 *)&bf[j]; 596 596 - *bebits = cpu_to_be64(byte_count << 3); 597 597 - j += 2; 461 461 + j = hash_pad(bf, 4096, j, byte_count, false, bs); 598 462 break; 599 463 } 600 600 - 601 601 - addr_pad = dma_map_single(ss->dev, pad, j * 4, DMA_TO_DEVICE); 602 602 - rctx->t_src[i].addr = addr_pad; 603 603 - rctx->t_src[i].len = j; 604 604 - rctx->t_dst[i].addr = addr_res; 605 605 - rctx->t_dst[i].len = digestsize / 4; 606 606 - if (dma_mapping_error(ss->dev, addr_pad)) { 607 607 - dev_err(ss->dev, "DMA error on padding SG\n"); 464 464 + if (!j) { 608 465 err = -EINVAL; 609 466 goto theend; 610 467 } 611 468 469 469 + addr_pad = dma_map_single(ss->dev, pad, j * 4, DMA_TO_DEVICE); 470 470 + if (dma_mapping_error(ss->dev, addr_pad)) { 471 471 + dev_err(ss->dev, "DMA error on padding SG\n"); 472 472 + err = -EINVAL; 473 473 + goto err_dma_pad; 474 474 + } 475 475 + rctx->t_src[i].addr = addr_pad; 476 476 + rctx->t_src[i].len = j; 477 477 + rctx->t_dst[i].addr = addr_res; 478 478 + rctx->t_dst[i].len = digestsize / 4; 479 479 + 612 480 err = sun8i_ss_run_hash_task(ss, rctx, crypto_tfm_alg_name(areq->base.tfm)); 613 481 614 614 - dma_unmap_single(ss->dev, addr_pad, j * 4, DMA_TO_DEVICE); 615 615 - dma_unmap_sg(ss->dev, areq->src, sg_nents(areq->src), 616 616 - DMA_TO_DEVICE); 617 617 - dma_unmap_single(ss->dev, addr_res, digestsize, DMA_FROM_DEVICE); 482 482 + /* 483 483 + * mini helper for checking dma map/unmap 484 484 + * flow start for hmac = 0 (and HMAC = 1) 485 485 + * HMAC = 0 486 486 + * MAP src 487 487 + * MAP res 488 488 + * 489 489 + * retry: 490 490 + * if hmac then hmac = 1 491 491 + * MAP xpad (ipad) 492 492 + * if hmac == 2 493 493 + * MAP res 494 494 + * MAP xpad (opad) 495 495 + * MAP pad 496 496 + * ACTION! 497 497 + * UNMAP pad 498 498 + * if hmac 499 499 + * UNMAP xpad 500 500 + * UNMAP res 501 501 + * if hmac < 2 502 502 + * UNMAP SRC 503 503 + * 504 504 + * if hmac = 1 then hmac = 2 goto retry 505 505 + */ 618 506 619 619 - memcpy(areq->result, result, algt->alg.hash.halg.digestsize); 507 507 + dma_unmap_single(ss->dev, addr_pad, j * 4, DMA_TO_DEVICE); 508 508 + 509 509 + err_dma_pad: 510 510 + if (hmac > 0) 511 511 + dma_unmap_single(ss->dev, addr_xpad, bs, DMA_TO_DEVICE); 512 512 + err_dma_xpad: 513 513 + dma_unmap_single(ss->dev, addr_res, digestsize, DMA_FROM_DEVICE); 514 514 + err_dma_result: 515 515 + if (hmac < 2) 516 516 + dma_unmap_sg(ss->dev, areq->src, sg_nents(areq->src), 517 517 + DMA_TO_DEVICE); 518 518 + if (hmac == 1 && !err) { 519 519 + hmac = 2; 520 520 + goto retry; 521 521 + } 522 522 + 523 523 + if (!err) 524 524 + memcpy(areq->result, result, algt->alg.hash.halg.digestsize); 620 525 theend: 621 621 - kfree(pad); 622 622 - kfree(result); 623 526 local_bh_disable(); 624 527 crypto_finalize_hash_request(engine, breq, err); 625 528 local_bh_enable();

+2 -4

drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c

reviewed

··· 112 112 goto err_iv; 113 113 } 114 114 115 115 - err = pm_runtime_get_sync(ss->dev); 116 116 - if (err < 0) { 117 117 - pm_runtime_put_noidle(ss->dev); 115 115 + err = pm_runtime_resume_and_get(ss->dev); 116 116 + if (err < 0) 118 117 goto err_pm; 119 119 - } 120 118 err = 0; 121 119 122 120 mutex_lock(&ss->mlock);

+26 -7

drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h

reviewed

··· 82 82 #define PRNG_DATA_SIZE (160 / 8) 83 83 #define PRNG_SEED_SIZE DIV_ROUND_UP(175, 8) 84 84 85 85 + #define MAX_PAD_SIZE 4096 86 86 + 85 87 /* 86 88 * struct ss_clock - Describe clocks used by sun8i-ss 87 89 * @name: Name of clock needed by this variant ··· 123 121 * @complete: completion for the current task on this flow 124 122 * @status: set to 1 by interrupt if task is done 125 123 * @stat_req: number of request done by this flow 124 124 + * @iv: list of IV to use for each step 125 125 + * @biv: buffer which contain the backuped IV 126 126 + * @pad: padding buffer for hash operations 127 127 + * @result: buffer for storing the result of hash operations 126 128 */ 127 129 struct sun8i_ss_flow { 128 130 struct crypto_engine *engine; 129 131 struct completion complete; 130 132 int status; 133 133 + u8 *iv[MAX_SG]; 134 134 + u8 *biv; 135 135 + void *pad; 136 136 + void *result; 131 137 #ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_DEBUG 132 138 unsigned long stat_req; 133 139 #endif ··· 174 164 * @t_src: list of mapped SGs with their size 175 165 * @t_dst: list of mapped SGs with their size 176 166 * @p_key: DMA address of the key 177 177 - * @p_iv: DMA address of the IV 167 167 + * @p_iv: DMA address of the IVs 168 168 + * @niv: Number of IVs DMA mapped 178 169 * @method: current algorithm for this request 179 170 * @op_mode: op_mode for this request 180 171 * @op_dir: direction (encrypt vs decrypt) for this request 181 172 * @flow: the flow to use for this request 182 182 - * @ivlen: size of biv 173 173 + * @ivlen: size of IVs 183 174 * @keylen: keylen for this request 184 184 - * @biv: buffer which contain the IV 185 175 * @fallback_req: request struct for invoking the fallback skcipher TFM 186 176 */ 187 177 struct sun8i_cipher_req_ctx { 188 178 struct sginfo t_src[MAX_SG]; 189 179 struct sginfo t_dst[MAX_SG]; 190 180 u32 p_key; 191 191 - u32 p_iv; 181 181 + u32 p_iv[MAX_SG]; 182 182 + int niv; 192 183 u32 method; 193 184 u32 op_mode; 194 185 u32 op_dir; 195 186 int flow; 196 187 unsigned int ivlen; 197 188 unsigned int keylen; 198 198 - void *biv; 199 189 struct skcipher_request fallback_req; // keep at the end 200 190 }; 201 191 ··· 239 229 struct crypto_engine_ctx enginectx; 240 230 struct crypto_ahash *fallback_tfm; 241 231 struct sun8i_ss_dev *ss; 232 232 + u8 *ipad; 233 233 + u8 *opad; 234 234 + u8 key[SHA256_BLOCK_SIZE]; 235 235 + int keylen; 242 236 }; 243 237 244 238 /* ··· 283 269 struct rng_alg rng; 284 270 struct ahash_alg hash; 285 271 } alg; 286 286 - #ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_DEBUG 287 272 unsigned long stat_req; 288 273 unsigned long stat_fb; 289 274 unsigned long stat_bytes; 290 290 - #endif 275 275 + unsigned long stat_fb_len; 276 276 + unsigned long stat_fb_sglen; 277 277 + unsigned long stat_fb_align; 278 278 + unsigned long stat_fb_sgnum; 279 279 + char fbname[CRYPTO_MAX_ALG_NAME]; 291 280 }; 292 281 293 282 int sun8i_ss_enqueue(struct crypto_async_request *areq, u32 type); ··· 323 306 int sun8i_ss_hash_finup(struct ahash_request *areq); 324 307 int sun8i_ss_hash_digest(struct ahash_request *areq); 325 308 int sun8i_ss_hash_run(struct crypto_engine *engine, void *breq); 309 309 + int sun8i_ss_hmac_setkey(struct crypto_ahash *ahash, const u8 *key, 310 310 + unsigned int keylen);

+1 -1

drivers/crypto/atmel-ecc.c

reviewed

··· 398 398 399 399 static void __exit atmel_ecc_exit(void) 400 400 { 401 401 - flush_scheduled_work(); 401 401 + atmel_i2c_flush_queue(); 402 402 i2c_del_driver(&atmel_ecc_driver); 403 403 } 404 404

+24 -6

drivers/crypto/atmel-i2c.c

reviewed

··· 263 263 work_data->cbk(work_data, work_data->areq, status); 264 264 } 265 265 266 266 + static struct workqueue_struct *atmel_wq; 267 267 + 266 268 void atmel_i2c_enqueue(struct atmel_i2c_work_data *work_data, 267 269 void (*cbk)(struct atmel_i2c_work_data *work_data, 268 270 void *areq, int status), ··· 274 272 work_data->areq = areq; 275 273 276 274 INIT_WORK(&work_data->work, atmel_i2c_work_handler); 277 277 - schedule_work(&work_data->work); 275 275 + queue_work(atmel_wq, &work_data->work); 278 276 } 279 277 EXPORT_SYMBOL(atmel_i2c_enqueue); 278 278 + 279 279 + void atmel_i2c_flush_queue(void) 280 280 + { 281 281 + flush_workqueue(atmel_wq); 282 282 + } 283 283 + EXPORT_SYMBOL(atmel_i2c_flush_queue); 280 284 281 285 static inline size_t atmel_i2c_wake_token_sz(u32 bus_clk_rate) 282 286 { ··· 372 364 373 365 i2c_set_clientdata(client, i2c_priv); 374 366 375 375 - ret = device_sanity_check(client); 376 376 - if (ret) 377 377 - return ret; 378 378 - 379 379 - return 0; 367 367 + return device_sanity_check(client); 380 368 } 381 369 EXPORT_SYMBOL(atmel_i2c_probe); 370 370 + 371 371 + static int __init atmel_i2c_init(void) 372 372 + { 373 373 + atmel_wq = alloc_workqueue("atmel_wq", 0, 0); 374 374 + return atmel_wq ? 0 : -ENOMEM; 375 375 + } 376 376 + 377 377 + static void __exit atmel_i2c_exit(void) 378 378 + { 379 379 + destroy_workqueue(atmel_wq); 380 380 + } 381 381 + 382 382 + module_init(atmel_i2c_init); 383 383 + module_exit(atmel_i2c_exit); 382 384 383 385 MODULE_AUTHOR("Tudor Ambarus <tudor.ambarus@microchip.com>"); 384 386 MODULE_DESCRIPTION("Microchip / Atmel ECC (I2C) driver");

+1

drivers/crypto/atmel-i2c.h

reviewed

··· 173 173 void (*cbk)(struct atmel_i2c_work_data *work_data, 174 174 void *areq, int status), 175 175 void *areq); 176 176 + void atmel_i2c_flush_queue(void); 176 177 177 178 int atmel_i2c_send_receive(struct i2c_client *client, struct atmel_i2c_cmd *cmd); 178 179

+6 -5

drivers/crypto/atmel-sha204a.c

reviewed

··· 121 121 struct atmel_i2c_client_priv *i2c_priv = i2c_get_clientdata(client); 122 122 123 123 if (atomic_read(&i2c_priv->tfm_count)) { 124 124 - dev_err(&client->dev, "Device is busy\n"); 125 125 - return -EBUSY; 124 124 + dev_emerg(&client->dev, "Device is busy, will remove it anyhow\n"); 125 125 + return 0; 126 126 } 127 127 128 128 - if (i2c_priv->hwrng.priv) 129 129 - kfree((void *)i2c_priv->hwrng.priv); 128 128 + kfree((void *)i2c_priv->hwrng.priv); 130 129 131 130 return 0; 132 131 } 133 132 134 133 static const struct of_device_id atmel_sha204a_dt_ids[] = { 134 134 + { .compatible = "atmel,atsha204", }, 135 135 { .compatible = "atmel,atsha204a", }, 136 136 { /* sentinel */ } 137 137 }; 138 138 MODULE_DEVICE_TABLE(of, atmel_sha204a_dt_ids); 139 139 140 140 static const struct i2c_device_id atmel_sha204a_id[] = { 141 141 + { "atsha204", 0 }, 141 142 { "atsha204a", 0 }, 142 143 { /* sentinel */ } 143 144 }; ··· 160 159 161 160 static void __exit atmel_sha204a_exit(void) 162 161 { 163 163 - flush_scheduled_work(); 162 162 + atmel_i2c_flush_queue(); 164 163 i2c_del_driver(&atmel_sha204a_driver); 165 164 } 166 165

+8

drivers/crypto/caam/Kconfig

reviewed

··· 151 151 Selecting this will register the SEC4 hardware rng to 152 152 the hw_random API for supplying the kernel entropy pool. 153 153 154 154 + config CRYPTO_DEV_FSL_CAAM_PRNG_API 155 155 + bool "Register Pseudo random number generation implementation with Crypto API" 156 156 + default y 157 157 + select CRYPTO_RNG 158 158 + help 159 159 + Selecting this will register the SEC hardware prng to 160 160 + the Crypto API. 161 161 + 154 162 config CRYPTO_DEV_FSL_CAAM_BLOB_GEN 155 163 bool 156 164

+1

drivers/crypto/caam/Makefile

reviewed

··· 20 20 caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI) += caamalg_qi.o 21 21 caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o 22 22 caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o 23 23 + caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API) += caamprng.o 23 24 caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API) += caampkc.o pkc_desc.o 24 25 caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_BLOB_GEN) += blob_gen.o 25 26

+235

drivers/crypto/caam/caamprng.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0+ 2 2 + /* 3 3 + * Driver to expose SEC4 PRNG via crypto RNG API 4 4 + * 5 5 + * Copyright 2022 NXP 6 6 + * 7 7 + */ 8 8 + 9 9 + #include <linux/completion.h> 10 10 + #include <crypto/internal/rng.h> 11 11 + #include "compat.h" 12 12 + #include "regs.h" 13 13 + #include "intern.h" 14 14 + #include "desc_constr.h" 15 15 + #include "jr.h" 16 16 + #include "error.h" 17 17 + 18 18 + /* 19 19 + * Length of used descriptors, see caam_init_desc() 20 20 + */ 21 21 + #define CAAM_PRNG_MAX_DESC_LEN (CAAM_CMD_SZ + \ 22 22 + CAAM_CMD_SZ + \ 23 23 + CAAM_CMD_SZ + CAAM_PTR_SZ_MAX) 24 24 + 25 25 + /* prng per-device context */ 26 26 + struct caam_prng_ctx { 27 27 + int err; 28 28 + struct completion done; 29 29 + }; 30 30 + 31 31 + struct caam_prng_alg { 32 32 + struct rng_alg rng; 33 33 + bool registered; 34 34 + }; 35 35 + 36 36 + static void caam_prng_done(struct device *jrdev, u32 *desc, u32 err, 37 37 + void *context) 38 38 + { 39 39 + struct caam_prng_ctx *jctx = context; 40 40 + 41 41 + jctx->err = err ? caam_jr_strstatus(jrdev, err) : 0; 42 42 + 43 43 + complete(&jctx->done); 44 44 + } 45 45 + 46 46 + static u32 *caam_init_reseed_desc(u32 *desc) 47 47 + { 48 48 + init_job_desc(desc, 0); /* + 1 cmd_sz */ 49 49 + /* Generate random bytes: + 1 cmd_sz */ 50 50 + append_operation(desc, OP_TYPE_CLASS1_ALG | OP_ALG_ALGSEL_RNG | 51 51 + OP_ALG_AS_FINALIZE); 52 52 + 53 53 + print_hex_dump_debug("prng reseed desc@: ", DUMP_PREFIX_ADDRESS, 54 54 + 16, 4, desc, desc_bytes(desc), 1); 55 55 + 56 56 + return desc; 57 57 + } 58 58 + 59 59 + static u32 *caam_init_prng_desc(u32 *desc, dma_addr_t dst_dma, u32 len) 60 60 + { 61 61 + init_job_desc(desc, 0); /* + 1 cmd_sz */ 62 62 + /* Generate random bytes: + 1 cmd_sz */ 63 63 + append_operation(desc, OP_ALG_ALGSEL_RNG | OP_TYPE_CLASS1_ALG); 64 64 + /* Store bytes: + 1 cmd_sz + caam_ptr_sz */ 65 65 + append_fifo_store(desc, dst_dma, 66 66 + len, FIFOST_TYPE_RNGSTORE); 67 67 + 68 68 + print_hex_dump_debug("prng job desc@: ", DUMP_PREFIX_ADDRESS, 69 69 + 16, 4, desc, desc_bytes(desc), 1); 70 70 + 71 71 + return desc; 72 72 + } 73 73 + 74 74 + static int caam_prng_generate(struct crypto_rng *tfm, 75 75 + const u8 *src, unsigned int slen, 76 76 + u8 *dst, unsigned int dlen) 77 77 + { 78 78 + struct caam_prng_ctx ctx; 79 79 + struct device *jrdev; 80 80 + dma_addr_t dst_dma; 81 81 + u32 *desc; 82 82 + u8 *buf; 83 83 + int ret; 84 84 + 85 85 + buf = kzalloc(dlen, GFP_KERNEL); 86 86 + if (!buf) 87 87 + return -ENOMEM; 88 88 + 89 89 + jrdev = caam_jr_alloc(); 90 90 + ret = PTR_ERR_OR_ZERO(jrdev); 91 91 + if (ret) { 92 92 + pr_err("Job Ring Device allocation failed\n"); 93 93 + kfree(buf); 94 94 + return ret; 95 95 + } 96 96 + 97 97 + desc = kzalloc(CAAM_PRNG_MAX_DESC_LEN, GFP_KERNEL | GFP_DMA); 98 98 + if (!desc) { 99 99 + ret = -ENOMEM; 100 100 + goto out1; 101 101 + } 102 102 + 103 103 + dst_dma = dma_map_single(jrdev, buf, dlen, DMA_FROM_DEVICE); 104 104 + if (dma_mapping_error(jrdev, dst_dma)) { 105 105 + dev_err(jrdev, "Failed to map destination buffer memory\n"); 106 106 + ret = -ENOMEM; 107 107 + goto out; 108 108 + } 109 109 + 110 110 + init_completion(&ctx.done); 111 111 + ret = caam_jr_enqueue(jrdev, 112 112 + caam_init_prng_desc(desc, dst_dma, dlen), 113 113 + caam_prng_done, &ctx); 114 114 + 115 115 + if (ret == -EINPROGRESS) { 116 116 + wait_for_completion(&ctx.done); 117 117 + ret = ctx.err; 118 118 + } 119 119 + 120 120 + dma_unmap_single(jrdev, dst_dma, dlen, DMA_FROM_DEVICE); 121 121 + 122 122 + if (!ret) 123 123 + memcpy(dst, buf, dlen); 124 124 + out: 125 125 + kfree(desc); 126 126 + out1: 127 127 + caam_jr_free(jrdev); 128 128 + kfree(buf); 129 129 + return ret; 130 130 + } 131 131 + 132 132 + static void caam_prng_exit(struct crypto_tfm *tfm) {} 133 133 + 134 134 + static int caam_prng_init(struct crypto_tfm *tfm) 135 135 + { 136 136 + return 0; 137 137 + } 138 138 + 139 139 + static int caam_prng_seed(struct crypto_rng *tfm, 140 140 + const u8 *seed, unsigned int slen) 141 141 + { 142 142 + struct caam_prng_ctx ctx; 143 143 + struct device *jrdev; 144 144 + u32 *desc; 145 145 + int ret; 146 146 + 147 147 + if (slen) { 148 148 + pr_err("Seed length should be zero\n"); 149 149 + return -EINVAL; 150 150 + } 151 151 + 152 152 + jrdev = caam_jr_alloc(); 153 153 + ret = PTR_ERR_OR_ZERO(jrdev); 154 154 + if (ret) { 155 155 + pr_err("Job Ring Device allocation failed\n"); 156 156 + return ret; 157 157 + } 158 158 + 159 159 + desc = kzalloc(CAAM_PRNG_MAX_DESC_LEN, GFP_KERNEL | GFP_DMA); 160 160 + if (!desc) { 161 161 + caam_jr_free(jrdev); 162 162 + return -ENOMEM; 163 163 + } 164 164 + 165 165 + init_completion(&ctx.done); 166 166 + ret = caam_jr_enqueue(jrdev, 167 167 + caam_init_reseed_desc(desc), 168 168 + caam_prng_done, &ctx); 169 169 + 170 170 + if (ret == -EINPROGRESS) { 171 171 + wait_for_completion(&ctx.done); 172 172 + ret = ctx.err; 173 173 + } 174 174 + 175 175 + kfree(desc); 176 176 + caam_jr_free(jrdev); 177 177 + return ret; 178 178 + } 179 179 + 180 180 + static struct caam_prng_alg caam_prng_alg = { 181 181 + .rng = { 182 182 + .generate = caam_prng_generate, 183 183 + .seed = caam_prng_seed, 184 184 + .seedsize = 0, 185 185 + .base = { 186 186 + .cra_name = "stdrng", 187 187 + .cra_driver_name = "prng-caam", 188 188 + .cra_priority = 500, 189 189 + .cra_ctxsize = sizeof(struct caam_prng_ctx), 190 190 + .cra_module = THIS_MODULE, 191 191 + .cra_init = caam_prng_init, 192 192 + .cra_exit = caam_prng_exit, 193 193 + }, 194 194 + } 195 195 + }; 196 196 + 197 197 + void caam_prng_unregister(void *data) 198 198 + { 199 199 + if (caam_prng_alg.registered) 200 200 + crypto_unregister_rng(&caam_prng_alg.rng); 201 201 + } 202 202 + 203 203 + int caam_prng_register(struct device *ctrldev) 204 204 + { 205 205 + struct caam_drv_private *priv = dev_get_drvdata(ctrldev); 206 206 + u32 rng_inst; 207 207 + int ret = 0; 208 208 + 209 209 + /* Check for available RNG blocks before registration */ 210 210 + if (priv->era < 10) 211 211 + rng_inst = (rd_reg32(&priv->jr[0]->perfmon.cha_num_ls) & 212 212 + CHA_ID_LS_RNG_MASK) >> CHA_ID_LS_RNG_SHIFT; 213 213 + else 214 214 + rng_inst = rd_reg32(&priv->jr[0]->vreg.rng) & CHA_VER_NUM_MASK; 215 215 + 216 216 + if (!rng_inst) { 217 217 + dev_dbg(ctrldev, "RNG block is not available... skipping registering algorithm\n"); 218 218 + return ret; 219 219 + } 220 220 + 221 221 + ret = crypto_register_rng(&caam_prng_alg.rng); 222 222 + if (ret) { 223 223 + dev_err(ctrldev, 224 224 + "couldn't register rng crypto alg: %d\n", 225 225 + ret); 226 226 + return ret; 227 227 + } 228 228 + 229 229 + caam_prng_alg.registered = true; 230 230 + 231 231 + dev_info(ctrldev, 232 232 + "rng crypto API alg registered %s\n", caam_prng_alg.rng.base.cra_driver_name); 233 233 + 234 234 + return 0; 235 235 + }

+18

drivers/crypto/caam/ctrl.c

reviewed

··· 609 609 } 610 610 #endif 611 611 612 612 + static bool needs_entropy_delay_adjustment(void) 613 613 + { 614 614 + if (of_machine_is_compatible("fsl,imx6sx")) 615 615 + return true; 616 616 + return false; 617 617 + } 618 618 + 612 619 /* Probe routine for CAAM top (controller) level */ 613 620 static int caam_probe(struct platform_device *pdev) 614 621 { ··· 875 868 * Also, if a handle was instantiated, do not change 876 869 * the TRNG parameters. 877 870 */ 871 871 + if (needs_entropy_delay_adjustment()) 872 872 + ent_delay = 12000; 878 873 if (!(ctrlpriv->rng4_sh_init || inst_handles)) { 879 874 dev_info(dev, 880 875 "Entropy delay = %u\n", ··· 893 884 */ 894 885 ret = instantiate_rng(dev, inst_handles, 895 886 gen_sk); 887 887 + /* 888 888 + * Entropy delay is determined via TRNG characterization. 889 889 + * TRNG characterization is run across different voltages 890 890 + * and temperatures. 891 891 + * If worst case value for ent_dly is identified, 892 892 + * the loop can be skipped for that platform. 893 893 + */ 894 894 + if (needs_entropy_delay_adjustment()) 895 895 + break; 896 896 if (ret == -EAGAIN) 897 897 /* 898 898 * if here, the loop will rerun,

+15

drivers/crypto/caam/intern.h

reviewed

··· 186 186 187 187 #endif /* CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API */ 188 188 189 189 + #ifdef CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API 190 190 + 191 191 + int caam_prng_register(struct device *dev); 192 192 + void caam_prng_unregister(void *data); 193 193 + 194 194 + #else 195 195 + 196 196 + static inline int caam_prng_register(struct device *dev) 197 197 + { 198 198 + return 0; 199 199 + } 200 200 + 201 201 + static inline void caam_prng_unregister(void *data) {} 202 202 + #endif /* CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API */ 203 203 + 189 204 #ifdef CONFIG_CAAM_QI 190 205 191 206 int caam_qi_algapi_init(struct device *dev);

+2 -1

drivers/crypto/caam/jr.c

reviewed

··· 39 39 caam_algapi_hash_init(dev); 40 40 caam_pkc_init(dev); 41 41 jrpriv->hwrng = !caam_rng_init(dev); 42 42 + caam_prng_register(dev); 42 43 caam_qi_algapi_init(dev); 43 44 44 45 algs_unlock: ··· 54 53 goto algs_unlock; 55 54 56 55 caam_qi_algapi_exit(); 57 57 - 56 56 + caam_prng_unregister(NULL); 58 57 caam_pkc_exit(); 59 58 caam_algapi_hash_exit(); 60 59 caam_algapi_exit();

+6 -4

drivers/crypto/cavium/nitrox/nitrox_main.c

reviewed

··· 269 269 270 270 struct nitrox_device *nitrox_get_first_device(void) 271 271 { 272 272 - struct nitrox_device *ndev; 272 272 + struct nitrox_device *ndev = NULL, *iter; 273 273 274 274 mutex_lock(&devlist_lock); 275 275 - list_for_each_entry(ndev, &ndevlist, list) { 276 276 - if (nitrox_ready(ndev)) 275 275 + list_for_each_entry(iter, &ndevlist, list) { 276 276 + if (nitrox_ready(iter)) { 277 277 + ndev = iter; 277 278 break; 279 279 + } 278 280 } 279 281 mutex_unlock(&devlist_lock); 280 280 - if (&ndev->list == &ndevlist) 282 282 + if (!ndev) 281 283 return NULL; 282 284 283 285 refcount_inc(&ndev->refcnt);

+18 -31

drivers/crypto/ccp/psp-dev.c

reviewed

··· 70 70 */ 71 71 if (val == 0xffffffff) { 72 72 dev_notice(psp->dev, "psp: unable to access the device: you might be running a broken BIOS.\n"); 73 73 - return 0; 73 73 + return -ENODEV; 74 74 } 75 75 + psp->capability = val; 75 76 76 76 - return val; 77 77 + /* Detect if TSME and SME are both enabled */ 78 78 + if (psp->capability & PSP_CAPABILITY_PSP_SECURITY_REPORTING && 79 79 + psp->capability & (PSP_SECURITY_TSME_STATUS << PSP_CAPABILITY_PSP_SECURITY_OFFSET) && 80 80 + cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) 81 81 + dev_notice(psp->dev, "psp: Both TSME and SME are active, SME is unnecessary when TSME is active.\n"); 82 82 + 83 83 + return 0; 77 84 } 78 85 79 79 - static int psp_check_sev_support(struct psp_device *psp, 80 80 - unsigned int capability) 86 86 + static int psp_check_sev_support(struct psp_device *psp) 81 87 { 82 88 /* Check if device supports SEV feature */ 83 83 - if (!(capability & 1)) { 89 89 + if (!(psp->capability & PSP_CAPABILITY_SEV)) { 84 90 dev_dbg(psp->dev, "psp does not support SEV\n"); 85 91 return -ENODEV; 86 92 } ··· 94 88 return 0; 95 89 } 96 90 97 97 - static int psp_check_tee_support(struct psp_device *psp, 98 98 - unsigned int capability) 91 91 + static int psp_check_tee_support(struct psp_device *psp) 99 92 { 100 93 /* Check if device supports TEE feature */ 101 101 - if (!(capability & 2)) { 94 94 + if (!(psp->capability & PSP_CAPABILITY_TEE)) { 102 95 dev_dbg(psp->dev, "psp does not support TEE\n"); 103 96 return -ENODEV; 104 97 } ··· 105 100 return 0; 106 101 } 107 102 108 108 - static int psp_check_support(struct psp_device *psp, 109 109 - unsigned int capability) 110 110 - { 111 111 - int sev_support = psp_check_sev_support(psp, capability); 112 112 - int tee_support = psp_check_tee_support(psp, capability); 113 113 - 114 114 - /* Return error if device neither supports SEV nor TEE */ 115 115 - if (sev_support && tee_support) 116 116 - return -ENODEV; 117 117 - 118 118 - return 0; 119 119 - } 120 120 - 121 121 - static int psp_init(struct psp_device *psp, unsigned int capability) 103 103 + static int psp_init(struct psp_device *psp) 122 104 { 123 105 int ret; 124 106 125 125 - if (!psp_check_sev_support(psp, capability)) { 107 107 + if (!psp_check_sev_support(psp)) { 126 108 ret = sev_dev_init(psp); 127 109 if (ret) 128 110 return ret; 129 111 } 130 112 131 131 - if (!psp_check_tee_support(psp, capability)) { 113 113 + if (!psp_check_tee_support(psp)) { 132 114 ret = tee_dev_init(psp); 133 115 if (ret) 134 116 return ret; ··· 128 136 { 129 137 struct device *dev = sp->dev; 130 138 struct psp_device *psp; 131 131 - unsigned int capability; 132 139 int ret; 133 140 134 141 ret = -ENOMEM; ··· 146 155 147 156 psp->io_regs = sp->io_map; 148 157 149 149 - capability = psp_get_capability(psp); 150 150 - if (!capability) 151 151 - goto e_disable; 152 152 - 153 153 - ret = psp_check_support(psp, capability); 158 158 + ret = psp_get_capability(psp); 154 159 if (ret) 155 160 goto e_disable; 156 161 ··· 161 174 goto e_err; 162 175 } 163 176 164 164 - ret = psp_init(psp, capability); 177 177 + ret = psp_init(psp); 165 178 if (ret) 166 179 goto e_irq; 167 180

+22

drivers/crypto/ccp/psp-dev.h

reviewed

··· 45 45 46 46 void *sev_data; 47 47 void *tee_data; 48 48 + 49 49 + unsigned int capability; 48 50 }; 49 51 50 52 void psp_set_sev_irq_handler(struct psp_device *psp, psp_irq_handler_t handler, ··· 58 56 void psp_clear_tee_irq_handler(struct psp_device *psp); 59 57 60 58 struct psp_device *psp_get_master_device(void); 59 59 + 60 60 + #define PSP_CAPABILITY_SEV BIT(0) 61 61 + #define PSP_CAPABILITY_TEE BIT(1) 62 62 + #define PSP_CAPABILITY_PSP_SECURITY_REPORTING BIT(7) 63 63 + 64 64 + #define PSP_CAPABILITY_PSP_SECURITY_OFFSET 8 65 65 + /* 66 66 + * The PSP doesn't directly store these bits in the capability register 67 67 + * but instead copies them from the results of query command. 68 68 + * 69 69 + * The offsets from the query command are below, and shifted when used. 70 70 + */ 71 71 + #define PSP_SECURITY_FUSED_PART BIT(0) 72 72 + #define PSP_SECURITY_DEBUG_LOCK_ON BIT(2) 73 73 + #define PSP_SECURITY_TSME_STATUS BIT(5) 74 74 + #define PSP_SECURITY_ANTI_ROLLBACK_STATUS BIT(7) 75 75 + #define PSP_SECURITY_RPMC_PRODUCTION_ENABLED BIT(8) 76 76 + #define PSP_SECURITY_RPMC_SPIROM_AVAILABLE BIT(9) 77 77 + #define PSP_SECURITY_HSP_TPM_AVAILABLE BIT(10) 78 78 + #define PSP_SECURITY_ROM_ARMOR_ENFORCED BIT(11) 61 79 62 80 #endif /* __PSP_DEV_H */

+29 -3

drivers/crypto/ccp/sev-dev.c

reviewed

··· 23 23 #include <linux/gfp.h> 24 24 #include <linux/cpufeature.h> 25 25 #include <linux/fs.h> 26 26 + #include <linux/fs_struct.h> 26 27 27 28 #include <asm/smp.h> 28 29 ··· 171 170 return page_address(page); 172 171 } 173 172 173 173 + static struct file *open_file_as_root(const char *filename, int flags, umode_t mode) 174 174 + { 175 175 + struct file *fp; 176 176 + struct path root; 177 177 + struct cred *cred; 178 178 + const struct cred *old_cred; 179 179 + 180 180 + task_lock(&init_task); 181 181 + get_fs_root(init_task.fs, &root); 182 182 + task_unlock(&init_task); 183 183 + 184 184 + cred = prepare_creds(); 185 185 + if (!cred) 186 186 + return ERR_PTR(-ENOMEM); 187 187 + cred->fsuid = GLOBAL_ROOT_UID; 188 188 + old_cred = override_creds(cred); 189 189 + 190 190 + fp = file_open_root(&root, filename, flags, mode); 191 191 + path_put(&root); 192 192 + 193 193 + revert_creds(old_cred); 194 194 + 195 195 + return fp; 196 196 + } 197 197 + 174 198 static int sev_read_init_ex_file(void) 175 199 { 176 200 struct sev_device *sev = psp_master->sev_data; ··· 207 181 if (!sev_init_ex_buffer) 208 182 return -EOPNOTSUPP; 209 183 210 210 - fp = filp_open(init_ex_path, O_RDONLY, 0); 184 184 + fp = open_file_as_root(init_ex_path, O_RDONLY, 0); 211 185 if (IS_ERR(fp)) { 212 186 int ret = PTR_ERR(fp); 213 187 ··· 243 217 if (!sev_init_ex_buffer) 244 218 return; 245 219 246 246 - fp = filp_open(init_ex_path, O_CREAT | O_WRONLY, 0600); 220 220 + fp = open_file_as_root(init_ex_path, O_CREAT | O_WRONLY, 0600); 247 221 if (IS_ERR(fp)) { 248 222 dev_err(sev->dev, 249 223 "SEV: could not open file for write, error %ld\n", ··· 461 435 * initialization function should succeed by replacing the state 462 436 * with a reset state. 463 437 */ 464 464 - dev_dbg(sev->dev, "SEV: retrying INIT command"); 438 438 + dev_err(sev->dev, "SEV: retrying INIT command because of SECURE_DATA_INVALID error. Retrying once to reset PSP SEV state."); 465 439 rc = init_function(&psp_ret); 466 440 } 467 441 if (error)

+62

drivers/crypto/ccp/sp-pci.c

reviewed

··· 32 32 }; 33 33 static struct sp_device *sp_dev_master; 34 34 35 35 + #define attribute_show(name, def) \ 36 36 + static ssize_t name##_show(struct device *d, struct device_attribute *attr, \ 37 37 + char *buf) \ 38 38 + { \ 39 39 + struct sp_device *sp = dev_get_drvdata(d); \ 40 40 + struct psp_device *psp = sp->psp_data; \ 41 41 + int bit = PSP_SECURITY_##def << PSP_CAPABILITY_PSP_SECURITY_OFFSET; \ 42 42 + return sysfs_emit(buf, "%d\n", (psp->capability & bit) > 0); \ 43 43 + } 44 44 + 45 45 + attribute_show(fused_part, FUSED_PART) 46 46 + static DEVICE_ATTR_RO(fused_part); 47 47 + attribute_show(debug_lock_on, DEBUG_LOCK_ON) 48 48 + static DEVICE_ATTR_RO(debug_lock_on); 49 49 + attribute_show(tsme_status, TSME_STATUS) 50 50 + static DEVICE_ATTR_RO(tsme_status); 51 51 + attribute_show(anti_rollback_status, ANTI_ROLLBACK_STATUS) 52 52 + static DEVICE_ATTR_RO(anti_rollback_status); 53 53 + attribute_show(rpmc_production_enabled, RPMC_PRODUCTION_ENABLED) 54 54 + static DEVICE_ATTR_RO(rpmc_production_enabled); 55 55 + attribute_show(rpmc_spirom_available, RPMC_SPIROM_AVAILABLE) 56 56 + static DEVICE_ATTR_RO(rpmc_spirom_available); 57 57 + attribute_show(hsp_tpm_available, HSP_TPM_AVAILABLE) 58 58 + static DEVICE_ATTR_RO(hsp_tpm_available); 59 59 + attribute_show(rom_armor_enforced, ROM_ARMOR_ENFORCED) 60 60 + static DEVICE_ATTR_RO(rom_armor_enforced); 61 61 + 62 62 + static struct attribute *psp_attrs[] = { 63 63 + &dev_attr_fused_part.attr, 64 64 + &dev_attr_debug_lock_on.attr, 65 65 + &dev_attr_tsme_status.attr, 66 66 + &dev_attr_anti_rollback_status.attr, 67 67 + &dev_attr_rpmc_production_enabled.attr, 68 68 + &dev_attr_rpmc_spirom_available.attr, 69 69 + &dev_attr_hsp_tpm_available.attr, 70 70 + &dev_attr_rom_armor_enforced.attr, 71 71 + NULL 72 72 + }; 73 73 + 74 74 + static umode_t psp_security_is_visible(struct kobject *kobj, struct attribute *attr, int idx) 75 75 + { 76 76 + struct device *dev = kobj_to_dev(kobj); 77 77 + struct sp_device *sp = dev_get_drvdata(dev); 78 78 + struct psp_device *psp = sp->psp_data; 79 79 + 80 80 + if (psp && (psp->capability & PSP_CAPABILITY_PSP_SECURITY_REPORTING)) 81 81 + return 0444; 82 82 + 83 83 + return 0; 84 84 + } 85 85 + 86 86 + static struct attribute_group psp_attr_group = { 87 87 + .attrs = psp_attrs, 88 88 + .is_visible = psp_security_is_visible, 89 89 + }; 90 90 + 91 91 + static const struct attribute_group *psp_groups[] = { 92 92 + &psp_attr_group, 93 93 + NULL, 94 94 + }; 95 95 + 35 96 static int sp_get_msix_irqs(struct sp_device *sp) 36 97 { 37 98 struct sp_pci *sp_pci = sp->dev_specific; ··· 452 391 .remove = sp_pci_remove, 453 392 .shutdown = sp_pci_shutdown, 454 393 .driver.pm = &sp_pci_pm_ops, 394 394 + .dev_groups = psp_groups, 455 395 }; 456 396 457 397 int sp_pci_init(void)

+15 -12

drivers/crypto/ccree/cc_buffer_mgr.c

reviewed

··· 356 356 req_ctx->mlli_params.mlli_dma_addr); 357 357 } 358 358 359 359 - dma_unmap_sg(dev, src, req_ctx->in_nents, DMA_BIDIRECTIONAL); 360 360 - dev_dbg(dev, "Unmapped req->src=%pK\n", sg_virt(src)); 361 361 - 362 359 if (src != dst) { 363 363 - dma_unmap_sg(dev, dst, req_ctx->out_nents, DMA_BIDIRECTIONAL); 360 360 + dma_unmap_sg(dev, src, req_ctx->in_nents, DMA_TO_DEVICE); 361 361 + dma_unmap_sg(dev, dst, req_ctx->out_nents, DMA_FROM_DEVICE); 364 362 dev_dbg(dev, "Unmapped req->dst=%pK\n", sg_virt(dst)); 363 363 + dev_dbg(dev, "Unmapped req->src=%pK\n", sg_virt(src)); 364 364 + } else { 365 365 + dma_unmap_sg(dev, src, req_ctx->in_nents, DMA_BIDIRECTIONAL); 366 366 + dev_dbg(dev, "Unmapped req->src=%pK\n", sg_virt(src)); 365 367 } 366 368 } 367 369 ··· 379 377 u32 dummy = 0; 380 378 int rc = 0; 381 379 u32 mapped_nents = 0; 380 380 + int src_direction = (src != dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL); 382 381 383 382 req_ctx->dma_buf_type = CC_DMA_BUF_DLLI; 384 383 mlli_params->curr_pool = NULL; ··· 402 399 } 403 400 404 401 /* Map the src SGL */ 405 405 - rc = cc_map_sg(dev, src, nbytes, DMA_BIDIRECTIONAL, &req_ctx->in_nents, 402 402 + rc = cc_map_sg(dev, src, nbytes, src_direction, &req_ctx->in_nents, 406 403 LLI_MAX_NUM_OF_DATA_ENTRIES, &dummy, &mapped_nents); 407 404 if (rc) 408 405 goto cipher_exit; ··· 419 416 } 420 417 } else { 421 418 /* Map the dst sg */ 422 422 - rc = cc_map_sg(dev, dst, nbytes, DMA_BIDIRECTIONAL, 419 419 + rc = cc_map_sg(dev, dst, nbytes, DMA_FROM_DEVICE, 423 420 &req_ctx->out_nents, LLI_MAX_NUM_OF_DATA_ENTRIES, 424 421 &dummy, &mapped_nents); 425 422 if (rc) ··· 459 456 struct aead_req_ctx *areq_ctx = aead_request_ctx(req); 460 457 unsigned int hw_iv_size = areq_ctx->hw_iv_size; 461 458 struct cc_drvdata *drvdata = dev_get_drvdata(dev); 459 459 + int src_direction = (req->src != req->dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL); 462 460 463 461 if (areq_ctx->mac_buf_dma_addr) { 464 462 dma_unmap_single(dev, areq_ctx->mac_buf_dma_addr, ··· 518 514 sg_virt(req->src), areq_ctx->src.nents, areq_ctx->assoc.nents, 519 515 areq_ctx->assoclen, req->cryptlen); 520 516 521 521 - dma_unmap_sg(dev, req->src, areq_ctx->src.mapped_nents, 522 522 - DMA_BIDIRECTIONAL); 517 517 + dma_unmap_sg(dev, req->src, areq_ctx->src.mapped_nents, src_direction); 523 518 if (req->src != req->dst) { 524 519 dev_dbg(dev, "Unmapping dst sgl: req->dst=%pK\n", 525 520 sg_virt(req->dst)); 526 526 - dma_unmap_sg(dev, req->dst, areq_ctx->dst.mapped_nents, 527 527 - DMA_BIDIRECTIONAL); 521 521 + dma_unmap_sg(dev, req->dst, areq_ctx->dst.mapped_nents, DMA_FROM_DEVICE); 528 522 } 529 523 if (drvdata->coherent && 530 524 areq_ctx->gen_ctx.op_type == DRV_CRYPTO_DIRECTION_DECRYPT && ··· 845 843 else 846 844 size_for_map -= authsize; 847 845 848 848 - rc = cc_map_sg(dev, req->dst, size_for_map, DMA_BIDIRECTIONAL, 846 846 + rc = cc_map_sg(dev, req->dst, size_for_map, DMA_FROM_DEVICE, 849 847 &areq_ctx->dst.mapped_nents, 850 848 LLI_MAX_NUM_OF_DATA_ENTRIES, &dst_last_bytes, 851 849 &dst_mapped_nents); ··· 1058 1056 size_to_map += authsize; 1059 1057 } 1060 1058 1061 1061 - rc = cc_map_sg(dev, req->src, size_to_map, DMA_BIDIRECTIONAL, 1059 1059 + rc = cc_map_sg(dev, req->src, size_to_map, 1060 1060 + (req->src != req->dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL), 1062 1061 &areq_ctx->src.mapped_nents, 1063 1062 (LLI_MAX_NUM_OF_ASSOC_DATA_ENTRIES + 1064 1063 LLI_MAX_NUM_OF_DATA_ENTRIES),

+14 -12

drivers/crypto/ccree/cc_driver.c

reviewed

··· 529 529 goto post_req_mgr_err; 530 530 } 531 531 532 532 + /* hash must be allocated first due to use of send_request_init() 533 533 + * and dependency of AEAD on it 534 534 + */ 535 535 + rc = cc_hash_alloc(new_drvdata); 536 536 + if (rc) { 537 537 + dev_err(dev, "cc_hash_alloc failed\n"); 538 538 + goto post_buf_mgr_err; 539 539 + } 540 540 + 532 541 /* Allocate crypto algs */ 533 542 rc = cc_cipher_alloc(new_drvdata); 534 543 if (rc) { 535 544 dev_err(dev, "cc_cipher_alloc failed\n"); 536 536 - goto post_buf_mgr_err; 537 537 - } 538 538 - 539 539 - /* hash must be allocated before aead since hash exports APIs */ 540 540 - rc = cc_hash_alloc(new_drvdata); 541 541 - if (rc) { 542 542 - dev_err(dev, "cc_hash_alloc failed\n"); 543 543 - goto post_cipher_err; 545 545 + goto post_hash_err; 544 546 } 545 547 546 548 rc = cc_aead_alloc(new_drvdata); 547 549 if (rc) { 548 550 dev_err(dev, "cc_aead_alloc failed\n"); 549 549 - goto post_hash_err; 551 551 + goto post_cipher_err; 550 552 } 551 553 552 554 /* If we got here and FIPS mode is enabled ··· 560 558 pm_runtime_put(dev); 561 559 return 0; 562 560 563 563 - post_hash_err: 564 564 - cc_hash_free(new_drvdata); 565 561 post_cipher_err: 566 562 cc_cipher_free(new_drvdata); 563 563 + post_hash_err: 564 564 + cc_hash_free(new_drvdata); 567 565 post_buf_mgr_err: 568 566 cc_buffer_mgr_fini(new_drvdata); 569 567 post_req_mgr_err: ··· 595 593 (struct cc_drvdata *)platform_get_drvdata(plat_dev); 596 594 597 595 cc_aead_free(drvdata); 598 598 - cc_hash_free(drvdata); 599 596 cc_cipher_free(drvdata); 597 597 + cc_hash_free(drvdata); 600 598 cc_buffer_mgr_fini(drvdata); 601 599 cc_req_mgr_fini(drvdata); 602 600 cc_fips_fini(drvdata);

+1

drivers/crypto/hisilicon/Kconfig

reviewed

··· 26 26 select CRYPTO_SHA1 27 27 select CRYPTO_SHA256 28 28 select CRYPTO_SHA512 29 29 + select CRYPTO_SM4 29 30 depends on PCI && PCI_MSI 30 31 depends on UACCE || UACCE=n 31 32 depends on ARM64 || (COMPILE_TEST && 64BIT)

+184 -38

drivers/crypto/hisilicon/hpre/hpre_main.c

reviewed

··· 36 36 #define HPRE_DATA_WUSER_CFG 0x301040 37 37 #define HPRE_INT_MASK 0x301400 38 38 #define HPRE_INT_STATUS 0x301800 39 39 + #define HPRE_HAC_INT_MSK 0x301400 40 40 + #define HPRE_HAC_RAS_CE_ENB 0x301410 41 41 + #define HPRE_HAC_RAS_NFE_ENB 0x301414 42 42 + #define HPRE_HAC_RAS_FE_ENB 0x301418 43 43 + #define HPRE_HAC_INT_SET 0x301500 44 44 + #define HPRE_RNG_TIMEOUT_NUM 0x301A34 39 45 #define HPRE_CORE_INT_ENABLE 0 40 46 #define HPRE_CORE_INT_DISABLE GENMASK(21, 0) 41 47 #define HPRE_RDCHN_INI_ST 0x301a00 ··· 112 106 #define HPRE_VIA_MSI_DSM 1 113 107 #define HPRE_SQE_MASK_OFFSET 8 114 108 #define HPRE_SQE_MASK_LEN 24 109 109 + 110 110 + #define HPRE_DFX_BASE 0x301000 111 111 + #define HPRE_DFX_COMMON1 0x301400 112 112 + #define HPRE_DFX_COMMON2 0x301A00 113 113 + #define HPRE_DFX_CORE 0x302000 114 114 + #define HPRE_DFX_BASE_LEN 0x55 115 115 + #define HPRE_DFX_COMMON1_LEN 0x41 116 116 + #define HPRE_DFX_COMMON2_LEN 0xE 117 117 + #define HPRE_DFX_CORE_LEN 0x43 115 118 116 119 static const char hpre_name[] = "hisi_hpre"; 117 120 static struct dentry *hpre_debugfs_root; ··· 207 192 }; 208 193 209 194 static const struct debugfs_reg32 hpre_cluster_dfx_regs[] = { 210 210 - {"CORES_EN_STATUS ", HPRE_CORE_EN_OFFSET}, 211 211 - {"CORES_INI_CFG ", HPRE_CORE_INI_CFG_OFFSET}, 212 212 - {"CORES_INI_STATUS ", HPRE_CORE_INI_STATUS_OFFSET}, 213 213 - {"CORES_HTBT_WARN ", HPRE_CORE_HTBT_WARN_OFFSET}, 214 214 - {"CORES_IS_SCHD ", HPRE_CORE_IS_SCHD_OFFSET}, 195 195 + {"CORES_EN_STATUS ", HPRE_CORE_EN_OFFSET}, 196 196 + {"CORES_INI_CFG ", HPRE_CORE_INI_CFG_OFFSET}, 197 197 + {"CORES_INI_STATUS ", HPRE_CORE_INI_STATUS_OFFSET}, 198 198 + {"CORES_HTBT_WARN ", HPRE_CORE_HTBT_WARN_OFFSET}, 199 199 + {"CORES_IS_SCHD ", HPRE_CORE_IS_SCHD_OFFSET}, 215 200 }; 216 201 217 202 static const struct debugfs_reg32 hpre_com_dfx_regs[] = { 218 218 - {"READ_CLR_EN ", HPRE_CTRL_CNT_CLR_CE}, 219 219 - {"AXQOS ", HPRE_VFG_AXQOS}, 220 220 - {"AWUSR_CFG ", HPRE_AWUSR_FP_CFG}, 221 221 - {"QM_ARUSR_MCFG1 ", QM_ARUSER_M_CFG_1}, 222 222 - {"QM_AWUSR_MCFG1 ", QM_AWUSER_M_CFG_1}, 223 223 - {"BD_ENDIAN ", HPRE_BD_ENDIAN}, 224 224 - {"ECC_CHECK_CTRL ", HPRE_ECC_BYPASS}, 225 225 - {"RAS_INT_WIDTH ", HPRE_RAS_WIDTH_CFG}, 226 226 - {"POISON_BYPASS ", HPRE_POISON_BYPASS}, 227 227 - {"BD_ARUSER ", HPRE_BD_ARUSR_CFG}, 228 228 - {"BD_AWUSER ", HPRE_BD_AWUSR_CFG}, 229 229 - {"DATA_ARUSER ", HPRE_DATA_RUSER_CFG}, 230 230 - {"DATA_AWUSER ", HPRE_DATA_WUSER_CFG}, 231 231 - {"INT_STATUS ", HPRE_INT_STATUS}, 203 203 + {"READ_CLR_EN ", HPRE_CTRL_CNT_CLR_CE}, 204 204 + {"AXQOS ", HPRE_VFG_AXQOS}, 205 205 + {"AWUSR_CFG ", HPRE_AWUSR_FP_CFG}, 206 206 + {"BD_ENDIAN ", HPRE_BD_ENDIAN}, 207 207 + {"ECC_CHECK_CTRL ", HPRE_ECC_BYPASS}, 208 208 + {"RAS_INT_WIDTH ", HPRE_RAS_WIDTH_CFG}, 209 209 + {"POISON_BYPASS ", HPRE_POISON_BYPASS}, 210 210 + {"BD_ARUSER ", HPRE_BD_ARUSR_CFG}, 211 211 + {"BD_AWUSER ", HPRE_BD_AWUSR_CFG}, 212 212 + {"DATA_ARUSER ", HPRE_DATA_RUSER_CFG}, 213 213 + {"DATA_AWUSER ", HPRE_DATA_WUSER_CFG}, 214 214 + {"INT_STATUS ", HPRE_INT_STATUS}, 215 215 + {"INT_MASK ", HPRE_HAC_INT_MSK}, 216 216 + {"RAS_CE_ENB ", HPRE_HAC_RAS_CE_ENB}, 217 217 + {"RAS_NFE_ENB ", HPRE_HAC_RAS_NFE_ENB}, 218 218 + {"RAS_FE_ENB ", HPRE_HAC_RAS_FE_ENB}, 219 219 + {"INT_SET ", HPRE_HAC_INT_SET}, 220 220 + {"RNG_TIMEOUT_NUM ", HPRE_RNG_TIMEOUT_NUM}, 232 221 }; 233 222 234 223 static const char *hpre_dfx_files[HPRE_DFX_FILE_NUM] = { ··· 244 225 "overtime_thrhld", 245 226 "invalid_req_cnt" 246 227 }; 228 228 + 229 229 + /* define the HPRE's dfx regs region and region length */ 230 230 + static struct dfx_diff_registers hpre_diff_regs[] = { 231 231 + { 232 232 + .reg_offset = HPRE_DFX_BASE, 233 233 + .reg_len = HPRE_DFX_BASE_LEN, 234 234 + }, { 235 235 + .reg_offset = HPRE_DFX_COMMON1, 236 236 + .reg_len = HPRE_DFX_COMMON1_LEN, 237 237 + }, { 238 238 + .reg_offset = HPRE_DFX_COMMON2, 239 239 + .reg_len = HPRE_DFX_COMMON2_LEN, 240 240 + }, { 241 241 + .reg_offset = HPRE_DFX_CORE, 242 242 + .reg_len = HPRE_DFX_CORE_LEN, 243 243 + }, 244 244 + }; 245 245 + 246 246 + static int hpre_diff_regs_show(struct seq_file *s, void *unused) 247 247 + { 248 248 + struct hisi_qm *qm = s->private; 249 249 + 250 250 + hisi_qm_acc_diff_regs_dump(qm, s, qm->debug.acc_diff_regs, 251 251 + ARRAY_SIZE(hpre_diff_regs)); 252 252 + 253 253 + return 0; 254 254 + } 255 255 + 256 256 + DEFINE_SHOW_ATTRIBUTE(hpre_diff_regs); 257 257 + 258 258 + static int hpre_com_regs_show(struct seq_file *s, void *unused) 259 259 + { 260 260 + hisi_qm_regs_dump(s, s->private); 261 261 + 262 262 + return 0; 263 263 + } 264 264 + 265 265 + DEFINE_SHOW_ATTRIBUTE(hpre_com_regs); 266 266 + 267 267 + static int hpre_cluster_regs_show(struct seq_file *s, void *unused) 268 268 + { 269 269 + hisi_qm_regs_dump(s, s->private); 270 270 + 271 271 + return 0; 272 272 + } 273 273 + 274 274 + DEFINE_SHOW_ATTRIBUTE(hpre_cluster_regs); 247 275 248 276 static const struct kernel_param_ops hpre_uacce_mode_ops = { 249 277 .set = uacce_mode_set, ··· 845 779 DEFINE_DEBUGFS_ATTRIBUTE(hpre_atomic64_ops, hpre_debugfs_atomic64_get, 846 780 hpre_debugfs_atomic64_set, "%llu\n"); 847 781 848 848 - static int hpre_com_regs_show(struct seq_file *s, void *unused) 849 849 - { 850 850 - hisi_qm_regs_dump(s, s->private); 851 851 - 852 852 - return 0; 853 853 - } 854 854 - 855 855 - DEFINE_SHOW_ATTRIBUTE(hpre_com_regs); 856 856 - 857 857 - static int hpre_cluster_regs_show(struct seq_file *s, void *unused) 858 858 - { 859 859 - hisi_qm_regs_dump(s, s->private); 860 860 - 861 861 - return 0; 862 862 - } 863 863 - 864 864 - DEFINE_SHOW_ATTRIBUTE(hpre_cluster_regs); 865 865 - 866 782 static int hpre_create_debugfs_file(struct hisi_qm *qm, struct dentry *dir, 867 783 enum hpre_ctrl_dbgfs_file type, int indx) 868 784 { ··· 943 895 944 896 static void hpre_dfx_debug_init(struct hisi_qm *qm) 945 897 { 898 898 + struct dfx_diff_registers *hpre_regs = qm->debug.acc_diff_regs; 946 899 struct hpre *hpre = container_of(qm, struct hpre, qm); 947 900 struct hpre_dfx *dfx = hpre->debug.dfx; 948 901 struct dentry *parent; ··· 955 906 debugfs_create_file(hpre_dfx_files[i], 0644, parent, &dfx[i], 956 907 &hpre_atomic64_ops); 957 908 } 909 909 + 910 910 + if (qm->fun_type == QM_HW_PF && hpre_regs) 911 911 + debugfs_create_file("diff_regs", 0444, parent, 912 912 + qm, &hpre_diff_regs_fops); 958 913 } 959 914 960 915 static int hpre_debugfs_init(struct hisi_qm *qm) ··· 971 918 972 919 qm->debug.sqe_mask_offset = HPRE_SQE_MASK_OFFSET; 973 920 qm->debug.sqe_mask_len = HPRE_SQE_MASK_LEN; 921 921 + ret = hisi_qm_diff_regs_init(qm, hpre_diff_regs, 922 922 + ARRAY_SIZE(hpre_diff_regs)); 923 923 + if (ret) { 924 924 + dev_warn(dev, "Failed to init HPRE diff regs!\n"); 925 925 + goto debugfs_remove; 926 926 + } 927 927 + 974 928 hisi_qm_debug_init(qm); 975 929 976 930 if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_HPRE_PF) { ··· 991 931 return 0; 992 932 993 933 failed_to_create: 934 934 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(hpre_diff_regs)); 935 935 + debugfs_remove: 994 936 debugfs_remove_recursive(qm->debug.debug_root); 995 937 return ret; 996 938 } 997 939 998 940 static void hpre_debugfs_exit(struct hisi_qm *qm) 999 941 { 942 942 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(hpre_diff_regs)); 943 943 + 1000 944 debugfs_remove_recursive(qm->debug.debug_root); 1001 945 } 1002 946 ··· 1031 967 } 1032 968 1033 969 return hisi_qm_init(qm); 970 970 + } 971 971 + 972 972 + static int hpre_show_last_regs_init(struct hisi_qm *qm) 973 973 + { 974 974 + int cluster_dfx_regs_num = ARRAY_SIZE(hpre_cluster_dfx_regs); 975 975 + int com_dfx_regs_num = ARRAY_SIZE(hpre_com_dfx_regs); 976 976 + u8 clusters_num = hpre_cluster_num(qm); 977 977 + struct qm_debug *debug = &qm->debug; 978 978 + void __iomem *io_base; 979 979 + int i, j, idx; 980 980 + 981 981 + debug->last_words = kcalloc(cluster_dfx_regs_num * clusters_num + 982 982 + com_dfx_regs_num, sizeof(unsigned int), GFP_KERNEL); 983 983 + if (!debug->last_words) 984 984 + return -ENOMEM; 985 985 + 986 986 + for (i = 0; i < com_dfx_regs_num; i++) 987 987 + debug->last_words[i] = readl_relaxed(qm->io_base + 988 988 + hpre_com_dfx_regs[i].offset); 989 989 + 990 990 + for (i = 0; i < clusters_num; i++) { 991 991 + io_base = qm->io_base + hpre_cluster_offsets[i]; 992 992 + for (j = 0; j < cluster_dfx_regs_num; j++) { 993 993 + idx = com_dfx_regs_num + i * cluster_dfx_regs_num + j; 994 994 + debug->last_words[idx] = readl_relaxed( 995 995 + io_base + hpre_cluster_dfx_regs[j].offset); 996 996 + } 997 997 + } 998 998 + 999 999 + return 0; 1000 1000 + } 1001 1001 + 1002 1002 + static void hpre_show_last_regs_uninit(struct hisi_qm *qm) 1003 1003 + { 1004 1004 + struct qm_debug *debug = &qm->debug; 1005 1005 + 1006 1006 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 1007 1007 + return; 1008 1008 + 1009 1009 + kfree(debug->last_words); 1010 1010 + debug->last_words = NULL; 1011 1011 + } 1012 1012 + 1013 1013 + static void hpre_show_last_dfx_regs(struct hisi_qm *qm) 1014 1014 + { 1015 1015 + int cluster_dfx_regs_num = ARRAY_SIZE(hpre_cluster_dfx_regs); 1016 1016 + int com_dfx_regs_num = ARRAY_SIZE(hpre_com_dfx_regs); 1017 1017 + u8 clusters_num = hpre_cluster_num(qm); 1018 1018 + struct qm_debug *debug = &qm->debug; 1019 1019 + struct pci_dev *pdev = qm->pdev; 1020 1020 + void __iomem *io_base; 1021 1021 + int i, j, idx; 1022 1022 + u32 val; 1023 1023 + 1024 1024 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 1025 1025 + return; 1026 1026 + 1027 1027 + /* dumps last word of the debugging registers during controller reset */ 1028 1028 + for (i = 0; i < com_dfx_regs_num; i++) { 1029 1029 + val = readl_relaxed(qm->io_base + hpre_com_dfx_regs[i].offset); 1030 1030 + if (debug->last_words[i] != val) 1031 1031 + pci_info(pdev, "Common_core:%s \t= 0x%08x => 0x%08x\n", 1032 1032 + hpre_com_dfx_regs[i].name, debug->last_words[i], val); 1033 1033 + } 1034 1034 + 1035 1035 + for (i = 0; i < clusters_num; i++) { 1036 1036 + io_base = qm->io_base + hpre_cluster_offsets[i]; 1037 1037 + for (j = 0; j < cluster_dfx_regs_num; j++) { 1038 1038 + val = readl_relaxed(io_base + 1039 1039 + hpre_cluster_dfx_regs[j].offset); 1040 1040 + idx = com_dfx_regs_num + i * cluster_dfx_regs_num + j; 1041 1041 + if (debug->last_words[idx] != val) 1042 1042 + pci_info(pdev, "cluster-%d:%s \t= 0x%08x => 0x%08x\n", 1043 1043 + i, hpre_cluster_dfx_regs[j].name, debug->last_words[idx], val); 1044 1044 + } 1045 1045 + } 1034 1046 } 1035 1047 1036 1048 static void hpre_log_hw_error(struct hisi_qm *qm, u32 err_sts) ··· 1167 1027 .open_axi_master_ooo = hpre_open_axi_master_ooo, 1168 1028 .open_sva_prefetch = hpre_open_sva_prefetch, 1169 1029 .close_sva_prefetch = hpre_close_sva_prefetch, 1030 1030 + .show_last_dfx_regs = hpre_show_last_dfx_regs, 1170 1031 .err_info_init = hpre_err_info_init, 1171 1032 }; 1172 1033 ··· 1185 1044 qm->err_ini = &hpre_err_ini; 1186 1045 qm->err_ini->err_info_init(qm); 1187 1046 hisi_qm_dev_err_init(qm); 1047 1047 + ret = hpre_show_last_regs_init(qm); 1048 1048 + if (ret) 1049 1049 + pci_err(qm->pdev, "Failed to init last word regs!\n"); 1188 1050 1189 1189 - return 0; 1051 1051 + return ret; 1190 1052 } 1191 1053 1192 1054 static int hpre_probe_init(struct hpre *hpre) ··· 1275 1131 hisi_qm_stop(qm, QM_NORMAL); 1276 1132 1277 1133 err_with_err_init: 1134 1134 + hpre_show_last_regs_uninit(qm); 1278 1135 hisi_qm_dev_err_uninit(qm); 1279 1136 1280 1137 err_with_qm_init: ··· 1306 1161 if (qm->fun_type == QM_HW_PF) { 1307 1162 hpre_cnt_regs_clear(qm); 1308 1163 qm->debug.curr_qm_qp_num = 0; 1164 1164 + hpre_show_last_regs_uninit(qm); 1309 1165 hisi_qm_dev_err_uninit(qm); 1310 1166 } 1311 1167

+253 -29

drivers/crypto/hisilicon/qm.c

reviewed

··· 253 253 #define QM_QOS_MAX_CIR_U 6 254 254 #define QM_QOS_MAX_CIR_S 11 255 255 #define QM_QOS_VAL_MAX_LEN 32 256 256 - 256 256 + #define QM_DFX_BASE 0x0100000 257 257 + #define QM_DFX_STATE1 0x0104000 258 258 + #define QM_DFX_STATE2 0x01040C8 259 259 + #define QM_DFX_COMMON 0x0000 260 260 + #define QM_DFX_BASE_LEN 0x5A 261 261 + #define QM_DFX_STATE1_LEN 0x2E 262 262 + #define QM_DFX_STATE2_LEN 0x11 263 263 + #define QM_DFX_COMMON_LEN 0xC3 264 264 + #define QM_DFX_REGS_LEN 4UL 257 265 #define QM_AUTOSUSPEND_DELAY 3000 258 266 259 267 #define QM_MK_CQC_DW3_V1(hop_num, pg_sz, buf_sz, cqe_sz) \ ··· 473 465 { .int_msk = BIT(13), .msg = "qm_mailbox_timeout" }, 474 466 { .int_msk = BIT(14), .msg = "qm_flr_timeout" }, 475 467 { /* sentinel */ } 468 468 + }; 469 469 + 470 470 + /* define the QM's dfx regs region and region length */ 471 471 + static struct dfx_diff_registers qm_diff_regs[] = { 472 472 + { 473 473 + .reg_offset = QM_DFX_BASE, 474 474 + .reg_len = QM_DFX_BASE_LEN, 475 475 + }, { 476 476 + .reg_offset = QM_DFX_STATE1, 477 477 + .reg_len = QM_DFX_STATE1_LEN, 478 478 + }, { 479 479 + .reg_offset = QM_DFX_STATE2, 480 480 + .reg_len = QM_DFX_STATE2_LEN, 481 481 + }, { 482 482 + .reg_offset = QM_DFX_COMMON, 483 483 + .reg_len = QM_DFX_COMMON_LEN, 484 484 + }, 476 485 }; 477 486 478 487 static const char * const qm_db_timeout[] = { ··· 712 687 713 688 if (!IS_ENABLED(CONFIG_ARM64)) { 714 689 memcpy_toio(fun_base, src, 16); 715 715 - wmb(); 690 690 + dma_wmb(); 716 691 return; 717 692 } 718 693 719 694 asm volatile("ldp %0, %1, %3\n" 720 695 "stp %0, %1, %2\n" 721 721 - "dsb sy\n" 696 696 + "dmb oshst\n" 722 697 : "=&r" (tmp0), 723 698 "=&r" (tmp1), 724 699 "+Q" (*((char __iomem *)fun_base)) ··· 1007 982 *addr = 1; 1008 983 1009 984 /* make sure setup is completed */ 1010 1010 - mb(); 985 985 + smp_wmb(); 1011 986 } 1012 987 1013 988 static void qm_disable_qp(struct hisi_qm *qm, u32 qp_id) ··· 1649 1624 } 1650 1625 1651 1626 DEFINE_SHOW_ATTRIBUTE(qm_regs); 1627 1627 + 1628 1628 + static struct dfx_diff_registers *dfx_regs_init(struct hisi_qm *qm, 1629 1629 + const struct dfx_diff_registers *cregs, int reg_len) 1630 1630 + { 1631 1631 + struct dfx_diff_registers *diff_regs; 1632 1632 + u32 j, base_offset; 1633 1633 + int i; 1634 1634 + 1635 1635 + diff_regs = kcalloc(reg_len, sizeof(*diff_regs), GFP_KERNEL); 1636 1636 + if (!diff_regs) 1637 1637 + return ERR_PTR(-ENOMEM); 1638 1638 + 1639 1639 + for (i = 0; i < reg_len; i++) { 1640 1640 + if (!cregs[i].reg_len) 1641 1641 + continue; 1642 1642 + 1643 1643 + diff_regs[i].reg_offset = cregs[i].reg_offset; 1644 1644 + diff_regs[i].reg_len = cregs[i].reg_len; 1645 1645 + diff_regs[i].regs = kcalloc(QM_DFX_REGS_LEN, cregs[i].reg_len, 1646 1646 + GFP_KERNEL); 1647 1647 + if (!diff_regs[i].regs) 1648 1648 + goto alloc_error; 1649 1649 + 1650 1650 + for (j = 0; j < diff_regs[i].reg_len; j++) { 1651 1651 + base_offset = diff_regs[i].reg_offset + 1652 1652 + j * QM_DFX_REGS_LEN; 1653 1653 + diff_regs[i].regs[j] = readl(qm->io_base + base_offset); 1654 1654 + } 1655 1655 + } 1656 1656 + 1657 1657 + return diff_regs; 1658 1658 + 1659 1659 + alloc_error: 1660 1660 + while (i > 0) { 1661 1661 + i--; 1662 1662 + kfree(diff_regs[i].regs); 1663 1663 + } 1664 1664 + kfree(diff_regs); 1665 1665 + return ERR_PTR(-ENOMEM); 1666 1666 + } 1667 1667 + 1668 1668 + static void dfx_regs_uninit(struct hisi_qm *qm, 1669 1669 + struct dfx_diff_registers *dregs, int reg_len) 1670 1670 + { 1671 1671 + int i; 1672 1672 + 1673 1673 + /* Setting the pointer is NULL to prevent double free */ 1674 1674 + for (i = 0; i < reg_len; i++) { 1675 1675 + kfree(dregs[i].regs); 1676 1676 + dregs[i].regs = NULL; 1677 1677 + } 1678 1678 + kfree(dregs); 1679 1679 + dregs = NULL; 1680 1680 + } 1681 1681 + 1682 1682 + /** 1683 1683 + * hisi_qm_diff_regs_init() - Allocate memory for registers. 1684 1684 + * @qm: device qm handle. 1685 1685 + * @dregs: diff registers handle. 1686 1686 + * @reg_len: diff registers region length. 1687 1687 + */ 1688 1688 + int hisi_qm_diff_regs_init(struct hisi_qm *qm, 1689 1689 + struct dfx_diff_registers *dregs, int reg_len) 1690 1690 + { 1691 1691 + if (!qm || !dregs || reg_len <= 0) 1692 1692 + return -EINVAL; 1693 1693 + 1694 1694 + if (qm->fun_type != QM_HW_PF) 1695 1695 + return 0; 1696 1696 + 1697 1697 + qm->debug.qm_diff_regs = dfx_regs_init(qm, qm_diff_regs, 1698 1698 + ARRAY_SIZE(qm_diff_regs)); 1699 1699 + if (IS_ERR(qm->debug.qm_diff_regs)) 1700 1700 + return PTR_ERR(qm->debug.qm_diff_regs); 1701 1701 + 1702 1702 + qm->debug.acc_diff_regs = dfx_regs_init(qm, dregs, reg_len); 1703 1703 + if (IS_ERR(qm->debug.acc_diff_regs)) { 1704 1704 + dfx_regs_uninit(qm, qm->debug.qm_diff_regs, 1705 1705 + ARRAY_SIZE(qm_diff_regs)); 1706 1706 + return PTR_ERR(qm->debug.acc_diff_regs); 1707 1707 + } 1708 1708 + 1709 1709 + return 0; 1710 1710 + } 1711 1711 + EXPORT_SYMBOL_GPL(hisi_qm_diff_regs_init); 1712 1712 + 1713 1713 + /** 1714 1714 + * hisi_qm_diff_regs_uninit() - Free memory for registers. 1715 1715 + * @qm: device qm handle. 1716 1716 + * @reg_len: diff registers region length. 1717 1717 + */ 1718 1718 + void hisi_qm_diff_regs_uninit(struct hisi_qm *qm, int reg_len) 1719 1719 + { 1720 1720 + if (!qm || reg_len <= 0 || qm->fun_type != QM_HW_PF) 1721 1721 + return; 1722 1722 + 1723 1723 + dfx_regs_uninit(qm, qm->debug.acc_diff_regs, reg_len); 1724 1724 + dfx_regs_uninit(qm, qm->debug.qm_diff_regs, ARRAY_SIZE(qm_diff_regs)); 1725 1725 + } 1726 1726 + EXPORT_SYMBOL_GPL(hisi_qm_diff_regs_uninit); 1727 1727 + 1728 1728 + /** 1729 1729 + * hisi_qm_acc_diff_regs_dump() - Dump registers's value. 1730 1730 + * @qm: device qm handle. 1731 1731 + * @s: Debugfs file handle. 1732 1732 + * @dregs: diff registers handle. 1733 1733 + * @regs_len: diff registers region length. 1734 1734 + */ 1735 1735 + void hisi_qm_acc_diff_regs_dump(struct hisi_qm *qm, struct seq_file *s, 1736 1736 + struct dfx_diff_registers *dregs, int regs_len) 1737 1737 + { 1738 1738 + u32 j, val, base_offset; 1739 1739 + int i, ret; 1740 1740 + 1741 1741 + if (!qm || !s || !dregs || regs_len <= 0) 1742 1742 + return; 1743 1743 + 1744 1744 + ret = hisi_qm_get_dfx_access(qm); 1745 1745 + if (ret) 1746 1746 + return; 1747 1747 + 1748 1748 + down_read(&qm->qps_lock); 1749 1749 + for (i = 0; i < regs_len; i++) { 1750 1750 + if (!dregs[i].reg_len) 1751 1751 + continue; 1752 1752 + 1753 1753 + for (j = 0; j < dregs[i].reg_len; j++) { 1754 1754 + base_offset = dregs[i].reg_offset + j * QM_DFX_REGS_LEN; 1755 1755 + val = readl(qm->io_base + base_offset); 1756 1756 + if (val != dregs[i].regs[j]) 1757 1757 + seq_printf(s, "0x%08x = 0x%08x ---> 0x%08x\n", 1758 1758 + base_offset, dregs[i].regs[j], val); 1759 1759 + } 1760 1760 + } 1761 1761 + up_read(&qm->qps_lock); 1762 1762 + 1763 1763 + hisi_qm_put_dfx_access(qm); 1764 1764 + } 1765 1765 + EXPORT_SYMBOL_GPL(hisi_qm_acc_diff_regs_dump); 1766 1766 + 1767 1767 + static int qm_diff_regs_show(struct seq_file *s, void *unused) 1768 1768 + { 1769 1769 + struct hisi_qm *qm = s->private; 1770 1770 + 1771 1771 + hisi_qm_acc_diff_regs_dump(qm, s, qm->debug.qm_diff_regs, 1772 1772 + ARRAY_SIZE(qm_diff_regs)); 1773 1773 + 1774 1774 + return 0; 1775 1775 + } 1776 1776 + DEFINE_SHOW_ATTRIBUTE(qm_diff_regs); 1652 1777 1653 1778 static ssize_t qm_cmd_read(struct file *filp, char __user *buffer, 1654 1779 size_t count, loff_t *pos) ··· 2835 2660 * return created qp, -EBUSY if all qps in qm allocated, -ENOMEM if allocating 2836 2661 * qp memory fails. 2837 2662 */ 2838 2838 - struct hisi_qp *hisi_qm_create_qp(struct hisi_qm *qm, u8 alg_type) 2663 2663 + static struct hisi_qp *hisi_qm_create_qp(struct hisi_qm *qm, u8 alg_type) 2839 2664 { 2840 2665 struct hisi_qp *qp; 2841 2666 int ret; ··· 2853 2678 2854 2679 return qp; 2855 2680 } 2856 2856 - EXPORT_SYMBOL_GPL(hisi_qm_create_qp); 2857 2681 2858 2682 /** 2859 2683 * hisi_qm_release_qp() - Release a qp back to its qm. ··· 2860 2686 * 2861 2687 * This function releases the resource of a qp. 2862 2688 */ 2863 2863 - void hisi_qm_release_qp(struct hisi_qp *qp) 2689 2689 + static void hisi_qm_release_qp(struct hisi_qp *qp) 2864 2690 { 2865 2691 struct hisi_qm *qm = qp->qm; 2866 2692 ··· 2878 2704 2879 2705 qm_pm_put_sync(qm); 2880 2706 } 2881 2881 - EXPORT_SYMBOL_GPL(hisi_qm_release_qp); 2882 2707 2883 2708 static int qm_sq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid) 2884 2709 { ··· 3226 3053 wake_up_interruptible(&qp->uacce_q->wait); 3227 3054 } 3228 3055 3056 3056 + /* This function returns free number of qp in qm. */ 3229 3057 static int hisi_qm_get_available_instances(struct uacce_device *uacce) 3230 3058 { 3231 3231 - return hisi_qm_get_free_qp_num(uacce->priv); 3059 3059 + struct hisi_qm *qm = uacce->priv; 3060 3060 + int ret; 3061 3061 + 3062 3062 + down_read(&qm->qps_lock); 3063 3063 + ret = qm->qp_num - qm->qp_in_used; 3064 3064 + up_read(&qm->qps_lock); 3065 3065 + 3066 3066 + return ret; 3232 3067 } 3233 3068 3234 3069 static void hisi_qm_set_hw_reset(struct hisi_qm *qm, int offset) ··· 3548 3367 } 3549 3368 EXPORT_SYMBOL_GPL(hisi_qm_wait_task_finish); 3550 3369 3551 3551 - /** 3552 3552 - * hisi_qm_get_free_qp_num() - Get free number of qp in qm. 3553 3553 - * @qm: The qm which want to get free qp. 3554 3554 - * 3555 3555 - * This function return free number of qp in qm. 3556 3556 - */ 3557 3557 - int hisi_qm_get_free_qp_num(struct hisi_qm *qm) 3558 3558 - { 3559 3559 - int ret; 3560 3560 - 3561 3561 - down_read(&qm->qps_lock); 3562 3562 - ret = qm->qp_num - qm->qp_in_used; 3563 3563 - up_read(&qm->qps_lock); 3564 3564 - 3565 3565 - return ret; 3566 3566 - } 3567 3567 - EXPORT_SYMBOL_GPL(hisi_qm_get_free_qp_num); 3568 3568 - 3569 3370 static void hisi_qp_memory_uninit(struct hisi_qm *qm, int num) 3570 3371 { 3571 3372 struct device *dev = &qm->pdev->dev; ··· 3661 3498 writel(state, qm->io_base + QM_VF_STATE); 3662 3499 } 3663 3500 3501 3501 + static void qm_last_regs_uninit(struct hisi_qm *qm) 3502 3502 + { 3503 3503 + struct qm_debug *debug = &qm->debug; 3504 3504 + 3505 3505 + if (qm->fun_type == QM_HW_VF || !debug->qm_last_words) 3506 3506 + return; 3507 3507 + 3508 3508 + kfree(debug->qm_last_words); 3509 3509 + debug->qm_last_words = NULL; 3510 3510 + } 3511 3511 + 3664 3512 /** 3665 3513 * hisi_qm_uninit() - Uninitialize qm. 3666 3514 * @qm: The qm needed uninit. ··· 3682 3508 { 3683 3509 struct pci_dev *pdev = qm->pdev; 3684 3510 struct device *dev = &pdev->dev; 3511 3511 + 3512 3512 + qm_last_regs_uninit(qm); 3685 3513 3686 3514 qm_cmd_uninit(qm); 3687 3515 kfree(qm->factor); ··· 3726 3550 * 3727 3551 * qm hw v1 does not support this interface. 3728 3552 */ 3729 3729 - int hisi_qm_get_vft(struct hisi_qm *qm, u32 *base, u32 *number) 3553 3553 + static int hisi_qm_get_vft(struct hisi_qm *qm, u32 *base, u32 *number) 3730 3554 { 3731 3555 if (!base || !number) 3732 3556 return -EINVAL; ··· 3738 3562 3739 3563 return qm->ops->get_vft(qm, base, number); 3740 3564 } 3741 3741 - EXPORT_SYMBOL_GPL(hisi_qm_get_vft); 3742 3565 3743 3566 /** 3744 3567 * hisi_qm_set_vft() - Set vft to a qm. ··· 4659 4484 */ 4660 4485 void hisi_qm_debug_init(struct hisi_qm *qm) 4661 4486 { 4487 4487 + struct dfx_diff_registers *qm_regs = qm->debug.qm_diff_regs; 4662 4488 struct qm_dfx *dfx = &qm->debug.dfx; 4663 4489 struct dentry *qm_d; 4664 4490 void *data; ··· 4674 4498 for (i = CURRENT_Q; i < DEBUG_FILE_NUM; i++) 4675 4499 qm_create_debugfs_file(qm, qm->debug.qm_d, i); 4676 4500 } 4501 4501 + 4502 4502 + if (qm_regs) 4503 4503 + debugfs_create_file("diff_regs", 0444, qm->debug.qm_d, 4504 4504 + qm, &qm_diff_regs_fops); 4677 4505 4678 4506 debugfs_create_file("regs", 0444, qm->debug.qm_d, qm, &qm_regs_fops); 4679 4507 ··· 5361 5181 return 0; 5362 5182 } 5363 5183 5184 5184 + static void qm_show_last_dfx_regs(struct hisi_qm *qm) 5185 5185 + { 5186 5186 + struct qm_debug *debug = &qm->debug; 5187 5187 + struct pci_dev *pdev = qm->pdev; 5188 5188 + u32 val; 5189 5189 + int i; 5190 5190 + 5191 5191 + if (qm->fun_type == QM_HW_VF || !debug->qm_last_words) 5192 5192 + return; 5193 5193 + 5194 5194 + for (i = 0; i < ARRAY_SIZE(qm_dfx_regs); i++) { 5195 5195 + val = readl_relaxed(qm->io_base + qm_dfx_regs[i].offset); 5196 5196 + if (debug->qm_last_words[i] != val) 5197 5197 + pci_info(pdev, "%s \t= 0x%08x => 0x%08x\n", 5198 5198 + qm_dfx_regs[i].name, debug->qm_last_words[i], val); 5199 5199 + } 5200 5200 + } 5201 5201 + 5364 5202 static int qm_controller_reset(struct hisi_qm *qm) 5365 5203 { 5366 5204 struct pci_dev *pdev = qm->pdev; ··· 5393 5195 clear_bit(QM_RST_SCHED, &qm->misc_ctl); 5394 5196 return ret; 5395 5197 } 5198 5198 + 5199 5199 + qm_show_last_dfx_regs(qm); 5200 5200 + if (qm->err_ini->show_last_dfx_regs) 5201 5201 + qm->err_ini->show_last_dfx_regs(qm); 5396 5202 5397 5203 ret = qm_soft_reset(qm); 5398 5204 if (ret) { ··· 6108 5906 return ret; 6109 5907 } 6110 5908 5909 5909 + static void qm_last_regs_init(struct hisi_qm *qm) 5910 5910 + { 5911 5911 + int dfx_regs_num = ARRAY_SIZE(qm_dfx_regs); 5912 5912 + struct qm_debug *debug = &qm->debug; 5913 5913 + int i; 5914 5914 + 5915 5915 + if (qm->fun_type == QM_HW_VF) 5916 5916 + return; 5917 5917 + 5918 5918 + debug->qm_last_words = kcalloc(dfx_regs_num, sizeof(unsigned int), 5919 5919 + GFP_KERNEL); 5920 5920 + if (!debug->qm_last_words) 5921 5921 + return; 5922 5922 + 5923 5923 + for (i = 0; i < dfx_regs_num; i++) { 5924 5924 + debug->qm_last_words[i] = readl_relaxed(qm->io_base + 5925 5925 + qm_dfx_regs[i].offset); 5926 5926 + } 5927 5927 + } 5928 5928 + 6111 5929 /** 6112 5930 * hisi_qm_init() - Initialize configures about qm. 6113 5931 * @qm: The qm needing init. ··· 6179 5957 hisi_qm_init_work(qm); 6180 5958 qm_cmd_init(qm); 6181 5959 atomic_set(&qm->status.flags, QM_INIT); 5960 5960 + 5961 5961 + qm_last_regs_init(qm); 6182 5962 6183 5963 return 0; 6184 5964

-2

drivers/crypto/hisilicon/sec2/sec_crypto.c

reviewed

··· 2113 2113 .cra_driver_name = "hisi_sec_"sec_cra_name,\ 2114 2114 .cra_priority = SEC_PRIORITY,\ 2115 2115 .cra_flags = CRYPTO_ALG_ASYNC |\ 2116 2116 - CRYPTO_ALG_ALLOCATES_MEMORY |\ 2117 2116 CRYPTO_ALG_NEED_FALLBACK,\ 2118 2117 .cra_blocksize = blk_size,\ 2119 2118 .cra_ctxsize = sizeof(struct sec_ctx),\ ··· 2365 2366 .cra_driver_name = "hisi_sec_"sec_cra_name,\ 2366 2367 .cra_priority = SEC_PRIORITY,\ 2367 2368 .cra_flags = CRYPTO_ALG_ASYNC |\ 2368 2368 - CRYPTO_ALG_ALLOCATES_MEMORY |\ 2369 2369 CRYPTO_ALG_NEED_FALLBACK,\ 2370 2370 .cra_blocksize = blk_size,\ 2371 2371 .cra_ctxsize = sizeof(struct sec_ctx),\

+107 -1

drivers/crypto/hisilicon/sec2/sec_main.c

reviewed

··· 110 110 #define SEC_SQE_MASK_LEN 48 111 111 #define SEC_SHAPER_TYPE_RATE 400 112 112 113 113 + #define SEC_DFX_BASE 0x301000 114 114 + #define SEC_DFX_CORE 0x302100 115 115 + #define SEC_DFX_COMMON1 0x301600 116 116 + #define SEC_DFX_COMMON2 0x301C00 117 117 + #define SEC_DFX_BASE_LEN 0x9D 118 118 + #define SEC_DFX_CORE_LEN 0x32B 119 119 + #define SEC_DFX_COMMON1_LEN 0x45 120 120 + #define SEC_DFX_COMMON2_LEN 0xBA 121 121 + 113 122 struct sec_hw_error { 114 123 u32 int_msk; 115 124 const char *msg; ··· 234 225 {"SEC_BD_SAA7 ", 0x301C3C}, 235 226 {"SEC_BD_SAA8 ", 0x301C40}, 236 227 }; 228 228 + 229 229 + /* define the SEC's dfx regs region and region length */ 230 230 + static struct dfx_diff_registers sec_diff_regs[] = { 231 231 + { 232 232 + .reg_offset = SEC_DFX_BASE, 233 233 + .reg_len = SEC_DFX_BASE_LEN, 234 234 + }, { 235 235 + .reg_offset = SEC_DFX_COMMON1, 236 236 + .reg_len = SEC_DFX_COMMON1_LEN, 237 237 + }, { 238 238 + .reg_offset = SEC_DFX_COMMON2, 239 239 + .reg_len = SEC_DFX_COMMON2_LEN, 240 240 + }, { 241 241 + .reg_offset = SEC_DFX_CORE, 242 242 + .reg_len = SEC_DFX_CORE_LEN, 243 243 + }, 244 244 + }; 245 245 + 246 246 + static int sec_diff_regs_show(struct seq_file *s, void *unused) 247 247 + { 248 248 + struct hisi_qm *qm = s->private; 249 249 + 250 250 + hisi_qm_acc_diff_regs_dump(qm, s, qm->debug.acc_diff_regs, 251 251 + ARRAY_SIZE(sec_diff_regs)); 252 252 + 253 253 + return 0; 254 254 + } 255 255 + DEFINE_SHOW_ATTRIBUTE(sec_diff_regs); 237 256 238 257 static int sec_pf_q_num_set(const char *val, const struct kernel_param *kp) 239 258 { ··· 766 729 767 730 static int sec_core_debug_init(struct hisi_qm *qm) 768 731 { 732 732 + struct dfx_diff_registers *sec_regs = qm->debug.acc_diff_regs; 769 733 struct sec_dev *sec = container_of(qm, struct sec_dev, qm); 770 734 struct device *dev = &qm->pdev->dev; 771 735 struct sec_dfx *dfx = &sec->debug.dfx; ··· 787 749 788 750 if (qm->pdev->device == PCI_DEVICE_ID_HUAWEI_SEC_PF) 789 751 debugfs_create_file("regs", 0444, tmp_d, regset, &sec_regs_fops); 752 752 + if (qm->fun_type == QM_HW_PF && sec_regs) 753 753 + debugfs_create_file("diff_regs", 0444, tmp_d, 754 754 + qm, &sec_diff_regs_fops); 790 755 791 756 for (i = 0; i < ARRAY_SIZE(sec_dfx_labels); i++) { 792 757 atomic64_t *data = (atomic64_t *)((uintptr_t)dfx + ··· 831 790 sec_debugfs_root); 832 791 qm->debug.sqe_mask_offset = SEC_SQE_MASK_OFFSET; 833 792 qm->debug.sqe_mask_len = SEC_SQE_MASK_LEN; 793 793 + 794 794 + ret = hisi_qm_diff_regs_init(qm, sec_diff_regs, 795 795 + ARRAY_SIZE(sec_diff_regs)); 796 796 + if (ret) { 797 797 + dev_warn(dev, "Failed to init SEC diff regs!\n"); 798 798 + goto debugfs_remove; 799 799 + } 800 800 + 834 801 hisi_qm_debug_init(qm); 835 802 836 803 ret = sec_debug_init(qm); ··· 848 799 return 0; 849 800 850 801 failed_to_create: 802 802 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(sec_diff_regs)); 803 803 + debugfs_remove: 851 804 debugfs_remove_recursive(sec_debugfs_root); 852 805 return ret; 853 806 } 854 807 855 808 static void sec_debugfs_exit(struct hisi_qm *qm) 856 809 { 810 810 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(sec_diff_regs)); 811 811 + 857 812 debugfs_remove_recursive(qm->debug.debug_root); 813 813 + } 814 814 + 815 815 + static int sec_show_last_regs_init(struct hisi_qm *qm) 816 816 + { 817 817 + struct qm_debug *debug = &qm->debug; 818 818 + int i; 819 819 + 820 820 + debug->last_words = kcalloc(ARRAY_SIZE(sec_dfx_regs), 821 821 + sizeof(unsigned int), GFP_KERNEL); 822 822 + if (!debug->last_words) 823 823 + return -ENOMEM; 824 824 + 825 825 + for (i = 0; i < ARRAY_SIZE(sec_dfx_regs); i++) 826 826 + debug->last_words[i] = readl_relaxed(qm->io_base + 827 827 + sec_dfx_regs[i].offset); 828 828 + 829 829 + return 0; 830 830 + } 831 831 + 832 832 + static void sec_show_last_regs_uninit(struct hisi_qm *qm) 833 833 + { 834 834 + struct qm_debug *debug = &qm->debug; 835 835 + 836 836 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 837 837 + return; 838 838 + 839 839 + kfree(debug->last_words); 840 840 + debug->last_words = NULL; 841 841 + } 842 842 + 843 843 + static void sec_show_last_dfx_regs(struct hisi_qm *qm) 844 844 + { 845 845 + struct qm_debug *debug = &qm->debug; 846 846 + struct pci_dev *pdev = qm->pdev; 847 847 + u32 val; 848 848 + int i; 849 849 + 850 850 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 851 851 + return; 852 852 + 853 853 + /* dumps last word of the debugging registers during controller reset */ 854 854 + for (i = 0; i < ARRAY_SIZE(sec_dfx_regs); i++) { 855 855 + val = readl_relaxed(qm->io_base + sec_dfx_regs[i].offset); 856 856 + if (val != debug->last_words[i]) 857 857 + pci_info(pdev, "%s \t= 0x%08x => 0x%08x\n", 858 858 + sec_dfx_regs[i].name, debug->last_words[i], val); 859 859 + } 858 860 } 859 861 860 862 static void sec_log_hw_error(struct hisi_qm *qm, u32 err_sts) ··· 974 874 .open_axi_master_ooo = sec_open_axi_master_ooo, 975 875 .open_sva_prefetch = sec_open_sva_prefetch, 976 876 .close_sva_prefetch = sec_close_sva_prefetch, 877 877 + .show_last_dfx_regs = sec_show_last_dfx_regs, 977 878 .err_info_init = sec_err_info_init, 978 879 }; 979 880 ··· 993 892 sec_open_sva_prefetch(qm); 994 893 hisi_qm_dev_err_init(qm); 995 894 sec_debug_regs_clear(qm); 895 895 + ret = sec_show_last_regs_init(qm); 896 896 + if (ret) 897 897 + pci_err(qm->pdev, "Failed to init last word regs!\n"); 996 898 997 997 - return 0; 899 899 + return ret; 998 900 } 999 901 1000 902 static int sec_qm_init(struct hisi_qm *qm, struct pci_dev *pdev) ··· 1171 1067 sec_debugfs_exit(qm); 1172 1068 hisi_qm_stop(qm, QM_NORMAL); 1173 1069 err_probe_uninit: 1070 1070 + sec_show_last_regs_uninit(qm); 1174 1071 sec_probe_uninit(qm); 1175 1072 err_qm_uninit: 1176 1073 sec_qm_uninit(qm); ··· 1196 1091 1197 1092 if (qm->fun_type == QM_HW_PF) 1198 1093 sec_debug_regs_clear(qm); 1094 1094 + sec_show_last_regs_uninit(qm); 1199 1095 1200 1096 sec_probe_uninit(qm); 1201 1097

+4 -2

drivers/crypto/hisilicon/sgl.c

reviewed

··· 1 1 // SPDX-License-Identifier: GPL-2.0 2 2 /* Copyright (c) 2019 HiSilicon Limited. */ 3 3 + #include <linux/align.h> 3 4 #include <linux/dma-mapping.h> 4 5 #include <linux/hisi_acc_qm.h> 5 6 #include <linux/module.h> ··· 65 64 if (!dev || !count || !sge_nr || sge_nr > HISI_ACC_SGL_SGE_NR_MAX) 66 65 return ERR_PTR(-EINVAL); 67 66 68 68 - sgl_size = sizeof(struct acc_hw_sge) * sge_nr + 69 69 - sizeof(struct hisi_acc_hw_sgl); 67 67 + sgl_size = ALIGN(sizeof(struct acc_hw_sge) * sge_nr + 68 68 + sizeof(struct hisi_acc_hw_sgl), 69 69 + HISI_ACC_SGL_ALIGN_SIZE); 70 70 71 71 /* 72 72 * the pool may allocate a block of memory of size PAGE_SIZE * 2^(MAX_ORDER - 1),

+1 -1

drivers/crypto/hisilicon/zip/zip_crypto.c

reviewed

··· 521 521 static void hisi_zip_release_qp(struct hisi_zip_qp_ctx *ctx) 522 522 { 523 523 hisi_qm_stop_qp(ctx->qp); 524 524 - hisi_qm_release_qp(ctx->qp); 524 524 + hisi_qm_free_qps(&ctx->qp, 1); 525 525 } 526 526 527 527 static const struct hisi_zip_sqe_ops hisi_zip_ops_v1 = {

+176 -9

drivers/crypto/hisilicon/zip/zip_main.c

reviewed

··· 49 49 50 50 #define HZIP_QM_IDEL_STATUS 0x3040e4 51 51 52 52 - #define HZIP_CORE_DEBUG_COMP_0 0x302000 53 53 - #define HZIP_CORE_DEBUG_COMP_1 0x303000 54 54 - #define HZIP_CORE_DEBUG_DECOMP_0 0x304000 55 55 - #define HZIP_CORE_DEBUG_DECOMP_1 0x305000 56 56 - #define HZIP_CORE_DEBUG_DECOMP_2 0x306000 57 57 - #define HZIP_CORE_DEBUG_DECOMP_3 0x307000 58 58 - #define HZIP_CORE_DEBUG_DECOMP_4 0x308000 59 59 - #define HZIP_CORE_DEBUG_DECOMP_5 0x309000 52 52 + #define HZIP_CORE_DFX_BASE 0x301000 53 53 + #define HZIP_CLOCK_GATED_CONTL 0X301004 54 54 + #define HZIP_CORE_DFX_COMP_0 0x302000 55 55 + #define HZIP_CORE_DFX_COMP_1 0x303000 56 56 + #define HZIP_CORE_DFX_DECOMP_0 0x304000 57 57 + #define HZIP_CORE_DFX_DECOMP_1 0x305000 58 58 + #define HZIP_CORE_DFX_DECOMP_2 0x306000 59 59 + #define HZIP_CORE_DFX_DECOMP_3 0x307000 60 60 + #define HZIP_CORE_DFX_DECOMP_4 0x308000 61 61 + #define HZIP_CORE_DFX_DECOMP_5 0x309000 62 62 + #define HZIP_CORE_REGS_BASE_LEN 0xB0 63 63 + #define HZIP_CORE_REGS_DFX_LEN 0x28 60 64 61 65 #define HZIP_CORE_INT_SOURCE 0x3010A0 62 66 #define HZIP_CORE_INT_MASK_REG 0x3010A4 ··· 234 230 {"HZIP_DECOMP_LZ77_CURR_ST ", 0x9cull}, 235 231 }; 236 232 233 233 + static const struct debugfs_reg32 hzip_com_dfx_regs[] = { 234 234 + {"HZIP_CLOCK_GATE_CTRL ", 0x301004}, 235 235 + {"HZIP_CORE_INT_RAS_CE_ENB ", 0x301160}, 236 236 + {"HZIP_CORE_INT_RAS_NFE_ENB ", 0x301164}, 237 237 + {"HZIP_CORE_INT_RAS_FE_ENB ", 0x301168}, 238 238 + {"HZIP_UNCOM_ERR_RAS_CTRL ", 0x30116C}, 239 239 + }; 240 240 + 241 241 + static const struct debugfs_reg32 hzip_dump_dfx_regs[] = { 242 242 + {"HZIP_GET_BD_NUM ", 0x00ull}, 243 243 + {"HZIP_GET_RIGHT_BD ", 0x04ull}, 244 244 + {"HZIP_GET_ERROR_BD ", 0x08ull}, 245 245 + {"HZIP_DONE_BD_NUM ", 0x0cull}, 246 246 + {"HZIP_MAX_DELAY ", 0x20ull}, 247 247 + }; 248 248 + 249 249 + /* define the ZIP's dfx regs region and region length */ 250 250 + static struct dfx_diff_registers hzip_diff_regs[] = { 251 251 + { 252 252 + .reg_offset = HZIP_CORE_DFX_BASE, 253 253 + .reg_len = HZIP_CORE_REGS_BASE_LEN, 254 254 + }, { 255 255 + .reg_offset = HZIP_CORE_DFX_COMP_0, 256 256 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 257 257 + }, { 258 258 + .reg_offset = HZIP_CORE_DFX_COMP_1, 259 259 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 260 260 + }, { 261 261 + .reg_offset = HZIP_CORE_DFX_DECOMP_0, 262 262 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 263 263 + }, { 264 264 + .reg_offset = HZIP_CORE_DFX_DECOMP_1, 265 265 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 266 266 + }, { 267 267 + .reg_offset = HZIP_CORE_DFX_DECOMP_2, 268 268 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 269 269 + }, { 270 270 + .reg_offset = HZIP_CORE_DFX_DECOMP_3, 271 271 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 272 272 + }, { 273 273 + .reg_offset = HZIP_CORE_DFX_DECOMP_4, 274 274 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 275 275 + }, { 276 276 + .reg_offset = HZIP_CORE_DFX_DECOMP_5, 277 277 + .reg_len = HZIP_CORE_REGS_DFX_LEN, 278 278 + }, 279 279 + }; 280 280 + 281 281 + static int hzip_diff_regs_show(struct seq_file *s, void *unused) 282 282 + { 283 283 + struct hisi_qm *qm = s->private; 284 284 + 285 285 + hisi_qm_acc_diff_regs_dump(qm, s, qm->debug.acc_diff_regs, 286 286 + ARRAY_SIZE(hzip_diff_regs)); 287 287 + 288 288 + return 0; 289 289 + } 290 290 + DEFINE_SHOW_ATTRIBUTE(hzip_diff_regs); 237 291 static const struct kernel_param_ops zip_uacce_mode_ops = { 238 292 .set = uacce_mode_set, 239 293 .get = param_get_int, ··· 683 621 684 622 static void hisi_zip_dfx_debug_init(struct hisi_qm *qm) 685 623 { 624 624 + struct dfx_diff_registers *hzip_regs = qm->debug.acc_diff_regs; 686 625 struct hisi_zip *zip = container_of(qm, struct hisi_zip, qm); 687 626 struct hisi_zip_dfx *dfx = &zip->dfx; 688 627 struct dentry *tmp_dir; ··· 697 634 0644, tmp_dir, data, 698 635 &zip_atomic64_ops); 699 636 } 637 637 + 638 638 + if (qm->fun_type == QM_HW_PF && hzip_regs) 639 639 + debugfs_create_file("diff_regs", 0444, tmp_dir, 640 640 + qm, &hzip_diff_regs_fops); 700 641 } 701 642 702 643 static int hisi_zip_ctrl_debug_init(struct hisi_qm *qm) ··· 733 666 qm->debug.sqe_mask_offset = HZIP_SQE_MASK_OFFSET; 734 667 qm->debug.sqe_mask_len = HZIP_SQE_MASK_LEN; 735 668 qm->debug.debug_root = dev_d; 669 669 + ret = hisi_qm_diff_regs_init(qm, hzip_diff_regs, 670 670 + ARRAY_SIZE(hzip_diff_regs)); 671 671 + if (ret) { 672 672 + dev_warn(dev, "Failed to init ZIP diff regs!\n"); 673 673 + goto debugfs_remove; 674 674 + } 675 675 + 736 676 hisi_qm_debug_init(qm); 737 677 738 678 if (qm->fun_type == QM_HW_PF) { ··· 753 679 return 0; 754 680 755 681 failed_to_create: 682 682 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(hzip_diff_regs)); 683 683 + debugfs_remove: 756 684 debugfs_remove_recursive(hzip_debugfs_root); 757 685 return ret; 758 686 } ··· 779 703 780 704 static void hisi_zip_debugfs_exit(struct hisi_qm *qm) 781 705 { 706 706 + hisi_qm_diff_regs_uninit(qm, ARRAY_SIZE(hzip_diff_regs)); 707 707 + 782 708 debugfs_remove_recursive(qm->debug.debug_root); 783 709 784 710 if (qm->fun_type == QM_HW_PF) { 785 711 hisi_zip_debug_regs_clear(qm); 786 712 qm->debug.curr_qm_qp_num = 0; 713 713 + } 714 714 + } 715 715 + 716 716 + static int hisi_zip_show_last_regs_init(struct hisi_qm *qm) 717 717 + { 718 718 + int core_dfx_regs_num = ARRAY_SIZE(hzip_dump_dfx_regs); 719 719 + int com_dfx_regs_num = ARRAY_SIZE(hzip_com_dfx_regs); 720 720 + struct qm_debug *debug = &qm->debug; 721 721 + void __iomem *io_base; 722 722 + int i, j, idx; 723 723 + 724 724 + debug->last_words = kcalloc(core_dfx_regs_num * HZIP_CORE_NUM + 725 725 + com_dfx_regs_num, sizeof(unsigned int), GFP_KERNEL); 726 726 + if (!debug->last_words) 727 727 + return -ENOMEM; 728 728 + 729 729 + for (i = 0; i < com_dfx_regs_num; i++) { 730 730 + io_base = qm->io_base + hzip_com_dfx_regs[i].offset; 731 731 + debug->last_words[i] = readl_relaxed(io_base); 732 732 + } 733 733 + 734 734 + for (i = 0; i < HZIP_CORE_NUM; i++) { 735 735 + io_base = qm->io_base + core_offsets[i]; 736 736 + for (j = 0; j < core_dfx_regs_num; j++) { 737 737 + idx = com_dfx_regs_num + i * core_dfx_regs_num + j; 738 738 + debug->last_words[idx] = readl_relaxed( 739 739 + io_base + hzip_dump_dfx_regs[j].offset); 740 740 + } 741 741 + } 742 742 + 743 743 + return 0; 744 744 + } 745 745 + 746 746 + static void hisi_zip_show_last_regs_uninit(struct hisi_qm *qm) 747 747 + { 748 748 + struct qm_debug *debug = &qm->debug; 749 749 + 750 750 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 751 751 + return; 752 752 + 753 753 + kfree(debug->last_words); 754 754 + debug->last_words = NULL; 755 755 + } 756 756 + 757 757 + static void hisi_zip_show_last_dfx_regs(struct hisi_qm *qm) 758 758 + { 759 759 + int core_dfx_regs_num = ARRAY_SIZE(hzip_dump_dfx_regs); 760 760 + int com_dfx_regs_num = ARRAY_SIZE(hzip_com_dfx_regs); 761 761 + struct qm_debug *debug = &qm->debug; 762 762 + char buf[HZIP_BUF_SIZE]; 763 763 + void __iomem *base; 764 764 + int i, j, idx; 765 765 + u32 val; 766 766 + 767 767 + if (qm->fun_type == QM_HW_VF || !debug->last_words) 768 768 + return; 769 769 + 770 770 + for (i = 0; i < com_dfx_regs_num; i++) { 771 771 + val = readl_relaxed(qm->io_base + hzip_com_dfx_regs[i].offset); 772 772 + if (debug->last_words[i] != val) 773 773 + pci_info(qm->pdev, "com_dfx: %s \t= 0x%08x => 0x%08x\n", 774 774 + hzip_com_dfx_regs[i].name, debug->last_words[i], val); 775 775 + } 776 776 + 777 777 + for (i = 0; i < HZIP_CORE_NUM; i++) { 778 778 + if (i < HZIP_COMP_CORE_NUM) 779 779 + scnprintf(buf, sizeof(buf), "Comp_core-%d", i); 780 780 + else 781 781 + scnprintf(buf, sizeof(buf), "Decomp_core-%d", 782 782 + i - HZIP_COMP_CORE_NUM); 783 783 + base = qm->io_base + core_offsets[i]; 784 784 + 785 785 + pci_info(qm->pdev, "==>%s:\n", buf); 786 786 + /* dump last word for dfx regs during control resetting */ 787 787 + for (j = 0; j < core_dfx_regs_num; j++) { 788 788 + idx = com_dfx_regs_num + i * core_dfx_regs_num + j; 789 789 + val = readl_relaxed(base + hzip_dump_dfx_regs[j].offset); 790 790 + if (debug->last_words[idx] != val) 791 791 + pci_info(qm->pdev, "%s \t= 0x%08x => 0x%08x\n", 792 792 + hzip_dump_dfx_regs[j].name, debug->last_words[idx], val); 793 793 + } 787 794 } 788 795 } 789 796 ··· 957 798 .close_axi_master_ooo = hisi_zip_close_axi_master_ooo, 958 799 .open_sva_prefetch = hisi_zip_open_sva_prefetch, 959 800 .close_sva_prefetch = hisi_zip_close_sva_prefetch, 801 801 + .show_last_dfx_regs = hisi_zip_show_last_dfx_regs, 960 802 .err_info_init = hisi_zip_err_info_init, 961 803 }; 962 804 ··· 965 805 { 966 806 struct hisi_qm *qm = &hisi_zip->qm; 967 807 struct hisi_zip_ctrl *ctrl; 808 808 + int ret; 968 809 969 810 ctrl = devm_kzalloc(&qm->pdev->dev, sizeof(*ctrl), GFP_KERNEL); 970 811 if (!ctrl) ··· 981 820 hisi_qm_dev_err_init(qm); 982 821 hisi_zip_debug_regs_clear(qm); 983 822 984 984 - return 0; 823 823 + ret = hisi_zip_show_last_regs_init(qm); 824 824 + if (ret) 825 825 + pci_err(qm->pdev, "Failed to init last word regs!\n"); 826 826 + 827 827 + return ret; 985 828 } 986 829 987 830 static int hisi_zip_qm_init(struct hisi_qm *qm, struct pci_dev *pdev) ··· 1129 964 hisi_qm_stop(qm, QM_NORMAL); 1130 965 1131 966 err_dev_err_uninit: 967 967 + hisi_zip_show_last_regs_uninit(qm); 1132 968 hisi_qm_dev_err_uninit(qm); 1133 969 1134 970 err_qm_uninit: ··· 1151 985 1152 986 hisi_zip_debugfs_exit(qm); 1153 987 hisi_qm_stop(qm, QM_NORMAL); 988 988 + hisi_zip_show_last_regs_uninit(qm); 1154 989 hisi_qm_dev_err_uninit(qm); 1155 990 hisi_zip_qm_uninit(qm); 1156 991 }

+9

drivers/crypto/inside-secure/safexcel.c

reviewed

··· 1997 1997 MODULE_DESCRIPTION("Support for SafeXcel cryptographic engines: EIP97 & EIP197"); 1998 1998 MODULE_LICENSE("GPL v2"); 1999 1999 MODULE_IMPORT_NS(CRYPTO_INTERNAL); 2000 2000 + 2001 2001 + MODULE_FIRMWARE("ifpp.bin"); 2002 2002 + MODULE_FIRMWARE("ipue.bin"); 2003 2003 + MODULE_FIRMWARE("inside-secure/eip197b/ifpp.bin"); 2004 2004 + MODULE_FIRMWARE("inside-secure/eip197b/ipue.bin"); 2005 2005 + MODULE_FIRMWARE("inside-secure/eip197d/ifpp.bin"); 2006 2006 + MODULE_FIRMWARE("inside-secure/eip197d/ipue.bin"); 2007 2007 + MODULE_FIRMWARE("inside-secure/eip197_minifw/ifpp.bin"); 2008 2008 + MODULE_FIRMWARE("inside-secure/eip197_minifw/ipue.bin");

+1 -8

drivers/crypto/keembay/keembay-ocs-aes-core.c

reviewed

··· 1598 1598 { 1599 1599 struct device *dev = &pdev->dev; 1600 1600 struct ocs_aes_dev *aes_dev; 1601 1601 - struct resource *aes_mem; 1602 1601 int rc; 1603 1602 1604 1603 aes_dev = devm_kzalloc(dev, sizeof(*aes_dev), GFP_KERNEL); ··· 1615 1616 } 1616 1617 1617 1618 /* Get base register address. */ 1618 1618 - aes_mem = platform_get_resource(pdev, IORESOURCE_MEM, 0); 1619 1619 - if (!aes_mem) { 1620 1620 - dev_err(dev, "Could not retrieve io mem resource\n"); 1621 1621 - return -ENODEV; 1622 1622 - } 1623 1623 - 1624 1624 - aes_dev->base_reg = devm_ioremap_resource(&pdev->dev, aes_mem); 1619 1619 + aes_dev->base_reg = devm_platform_ioremap_resource(pdev, 0); 1625 1620 if (IS_ERR(aes_dev->base_reg)) 1626 1621 return PTR_ERR(aes_dev->base_reg); 1627 1622

-1

drivers/crypto/marvell/cesa/cipher.c

reviewed

··· 624 624 .decrypt = mv_cesa_ecb_des3_ede_decrypt, 625 625 .min_keysize = DES3_EDE_KEY_SIZE, 626 626 .max_keysize = DES3_EDE_KEY_SIZE, 627 627 - .ivsize = DES3_EDE_BLOCK_SIZE, 628 627 .base = { 629 628 .cra_name = "ecb(des3_ede)", 630 629 .cra_driver_name = "mv-ecb-des3-ede",

+1 -6

drivers/crypto/marvell/octeontx2/otx2_cptvf_algs.c

reviewed

··· 896 896 struct crypto_authenc_key_param *param; 897 897 int enckeylen = 0, authkeylen = 0; 898 898 struct rtattr *rta = (void *)key; 899 899 - int status; 900 899 901 900 if (!RTA_OK(rta, keylen)) 902 901 return -EINVAL; ··· 937 938 ctx->enc_key_len = enckeylen; 938 939 ctx->auth_key_len = authkeylen; 939 940 940 940 - status = aead_hmac_init(cipher); 941 941 - if (status) 942 942 - return status; 943 943 - 944 944 - return 0; 941 941 + return aead_hmac_init(cipher); 945 942 } 946 943 947 944 static int otx2_cpt_aead_ecb_null_sha_setkey(struct crypto_aead *cipher,

+1 -7

drivers/crypto/qat/qat_4xxx/adf_drv.c

reviewed

··· 14 14 15 15 static const struct pci_device_id adf_pci_tbl[] = { 16 16 { PCI_VDEVICE(INTEL, ADF_4XXX_PCI_DEVICE_ID), }, 17 17 + { PCI_VDEVICE(INTEL, ADF_401XX_PCI_DEVICE_ID), }, 17 18 { } 18 19 }; 19 20 MODULE_DEVICE_TABLE(pci, adf_pci_tbl); ··· 75 74 ret = adf_cfg_section_add(accel_dev, "Accelerator0"); 76 75 if (ret) 77 76 goto err; 78 78 - 79 79 - /* Temporarily set the number of crypto instances to zero to avoid 80 80 - * registering the crypto algorithms. 81 81 - * This will be removed when the algorithms will support the 82 82 - * CRYPTO_TFM_REQ_MAY_BACKLOG flag 83 83 - */ 84 84 - instances = 0; 85 77 86 78 for (i = 0; i < instances; i++) { 87 79 val = i;

+1 -14

drivers/crypto/qat/qat_c3xxx/adf_c3xxx_hw_data.c

reviewed

··· 78 78 return thrd_to_arb_map; 79 79 } 80 80 81 81 - static void adf_enable_ints(struct adf_accel_dev *accel_dev) 82 82 - { 83 83 - void __iomem *addr; 84 84 - 85 85 - addr = (&GET_BARS(accel_dev)[ADF_C3XXX_PMISC_BAR])->virt_addr; 86 86 - 87 87 - /* Enable bundle and misc interrupts */ 88 88 - ADF_CSR_WR(addr, ADF_C3XXX_SMIAPF0_MASK_OFFSET, 89 89 - ADF_C3XXX_SMIA0_MASK); 90 90 - ADF_CSR_WR(addr, ADF_C3XXX_SMIAPF1_MASK_OFFSET, 91 91 - ADF_C3XXX_SMIA1_MASK); 92 92 - } 93 93 - 94 81 static void configure_iov_threads(struct adf_accel_dev *accel_dev, bool enable) 95 82 { 96 83 adf_gen2_cfg_iov_thds(accel_dev, enable, ··· 120 133 hw_data->init_arb = adf_init_arb; 121 134 hw_data->exit_arb = adf_exit_arb; 122 135 hw_data->get_arb_mapping = adf_get_arbiter_mapping; 123 123 - hw_data->enable_ints = adf_enable_ints; 136 136 + hw_data->enable_ints = adf_gen2_enable_ints; 124 137 hw_data->reset_device = adf_reset_flr; 125 138 hw_data->set_ssm_wdtimer = adf_gen2_set_ssm_wdtimer; 126 139 hw_data->disable_iov = adf_disable_sriov;

-4

drivers/crypto/qat/qat_c3xxx/adf_c3xxx_hw_data.h

reviewed

··· 13 13 #define ADF_C3XXX_ACCELERATORS_MASK 0x7 14 14 #define ADF_C3XXX_ACCELENGINES_MASK 0x3F 15 15 #define ADF_C3XXX_ETR_MAX_BANKS 16 16 16 - #define ADF_C3XXX_SMIAPF0_MASK_OFFSET (0x3A000 + 0x28) 17 17 - #define ADF_C3XXX_SMIAPF1_MASK_OFFSET (0x3A000 + 0x30) 18 18 - #define ADF_C3XXX_SMIA0_MASK 0xFFFF 19 19 - #define ADF_C3XXX_SMIA1_MASK 0x1 20 16 #define ADF_C3XXX_SOFTSTRAP_CSR_OFFSET 0x2EC 21 17 22 18 /* AE to function mapping */

+1 -14

drivers/crypto/qat/qat_c62x/adf_c62x_hw_data.c

reviewed

··· 80 80 return thrd_to_arb_map; 81 81 } 82 82 83 83 - static void adf_enable_ints(struct adf_accel_dev *accel_dev) 84 84 - { 85 85 - void __iomem *addr; 86 86 - 87 87 - addr = (&GET_BARS(accel_dev)[ADF_C62X_PMISC_BAR])->virt_addr; 88 88 - 89 89 - /* Enable bundle and misc interrupts */ 90 90 - ADF_CSR_WR(addr, ADF_C62X_SMIAPF0_MASK_OFFSET, 91 91 - ADF_C62X_SMIA0_MASK); 92 92 - ADF_CSR_WR(addr, ADF_C62X_SMIAPF1_MASK_OFFSET, 93 93 - ADF_C62X_SMIA1_MASK); 94 94 - } 95 95 - 96 83 static void configure_iov_threads(struct adf_accel_dev *accel_dev, bool enable) 97 84 { 98 85 adf_gen2_cfg_iov_thds(accel_dev, enable, ··· 122 135 hw_data->init_arb = adf_init_arb; 123 136 hw_data->exit_arb = adf_exit_arb; 124 137 hw_data->get_arb_mapping = adf_get_arbiter_mapping; 125 125 - hw_data->enable_ints = adf_enable_ints; 138 138 + hw_data->enable_ints = adf_gen2_enable_ints; 126 139 hw_data->reset_device = adf_reset_flr; 127 140 hw_data->set_ssm_wdtimer = adf_gen2_set_ssm_wdtimer; 128 141 hw_data->disable_iov = adf_disable_sriov;

-4

drivers/crypto/qat/qat_c62x/adf_c62x_hw_data.h

reviewed

··· 13 13 #define ADF_C62X_ACCELERATORS_MASK 0x1F 14 14 #define ADF_C62X_ACCELENGINES_MASK 0x3FF 15 15 #define ADF_C62X_ETR_MAX_BANKS 16 16 16 - #define ADF_C62X_SMIAPF0_MASK_OFFSET (0x3A000 + 0x28) 17 17 - #define ADF_C62X_SMIAPF1_MASK_OFFSET (0x3A000 + 0x30) 18 18 - #define ADF_C62X_SMIA0_MASK 0xFFFF 19 19 - #define ADF_C62X_SMIA1_MASK 0x1 20 16 #define ADF_C62X_SOFTSTRAP_CSR_OFFSET 0x2EC 21 17 22 18 /* AE to function mapping */

+1

drivers/crypto/qat/qat_common/Makefile

reviewed

··· 16 16 qat_crypto.o \ 17 17 qat_algs.o \ 18 18 qat_asym_algs.o \ 19 19 + qat_algs_send.o \ 19 20 qat_uclo.o \ 20 21 qat_hal.o 21 22

+4 -2

drivers/crypto/qat/qat_common/adf_accel_devices.h

reviewed

··· 19 19 #define ADF_4XXX_DEVICE_NAME "4xxx" 20 20 #define ADF_4XXX_PCI_DEVICE_ID 0x4940 21 21 #define ADF_4XXXIOV_PCI_DEVICE_ID 0x4941 22 22 + #define ADF_401XX_PCI_DEVICE_ID 0x4942 23 23 + #define ADF_401XXIOV_PCI_DEVICE_ID 0x4943 22 24 #define ADF_DEVICE_FUSECTL_OFFSET 0x40 23 25 #define ADF_DEVICE_LEGFUSE_OFFSET 0x4C 24 26 #define ADF_DEVICE_FUSECTL_MASK 0x80000000 ··· 154 152 int (*enable_comms)(struct adf_accel_dev *accel_dev); 155 153 u32 (*get_pf2vf_offset)(u32 i); 156 154 u32 (*get_vf2pf_offset)(u32 i); 157 157 - u32 (*get_vf2pf_sources)(void __iomem *pmisc_addr); 158 155 void (*enable_vf2pf_interrupts)(void __iomem *pmisc_addr, u32 vf_mask); 159 159 - void (*disable_vf2pf_interrupts)(void __iomem *pmisc_addr, u32 vf_mask); 156 156 + void (*disable_all_vf2pf_interrupts)(void __iomem *pmisc_addr); 157 157 + u32 (*disable_pending_vf2pf_interrupts)(void __iomem *pmisc_addr); 160 158 int (*send_msg)(struct adf_accel_dev *accel_dev, struct pfvf_message msg, 161 159 u32 pfvf_offset, struct mutex *csr_lock); 162 160 struct pfvf_message (*recv_msg)(struct adf_accel_dev *accel_dev,

+2 -16

drivers/crypto/qat/qat_common/adf_common_drv.h

reviewed

··· 195 195 #if defined(CONFIG_PCI_IOV) 196 196 int adf_sriov_configure(struct pci_dev *pdev, int numvfs); 197 197 void adf_disable_sriov(struct adf_accel_dev *accel_dev); 198 198 - void adf_disable_vf2pf_interrupts(struct adf_accel_dev *accel_dev, 199 199 - u32 vf_mask); 200 200 - void adf_enable_vf2pf_interrupts(struct adf_accel_dev *accel_dev, 201 201 - u32 vf_mask); 198 198 + void adf_enable_vf2pf_interrupts(struct adf_accel_dev *accel_dev, u32 vf_mask); 199 199 + void adf_disable_all_vf2pf_interrupts(struct adf_accel_dev *accel_dev); 202 200 bool adf_recv_and_handle_pf2vf_msg(struct adf_accel_dev *accel_dev); 203 201 bool adf_recv_and_handle_vf2pf_msg(struct adf_accel_dev *accel_dev, u32 vf_nr); 204 202 int adf_pf2vf_handle_pf_restarting(struct adf_accel_dev *accel_dev); ··· 215 217 { 216 218 } 217 219 218 218 - static inline void adf_enable_pf2vf_interrupts(struct adf_accel_dev *accel_dev) 219 219 - { 220 220 - } 221 221 - 222 222 - static inline void adf_disable_pf2vf_interrupts(struct adf_accel_dev *accel_dev) 223 223 - { 224 224 - } 225 225 - 226 220 static inline int adf_init_pf_wq(void) 227 221 { 228 222 return 0; ··· 230 240 } 231 241 232 242 static inline void adf_exit_vf_wq(void) 233 233 - { 234 234 - } 235 235 - 236 236 - static inline void adf_flush_vf_wq(struct adf_accel_dev *accel_dev) 237 243 { 238 244 } 239 245

+13

drivers/crypto/qat/qat_common/adf_gen2_hw_data.c

reviewed

··· 98 98 } 99 99 EXPORT_SYMBOL_GPL(adf_gen2_get_arb_info); 100 100 101 101 + void adf_gen2_enable_ints(struct adf_accel_dev *accel_dev) 102 102 + { 103 103 + void __iomem *addr = adf_get_pmisc_base(accel_dev); 104 104 + u32 val; 105 105 + 106 106 + val = accel_dev->pf.vf_info ? 0 : BIT_ULL(GET_MAX_BANKS(accel_dev)) - 1; 107 107 + 108 108 + /* Enable bundle and misc interrupts */ 109 109 + ADF_CSR_WR(addr, ADF_GEN2_SMIAPF0_MASK_OFFSET, val); 110 110 + ADF_CSR_WR(addr, ADF_GEN2_SMIAPF1_MASK_OFFSET, ADF_GEN2_SMIA1_MASK); 111 111 + } 112 112 + EXPORT_SYMBOL_GPL(adf_gen2_enable_ints); 113 113 + 101 114 static u64 build_csr_ring_base_addr(dma_addr_t addr, u32 size) 102 115 { 103 116 return BUILD_RING_BASE_ADDR(addr, size);

+6

drivers/crypto/qat/qat_common/adf_gen2_hw_data.h

reviewed

··· 145 145 #define ADF_GEN2_CERRSSMSH(i) ((i) * 0x4000 + 0x10) 146 146 #define ADF_GEN2_ERRSSMSH_EN BIT(3) 147 147 148 148 + /* Interrupts */ 149 149 + #define ADF_GEN2_SMIAPF0_MASK_OFFSET (0x3A000 + 0x28) 150 150 + #define ADF_GEN2_SMIAPF1_MASK_OFFSET (0x3A000 + 0x30) 151 151 + #define ADF_GEN2_SMIA1_MASK 0x1 152 152 + 148 153 u32 adf_gen2_get_num_accels(struct adf_hw_device_data *self); 149 154 u32 adf_gen2_get_num_aes(struct adf_hw_device_data *self); 150 155 void adf_gen2_enable_error_correction(struct adf_accel_dev *accel_dev); ··· 158 153 void adf_gen2_init_hw_csr_ops(struct adf_hw_csr_ops *csr_ops); 159 154 void adf_gen2_get_admin_info(struct admin_info *admin_csrs_info); 160 155 void adf_gen2_get_arb_info(struct arb_info *arb_info); 156 156 + void adf_gen2_enable_ints(struct adf_accel_dev *accel_dev); 161 157 u32 adf_gen2_get_accel_cap(struct adf_accel_dev *accel_dev); 162 158 void adf_gen2_set_ssm_wdtimer(struct adf_accel_dev *accel_dev); 163 159

+48 -30

drivers/crypto/qat/qat_common/adf_gen2_pfvf.c

reviewed

··· 13 13 #include "adf_pfvf_utils.h" 14 14 15 15 /* VF2PF interrupts */ 16 16 + #define ADF_GEN2_VF_MSK 0xFFFF 16 17 #define ADF_GEN2_ERR_REG_VF2PF(vf_src) (((vf_src) & 0x01FFFE00) >> 9) 17 17 - #define ADF_GEN2_ERR_MSK_VF2PF(vf_mask) (((vf_mask) & 0xFFFF) << 9) 18 18 + #define ADF_GEN2_ERR_MSK_VF2PF(vf_mask) (((vf_mask) & ADF_GEN2_VF_MSK) << 9) 18 19 19 20 #define ADF_GEN2_PF_PF2VF_OFFSET(i) (0x3A000 + 0x280 + ((i) * 0x04)) 20 21 #define ADF_GEN2_VF_PF2VF_OFFSET 0x200 ··· 51 50 return ADF_GEN2_VF_PF2VF_OFFSET; 52 51 } 53 52 54 54 - static u32 adf_gen2_get_vf2pf_sources(void __iomem *pmisc_addr) 55 55 - { 56 56 - u32 errsou3, errmsk3, vf_int_mask; 57 57 - 58 58 - /* Get the interrupt sources triggered by VFs */ 59 59 - errsou3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRSOU3); 60 60 - vf_int_mask = ADF_GEN2_ERR_REG_VF2PF(errsou3); 61 61 - 62 62 - /* To avoid adding duplicate entries to work queue, clear 63 63 - * vf_int_mask_sets bits that are already masked in ERRMSK register. 64 64 - */ 65 65 - errmsk3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3); 66 66 - vf_int_mask &= ~ADF_GEN2_ERR_REG_VF2PF(errmsk3); 67 67 - 68 68 - return vf_int_mask; 69 69 - } 70 70 - 71 71 - static void adf_gen2_enable_vf2pf_interrupts(void __iomem *pmisc_addr, 72 72 - u32 vf_mask) 53 53 + static void adf_gen2_enable_vf2pf_interrupts(void __iomem *pmisc_addr, u32 vf_mask) 73 54 { 74 55 /* Enable VF2PF Messaging Ints - VFs 0 through 15 per vf_mask[15:0] */ 75 75 - if (vf_mask & 0xFFFF) { 56 56 + if (vf_mask & ADF_GEN2_VF_MSK) { 76 57 u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3) 77 58 & ~ADF_GEN2_ERR_MSK_VF2PF(vf_mask); 78 59 ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, val); 79 60 } 80 61 } 81 62 82 82 - static void adf_gen2_disable_vf2pf_interrupts(void __iomem *pmisc_addr, 83 83 - u32 vf_mask) 63 63 + static void adf_gen2_disable_all_vf2pf_interrupts(void __iomem *pmisc_addr) 84 64 { 85 65 /* Disable VF2PF interrupts for VFs 0 through 15 per vf_mask[15:0] */ 86 86 - if (vf_mask & 0xFFFF) { 87 87 - u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3) 88 88 - | ADF_GEN2_ERR_MSK_VF2PF(vf_mask); 89 89 - ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, val); 90 90 - } 66 66 + u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3) 67 67 + | ADF_GEN2_ERR_MSK_VF2PF(ADF_GEN2_VF_MSK); 68 68 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, val); 69 69 + } 70 70 + 71 71 + static u32 adf_gen2_disable_pending_vf2pf_interrupts(void __iomem *pmisc_addr) 72 72 + { 73 73 + u32 sources, disabled, pending; 74 74 + u32 errsou3, errmsk3; 75 75 + 76 76 + /* Get the interrupt sources triggered by VFs */ 77 77 + errsou3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRSOU3); 78 78 + sources = ADF_GEN2_ERR_REG_VF2PF(errsou3); 79 79 + 80 80 + if (!sources) 81 81 + return 0; 82 82 + 83 83 + /* Get the already disabled interrupts */ 84 84 + errmsk3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3); 85 85 + disabled = ADF_GEN2_ERR_REG_VF2PF(errmsk3); 86 86 + 87 87 + pending = sources & ~disabled; 88 88 + if (!pending) 89 89 + return 0; 90 90 + 91 91 + /* Due to HW limitations, when disabling the interrupts, we can't 92 92 + * just disable the requested sources, as this would lead to missed 93 93 + * interrupts if ERRSOU3 changes just before writing to ERRMSK3. 94 94 + * To work around it, disable all and re-enable only the sources that 95 95 + * are not in vf_mask and were not already disabled. Re-enabling will 96 96 + * trigger a new interrupt for the sources that have changed in the 97 97 + * meantime, if any. 98 98 + */ 99 99 + errmsk3 |= ADF_GEN2_ERR_MSK_VF2PF(ADF_GEN2_VF_MSK); 100 100 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, errmsk3); 101 101 + 102 102 + errmsk3 &= ADF_GEN2_ERR_MSK_VF2PF(sources | disabled); 103 103 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, errmsk3); 104 104 + 105 105 + /* Return the sources of the (new) interrupt(s) */ 106 106 + return pending; 91 107 } 92 108 93 109 static u32 gen2_csr_get_int_bit(enum gen2_csr_pos offset) ··· 380 362 pfvf_ops->enable_comms = adf_enable_pf2vf_comms; 381 363 pfvf_ops->get_pf2vf_offset = adf_gen2_pf_get_pfvf_offset; 382 364 pfvf_ops->get_vf2pf_offset = adf_gen2_pf_get_pfvf_offset; 383 383 - pfvf_ops->get_vf2pf_sources = adf_gen2_get_vf2pf_sources; 384 365 pfvf_ops->enable_vf2pf_interrupts = adf_gen2_enable_vf2pf_interrupts; 385 385 - pfvf_ops->disable_vf2pf_interrupts = adf_gen2_disable_vf2pf_interrupts; 366 366 + pfvf_ops->disable_all_vf2pf_interrupts = adf_gen2_disable_all_vf2pf_interrupts; 367 367 + pfvf_ops->disable_pending_vf2pf_interrupts = adf_gen2_disable_pending_vf2pf_interrupts; 386 368 pfvf_ops->send_msg = adf_gen2_pf2vf_send; 387 369 pfvf_ops->recv_msg = adf_gen2_vf2pf_recv; 388 370 }

+43 -20

drivers/crypto/qat/qat_common/adf_gen4_pfvf.c

reviewed

··· 15 15 /* VF2PF interrupt source registers */ 16 16 #define ADF_4XXX_VM2PF_SOU 0x41A180 17 17 #define ADF_4XXX_VM2PF_MSK 0x41A1C0 18 18 + #define ADF_GEN4_VF_MSK 0xFFFF 18 19 19 20 #define ADF_PFVF_GEN4_MSGTYPE_SHIFT 2 20 21 #define ADF_PFVF_GEN4_MSGTYPE_MASK 0x3F ··· 37 36 return ADF_4XXX_VM2PF_OFFSET(i); 38 37 } 39 38 40 40 - static u32 adf_gen4_get_vf2pf_sources(void __iomem *pmisc_addr) 39 39 + static void adf_gen4_enable_vf2pf_interrupts(void __iomem *pmisc_addr, u32 vf_mask) 41 40 { 42 42 - u32 sou, mask; 43 43 - 44 44 - sou = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_SOU); 45 45 - mask = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_MSK); 46 46 - 47 47 - return sou & ~mask; 48 48 - } 49 49 - 50 50 - static void adf_gen4_enable_vf2pf_interrupts(void __iomem *pmisc_addr, 51 51 - u32 vf_mask) 52 52 - { 53 53 - unsigned int val; 41 41 + u32 val; 54 42 55 43 val = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_MSK) & ~vf_mask; 56 44 ADF_CSR_WR(pmisc_addr, ADF_4XXX_VM2PF_MSK, val); 57 45 } 58 46 59 59 - static void adf_gen4_disable_vf2pf_interrupts(void __iomem *pmisc_addr, 60 60 - u32 vf_mask) 47 47 + static void adf_gen4_disable_all_vf2pf_interrupts(void __iomem *pmisc_addr) 61 48 { 62 62 - unsigned int val; 49 49 + ADF_CSR_WR(pmisc_addr, ADF_4XXX_VM2PF_MSK, ADF_GEN4_VF_MSK); 50 50 + } 63 51 64 64 - val = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_MSK) | vf_mask; 65 65 - ADF_CSR_WR(pmisc_addr, ADF_4XXX_VM2PF_MSK, val); 52 52 + static u32 adf_gen4_disable_pending_vf2pf_interrupts(void __iomem *pmisc_addr) 53 53 + { 54 54 + u32 sources, disabled, pending; 55 55 + 56 56 + /* Get the interrupt sources triggered by VFs */ 57 57 + sources = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_SOU); 58 58 + if (!sources) 59 59 + return 0; 60 60 + 61 61 + /* Get the already disabled interrupts */ 62 62 + disabled = ADF_CSR_RD(pmisc_addr, ADF_4XXX_VM2PF_MSK); 63 63 + 64 64 + pending = sources & ~disabled; 65 65 + if (!pending) 66 66 + return 0; 67 67 + 68 68 + /* Due to HW limitations, when disabling the interrupts, we can't 69 69 + * just disable the requested sources, as this would lead to missed 70 70 + * interrupts if VM2PF_SOU changes just before writing to VM2PF_MSK. 71 71 + * To work around it, disable all and re-enable only the sources that 72 72 + * are not in vf_mask and were not already disabled. Re-enabling will 73 73 + * trigger a new interrupt for the sources that have changed in the 74 74 + * meantime, if any. 75 75 + */ 76 76 + ADF_CSR_WR(pmisc_addr, ADF_4XXX_VM2PF_MSK, ADF_GEN4_VF_MSK); 77 77 + ADF_CSR_WR(pmisc_addr, ADF_4XXX_VM2PF_MSK, disabled | sources); 78 78 + 79 79 + /* Return the sources of the (new) interrupt(s) */ 80 80 + return pending; 66 81 } 67 82 68 83 static int adf_gen4_pfvf_send(struct adf_accel_dev *accel_dev, ··· 113 96 u32 pfvf_offset, u8 compat_ver) 114 97 { 115 98 void __iomem *pmisc_addr = adf_get_pmisc_base(accel_dev); 99 99 + struct pfvf_message msg = { 0 }; 116 100 u32 csr_val; 117 101 118 102 /* Read message from the CSR */ 119 103 csr_val = ADF_CSR_RD(pmisc_addr, pfvf_offset); 104 104 + if (!(csr_val & ADF_PFVF_INT)) { 105 105 + dev_info(&GET_DEV(accel_dev), 106 106 + "Spurious PFVF interrupt, msg 0x%.8x. Ignored\n", csr_val); 107 107 + return msg; 108 108 + } 120 109 121 110 /* We can now acknowledge the message reception by clearing the 122 111 * interrupt bit ··· 138 115 pfvf_ops->enable_comms = adf_enable_pf2vf_comms; 139 116 pfvf_ops->get_pf2vf_offset = adf_gen4_pf_get_pf2vf_offset; 140 117 pfvf_ops->get_vf2pf_offset = adf_gen4_pf_get_vf2pf_offset; 141 141 - pfvf_ops->get_vf2pf_sources = adf_gen4_get_vf2pf_sources; 142 118 pfvf_ops->enable_vf2pf_interrupts = adf_gen4_enable_vf2pf_interrupts; 143 143 - pfvf_ops->disable_vf2pf_interrupts = adf_gen4_disable_vf2pf_interrupts; 119 119 + pfvf_ops->disable_all_vf2pf_interrupts = adf_gen4_disable_all_vf2pf_interrupts; 120 120 + pfvf_ops->disable_pending_vf2pf_interrupts = adf_gen4_disable_pending_vf2pf_interrupts; 144 121 pfvf_ops->send_msg = adf_gen4_pfvf_send; 145 122 pfvf_ops->recv_msg = adf_gen4_pfvf_recv; 146 123 }

+9 -12

drivers/crypto/qat/qat_common/adf_isr.c

reviewed

··· 66 66 spin_unlock_irqrestore(&accel_dev->pf.vf2pf_ints_lock, flags); 67 67 } 68 68 69 69 - void adf_disable_vf2pf_interrupts(struct adf_accel_dev *accel_dev, u32 vf_mask) 69 69 + void adf_disable_all_vf2pf_interrupts(struct adf_accel_dev *accel_dev) 70 70 { 71 71 void __iomem *pmisc_addr = adf_get_pmisc_base(accel_dev); 72 72 unsigned long flags; 73 73 74 74 spin_lock_irqsave(&accel_dev->pf.vf2pf_ints_lock, flags); 75 75 - GET_PFVF_OPS(accel_dev)->disable_vf2pf_interrupts(pmisc_addr, vf_mask); 75 75 + GET_PFVF_OPS(accel_dev)->disable_all_vf2pf_interrupts(pmisc_addr); 76 76 spin_unlock_irqrestore(&accel_dev->pf.vf2pf_ints_lock, flags); 77 77 } 78 78 79 79 - static void adf_disable_vf2pf_interrupts_irq(struct adf_accel_dev *accel_dev, 80 80 - u32 vf_mask) 79 79 + static u32 adf_disable_pending_vf2pf_interrupts(struct adf_accel_dev *accel_dev) 81 80 { 82 81 void __iomem *pmisc_addr = adf_get_pmisc_base(accel_dev); 82 82 + u32 pending; 83 83 84 84 spin_lock(&accel_dev->pf.vf2pf_ints_lock); 85 85 - GET_PFVF_OPS(accel_dev)->disable_vf2pf_interrupts(pmisc_addr, vf_mask); 85 85 + pending = GET_PFVF_OPS(accel_dev)->disable_pending_vf2pf_interrupts(pmisc_addr); 86 86 spin_unlock(&accel_dev->pf.vf2pf_ints_lock); 87 87 + 88 88 + return pending; 87 89 } 88 90 89 91 static bool adf_handle_vf2pf_int(struct adf_accel_dev *accel_dev) 90 92 { 91 91 - void __iomem *pmisc_addr = adf_get_pmisc_base(accel_dev); 92 93 bool irq_handled = false; 93 94 unsigned long vf_mask; 94 95 95 95 - /* Get the interrupt sources triggered by VFs */ 96 96 - vf_mask = GET_PFVF_OPS(accel_dev)->get_vf2pf_sources(pmisc_addr); 97 97 - 96 96 + /* Get the interrupt sources triggered by VFs, except for those already disabled */ 97 97 + vf_mask = adf_disable_pending_vf2pf_interrupts(accel_dev); 98 98 if (vf_mask) { 99 99 struct adf_accel_vf_info *vf_info; 100 100 int i; 101 101 - 102 102 - /* Disable VF2PF interrupts for VFs with pending ints */ 103 103 - adf_disable_vf2pf_interrupts_irq(accel_dev, vf_mask); 104 101 105 102 /* 106 103 * Handle VF2PF interrupt unless the VF is malicious and

+2 -2

drivers/crypto/qat/qat_common/adf_pfvf_msg.h

reviewed

··· 8 8 /* 9 9 * PF<->VF Gen2 Messaging format 10 10 * 11 11 - * The PF has an array of 32-bit PF2VF registers, one for each VF. The 12 12 - * PF can access all these registers; each VF can access only the one 11 11 + * The PF has an array of 32-bit PF2VF registers, one for each VF. The 12 12 + * PF can access all these registers while each VF can access only the one 13 13 * register associated with that particular VF. 14 14 * 15 15 * The register functionally is split into two parts:

+4 -2

drivers/crypto/qat/qat_common/adf_pfvf_pf_proto.c

reviewed

··· 154 154 if (FIELD_GET(ADF_VF2PF_BLOCK_CRC_REQ_MASK, req.data)) { 155 155 dev_dbg(&GET_DEV(vf_info->accel_dev), 156 156 "BlockMsg of type %d for CRC over %d bytes received from VF%d\n", 157 157 - blk_type, blk_byte, vf_info->vf_nr); 157 157 + blk_type, blk_byte + 1, vf_info->vf_nr); 158 158 159 159 if (!adf_pf2vf_blkmsg_get_data(vf_info, blk_type, blk_byte, 160 160 byte_max, &resp_data, ··· 242 242 "VersionRequest received from VF%d (vers %d) to PF (vers %d)\n", 243 243 vf_nr, vf_compat_ver, ADF_PFVF_COMPAT_THIS_VERSION); 244 244 245 245 - if (vf_compat_ver <= ADF_PFVF_COMPAT_THIS_VERSION) 245 245 + if (vf_compat_ver == 0) 246 246 + compat = ADF_PF2VF_VF_INCOMPATIBLE; 247 247 + else if (vf_compat_ver <= ADF_PFVF_COMPAT_THIS_VERSION) 246 248 compat = ADF_PF2VF_VF_COMPATIBLE; 247 249 else 248 250 compat = ADF_PF2VF_VF_COMPAT_UNKNOWN;

+5 -11

drivers/crypto/qat/qat_common/adf_sriov.c

reviewed

··· 3 3 #include <linux/workqueue.h> 4 4 #include <linux/pci.h> 5 5 #include <linux/device.h> 6 6 - #include <linux/iommu.h> 7 6 #include "adf_common_drv.h" 8 7 #include "adf_cfg.h" 9 8 #include "adf_pfvf_pf_msg.h" ··· 73 74 hw_data->configure_iov_threads(accel_dev, true); 74 75 75 76 /* Enable VF to PF interrupts for all VFs */ 76 76 - if (hw_data->pfvf_ops.get_pf2vf_offset) 77 77 - adf_enable_vf2pf_interrupts(accel_dev, BIT_ULL(totalvfs) - 1); 77 77 + adf_enable_vf2pf_interrupts(accel_dev, BIT_ULL(totalvfs) - 1); 78 78 79 79 /* 80 80 * Due to the hardware design, when SR-IOV and the ring arbiter ··· 102 104 if (!accel_dev->pf.vf_info) 103 105 return; 104 106 105 105 - if (hw_data->pfvf_ops.get_pf2vf_offset) 106 106 - adf_pf2vf_notify_restarting(accel_dev); 107 107 - 107 107 + adf_pf2vf_notify_restarting(accel_dev); 108 108 pci_disable_sriov(accel_to_pci_dev(accel_dev)); 109 109 110 110 /* Disable VF to PF interrupts */ 111 111 - if (hw_data->pfvf_ops.get_pf2vf_offset) 112 112 - adf_disable_vf2pf_interrupts(accel_dev, GENMASK(31, 0)); 111 111 + adf_disable_all_vf2pf_interrupts(accel_dev); 113 112 114 113 /* Clear Valid bits in AE Thread to PCIe Function Mapping */ 115 114 if (hw_data->configure_iov_threads) 116 115 hw_data->configure_iov_threads(accel_dev, false); 117 116 118 118 - for (i = 0, vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) { 117 117 + for (i = 0, vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) 119 118 mutex_destroy(&vf->pf2vf_lock); 120 120 - } 121 119 122 120 kfree(accel_dev->pf.vf_info); 123 121 accel_dev->pf.vf_info = NULL; ··· 170 176 return -EFAULT; 171 177 } 172 178 173 173 - if (!iommu_present(&pci_bus_type)) 179 179 + if (!device_iommu_mapped(&pdev->dev)) 174 180 dev_warn(&pdev->dev, "IOMMU should be enabled for SR-IOV to work correctly\n"); 175 181 176 182 if (accel_dev->pf.vf_info) {

+11

drivers/crypto/qat/qat_common/adf_transport.c

reviewed

··· 8 8 #include "adf_cfg.h" 9 9 #include "adf_common_drv.h" 10 10 11 11 + #define ADF_MAX_RING_THRESHOLD 80 12 12 + #define ADF_PERCENT(tot, percent) (((tot) * (percent)) / 100) 13 13 + 11 14 static inline u32 adf_modulo(u32 data, u32 shift) 12 15 { 13 16 u32 div = data >> shift; ··· 78 75 spin_unlock_bh(&bank->lock); 79 76 csr_ops->write_csr_int_col_en(bank->csr_addr, bank->bank_number, 80 77 bank->irq_mask); 78 78 + } 79 79 + 80 80 + bool adf_ring_nearly_full(struct adf_etr_ring_data *ring) 81 81 + { 82 82 + return atomic_read(ring->inflights) > ring->threshold; 81 83 } 82 84 83 85 int adf_send_message(struct adf_etr_ring_data *ring, u32 *msg) ··· 225 217 struct adf_etr_bank_data *bank; 226 218 struct adf_etr_ring_data *ring; 227 219 char val[ADF_CFG_MAX_VAL_LEN_IN_BYTES]; 220 220 + int max_inflights; 228 221 u32 ring_num; 229 222 int ret; 230 223 ··· 272 263 ring->ring_size = adf_verify_ring_size(msg_size, num_msgs); 273 264 ring->head = 0; 274 265 ring->tail = 0; 266 266 + max_inflights = ADF_MAX_INFLIGHTS(ring->ring_size, ring->msg_size); 267 267 + ring->threshold = ADF_PERCENT(max_inflights, ADF_MAX_RING_THRESHOLD); 275 268 atomic_set(ring->inflights, 0); 276 269 ret = adf_init_ring(ring); 277 270 if (ret)

+1

drivers/crypto/qat/qat_common/adf_transport.h

reviewed

··· 14 14 const char *ring_name, adf_callback_fn callback, 15 15 int poll_mode, struct adf_etr_ring_data **ring_ptr); 16 16 17 17 + bool adf_ring_nearly_full(struct adf_etr_ring_data *ring); 17 18 int adf_send_message(struct adf_etr_ring_data *ring, u32 *msg); 18 19 void adf_remove_ring(struct adf_etr_ring_data *ring); 19 20 #endif

+1

drivers/crypto/qat/qat_common/adf_transport_internal.h

reviewed

··· 22 22 spinlock_t lock; /* protects ring data struct */ 23 23 u16 head; 24 24 u16 tail; 25 25 + u32 threshold; 25 26 u8 ring_number; 26 27 u8 ring_size; 27 28 u8 msg_size;

+1

drivers/crypto/qat/qat_common/adf_vf_isr.c

reviewed

··· 70 70 container_of(work, struct adf_vf_stop_data, work); 71 71 struct adf_accel_dev *accel_dev = stop_data->accel_dev; 72 72 73 73 + adf_dev_restarting_notify(accel_dev); 73 74 adf_dev_stop(accel_dev); 74 75 adf_dev_shutdown(accel_dev); 75 76

+85 -68

drivers/crypto/qat/qat_common/qat_algs.c

reviewed

··· 17 17 #include <crypto/xts.h> 18 18 #include <linux/dma-mapping.h> 19 19 #include "adf_accel_devices.h" 20 20 - #include "adf_transport.h" 20 20 + #include "qat_algs_send.h" 21 21 #include "adf_common_drv.h" 22 22 #include "qat_crypto.h" 23 23 #include "icp_qat_hw.h" ··· 45 45 46 46 static DEFINE_MUTEX(algs_lock); 47 47 static unsigned int active_devs; 48 48 - 49 49 - struct qat_alg_buf { 50 50 - u32 len; 51 51 - u32 resrvd; 52 52 - u64 addr; 53 53 - } __packed; 54 54 - 55 55 - struct qat_alg_buf_list { 56 56 - u64 resrvd; 57 57 - u32 num_bufs; 58 58 - u32 num_mapped_bufs; 59 59 - struct qat_alg_buf bufers[]; 60 60 - } __packed __aligned(64); 61 48 62 49 /* Common content descriptor */ 63 50 struct qat_alg_cd { ··· 680 693 bl->bufers[i].len, DMA_BIDIRECTIONAL); 681 694 682 695 dma_unmap_single(dev, blp, sz, DMA_TO_DEVICE); 683 683 - kfree(bl); 696 696 + 697 697 + if (!qat_req->buf.sgl_src_valid) 698 698 + kfree(bl); 699 699 + 684 700 if (blp != blpout) { 685 701 /* If out of place operation dma unmap only data */ 686 702 int bufless = blout->num_bufs - blout->num_mapped_bufs; ··· 694 704 DMA_BIDIRECTIONAL); 695 705 } 696 706 dma_unmap_single(dev, blpout, sz_out, DMA_TO_DEVICE); 697 697 - kfree(blout); 707 707 + 708 708 + if (!qat_req->buf.sgl_dst_valid) 709 709 + kfree(blout); 698 710 } 699 711 } 700 712 701 713 static int qat_alg_sgl_to_bufl(struct qat_crypto_instance *inst, 702 714 struct scatterlist *sgl, 703 715 struct scatterlist *sglout, 704 704 - struct qat_crypto_request *qat_req) 716 716 + struct qat_crypto_request *qat_req, 717 717 + gfp_t flags) 705 718 { 706 719 struct device *dev = &GET_DEV(inst->accel_dev); 707 720 int i, sg_nctr = 0; ··· 714 721 dma_addr_t blp = DMA_MAPPING_ERROR; 715 722 dma_addr_t bloutp = DMA_MAPPING_ERROR; 716 723 struct scatterlist *sg; 717 717 - size_t sz_out, sz = struct_size(bufl, bufers, n + 1); 724 724 + size_t sz_out, sz = struct_size(bufl, bufers, n); 725 725 + int node = dev_to_node(&GET_DEV(inst->accel_dev)); 718 726 719 727 if (unlikely(!n)) 720 728 return -EINVAL; 721 729 722 722 - bufl = kzalloc_node(sz, GFP_ATOMIC, 723 723 - dev_to_node(&GET_DEV(inst->accel_dev))); 724 724 - if (unlikely(!bufl)) 725 725 - return -ENOMEM; 730 730 + qat_req->buf.sgl_src_valid = false; 731 731 + qat_req->buf.sgl_dst_valid = false; 732 732 + 733 733 + if (n > QAT_MAX_BUFF_DESC) { 734 734 + bufl = kzalloc_node(sz, flags, node); 735 735 + if (unlikely(!bufl)) 736 736 + return -ENOMEM; 737 737 + } else { 738 738 + bufl = &qat_req->buf.sgl_src.sgl_hdr; 739 739 + memset(bufl, 0, sizeof(struct qat_alg_buf_list)); 740 740 + qat_req->buf.sgl_src_valid = true; 741 741 + } 726 742 727 743 for_each_sg(sgl, sg, n, i) 728 744 bufl->bufers[i].addr = DMA_MAPPING_ERROR; ··· 762 760 struct qat_alg_buf *bufers; 763 761 764 762 n = sg_nents(sglout); 765 765 - sz_out = struct_size(buflout, bufers, n + 1); 763 763 + sz_out = struct_size(buflout, bufers, n); 766 764 sg_nctr = 0; 767 767 - buflout = kzalloc_node(sz_out, GFP_ATOMIC, 768 768 - dev_to_node(&GET_DEV(inst->accel_dev))); 769 769 - if (unlikely(!buflout)) 770 770 - goto err_in; 765 765 + 766 766 + if (n > QAT_MAX_BUFF_DESC) { 767 767 + buflout = kzalloc_node(sz_out, flags, node); 768 768 + if (unlikely(!buflout)) 769 769 + goto err_in; 770 770 + } else { 771 771 + buflout = &qat_req->buf.sgl_dst.sgl_hdr; 772 772 + memset(buflout, 0, sizeof(struct qat_alg_buf_list)); 773 773 + qat_req->buf.sgl_dst_valid = true; 774 774 + } 771 775 772 776 bufers = buflout->bufers; 773 777 for_each_sg(sglout, sg, n, i) ··· 818 810 dma_unmap_single(dev, buflout->bufers[i].addr, 819 811 buflout->bufers[i].len, 820 812 DMA_BIDIRECTIONAL); 821 821 - kfree(buflout); 813 813 + 814 814 + if (!qat_req->buf.sgl_dst_valid) 815 815 + kfree(buflout); 822 816 823 817 err_in: 824 818 if (!dma_mapping_error(dev, blp)) ··· 833 823 bufl->bufers[i].len, 834 824 DMA_BIDIRECTIONAL); 835 825 836 836 - kfree(bufl); 826 826 + if (!qat_req->buf.sgl_src_valid) 827 827 + kfree(bufl); 837 828 838 829 dev_err(dev, "Failed to map buf for dma\n"); 839 830 return -ENOMEM; ··· 936 925 struct icp_qat_fw_la_resp *qat_resp = resp; 937 926 struct qat_crypto_request *qat_req = 938 927 (void *)(__force long)qat_resp->opaque_data; 928 928 + struct qat_instance_backlog *backlog = qat_req->alg_req.backlog; 939 929 940 930 qat_req->cb(qat_resp, qat_req); 931 931 + 932 932 + qat_alg_send_backlog(backlog); 933 933 + } 934 934 + 935 935 + static int qat_alg_send_sym_message(struct qat_crypto_request *qat_req, 936 936 + struct qat_crypto_instance *inst, 937 937 + struct crypto_async_request *base) 938 938 + { 939 939 + struct qat_alg_req *alg_req = &qat_req->alg_req; 940 940 + 941 941 + alg_req->fw_req = (u32 *)&qat_req->req; 942 942 + alg_req->tx_ring = inst->sym_tx; 943 943 + alg_req->base = base; 944 944 + alg_req->backlog = &inst->backlog; 945 945 + 946 946 + return qat_alg_send_message(alg_req); 941 947 } 942 948 943 949 static int qat_alg_aead_dec(struct aead_request *areq) ··· 967 939 struct icp_qat_fw_la_auth_req_params *auth_param; 968 940 struct icp_qat_fw_la_bulk_req *msg; 969 941 int digst_size = crypto_aead_authsize(aead_tfm); 970 970 - int ret, ctr = 0; 942 942 + gfp_t f = qat_algs_alloc_flags(&areq->base); 943 943 + int ret; 971 944 u32 cipher_len; 972 945 973 946 cipher_len = areq->cryptlen - digst_size; 974 947 if (cipher_len % AES_BLOCK_SIZE != 0) 975 948 return -EINVAL; 976 949 977 977 - ret = qat_alg_sgl_to_bufl(ctx->inst, areq->src, areq->dst, qat_req); 950 950 + ret = qat_alg_sgl_to_bufl(ctx->inst, areq->src, areq->dst, qat_req, f); 978 951 if (unlikely(ret)) 979 952 return ret; 980 953 ··· 994 965 auth_param = (void *)((u8 *)cipher_param + sizeof(*cipher_param)); 995 966 auth_param->auth_off = 0; 996 967 auth_param->auth_len = areq->assoclen + cipher_param->cipher_length; 997 997 - do { 998 998 - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); 999 999 - } while (ret == -EAGAIN && ctr++ < 10); 1000 968 1001 1001 - if (ret == -EAGAIN) { 969 969 + ret = qat_alg_send_sym_message(qat_req, ctx->inst, &areq->base); 970 970 + if (ret == -ENOSPC) 1002 971 qat_alg_free_bufl(ctx->inst, qat_req); 1003 1003 - return -EBUSY; 1004 1004 - } 1005 1005 - return -EINPROGRESS; 972 972 + 973 973 + return ret; 1006 974 } 1007 975 1008 976 static int qat_alg_aead_enc(struct aead_request *areq) ··· 1010 984 struct qat_crypto_request *qat_req = aead_request_ctx(areq); 1011 985 struct icp_qat_fw_la_cipher_req_params *cipher_param; 1012 986 struct icp_qat_fw_la_auth_req_params *auth_param; 987 987 + gfp_t f = qat_algs_alloc_flags(&areq->base); 1013 988 struct icp_qat_fw_la_bulk_req *msg; 1014 989 u8 *iv = areq->iv; 1015 1015 - int ret, ctr = 0; 990 990 + int ret; 1016 991 1017 992 if (areq->cryptlen % AES_BLOCK_SIZE != 0) 1018 993 return -EINVAL; 1019 994 1020 1020 - ret = qat_alg_sgl_to_bufl(ctx->inst, areq->src, areq->dst, qat_req); 995 995 + ret = qat_alg_sgl_to_bufl(ctx->inst, areq->src, areq->dst, qat_req, f); 1021 996 if (unlikely(ret)) 1022 997 return ret; 1023 998 ··· 1040 1013 auth_param->auth_off = 0; 1041 1014 auth_param->auth_len = areq->assoclen + areq->cryptlen; 1042 1015 1043 1043 - do { 1044 1044 - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); 1045 1045 - } while (ret == -EAGAIN && ctr++ < 10); 1046 1046 - 1047 1047 - if (ret == -EAGAIN) { 1016 1016 + ret = qat_alg_send_sym_message(qat_req, ctx->inst, &areq->base); 1017 1017 + if (ret == -ENOSPC) 1048 1018 qat_alg_free_bufl(ctx->inst, qat_req); 1049 1049 - return -EBUSY; 1050 1050 - } 1051 1051 - return -EINPROGRESS; 1019 1019 + 1020 1020 + return ret; 1052 1021 } 1053 1022 1054 1023 static int qat_alg_skcipher_rekey(struct qat_alg_skcipher_ctx *ctx, ··· 1196 1173 struct qat_alg_skcipher_ctx *ctx = crypto_tfm_ctx(tfm); 1197 1174 struct qat_crypto_request *qat_req = skcipher_request_ctx(req); 1198 1175 struct icp_qat_fw_la_cipher_req_params *cipher_param; 1176 1176 + gfp_t f = qat_algs_alloc_flags(&req->base); 1199 1177 struct icp_qat_fw_la_bulk_req *msg; 1200 1200 - int ret, ctr = 0; 1178 1178 + int ret; 1201 1179 1202 1180 if (req->cryptlen == 0) 1203 1181 return 0; 1204 1182 1205 1205 - ret = qat_alg_sgl_to_bufl(ctx->inst, req->src, req->dst, qat_req); 1183 1183 + ret = qat_alg_sgl_to_bufl(ctx->inst, req->src, req->dst, qat_req, f); 1206 1184 if (unlikely(ret)) 1207 1185 return ret; 1208 1186 ··· 1222 1198 1223 1199 qat_alg_set_req_iv(qat_req); 1224 1200 1225 1225 - do { 1226 1226 - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); 1227 1227 - } while (ret == -EAGAIN && ctr++ < 10); 1228 1228 - 1229 1229 - if (ret == -EAGAIN) { 1201 1201 + ret = qat_alg_send_sym_message(qat_req, ctx->inst, &req->base); 1202 1202 + if (ret == -ENOSPC) 1230 1203 qat_alg_free_bufl(ctx->inst, qat_req); 1231 1231 - return -EBUSY; 1232 1232 - } 1233 1233 - return -EINPROGRESS; 1204 1204 + 1205 1205 + return ret; 1234 1206 } 1235 1207 1236 1208 static int qat_alg_skcipher_blk_encrypt(struct skcipher_request *req) ··· 1262 1242 struct qat_alg_skcipher_ctx *ctx = crypto_tfm_ctx(tfm); 1263 1243 struct qat_crypto_request *qat_req = skcipher_request_ctx(req); 1264 1244 struct icp_qat_fw_la_cipher_req_params *cipher_param; 1245 1245 + gfp_t f = qat_algs_alloc_flags(&req->base); 1265 1246 struct icp_qat_fw_la_bulk_req *msg; 1266 1266 - int ret, ctr = 0; 1247 1247 + int ret; 1267 1248 1268 1249 if (req->cryptlen == 0) 1269 1250 return 0; 1270 1251 1271 1271 - ret = qat_alg_sgl_to_bufl(ctx->inst, req->src, req->dst, qat_req); 1252 1252 + ret = qat_alg_sgl_to_bufl(ctx->inst, req->src, req->dst, qat_req, f); 1272 1253 if (unlikely(ret)) 1273 1254 return ret; 1274 1255 ··· 1289 1268 qat_alg_set_req_iv(qat_req); 1290 1269 qat_alg_update_iv(qat_req); 1291 1270 1292 1292 - do { 1293 1293 - ret = adf_send_message(ctx->inst->sym_tx, (u32 *)msg); 1294 1294 - } while (ret == -EAGAIN && ctr++ < 10); 1295 1295 - 1296 1296 - if (ret == -EAGAIN) { 1271 1271 + ret = qat_alg_send_sym_message(qat_req, ctx->inst, &req->base); 1272 1272 + if (ret == -ENOSPC) 1297 1273 qat_alg_free_bufl(ctx->inst, qat_req); 1298 1298 - return -EBUSY; 1299 1299 - } 1300 1300 - return -EINPROGRESS; 1274 1274 + 1275 1275 + return ret; 1301 1276 } 1302 1277 1303 1278 static int qat_alg_skcipher_blk_decrypt(struct skcipher_request *req)

+86

drivers/crypto/qat/qat_common/qat_algs_send.c

reviewed

··· 1 1 + // SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0-only) 2 2 + /* Copyright(c) 2022 Intel Corporation */ 3 3 + #include "adf_transport.h" 4 4 + #include "qat_algs_send.h" 5 5 + #include "qat_crypto.h" 6 6 + 7 7 + #define ADF_MAX_RETRIES 20 8 8 + 9 9 + static int qat_alg_send_message_retry(struct qat_alg_req *req) 10 10 + { 11 11 + int ret = 0, ctr = 0; 12 12 + 13 13 + do { 14 14 + ret = adf_send_message(req->tx_ring, req->fw_req); 15 15 + } while (ret == -EAGAIN && ctr++ < ADF_MAX_RETRIES); 16 16 + 17 17 + if (ret == -EAGAIN) 18 18 + return -ENOSPC; 19 19 + 20 20 + return -EINPROGRESS; 21 21 + } 22 22 + 23 23 + void qat_alg_send_backlog(struct qat_instance_backlog *backlog) 24 24 + { 25 25 + struct qat_alg_req *req, *tmp; 26 26 + 27 27 + spin_lock_bh(&backlog->lock); 28 28 + list_for_each_entry_safe(req, tmp, &backlog->list, list) { 29 29 + if (adf_send_message(req->tx_ring, req->fw_req)) { 30 30 + /* The HW ring is full. Do nothing. 31 31 + * qat_alg_send_backlog() will be invoked again by 32 32 + * another callback. 33 33 + */ 34 34 + break; 35 35 + } 36 36 + list_del(&req->list); 37 37 + req->base->complete(req->base, -EINPROGRESS); 38 38 + } 39 39 + spin_unlock_bh(&backlog->lock); 40 40 + } 41 41 + 42 42 + static void qat_alg_backlog_req(struct qat_alg_req *req, 43 43 + struct qat_instance_backlog *backlog) 44 44 + { 45 45 + INIT_LIST_HEAD(&req->list); 46 46 + 47 47 + spin_lock_bh(&backlog->lock); 48 48 + list_add_tail(&req->list, &backlog->list); 49 49 + spin_unlock_bh(&backlog->lock); 50 50 + } 51 51 + 52 52 + static int qat_alg_send_message_maybacklog(struct qat_alg_req *req) 53 53 + { 54 54 + struct qat_instance_backlog *backlog = req->backlog; 55 55 + struct adf_etr_ring_data *tx_ring = req->tx_ring; 56 56 + u32 *fw_req = req->fw_req; 57 57 + 58 58 + /* If any request is already backlogged, then add to backlog list */ 59 59 + if (!list_empty(&backlog->list)) 60 60 + goto enqueue; 61 61 + 62 62 + /* If ring is nearly full, then add to backlog list */ 63 63 + if (adf_ring_nearly_full(tx_ring)) 64 64 + goto enqueue; 65 65 + 66 66 + /* If adding request to HW ring fails, then add to backlog list */ 67 67 + if (adf_send_message(tx_ring, fw_req)) 68 68 + goto enqueue; 69 69 + 70 70 + return -EINPROGRESS; 71 71 + 72 72 + enqueue: 73 73 + qat_alg_backlog_req(req, backlog); 74 74 + 75 75 + return -EBUSY; 76 76 + } 77 77 + 78 78 + int qat_alg_send_message(struct qat_alg_req *req) 79 79 + { 80 80 + u32 flags = req->base->flags; 81 81 + 82 82 + if (flags & CRYPTO_TFM_REQ_MAY_BACKLOG) 83 83 + return qat_alg_send_message_maybacklog(req); 84 84 + else 85 85 + return qat_alg_send_message_retry(req); 86 86 + }

+11

drivers/crypto/qat/qat_common/qat_algs_send.h

reviewed

··· 1 1 + /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0-only) */ 2 2 + /* Copyright(c) 2022 Intel Corporation */ 3 3 + #ifndef QAT_ALGS_SEND_H 4 4 + #define QAT_ALGS_SEND_H 5 5 + 6 6 + #include "qat_crypto.h" 7 7 + 8 8 + int qat_alg_send_message(struct qat_alg_req *req); 9 9 + void qat_alg_send_backlog(struct qat_instance_backlog *backlog); 10 10 + 11 11 + #endif

+152 -155

drivers/crypto/qat/qat_common/qat_asym_algs.c

reviewed

··· 12 12 #include <crypto/scatterwalk.h> 13 13 #include "icp_qat_fw_pke.h" 14 14 #include "adf_accel_devices.h" 15 15 + #include "qat_algs_send.h" 15 16 #include "adf_transport.h" 16 17 #include "adf_common_drv.h" 17 18 #include "qat_crypto.h" ··· 136 135 } areq; 137 136 int err; 138 137 void (*cb)(struct icp_qat_fw_pke_resp *resp); 138 138 + struct qat_alg_req alg_req; 139 139 } __aligned(64); 140 140 + 141 141 + static int qat_alg_send_asym_message(struct qat_asym_request *qat_req, 142 142 + struct qat_crypto_instance *inst, 143 143 + struct crypto_async_request *base) 144 144 + { 145 145 + struct qat_alg_req *alg_req = &qat_req->alg_req; 146 146 + 147 147 + alg_req->fw_req = (u32 *)&qat_req->req; 148 148 + alg_req->tx_ring = inst->pke_tx; 149 149 + alg_req->base = base; 150 150 + alg_req->backlog = &inst->backlog; 151 151 + 152 152 + return qat_alg_send_message(alg_req); 153 153 + } 140 154 141 155 static void qat_dh_cb(struct icp_qat_fw_pke_resp *resp) 142 156 { ··· 164 148 err = (err == ICP_QAT_FW_COMN_STATUS_FLAG_OK) ? 0 : -EINVAL; 165 149 166 150 if (areq->src) { 167 167 - if (req->src_align) 168 168 - dma_free_coherent(dev, req->ctx.dh->p_size, 169 169 - req->src_align, req->in.dh.in.b); 170 170 - else 171 171 - dma_unmap_single(dev, req->in.dh.in.b, 172 172 - req->ctx.dh->p_size, DMA_TO_DEVICE); 151 151 + dma_unmap_single(dev, req->in.dh.in.b, req->ctx.dh->p_size, 152 152 + DMA_TO_DEVICE); 153 153 + kfree_sensitive(req->src_align); 173 154 } 174 155 175 156 areq->dst_len = req->ctx.dh->p_size; 176 157 if (req->dst_align) { 177 158 scatterwalk_map_and_copy(req->dst_align, areq->dst, 0, 178 159 areq->dst_len, 1); 179 179 - 180 180 - dma_free_coherent(dev, req->ctx.dh->p_size, req->dst_align, 181 181 - req->out.dh.r); 182 182 - } else { 183 183 - dma_unmap_single(dev, req->out.dh.r, req->ctx.dh->p_size, 184 184 - DMA_FROM_DEVICE); 160 160 + kfree_sensitive(req->dst_align); 185 161 } 162 162 + 163 163 + dma_unmap_single(dev, req->out.dh.r, req->ctx.dh->p_size, 164 164 + DMA_FROM_DEVICE); 186 165 187 166 dma_unmap_single(dev, req->phy_in, sizeof(struct qat_dh_input_params), 188 167 DMA_TO_DEVICE); ··· 224 213 struct qat_asym_request *qat_req = 225 214 PTR_ALIGN(kpp_request_ctx(req), 64); 226 215 struct icp_qat_fw_pke_request *msg = &qat_req->req; 227 227 - int ret, ctr = 0; 216 216 + gfp_t flags = qat_algs_alloc_flags(&req->base); 228 217 int n_input_params = 0; 218 218 + u8 *vaddr; 219 219 + int ret; 229 220 230 221 if (unlikely(!ctx->xa)) 231 222 return -EINVAL; ··· 236 223 req->dst_len = ctx->p_size; 237 224 return -EOVERFLOW; 238 225 } 226 226 + 227 227 + if (req->src_len > ctx->p_size) 228 228 + return -EINVAL; 229 229 + 239 230 memset(msg, '\0', sizeof(*msg)); 240 231 ICP_QAT_FW_PKE_HDR_VALID_FLAG_SET(msg->pke_hdr, 241 232 ICP_QAT_FW_COMN_REQ_FLAG_SET); ··· 288 271 */ 289 272 if (sg_is_last(req->src) && req->src_len == ctx->p_size) { 290 273 qat_req->src_align = NULL; 291 291 - qat_req->in.dh.in.b = dma_map_single(dev, 292 292 - sg_virt(req->src), 293 293 - req->src_len, 294 294 - DMA_TO_DEVICE); 295 295 - if (unlikely(dma_mapping_error(dev, 296 296 - qat_req->in.dh.in.b))) 297 297 - return ret; 298 298 - 274 274 + vaddr = sg_virt(req->src); 299 275 } else { 300 276 int shift = ctx->p_size - req->src_len; 301 277 302 302 - qat_req->src_align = dma_alloc_coherent(dev, 303 303 - ctx->p_size, 304 304 - &qat_req->in.dh.in.b, 305 305 - GFP_KERNEL); 278 278 + qat_req->src_align = kzalloc(ctx->p_size, flags); 306 279 if (unlikely(!qat_req->src_align)) 307 280 return ret; 308 281 309 282 scatterwalk_map_and_copy(qat_req->src_align + shift, 310 283 req->src, 0, req->src_len, 0); 284 284 + 285 285 + vaddr = qat_req->src_align; 311 286 } 287 287 + 288 288 + qat_req->in.dh.in.b = dma_map_single(dev, vaddr, ctx->p_size, 289 289 + DMA_TO_DEVICE); 290 290 + if (unlikely(dma_mapping_error(dev, qat_req->in.dh.in.b))) 291 291 + goto unmap_src; 312 292 } 313 293 /* 314 294 * dst can be of any size in valid range, but HW expects it to be the ··· 316 302 */ 317 303 if (sg_is_last(req->dst) && req->dst_len == ctx->p_size) { 318 304 qat_req->dst_align = NULL; 319 319 - qat_req->out.dh.r = dma_map_single(dev, sg_virt(req->dst), 320 320 - req->dst_len, 321 321 - DMA_FROM_DEVICE); 322 322 - 323 323 - if (unlikely(dma_mapping_error(dev, qat_req->out.dh.r))) 324 324 - goto unmap_src; 325 325 - 305 305 + vaddr = sg_virt(req->dst); 326 306 } else { 327 327 - qat_req->dst_align = dma_alloc_coherent(dev, ctx->p_size, 328 328 - &qat_req->out.dh.r, 329 329 - GFP_KERNEL); 307 307 + qat_req->dst_align = kzalloc(ctx->p_size, flags); 330 308 if (unlikely(!qat_req->dst_align)) 331 309 goto unmap_src; 310 310 + 311 311 + vaddr = qat_req->dst_align; 332 312 } 313 313 + qat_req->out.dh.r = dma_map_single(dev, vaddr, ctx->p_size, 314 314 + DMA_FROM_DEVICE); 315 315 + if (unlikely(dma_mapping_error(dev, qat_req->out.dh.r))) 316 316 + goto unmap_dst; 333 317 334 318 qat_req->in.dh.in_tab[n_input_params] = 0; 335 319 qat_req->out.dh.out_tab[1] = 0; ··· 350 338 msg->input_param_count = n_input_params; 351 339 msg->output_param_count = 1; 352 340 353 353 - do { 354 354 - ret = adf_send_message(ctx->inst->pke_tx, (u32 *)msg); 355 355 - } while (ret == -EBUSY && ctr++ < 100); 341 341 + ret = qat_alg_send_asym_message(qat_req, inst, &req->base); 342 342 + if (ret == -ENOSPC) 343 343 + goto unmap_all; 356 344 357 357 - if (!ret) 358 358 - return -EINPROGRESS; 345 345 + return ret; 359 346 347 347 + unmap_all: 360 348 if (!dma_mapping_error(dev, qat_req->phy_out)) 361 349 dma_unmap_single(dev, qat_req->phy_out, 362 350 sizeof(struct qat_dh_output_params), ··· 367 355 sizeof(struct qat_dh_input_params), 368 356 DMA_TO_DEVICE); 369 357 unmap_dst: 370 370 - if (qat_req->dst_align) 371 371 - dma_free_coherent(dev, ctx->p_size, qat_req->dst_align, 372 372 - qat_req->out.dh.r); 373 373 - else 374 374 - if (!dma_mapping_error(dev, qat_req->out.dh.r)) 375 375 - dma_unmap_single(dev, qat_req->out.dh.r, ctx->p_size, 376 376 - DMA_FROM_DEVICE); 358 358 + if (!dma_mapping_error(dev, qat_req->out.dh.r)) 359 359 + dma_unmap_single(dev, qat_req->out.dh.r, ctx->p_size, 360 360 + DMA_FROM_DEVICE); 361 361 + kfree_sensitive(qat_req->dst_align); 377 362 unmap_src: 378 363 if (req->src) { 379 379 - if (qat_req->src_align) 380 380 - dma_free_coherent(dev, ctx->p_size, qat_req->src_align, 381 381 - qat_req->in.dh.in.b); 382 382 - else 383 383 - if (!dma_mapping_error(dev, qat_req->in.dh.in.b)) 384 384 - dma_unmap_single(dev, qat_req->in.dh.in.b, 385 385 - ctx->p_size, 386 386 - DMA_TO_DEVICE); 364 364 + if (!dma_mapping_error(dev, qat_req->in.dh.in.b)) 365 365 + dma_unmap_single(dev, qat_req->in.dh.in.b, 366 366 + ctx->p_size, 367 367 + DMA_TO_DEVICE); 368 368 + kfree_sensitive(qat_req->src_align); 387 369 } 388 370 return ret; 389 371 } ··· 426 420 static void qat_dh_clear_ctx(struct device *dev, struct qat_dh_ctx *ctx) 427 421 { 428 422 if (ctx->g) { 423 423 + memset(ctx->g, 0, ctx->p_size); 429 424 dma_free_coherent(dev, ctx->p_size, ctx->g, ctx->dma_g); 430 425 ctx->g = NULL; 431 426 } 432 427 if (ctx->xa) { 428 428 + memset(ctx->xa, 0, ctx->p_size); 433 429 dma_free_coherent(dev, ctx->p_size, ctx->xa, ctx->dma_xa); 434 430 ctx->xa = NULL; 435 431 } 436 432 if (ctx->p) { 433 433 + memset(ctx->p, 0, ctx->p_size); 437 434 dma_free_coherent(dev, ctx->p_size, ctx->p, ctx->dma_p); 438 435 ctx->p = NULL; 439 436 } ··· 519 510 520 511 err = (err == ICP_QAT_FW_COMN_STATUS_FLAG_OK) ? 0 : -EINVAL; 521 512 522 522 - if (req->src_align) 523 523 - dma_free_coherent(dev, req->ctx.rsa->key_sz, req->src_align, 524 524 - req->in.rsa.enc.m); 525 525 - else 526 526 - dma_unmap_single(dev, req->in.rsa.enc.m, req->ctx.rsa->key_sz, 527 527 - DMA_TO_DEVICE); 513 513 + kfree_sensitive(req->src_align); 514 514 + 515 515 + dma_unmap_single(dev, req->in.rsa.enc.m, req->ctx.rsa->key_sz, 516 516 + DMA_TO_DEVICE); 528 517 529 518 areq->dst_len = req->ctx.rsa->key_sz; 530 519 if (req->dst_align) { 531 520 scatterwalk_map_and_copy(req->dst_align, areq->dst, 0, 532 521 areq->dst_len, 1); 533 522 534 534 - dma_free_coherent(dev, req->ctx.rsa->key_sz, req->dst_align, 535 535 - req->out.rsa.enc.c); 536 536 - } else { 537 537 - dma_unmap_single(dev, req->out.rsa.enc.c, req->ctx.rsa->key_sz, 538 538 - DMA_FROM_DEVICE); 523 523 + kfree_sensitive(req->dst_align); 539 524 } 525 525 + 526 526 + dma_unmap_single(dev, req->out.rsa.enc.c, req->ctx.rsa->key_sz, 527 527 + DMA_FROM_DEVICE); 540 528 541 529 dma_unmap_single(dev, req->phy_in, sizeof(struct qat_rsa_input_params), 542 530 DMA_TO_DEVICE); ··· 548 542 { 549 543 struct icp_qat_fw_pke_resp *resp = _resp; 550 544 struct qat_asym_request *areq = (void *)(__force long)resp->opaque; 545 545 + struct qat_instance_backlog *backlog = areq->alg_req.backlog; 551 546 552 547 areq->cb(resp); 548 548 + 549 549 + qat_alg_send_backlog(backlog); 553 550 } 554 551 555 552 #define PKE_RSA_EP_512 0x1c161b21 ··· 651 642 struct qat_asym_request *qat_req = 652 643 PTR_ALIGN(akcipher_request_ctx(req), 64); 653 644 struct icp_qat_fw_pke_request *msg = &qat_req->req; 654 654 - int ret, ctr = 0; 645 645 + gfp_t flags = qat_algs_alloc_flags(&req->base); 646 646 + u8 *vaddr; 647 647 + int ret; 655 648 656 649 if (unlikely(!ctx->n || !ctx->e)) 657 650 return -EINVAL; ··· 662 651 req->dst_len = ctx->key_sz; 663 652 return -EOVERFLOW; 664 653 } 654 654 + 655 655 + if (req->src_len > ctx->key_sz) 656 656 + return -EINVAL; 657 657 + 665 658 memset(msg, '\0', sizeof(*msg)); 666 659 ICP_QAT_FW_PKE_HDR_VALID_FLAG_SET(msg->pke_hdr, 667 660 ICP_QAT_FW_COMN_REQ_FLAG_SET); ··· 694 679 */ 695 680 if (sg_is_last(req->src) && req->src_len == ctx->key_sz) { 696 681 qat_req->src_align = NULL; 697 697 - qat_req->in.rsa.enc.m = dma_map_single(dev, sg_virt(req->src), 698 698 - req->src_len, DMA_TO_DEVICE); 699 699 - if (unlikely(dma_mapping_error(dev, qat_req->in.rsa.enc.m))) 700 700 - return ret; 701 701 - 682 682 + vaddr = sg_virt(req->src); 702 683 } else { 703 684 int shift = ctx->key_sz - req->src_len; 704 685 705 705 - qat_req->src_align = dma_alloc_coherent(dev, ctx->key_sz, 706 706 - &qat_req->in.rsa.enc.m, 707 707 - GFP_KERNEL); 686 686 + qat_req->src_align = kzalloc(ctx->key_sz, flags); 708 687 if (unlikely(!qat_req->src_align)) 709 688 return ret; 710 689 711 690 scatterwalk_map_and_copy(qat_req->src_align + shift, req->src, 712 691 0, req->src_len, 0); 692 692 + vaddr = qat_req->src_align; 713 693 } 694 694 + 695 695 + qat_req->in.rsa.enc.m = dma_map_single(dev, vaddr, ctx->key_sz, 696 696 + DMA_TO_DEVICE); 697 697 + if (unlikely(dma_mapping_error(dev, qat_req->in.rsa.enc.m))) 698 698 + goto unmap_src; 699 699 + 714 700 if (sg_is_last(req->dst) && req->dst_len == ctx->key_sz) { 715 701 qat_req->dst_align = NULL; 716 716 - qat_req->out.rsa.enc.c = dma_map_single(dev, sg_virt(req->dst), 717 717 - req->dst_len, 718 718 - DMA_FROM_DEVICE); 719 719 - 720 720 - if (unlikely(dma_mapping_error(dev, qat_req->out.rsa.enc.c))) 721 721 - goto unmap_src; 722 722 - 702 702 + vaddr = sg_virt(req->dst); 723 703 } else { 724 724 - qat_req->dst_align = dma_alloc_coherent(dev, ctx->key_sz, 725 725 - &qat_req->out.rsa.enc.c, 726 726 - GFP_KERNEL); 704 704 + qat_req->dst_align = kzalloc(ctx->key_sz, flags); 727 705 if (unlikely(!qat_req->dst_align)) 728 706 goto unmap_src; 729 729 - 707 707 + vaddr = qat_req->dst_align; 730 708 } 709 709 + 710 710 + qat_req->out.rsa.enc.c = dma_map_single(dev, vaddr, ctx->key_sz, 711 711 + DMA_FROM_DEVICE); 712 712 + if (unlikely(dma_mapping_error(dev, qat_req->out.rsa.enc.c))) 713 713 + goto unmap_dst; 714 714 + 731 715 qat_req->in.rsa.in_tab[3] = 0; 732 716 qat_req->out.rsa.out_tab[1] = 0; 733 717 qat_req->phy_in = dma_map_single(dev, &qat_req->in.rsa.enc.m, ··· 746 732 msg->pke_mid.opaque = (u64)(__force long)qat_req; 747 733 msg->input_param_count = 3; 748 734 msg->output_param_count = 1; 749 749 - do { 750 750 - ret = adf_send_message(ctx->inst->pke_tx, (u32 *)msg); 751 751 - } while (ret == -EBUSY && ctr++ < 100); 752 735 753 753 - if (!ret) 754 754 - return -EINPROGRESS; 736 736 + ret = qat_alg_send_asym_message(qat_req, inst, &req->base); 737 737 + if (ret == -ENOSPC) 738 738 + goto unmap_all; 755 739 740 740 + return ret; 741 741 + 742 742 + unmap_all: 756 743 if (!dma_mapping_error(dev, qat_req->phy_out)) 757 744 dma_unmap_single(dev, qat_req->phy_out, 758 745 sizeof(struct qat_rsa_output_params), ··· 764 749 sizeof(struct qat_rsa_input_params), 765 750 DMA_TO_DEVICE); 766 751 unmap_dst: 767 767 - if (qat_req->dst_align) 768 768 - dma_free_coherent(dev, ctx->key_sz, qat_req->dst_align, 769 769 - qat_req->out.rsa.enc.c); 770 770 - else 771 771 - if (!dma_mapping_error(dev, qat_req->out.rsa.enc.c)) 772 772 - dma_unmap_single(dev, qat_req->out.rsa.enc.c, 773 773 - ctx->key_sz, DMA_FROM_DEVICE); 752 752 + if (!dma_mapping_error(dev, qat_req->out.rsa.enc.c)) 753 753 + dma_unmap_single(dev, qat_req->out.rsa.enc.c, 754 754 + ctx->key_sz, DMA_FROM_DEVICE); 755 755 + kfree_sensitive(qat_req->dst_align); 774 756 unmap_src: 775 775 - if (qat_req->src_align) 776 776 - dma_free_coherent(dev, ctx->key_sz, qat_req->src_align, 777 777 - qat_req->in.rsa.enc.m); 778 778 - else 779 779 - if (!dma_mapping_error(dev, qat_req->in.rsa.enc.m)) 780 780 - dma_unmap_single(dev, qat_req->in.rsa.enc.m, 781 781 - ctx->key_sz, DMA_TO_DEVICE); 757 757 + if (!dma_mapping_error(dev, qat_req->in.rsa.enc.m)) 758 758 + dma_unmap_single(dev, qat_req->in.rsa.enc.m, ctx->key_sz, 759 759 + DMA_TO_DEVICE); 760 760 + kfree_sensitive(qat_req->src_align); 782 761 return ret; 783 762 } 784 763 ··· 785 776 struct qat_asym_request *qat_req = 786 777 PTR_ALIGN(akcipher_request_ctx(req), 64); 787 778 struct icp_qat_fw_pke_request *msg = &qat_req->req; 788 788 - int ret, ctr = 0; 779 779 + gfp_t flags = qat_algs_alloc_flags(&req->base); 780 780 + u8 *vaddr; 781 781 + int ret; 789 782 790 783 if (unlikely(!ctx->n || !ctx->d)) 791 784 return -EINVAL; ··· 796 785 req->dst_len = ctx->key_sz; 797 786 return -EOVERFLOW; 798 787 } 788 788 + 789 789 + if (req->src_len > ctx->key_sz) 790 790 + return -EINVAL; 791 791 + 799 792 memset(msg, '\0', sizeof(*msg)); 800 793 ICP_QAT_FW_PKE_HDR_VALID_FLAG_SET(msg->pke_hdr, 801 794 ICP_QAT_FW_COMN_REQ_FLAG_SET); ··· 838 823 */ 839 824 if (sg_is_last(req->src) && req->src_len == ctx->key_sz) { 840 825 qat_req->src_align = NULL; 841 841 - qat_req->in.rsa.dec.c = dma_map_single(dev, sg_virt(req->src), 842 842 - req->dst_len, DMA_TO_DEVICE); 843 843 - if (unlikely(dma_mapping_error(dev, qat_req->in.rsa.dec.c))) 844 844 - return ret; 845 845 - 826 826 + vaddr = sg_virt(req->src); 846 827 } else { 847 828 int shift = ctx->key_sz - req->src_len; 848 829 849 849 - qat_req->src_align = dma_alloc_coherent(dev, ctx->key_sz, 850 850 - &qat_req->in.rsa.dec.c, 851 851 - GFP_KERNEL); 830 830 + qat_req->src_align = kzalloc(ctx->key_sz, flags); 852 831 if (unlikely(!qat_req->src_align)) 853 832 return ret; 854 833 855 834 scatterwalk_map_and_copy(qat_req->src_align + shift, req->src, 856 835 0, req->src_len, 0); 836 836 + vaddr = qat_req->src_align; 857 837 } 838 838 + 839 839 + qat_req->in.rsa.dec.c = dma_map_single(dev, vaddr, ctx->key_sz, 840 840 + DMA_TO_DEVICE); 841 841 + if (unlikely(dma_mapping_error(dev, qat_req->in.rsa.dec.c))) 842 842 + goto unmap_src; 843 843 + 858 844 if (sg_is_last(req->dst) && req->dst_len == ctx->key_sz) { 859 845 qat_req->dst_align = NULL; 860 860 - qat_req->out.rsa.dec.m = dma_map_single(dev, sg_virt(req->dst), 861 861 - req->dst_len, 862 862 - DMA_FROM_DEVICE); 863 863 - 864 864 - if (unlikely(dma_mapping_error(dev, qat_req->out.rsa.dec.m))) 865 865 - goto unmap_src; 866 866 - 846 846 + vaddr = sg_virt(req->dst); 867 847 } else { 868 868 - qat_req->dst_align = dma_alloc_coherent(dev, ctx->key_sz, 869 869 - &qat_req->out.rsa.dec.m, 870 870 - GFP_KERNEL); 848 848 + qat_req->dst_align = kzalloc(ctx->key_sz, flags); 871 849 if (unlikely(!qat_req->dst_align)) 872 850 goto unmap_src; 873 873 - 851 851 + vaddr = qat_req->dst_align; 874 852 } 853 853 + qat_req->out.rsa.dec.m = dma_map_single(dev, vaddr, ctx->key_sz, 854 854 + DMA_FROM_DEVICE); 855 855 + if (unlikely(dma_mapping_error(dev, qat_req->out.rsa.dec.m))) 856 856 + goto unmap_dst; 875 857 876 858 if (ctx->crt_mode) 877 859 qat_req->in.rsa.in_tab[6] = 0; ··· 896 884 msg->input_param_count = 3; 897 885 898 886 msg->output_param_count = 1; 899 899 - do { 900 900 - ret = adf_send_message(ctx->inst->pke_tx, (u32 *)msg); 901 901 - } while (ret == -EBUSY && ctr++ < 100); 902 887 903 903 - if (!ret) 904 904 - return -EINPROGRESS; 888 888 + ret = qat_alg_send_asym_message(qat_req, inst, &req->base); 889 889 + if (ret == -ENOSPC) 890 890 + goto unmap_all; 905 891 892 892 + return ret; 893 893 + 894 894 + unmap_all: 906 895 if (!dma_mapping_error(dev, qat_req->phy_out)) 907 896 dma_unmap_single(dev, qat_req->phy_out, 908 897 sizeof(struct qat_rsa_output_params), ··· 914 901 sizeof(struct qat_rsa_input_params), 915 902 DMA_TO_DEVICE); 916 903 unmap_dst: 917 917 - if (qat_req->dst_align) 918 918 - dma_free_coherent(dev, ctx->key_sz, qat_req->dst_align, 919 919 - qat_req->out.rsa.dec.m); 920 920 - else 921 921 - if (!dma_mapping_error(dev, qat_req->out.rsa.dec.m)) 922 922 - dma_unmap_single(dev, qat_req->out.rsa.dec.m, 923 923 - ctx->key_sz, DMA_FROM_DEVICE); 904 904 + if (!dma_mapping_error(dev, qat_req->out.rsa.dec.m)) 905 905 + dma_unmap_single(dev, qat_req->out.rsa.dec.m, 906 906 + ctx->key_sz, DMA_FROM_DEVICE); 907 907 + kfree_sensitive(qat_req->dst_align); 924 908 unmap_src: 925 925 - if (qat_req->src_align) 926 926 - dma_free_coherent(dev, ctx->key_sz, qat_req->src_align, 927 927 - qat_req->in.rsa.dec.c); 928 928 - else 929 929 - if (!dma_mapping_error(dev, qat_req->in.rsa.dec.c)) 930 930 - dma_unmap_single(dev, qat_req->in.rsa.dec.c, 931 931 - ctx->key_sz, DMA_TO_DEVICE); 909 909 + if (!dma_mapping_error(dev, qat_req->in.rsa.dec.c)) 910 910 + dma_unmap_single(dev, qat_req->in.rsa.dec.c, ctx->key_sz, 911 911 + DMA_TO_DEVICE); 912 912 + kfree_sensitive(qat_req->src_align); 932 913 return ret; 933 914 } 934 915 ··· 1240 1233 struct qat_rsa_ctx *ctx = akcipher_tfm_ctx(tfm); 1241 1234 struct device *dev = &GET_DEV(ctx->inst->accel_dev); 1242 1235 1243 1243 - if (ctx->n) 1244 1244 - dma_free_coherent(dev, ctx->key_sz, ctx->n, ctx->dma_n); 1245 1245 - if (ctx->e) 1246 1246 - dma_free_coherent(dev, ctx->key_sz, ctx->e, ctx->dma_e); 1247 1247 - if (ctx->d) { 1248 1248 - memset(ctx->d, '\0', ctx->key_sz); 1249 1249 - dma_free_coherent(dev, ctx->key_sz, ctx->d, ctx->dma_d); 1250 1250 - } 1236 1236 + qat_rsa_clear_ctx(dev, ctx); 1251 1237 qat_crypto_put_instance(ctx->inst); 1252 1252 - ctx->n = NULL; 1253 1253 - ctx->e = NULL; 1254 1254 - ctx->d = NULL; 1255 1238 } 1256 1239 1257 1240 static struct akcipher_alg rsa = {

+3 -7

drivers/crypto/qat/qat_common/qat_crypto.c

reviewed

··· 161 161 if (ret) 162 162 goto err; 163 163 164 164 - /* Temporarily set the number of crypto instances to zero to avoid 165 165 - * registering the crypto algorithms. 166 166 - * This will be removed when the algorithms will support the 167 167 - * CRYPTO_TFM_REQ_MAY_BACKLOG flag 168 168 - */ 169 169 - instances = 0; 170 170 - 171 164 for (i = 0; i < instances; i++) { 172 165 val = i; 173 166 snprintf(key, sizeof(key), ADF_CY "%d" ADF_RING_ASYM_BANK_NUM, i); ··· 346 353 &inst->pke_rx); 347 354 if (ret) 348 355 goto err; 356 356 + 357 357 + INIT_LIST_HEAD(&inst->backlog.list); 358 358 + spin_lock_init(&inst->backlog.lock); 349 359 } 350 360 return 0; 351 361 err:

+44

drivers/crypto/qat/qat_common/qat_crypto.h

reviewed

··· 9 9 #include "adf_accel_devices.h" 10 10 #include "icp_qat_fw_la.h" 11 11 12 12 + struct qat_instance_backlog { 13 13 + struct list_head list; 14 14 + spinlock_t lock; /* protects backlog list */ 15 15 + }; 16 16 + 17 17 + struct qat_alg_req { 18 18 + u32 *fw_req; 19 19 + struct adf_etr_ring_data *tx_ring; 20 20 + struct crypto_async_request *base; 21 21 + struct list_head list; 22 22 + struct qat_instance_backlog *backlog; 23 23 + }; 24 24 + 12 25 struct qat_crypto_instance { 13 26 struct adf_etr_ring_data *sym_tx; 14 27 struct adf_etr_ring_data *sym_rx; ··· 32 19 unsigned long state; 33 20 int id; 34 21 atomic_t refctr; 22 22 + struct qat_instance_backlog backlog; 35 23 }; 24 24 + 25 25 + #define QAT_MAX_BUFF_DESC 4 26 26 + 27 27 + struct qat_alg_buf { 28 28 + u32 len; 29 29 + u32 resrvd; 30 30 + u64 addr; 31 31 + } __packed; 32 32 + 33 33 + struct qat_alg_buf_list { 34 34 + u64 resrvd; 35 35 + u32 num_bufs; 36 36 + u32 num_mapped_bufs; 37 37 + struct qat_alg_buf bufers[]; 38 38 + } __packed; 39 39 + 40 40 + struct qat_alg_fixed_buf_list { 41 41 + struct qat_alg_buf_list sgl_hdr; 42 42 + struct qat_alg_buf descriptors[QAT_MAX_BUFF_DESC]; 43 43 + } __packed __aligned(64); 36 44 37 45 struct qat_crypto_request_buffs { 38 46 struct qat_alg_buf_list *bl; ··· 62 28 dma_addr_t bloutp; 63 29 size_t sz; 64 30 size_t sz_out; 31 31 + bool sgl_src_valid; 32 32 + bool sgl_dst_valid; 33 33 + struct qat_alg_fixed_buf_list sgl_src; 34 34 + struct qat_alg_fixed_buf_list sgl_dst; 65 35 }; 66 36 67 37 struct qat_crypto_request; ··· 91 53 u8 iv[AES_BLOCK_SIZE]; 92 54 }; 93 55 bool encryption; 56 56 + struct qat_alg_req alg_req; 94 57 }; 95 58 96 59 static inline bool adf_hw_dev_has_crypto(struct adf_accel_dev *accel_dev) ··· 107 68 return false; 108 69 109 70 return true; 71 71 + } 72 72 + 73 73 + static inline gfp_t qat_algs_alloc_flags(struct crypto_async_request *req) 74 74 + { 75 75 + return req->flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL : GFP_ATOMIC; 110 76 } 111 77 112 78 #endif

+1

drivers/crypto/qat/qat_common/qat_hal.c

reviewed

··· 695 695 handle->pci_dev = pci_info->pci_dev; 696 696 switch (handle->pci_dev->device) { 697 697 case ADF_4XXX_PCI_DEVICE_ID: 698 698 + case ADF_401XX_PCI_DEVICE_ID: 698 699 handle->chip_info->mmp_sram_size = 0; 699 700 handle->chip_info->nn = false; 700 701 handle->chip_info->lm2lm3 = true;

+2 -1

drivers/crypto/qat/qat_common/qat_uclo.c

reviewed

··· 519 519 return NULL; 520 520 } 521 521 522 522 - static unsigned int 522 522 + static int 523 523 qat_uclo_check_image_compat(struct icp_qat_uof_encap_obj *encap_uof_obj, 524 524 struct icp_qat_uof_image *image) 525 525 { ··· 731 731 case PCI_DEVICE_ID_INTEL_QAT_C3XXX: 732 732 return ICP_QAT_AC_C3XXX_DEV_TYPE; 733 733 case ADF_4XXX_PCI_DEVICE_ID: 734 734 + case ADF_401XX_PCI_DEVICE_ID: 734 735 return ICP_QAT_AC_4XXX_A_DEV_TYPE; 735 736 default: 736 737 pr_err("QAT: unsupported device 0x%x\n",

+71 -55

drivers/crypto/qat/qat_dh895xcc/adf_dh895xcc_hw_data.c

reviewed

··· 7 7 #include "adf_dh895xcc_hw_data.h" 8 8 #include "icp_qat_hw.h" 9 9 10 10 + #define ADF_DH895XCC_VF_MSK 0xFFFFFFFF 11 11 + 10 12 /* Worker thread to service arbiter mappings */ 11 13 static const u32 thrd_to_arb_map[ADF_DH895XCC_MAX_ACCELENGINES] = { 12 14 0x12222AAA, 0x11666666, 0x12222AAA, 0x11666666, ··· 60 58 61 59 capabilities = ICP_ACCEL_CAPABILITIES_CRYPTO_SYMMETRIC | 62 60 ICP_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC | 63 63 - ICP_ACCEL_CAPABILITIES_AUTHENTICATION; 61 61 + ICP_ACCEL_CAPABILITIES_AUTHENTICATION | 62 62 + ICP_ACCEL_CAPABILITIES_CIPHER | 63 63 + ICP_ACCEL_CAPABILITIES_COMPRESSION; 64 64 65 65 /* Read accelerator capabilities mask */ 66 66 pci_read_config_dword(pdev, ADF_DEVICE_LEGFUSE_OFFSET, &legfuses); 67 67 68 68 - if (legfuses & ICP_ACCEL_MASK_CIPHER_SLICE) 68 68 + /* A set bit in legfuses means the feature is OFF in this SKU */ 69 69 + if (legfuses & ICP_ACCEL_MASK_CIPHER_SLICE) { 69 70 capabilities &= ~ICP_ACCEL_CAPABILITIES_CRYPTO_SYMMETRIC; 71 71 + capabilities &= ~ICP_ACCEL_CAPABILITIES_CIPHER; 72 72 + } 70 73 if (legfuses & ICP_ACCEL_MASK_PKE_SLICE) 71 74 capabilities &= ~ICP_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC; 72 72 - if (legfuses & ICP_ACCEL_MASK_AUTH_SLICE) 75 75 + if (legfuses & ICP_ACCEL_MASK_AUTH_SLICE) { 73 76 capabilities &= ~ICP_ACCEL_CAPABILITIES_AUTHENTICATION; 77 77 + capabilities &= ~ICP_ACCEL_CAPABILITIES_CIPHER; 78 78 + } 74 79 if (legfuses & ICP_ACCEL_MASK_COMPRESS_SLICE) 75 80 capabilities &= ~ICP_ACCEL_CAPABILITIES_COMPRESSION; 76 81 ··· 109 100 return thrd_to_arb_map; 110 101 } 111 102 112 112 - static void adf_enable_ints(struct adf_accel_dev *accel_dev) 113 113 - { 114 114 - void __iomem *addr; 115 115 - 116 116 - addr = (&GET_BARS(accel_dev)[ADF_DH895XCC_PMISC_BAR])->virt_addr; 117 117 - 118 118 - /* Enable bundle and misc interrupts */ 119 119 - ADF_CSR_WR(addr, ADF_DH895XCC_SMIAPF0_MASK_OFFSET, 120 120 - accel_dev->pf.vf_info ? 0 : 121 121 - BIT_ULL(GET_MAX_BANKS(accel_dev)) - 1); 122 122 - ADF_CSR_WR(addr, ADF_DH895XCC_SMIAPF1_MASK_OFFSET, 123 123 - ADF_DH895XCC_SMIA1_MASK); 124 124 - } 125 125 - 126 126 - static u32 get_vf2pf_sources(void __iomem *pmisc_bar) 127 127 - { 128 128 - u32 errsou3, errmsk3, errsou5, errmsk5, vf_int_mask; 129 129 - 130 130 - /* Get the interrupt sources triggered by VFs */ 131 131 - errsou3 = ADF_CSR_RD(pmisc_bar, ADF_GEN2_ERRSOU3); 132 132 - vf_int_mask = ADF_DH895XCC_ERR_REG_VF2PF_L(errsou3); 133 133 - 134 134 - /* To avoid adding duplicate entries to work queue, clear 135 135 - * vf_int_mask_sets bits that are already masked in ERRMSK register. 136 136 - */ 137 137 - errmsk3 = ADF_CSR_RD(pmisc_bar, ADF_GEN2_ERRMSK3); 138 138 - vf_int_mask &= ~ADF_DH895XCC_ERR_REG_VF2PF_L(errmsk3); 139 139 - 140 140 - /* Do the same for ERRSOU5 */ 141 141 - errsou5 = ADF_CSR_RD(pmisc_bar, ADF_GEN2_ERRSOU5); 142 142 - errmsk5 = ADF_CSR_RD(pmisc_bar, ADF_GEN2_ERRMSK5); 143 143 - vf_int_mask |= ADF_DH895XCC_ERR_REG_VF2PF_U(errsou5); 144 144 - vf_int_mask &= ~ADF_DH895XCC_ERR_REG_VF2PF_U(errmsk5); 145 145 - 146 146 - return vf_int_mask; 147 147 - } 148 148 - 149 103 static void enable_vf2pf_interrupts(void __iomem *pmisc_addr, u32 vf_mask) 150 104 { 151 105 /* Enable VF2PF Messaging Ints - VFs 0 through 15 per vf_mask[15:0] */ ··· 122 150 if (vf_mask >> 16) { 123 151 u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK5) 124 152 & ~ADF_DH895XCC_ERR_MSK_VF2PF_U(vf_mask); 125 125 - 126 153 ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK5, val); 127 154 } 128 155 } 129 156 130 130 - static void disable_vf2pf_interrupts(void __iomem *pmisc_addr, u32 vf_mask) 157 157 + static void disable_all_vf2pf_interrupts(void __iomem *pmisc_addr) 131 158 { 159 159 + u32 val; 160 160 + 132 161 /* Disable VF2PF interrupts for VFs 0 through 15 per vf_mask[15:0] */ 133 133 - if (vf_mask & 0xFFFF) { 134 134 - u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3) 135 135 - | ADF_DH895XCC_ERR_MSK_VF2PF_L(vf_mask); 136 136 - ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, val); 137 137 - } 162 162 + val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3) 163 163 + | ADF_DH895XCC_ERR_MSK_VF2PF_L(ADF_DH895XCC_VF_MSK); 164 164 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, val); 138 165 139 166 /* Disable VF2PF interrupts for VFs 16 through 31 per vf_mask[31:16] */ 140 140 - if (vf_mask >> 16) { 141 141 - u32 val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK5) 142 142 - | ADF_DH895XCC_ERR_MSK_VF2PF_U(vf_mask); 167 167 + val = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK5) 168 168 + | ADF_DH895XCC_ERR_MSK_VF2PF_U(ADF_DH895XCC_VF_MSK); 169 169 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK5, val); 170 170 + } 143 171 144 144 - ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK5, val); 145 145 - } 172 172 + static u32 disable_pending_vf2pf_interrupts(void __iomem *pmisc_addr) 173 173 + { 174 174 + u32 sources, pending, disabled; 175 175 + u32 errsou3, errmsk3; 176 176 + u32 errsou5, errmsk5; 177 177 + 178 178 + /* Get the interrupt sources triggered by VFs */ 179 179 + errsou3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRSOU3); 180 180 + errsou5 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRSOU5); 181 181 + sources = ADF_DH895XCC_ERR_REG_VF2PF_L(errsou3) 182 182 + | ADF_DH895XCC_ERR_REG_VF2PF_U(errsou5); 183 183 + 184 184 + if (!sources) 185 185 + return 0; 186 186 + 187 187 + /* Get the already disabled interrupts */ 188 188 + errmsk3 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK3); 189 189 + errmsk5 = ADF_CSR_RD(pmisc_addr, ADF_GEN2_ERRMSK5); 190 190 + disabled = ADF_DH895XCC_ERR_REG_VF2PF_L(errmsk3) 191 191 + | ADF_DH895XCC_ERR_REG_VF2PF_U(errmsk5); 192 192 + 193 193 + pending = sources & ~disabled; 194 194 + if (!pending) 195 195 + return 0; 196 196 + 197 197 + /* Due to HW limitations, when disabling the interrupts, we can't 198 198 + * just disable the requested sources, as this would lead to missed 199 199 + * interrupts if sources changes just before writing to ERRMSK3 and 200 200 + * ERRMSK5. 201 201 + * To work around it, disable all and re-enable only the sources that 202 202 + * are not in vf_mask and were not already disabled. Re-enabling will 203 203 + * trigger a new interrupt for the sources that have changed in the 204 204 + * meantime, if any. 205 205 + */ 206 206 + errmsk3 |= ADF_DH895XCC_ERR_MSK_VF2PF_L(ADF_DH895XCC_VF_MSK); 207 207 + errmsk5 |= ADF_DH895XCC_ERR_MSK_VF2PF_U(ADF_DH895XCC_VF_MSK); 208 208 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, errmsk3); 209 209 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK5, errmsk5); 210 210 + 211 211 + errmsk3 &= ADF_DH895XCC_ERR_MSK_VF2PF_L(sources | disabled); 212 212 + errmsk5 &= ADF_DH895XCC_ERR_MSK_VF2PF_U(sources | disabled); 213 213 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK3, errmsk3); 214 214 + ADF_CSR_WR(pmisc_addr, ADF_GEN2_ERRMSK5, errmsk5); 215 215 + 216 216 + /* Return the sources of the (new) interrupt(s) */ 217 217 + return pending; 146 218 } 147 219 148 220 static void configure_iov_threads(struct adf_accel_dev *accel_dev, bool enable) ··· 231 215 hw_data->init_arb = adf_init_arb; 232 216 hw_data->exit_arb = adf_exit_arb; 233 217 hw_data->get_arb_mapping = adf_get_arbiter_mapping; 234 234 - hw_data->enable_ints = adf_enable_ints; 218 218 + hw_data->enable_ints = adf_gen2_enable_ints; 235 219 hw_data->reset_device = adf_reset_sbr; 236 220 hw_data->disable_iov = adf_disable_sriov; 237 221 238 222 adf_gen2_init_pf_pfvf_ops(&hw_data->pfvf_ops); 239 239 - hw_data->pfvf_ops.get_vf2pf_sources = get_vf2pf_sources; 240 223 hw_data->pfvf_ops.enable_vf2pf_interrupts = enable_vf2pf_interrupts; 241 241 - hw_data->pfvf_ops.disable_vf2pf_interrupts = disable_vf2pf_interrupts; 224 224 + hw_data->pfvf_ops.disable_all_vf2pf_interrupts = disable_all_vf2pf_interrupts; 225 225 + hw_data->pfvf_ops.disable_pending_vf2pf_interrupts = disable_pending_vf2pf_interrupts; 242 226 adf_gen2_init_hw_csr_ops(&hw_data->csr_ops); 243 227 } 244 228

-4

drivers/crypto/qat/qat_dh895xcc/adf_dh895xcc_hw_data.h

reviewed

··· 19 19 #define ADF_DH895XCC_ACCELERATORS_MASK 0x3F 20 20 #define ADF_DH895XCC_ACCELENGINES_MASK 0xFFF 21 21 #define ADF_DH895XCC_ETR_MAX_BANKS 32 22 22 - #define ADF_DH895XCC_SMIAPF0_MASK_OFFSET (0x3A000 + 0x28) 23 23 - #define ADF_DH895XCC_SMIAPF1_MASK_OFFSET (0x3A000 + 0x30) 24 24 - #define ADF_DH895XCC_SMIA0_MASK 0xFFFFFFFF 25 25 - #define ADF_DH895XCC_SMIA1_MASK 0x1 26 22 27 23 /* Masks for VF2PF interrupts */ 28 24 #define ADF_DH895XCC_ERR_REG_VF2PF_L(vf_src) (((vf_src) & 0x01FFFE00) >> 9)

+1

drivers/crypto/sa2ul.c

reviewed

··· 2379 2379 { .compatible = "ti,j721e-sa2ul", .data = &am654_match_data, }, 2380 2380 { .compatible = "ti,am654-sa2ul", .data = &am654_match_data, }, 2381 2381 { .compatible = "ti,am64-sa2ul", .data = &am64_match_data, }, 2382 2382 + { .compatible = "ti,am62-sa3ul", .data = &am64_match_data, }, 2382 2383 {}, 2383 2384 }; 2384 2385 MODULE_DEVICE_TABLE(of, of_match);

+5 -5

drivers/crypto/talitos.c

reviewed

··· 1709 1709 struct talitos_desc *desc2 = (struct talitos_desc *) 1710 1710 (edesc->buf + edesc->dma_len); 1711 1711 1712 1712 - unmap_single_talitos_ptr(dev, &edesc->desc.ptr[5], DMA_FROM_DEVICE); 1712 1712 + unmap_single_talitos_ptr(dev, &desc->ptr[5], DMA_FROM_DEVICE); 1713 1713 if (desc->next_desc && 1714 1714 desc->ptr[5].ptr != desc2->ptr[5].ptr) 1715 1715 unmap_single_talitos_ptr(dev, &desc2->ptr[5], DMA_FROM_DEVICE); ··· 1721 1721 talitos_sg_unmap(dev, edesc, req_ctx->psrc, NULL, 0, 0); 1722 1722 1723 1723 /* When using hashctx-in, must unmap it. */ 1724 1724 - if (from_talitos_ptr_len(&edesc->desc.ptr[1], is_sec1)) 1725 1725 - unmap_single_talitos_ptr(dev, &edesc->desc.ptr[1], 1724 1724 + if (from_talitos_ptr_len(&desc->ptr[1], is_sec1)) 1725 1725 + unmap_single_talitos_ptr(dev, &desc->ptr[1], 1726 1726 DMA_TO_DEVICE); 1727 1727 else if (desc->next_desc) 1728 1728 unmap_single_talitos_ptr(dev, &desc2->ptr[1], ··· 1736 1736 dma_unmap_single(dev, edesc->dma_link_tbl, edesc->dma_len, 1737 1737 DMA_BIDIRECTIONAL); 1738 1738 1739 1739 - if (edesc->desc.next_desc) 1740 1740 - dma_unmap_single(dev, be32_to_cpu(edesc->desc.next_desc), 1739 1739 + if (desc->next_desc) 1740 1740 + dma_unmap_single(dev, be32_to_cpu(desc->next_desc), 1741 1741 TALITOS_DESC_SIZE, DMA_BIDIRECTIONAL); 1742 1742 } 1743 1743

+1 -3

drivers/crypto/ux500/hash/hash_core.c

reviewed

··· 877 877 __func__); 878 878 goto out; 879 879 } 880 880 - } 881 881 - 882 882 - if (!req_ctx->updated) { 880 880 + } else { 883 881 ret = hash_setconfiguration(device_data, &ctx->config); 884 882 if (ret) { 885 883 dev_err(device_data->dev,

+3 -14

drivers/crypto/vmx/Makefile

reviewed

··· 2 2 obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o 3 3 vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o 4 4 5 5 - ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) 6 6 - override flavour := linux-ppc64le 7 7 - else 8 8 - override flavour := linux-ppc64 9 9 - endif 10 10 - 11 11 - quiet_cmd_perl = PERL $@ 12 12 - cmd_perl = $(PERL) $(<) $(flavour) > $(@) 5 5 + quiet_cmd_perl = PERL $@ 6 6 + cmd_perl = $(PERL) $< $(if $(CONFIG_CPU_LITTLE_ENDIAN), linux-ppc64le, linux-ppc64) > $@ 13 7 14 8 targets += aesp8-ppc.S ghashp8-ppc.S 15 9 16 16 - $(obj)/aesp8-ppc.S: $(src)/aesp8-ppc.pl FORCE 10 10 + $(obj)/aesp8-ppc.S $(obj)/ghashp8-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE 17 11 $(call if_changed,perl) 18 18 - 19 19 - $(obj)/ghashp8-ppc.S: $(src)/ghashp8-ppc.pl FORCE 20 20 - $(call if_changed,perl) 21 21 - 22 22 - clean-files := aesp8-ppc.S ghashp8-ppc.S

+4

include/crypto/sm4.h

reviewed

··· 21 21 u32 rkey_dec[SM4_RKEY_WORDS]; 22 22 }; 23 23 24 24 + extern const u32 crypto_sm4_fk[]; 25 25 + extern const u32 crypto_sm4_ck[]; 26 26 + extern const u8 crypto_sm4_sbox[]; 27 27 + 24 28 /** 25 29 * sm4_expandkey - Expands the SM4 key as described in GB/T 32907-2016 26 30 * @ctx: The location where the computed key will be stored.

+18 -5

include/linux/hisi_acc_qm.h

reviewed

··· 168 168 QM_NOT_READY, 169 169 }; 170 170 171 171 + struct dfx_diff_registers { 172 172 + u32 *regs; 173 173 + u32 reg_offset; 174 174 + u32 reg_len; 175 175 + }; 176 176 + 171 177 struct qm_dfx { 172 178 atomic64_t err_irq_cnt; 173 179 atomic64_t aeq_irq_cnt; ··· 196 190 struct dentry *debug_root; 197 191 struct dentry *qm_d; 198 192 struct debugfs_file files[DEBUG_FILE_NUM]; 193 193 + unsigned int *qm_last_words; 194 194 + /* ACC engines recoreding last regs */ 195 195 + unsigned int *last_words; 196 196 + struct dfx_diff_registers *qm_diff_regs; 197 197 + struct dfx_diff_registers *acc_diff_regs; 199 198 }; 200 199 201 200 struct qm_shaper_factor { ··· 254 243 void (*open_sva_prefetch)(struct hisi_qm *qm); 255 244 void (*close_sva_prefetch)(struct hisi_qm *qm); 256 245 void (*log_dev_hw_err)(struct hisi_qm *qm, u32 err_sts); 246 246 + void (*show_last_dfx_regs)(struct hisi_qm *qm); 257 247 void (*err_info_init)(struct hisi_qm *qm); 258 248 }; 259 249 ··· 445 433 void hisi_qm_uninit(struct hisi_qm *qm); 446 434 int hisi_qm_start(struct hisi_qm *qm); 447 435 int hisi_qm_stop(struct hisi_qm *qm, enum qm_stop_reason r); 448 448 - struct hisi_qp *hisi_qm_create_qp(struct hisi_qm *qm, u8 alg_type); 449 436 int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg); 450 437 int hisi_qm_stop_qp(struct hisi_qp *qp); 451 451 - void hisi_qm_release_qp(struct hisi_qp *qp); 452 438 int hisi_qp_send(struct hisi_qp *qp, const void *msg); 453 453 - int hisi_qm_get_free_qp_num(struct hisi_qm *qm); 454 454 - int hisi_qm_get_vft(struct hisi_qm *qm, u32 *base, u32 *number); 455 439 void hisi_qm_debug_init(struct hisi_qm *qm); 456 456 - enum qm_hw_ver hisi_qm_get_hw_version(struct pci_dev *pdev); 457 440 void hisi_qm_debug_regs_clear(struct hisi_qm *qm); 458 441 int hisi_qm_sriov_enable(struct pci_dev *pdev, int max_vfs); 459 442 int hisi_qm_sriov_disable(struct pci_dev *pdev, bool is_frozen); 460 443 int hisi_qm_sriov_configure(struct pci_dev *pdev, int num_vfs); 461 444 void hisi_qm_dev_err_init(struct hisi_qm *qm); 462 445 void hisi_qm_dev_err_uninit(struct hisi_qm *qm); 446 446 + int hisi_qm_diff_regs_init(struct hisi_qm *qm, 447 447 + struct dfx_diff_registers *dregs, int reg_len); 448 448 + void hisi_qm_diff_regs_uninit(struct hisi_qm *qm, int reg_len); 449 449 + void hisi_qm_acc_diff_regs_dump(struct hisi_qm *qm, struct seq_file *s, 450 450 + struct dfx_diff_registers *dregs, int regs_len); 451 451 + 463 452 pci_ers_result_t hisi_qm_dev_err_detected(struct pci_dev *pdev, 464 453 pci_channel_state_t state); 465 454 pci_ers_result_t hisi_qm_dev_slot_reset(struct pci_dev *pdev);

-6

lib/crypto/Kconfig

reviewed

··· 124 124 config CRYPTO_LIB_SHA256 125 125 tristate 126 126 127 127 - config CRYPTO_LIB_SM3 128 128 - tristate 129 129 - 130 130 - config CRYPTO_LIB_SM4 131 131 - tristate 132 132 - 133 127 endmenu

-6

lib/crypto/Makefile

reviewed

··· 37 37 obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o 38 38 libsha256-y := sha256.o 39 39 40 40 - obj-$(CONFIG_CRYPTO_LIB_SM3) += libsm3.o 41 41 - libsm3-y := sm3.o 42 42 - 43 43 - obj-$(CONFIG_CRYPTO_LIB_SM4) += libsm4.o 44 44 - libsm4-y := sm4.o 45 45 - 46 40 ifneq ($(CONFIG_CRYPTO_MANAGER_DISABLE_TESTS),y) 47 41 libblake2s-y += blake2s-selftest.o 48 42 libchacha20poly1305-y += chacha20poly1305-selftest.o

lib/crypto/sm3.c crypto/sm3.c

reviewed

+9 -1

lib/crypto/sm4.c crypto/sm4.c

reviewed

··· 11 11 #include <asm/unaligned.h> 12 12 #include <crypto/sm4.h> 13 13 14 14 - static const u32 fk[4] = { 14 14 + static const u32 ____cacheline_aligned fk[4] = { 15 15 0xa3b1bac6, 0x56aa3350, 0x677d9197, 0xb27022dc 16 16 }; 17 17 ··· 60 60 0x18, 0xf0, 0x7d, 0xec, 0x3a, 0xdc, 0x4d, 0x20, 61 61 0x79, 0xee, 0x5f, 0x3e, 0xd7, 0xcb, 0x39, 0x48 62 62 }; 63 63 + 64 64 + extern const u32 crypto_sm4_fk[4] __alias(fk); 65 65 + extern const u32 crypto_sm4_ck[32] __alias(ck); 66 66 + extern const u8 crypto_sm4_sbox[256] __alias(sbox); 67 67 + 68 68 + EXPORT_SYMBOL(crypto_sm4_fk); 69 69 + EXPORT_SYMBOL(crypto_sm4_ck); 70 70 + EXPORT_SYMBOL(crypto_sm4_sbox); 63 71 64 72 static inline u32 sm4_t_non_lin_sub(u32 x) 65 73 {

+12

tools/testing/crypto/chacha20-s390/Makefile

reviewed

··· 1 1 + # SPDX-License-Identifier: GPL-2.0 2 2 + # 3 3 + # Copyright (C) 2022 Red Hat, Inc. 4 4 + # Author: Vladis Dronov <vdronoff@gmail.com> 5 5 + 6 6 + obj-m += test_cipher.o 7 7 + test_cipher-y := test-cipher.o 8 8 + 9 9 + all: 10 10 + make -C /lib/modules/$(shell uname -r)/build/ M=$(PWD) modules 11 11 + clean: 12 12 + make -C /lib/modules/$(shell uname -r)/build/ M=$(PWD) clean

+34

tools/testing/crypto/chacha20-s390/run-tests.sh

reviewed

··· 1 1 + #!/bin/bash 2 2 + # SPDX-License-Identifier: GPL-2.0 3 3 + # 4 4 + # Copyright (C) 2022 Red Hat, Inc. 5 5 + # Author: Vladis Dronov <vdronoff@gmail.com> 6 6 + # 7 7 + # This script runs (via instmod) test-cipher.ko module which invokes 8 8 + # generic and s390-native ChaCha20 encryprion algorithms with different 9 9 + # size of data. Check 'dmesg' for results. 10 10 + # 11 11 + # The insmod error is expected: 12 12 + # insmod: ERROR: could not insert module test_cipher.ko: Operation not permitted 13 13 + 14 14 + lsmod | grep chacha | cut -f1 -d' ' | xargs rmmod 15 15 + modprobe chacha_generic 16 16 + modprobe chacha_s390 17 17 + 18 18 + # run encryption for different data size, including whole block(s) +/- 1 19 19 + insmod test_cipher.ko size=63 20 20 + insmod test_cipher.ko size=64 21 21 + insmod test_cipher.ko size=65 22 22 + insmod test_cipher.ko size=127 23 23 + insmod test_cipher.ko size=128 24 24 + insmod test_cipher.ko size=129 25 25 + insmod test_cipher.ko size=511 26 26 + insmod test_cipher.ko size=512 27 27 + insmod test_cipher.ko size=513 28 28 + insmod test_cipher.ko size=4096 29 29 + insmod test_cipher.ko size=65611 30 30 + insmod test_cipher.ko size=6291456 31 31 + insmod test_cipher.ko size=62914560 32 32 + 33 33 + # print test logs 34 34 + dmesg | tail -170

+372

tools/testing/crypto/chacha20-s390/test-cipher.c

reviewed

··· 1 1 + /* SPDX-License-Identifier: GPL-2.0 2 2 + * 3 3 + * Copyright (C) 2022 Red Hat, Inc. 4 4 + * Author: Vladis Dronov <vdronoff@gmail.com> 5 5 + */ 6 6 + 7 7 + #include <asm/elf.h> 8 8 + #include <asm/uaccess.h> 9 9 + #include <asm/smp.h> 10 10 + #include <crypto/skcipher.h> 11 11 + #include <crypto/akcipher.h> 12 12 + #include <crypto/acompress.h> 13 13 + #include <crypto/rng.h> 14 14 + #include <crypto/drbg.h> 15 15 + #include <crypto/kpp.h> 16 16 + #include <crypto/internal/simd.h> 17 17 + #include <crypto/chacha.h> 18 18 + #include <crypto/aead.h> 19 19 + #include <crypto/hash.h> 20 20 + #include <linux/crypto.h> 21 21 + #include <linux/debugfs.h> 22 22 + #include <linux/delay.h> 23 23 + #include <linux/err.h> 24 24 + #include <linux/fs.h> 25 25 + #include <linux/fips.h> 26 26 + #include <linux/kernel.h> 27 27 + #include <linux/kthread.h> 28 28 + #include <linux/module.h> 29 29 + #include <linux/sched.h> 30 30 + #include <linux/scatterlist.h> 31 31 + #include <linux/time.h> 32 32 + #include <linux/vmalloc.h> 33 33 + #include <linux/zlib.h> 34 34 + #include <linux/once.h> 35 35 + #include <linux/random.h> 36 36 + #include <linux/slab.h> 37 37 + #include <linux/string.h> 38 38 + 39 39 + static unsigned int data_size __read_mostly = 256; 40 40 + static unsigned int debug __read_mostly = 0; 41 41 + 42 42 + /* tie all skcipher structures together */ 43 43 + struct skcipher_def { 44 44 + struct scatterlist sginp, sgout; 45 45 + struct crypto_skcipher *tfm; 46 46 + struct skcipher_request *req; 47 47 + struct crypto_wait wait; 48 48 + }; 49 49 + 50 50 + /* Perform cipher operations with the chacha lib */ 51 51 + static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain) 52 52 + { 53 53 + u32 chacha_state[CHACHA_STATE_WORDS]; 54 54 + u8 iv[16], key[32]; 55 55 + u64 start, end; 56 56 + 57 57 + memset(key, 'X', sizeof(key)); 58 58 + memset(iv, 'I', sizeof(iv)); 59 59 + 60 60 + if (debug) { 61 61 + print_hex_dump(KERN_INFO, "key: ", DUMP_PREFIX_OFFSET, 62 62 + 16, 1, key, 32, 1); 63 63 + 64 64 + print_hex_dump(KERN_INFO, "iv: ", DUMP_PREFIX_OFFSET, 65 65 + 16, 1, iv, 16, 1); 66 66 + } 67 67 + 68 68 + /* Encrypt */ 69 69 + chacha_init_arch(chacha_state, (u32*)key, iv); 70 70 + 71 71 + start = ktime_get_ns(); 72 72 + chacha_crypt_arch(chacha_state, cipher, plain, data_size, 20); 73 73 + end = ktime_get_ns(); 74 74 + 75 75 + 76 76 + if (debug) 77 77 + print_hex_dump(KERN_INFO, "encr:", DUMP_PREFIX_OFFSET, 78 78 + 16, 1, cipher, 79 79 + (data_size > 64 ? 64 : data_size), 1); 80 80 + 81 81 + pr_info("lib encryption took: %lld nsec", end - start); 82 82 + 83 83 + /* Decrypt */ 84 84 + chacha_init_arch(chacha_state, (u32 *)key, iv); 85 85 + 86 86 + start = ktime_get_ns(); 87 87 + chacha_crypt_arch(chacha_state, revert, cipher, data_size, 20); 88 88 + end = ktime_get_ns(); 89 89 + 90 90 + if (debug) 91 91 + print_hex_dump(KERN_INFO, "decr:", DUMP_PREFIX_OFFSET, 92 92 + 16, 1, revert, 93 93 + (data_size > 64 ? 64 : data_size), 1); 94 94 + 95 95 + pr_info("lib decryption took: %lld nsec", end - start); 96 96 + 97 97 + return 0; 98 98 + } 99 99 + 100 100 + /* Perform cipher operations with skcipher */ 101 101 + static unsigned int test_skcipher_encdec(struct skcipher_def *sk, 102 102 + int enc) 103 103 + { 104 104 + int rc; 105 105 + 106 106 + if (enc) { 107 107 + rc = crypto_wait_req(crypto_skcipher_encrypt(sk->req), 108 108 + &sk->wait); 109 109 + if (rc) 110 110 + pr_info("skcipher encrypt returned with result" 111 111 + "%d\n", rc); 112 112 + } 113 113 + else 114 114 + { 115 115 + rc = crypto_wait_req(crypto_skcipher_decrypt(sk->req), 116 116 + &sk->wait); 117 117 + if (rc) 118 118 + pr_info("skcipher decrypt returned with result" 119 119 + "%d\n", rc); 120 120 + } 121 121 + 122 122 + return rc; 123 123 + } 124 124 + 125 125 + /* Initialize and trigger cipher operations */ 126 126 + static int test_skcipher(char *name, u8 *revert, u8 *cipher, u8 *plain) 127 127 + { 128 128 + struct skcipher_def sk; 129 129 + struct crypto_skcipher *skcipher = NULL; 130 130 + struct skcipher_request *req = NULL; 131 131 + u8 iv[16], key[32]; 132 132 + u64 start, end; 133 133 + int ret = -EFAULT; 134 134 + 135 135 + skcipher = crypto_alloc_skcipher(name, 0, 0); 136 136 + if (IS_ERR(skcipher)) { 137 137 + pr_info("could not allocate skcipher %s handle\n", name); 138 138 + return PTR_ERR(skcipher); 139 139 + } 140 140 + 141 141 + req = skcipher_request_alloc(skcipher, GFP_KERNEL); 142 142 + if (!req) { 143 143 + pr_info("could not allocate skcipher request\n"); 144 144 + ret = -ENOMEM; 145 145 + goto out; 146 146 + } 147 147 + 148 148 + skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, 149 149 + crypto_req_done, 150 150 + &sk.wait); 151 151 + 152 152 + memset(key, 'X', sizeof(key)); 153 153 + memset(iv, 'I', sizeof(iv)); 154 154 + 155 155 + if (crypto_skcipher_setkey(skcipher, key, 32)) { 156 156 + pr_info("key could not be set\n"); 157 157 + ret = -EAGAIN; 158 158 + goto out; 159 159 + } 160 160 + 161 161 + if (debug) { 162 162 + print_hex_dump(KERN_INFO, "key: ", DUMP_PREFIX_OFFSET, 163 163 + 16, 1, key, 32, 1); 164 164 + 165 165 + print_hex_dump(KERN_INFO, "iv: ", DUMP_PREFIX_OFFSET, 166 166 + 16, 1, iv, 16, 1); 167 167 + } 168 168 + 169 169 + sk.tfm = skcipher; 170 170 + sk.req = req; 171 171 + 172 172 + /* Encrypt in one pass */ 173 173 + sg_init_one(&sk.sginp, plain, data_size); 174 174 + sg_init_one(&sk.sgout, cipher, data_size); 175 175 + skcipher_request_set_crypt(req, &sk.sginp, &sk.sgout, 176 176 + data_size, iv); 177 177 + crypto_init_wait(&sk.wait); 178 178 + 179 179 + /* Encrypt data */ 180 180 + start = ktime_get_ns(); 181 181 + ret = test_skcipher_encdec(&sk, 1); 182 182 + end = ktime_get_ns(); 183 183 + 184 184 + if (ret) 185 185 + goto out; 186 186 + 187 187 + pr_info("%s tfm encryption successful, took %lld nsec\n", name, end - start); 188 188 + 189 189 + if (debug) 190 190 + print_hex_dump(KERN_INFO, "encr:", DUMP_PREFIX_OFFSET, 191 191 + 16, 1, cipher, 192 192 + (data_size > 64 ? 64 : data_size), 1); 193 193 + 194 194 + /* Prepare for decryption */ 195 195 + memset(iv, 'I', sizeof(iv)); 196 196 + 197 197 + sg_init_one(&sk.sginp, cipher, data_size); 198 198 + sg_init_one(&sk.sgout, revert, data_size); 199 199 + skcipher_request_set_crypt(req, &sk.sginp, &sk.sgout, 200 200 + data_size, iv); 201 201 + crypto_init_wait(&sk.wait); 202 202 + 203 203 + /* Decrypt data */ 204 204 + start = ktime_get_ns(); 205 205 + ret = test_skcipher_encdec(&sk, 0); 206 206 + end = ktime_get_ns(); 207 207 + 208 208 + if (ret) 209 209 + goto out; 210 210 + 211 211 + pr_info("%s tfm decryption successful, took %lld nsec\n", name, end - start); 212 212 + 213 213 + if (debug) 214 214 + print_hex_dump(KERN_INFO, "decr:", DUMP_PREFIX_OFFSET, 215 215 + 16, 1, revert, 216 216 + (data_size > 64 ? 64 : data_size), 1); 217 217 + 218 218 + /* Dump some internal skcipher data */ 219 219 + if (debug) 220 220 + pr_info("skcipher %s: cryptlen %d blksize %d stride %d " 221 221 + "ivsize %d alignmask 0x%x\n", 222 222 + name, sk.req->cryptlen, 223 223 + crypto_skcipher_blocksize(sk.tfm), 224 224 + crypto_skcipher_alg(sk.tfm)->walksize, 225 225 + crypto_skcipher_ivsize(sk.tfm), 226 226 + crypto_skcipher_alignmask(sk.tfm)); 227 227 + 228 228 + out: 229 229 + if (skcipher) 230 230 + crypto_free_skcipher(skcipher); 231 231 + if (req) 232 232 + skcipher_request_free(req); 233 233 + return ret; 234 234 + } 235 235 + 236 236 + static int __init chacha_s390_test_init(void) 237 237 + { 238 238 + u8 *plain = NULL, *revert = NULL; 239 239 + u8 *cipher_generic = NULL, *cipher_s390 = NULL; 240 240 + int ret = -1; 241 241 + 242 242 + pr_info("s390 ChaCha20 test module: size=%d debug=%d\n", 243 243 + data_size, debug); 244 244 + 245 245 + /* Allocate and fill buffers */ 246 246 + plain = vmalloc(data_size); 247 247 + if (!plain) { 248 248 + pr_info("could not allocate plain buffer\n"); 249 249 + ret = -2; 250 250 + goto out; 251 251 + } 252 252 + memset(plain, 'a', data_size); 253 253 + get_random_bytes(plain, (data_size > 256 ? 256 : data_size)); 254 254 + 255 255 + cipher_generic = vmalloc(data_size); 256 256 + if (!cipher_generic) { 257 257 + pr_info("could not allocate cipher_generic buffer\n"); 258 258 + ret = -2; 259 259 + goto out; 260 260 + } 261 261 + memset(cipher_generic, 0, data_size); 262 262 + 263 263 + cipher_s390 = vmalloc(data_size); 264 264 + if (!cipher_s390) { 265 265 + pr_info("could not allocate cipher_s390 buffer\n"); 266 266 + ret = -2; 267 267 + goto out; 268 268 + } 269 269 + memset(cipher_s390, 0, data_size); 270 270 + 271 271 + revert = vmalloc(data_size); 272 272 + if (!revert) { 273 273 + pr_info("could not allocate revert buffer\n"); 274 274 + ret = -2; 275 275 + goto out; 276 276 + } 277 277 + memset(revert, 0, data_size); 278 278 + 279 279 + if (debug) 280 280 + print_hex_dump(KERN_INFO, "src: ", DUMP_PREFIX_OFFSET, 281 281 + 16, 1, plain, 282 282 + (data_size > 64 ? 64 : data_size), 1); 283 283 + 284 284 + /* Use chacha20 generic */ 285 285 + ret = test_skcipher("chacha20-generic", revert, cipher_generic, plain); 286 286 + if (ret) 287 287 + goto out; 288 288 + 289 289 + if (memcmp(plain, revert, data_size)) { 290 290 + pr_info("generic en/decryption check FAILED\n"); 291 291 + ret = -2; 292 292 + goto out; 293 293 + } 294 294 + else 295 295 + pr_info("generic en/decryption check OK\n"); 296 296 + 297 297 + memset(revert, 0, data_size); 298 298 + 299 299 + /* Use chacha20 s390 */ 300 300 + ret = test_skcipher("chacha20-s390", revert, cipher_s390, plain); 301 301 + if (ret) 302 302 + goto out; 303 303 + 304 304 + if (memcmp(plain, revert, data_size)) { 305 305 + pr_info("s390 en/decryption check FAILED\n"); 306 306 + ret = -2; 307 307 + goto out; 308 308 + } 309 309 + else 310 310 + pr_info("s390 en/decryption check OK\n"); 311 311 + 312 312 + if (memcmp(cipher_generic, cipher_s390, data_size)) { 313 313 + pr_info("s390 vs generic check FAILED\n"); 314 314 + ret = -2; 315 315 + goto out; 316 316 + } 317 317 + else 318 318 + pr_info("s390 vs generic check OK\n"); 319 319 + 320 320 + memset(cipher_s390, 0, data_size); 321 321 + memset(revert, 0, data_size); 322 322 + 323 323 + /* Use chacha20 lib */ 324 324 + test_lib_chacha(revert, cipher_s390, plain); 325 325 + 326 326 + if (memcmp(plain, revert, data_size)) { 327 327 + pr_info("lib en/decryption check FAILED\n"); 328 328 + ret = -2; 329 329 + goto out; 330 330 + } 331 331 + else 332 332 + pr_info("lib en/decryption check OK\n"); 333 333 + 334 334 + if (memcmp(cipher_generic, cipher_s390, data_size)) { 335 335 + pr_info("lib vs generic check FAILED\n"); 336 336 + ret = -2; 337 337 + goto out; 338 338 + } 339 339 + else 340 340 + pr_info("lib vs generic check OK\n"); 341 341 + 342 342 + pr_info("--- chacha20 s390 test end ---\n"); 343 343 + 344 344 + out: 345 345 + if (plain) 346 346 + vfree(plain); 347 347 + if (cipher_generic) 348 348 + vfree(cipher_generic); 349 349 + if (cipher_s390) 350 350 + vfree(cipher_s390); 351 351 + if (revert) 352 352 + vfree(revert); 353 353 + 354 354 + return -1; 355 355 + } 356 356 + 357 357 + static void __exit chacha_s390_test_exit(void) 358 358 + { 359 359 + pr_info("s390 ChaCha20 test module exit\n"); 360 360 + } 361 361 + 362 362 + module_param_named(size, data_size, uint, 0660); 363 363 + module_param(debug, int, 0660); 364 364 + MODULE_PARM_DESC(size, "Size of a plaintext"); 365 365 + MODULE_PARM_DESC(debug, "Debug level (0=off,1=on)"); 366 366 + 367 367 + module_init(chacha_s390_test_init); 368 368 + module_exit(chacha_s390_test_exit); 369 369 + 370 370 + MODULE_DESCRIPTION("s390 ChaCha20 self-test"); 371 371 + MODULE_AUTHOR("Vladis Dronov <vdronoff@gmail.com>"); 372 372 + MODULE_LICENSE("GPL v2");