Merge tag 'riscv-for-linus-6.11-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

+50

Documentation/arch/riscv/hwprobe.rst

··· 192 192 supported as defined in the RISC-V ISA manual starting from commit 193 193 d8ab5c78c207 ("Zihintpause is ratified"). 194 194 195 + * :c:macro:`RISCV_HWPROBE_EXT_ZVE32X`: The Vector sub-extension Zve32x is 196 + supported, as defined by version 1.0 of the RISC-V Vector extension manual. 197 + 198 + * :c:macro:`RISCV_HWPROBE_EXT_ZVE32F`: The Vector sub-extension Zve32f is 199 + supported, as defined by version 1.0 of the RISC-V Vector extension manual. 200 + 201 + * :c:macro:`RISCV_HWPROBE_EXT_ZVE64X`: The Vector sub-extension Zve64x is 202 + supported, as defined by version 1.0 of the RISC-V Vector extension manual. 203 + 204 + * :c:macro:`RISCV_HWPROBE_EXT_ZVE64F`: The Vector sub-extension Zve64f is 205 + supported, as defined by version 1.0 of the RISC-V Vector extension manual. 206 + 207 + * :c:macro:`RISCV_HWPROBE_EXT_ZVE64D`: The Vector sub-extension Zve64d is 208 + supported, as defined by version 1.0 of the RISC-V Vector extension manual. 209 + 210 + * :c:macro:`RISCV_HWPROBE_EXT_ZIMOP`: The Zimop May-Be-Operations extension is 211 + supported as defined in the RISC-V ISA manual starting from commit 212 + 58220614a5f ("Zimop is ratified/1.0"). 213 + 214 + * :c:macro:`RISCV_HWPROBE_EXT_ZCA`: The Zca extension part of Zc* standard 215 + extensions for code size reduction, as ratified in commit 8be3419c1c0 216 + ("Zcf doesn't exist on RV64 as it contains no instructions") of 217 + riscv-code-size-reduction. 218 + 219 + * :c:macro:`RISCV_HWPROBE_EXT_ZCB`: The Zcb extension part of Zc* standard 220 + extensions for code size reduction, as ratified in commit 8be3419c1c0 221 + ("Zcf doesn't exist on RV64 as it contains no instructions") of 222 + riscv-code-size-reduction. 223 + 224 + * :c:macro:`RISCV_HWPROBE_EXT_ZCD`: The Zcd extension part of Zc* standard 225 + extensions for code size reduction, as ratified in commit 8be3419c1c0 226 + ("Zcf doesn't exist on RV64 as it contains no instructions") of 227 + riscv-code-size-reduction. 228 + 229 + * :c:macro:`RISCV_HWPROBE_EXT_ZCF`: The Zcf extension part of Zc* standard 230 + extensions for code size reduction, as ratified in commit 8be3419c1c0 231 + ("Zcf doesn't exist on RV64 as it contains no instructions") of 232 + riscv-code-size-reduction. 233 + 234 + * :c:macro:`RISCV_HWPROBE_EXT_ZCMOP`: The Zcmop May-Be-Operations extension is 235 + supported as defined in the RISC-V ISA manual starting from commit 236 + c732a4f39a4 ("Zcmop is ratified/1.0"). 237 + 238 + * :c:macro:`RISCV_HWPROBE_EXT_ZAWRS`: The Zawrs extension is supported as 239 + ratified in commit 98918c844281 ("Merge pull request #1217 from 240 + riscv/zawrs") of riscv-isa-manual. 241 + 195 242 * :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: A bitmask that contains performance 196 243 information about the selected set of processors. 197 244 ··· 261 214 262 215 * :c:macro:`RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE`: An unsigned int which 263 216 represents the size of the Zicboz block in bytes. 217 + 218 + * :c:macro:`RISCV_HWPROBE_KEY_HIGHEST_VIRT_ADDRESS`: An unsigned long which 219 + represent the highest userspace virtual address usable.

-52

Documentation/devicetree/bindings/interrupt-controller/riscv,cpu-intc.txt

··· 1 - RISC-V Hart-Level Interrupt Controller (HLIC) 2 - --------------------------------------------- 3 - 4 - RISC-V cores include Control Status Registers (CSRs) which are local to each 5 - CPU core (HART in RISC-V terminology) and can be read or written by software. 6 - Some of these CSRs are used to control local interrupts connected to the core. 7 - Every interrupt is ultimately routed through a hart's HLIC before it 8 - interrupts that hart. 9 - 10 - The RISC-V supervisor ISA manual specifies three interrupt sources that are 11 - attached to every HLIC: software interrupts, the timer interrupt, and external 12 - interrupts. Software interrupts are used to send IPIs between cores. The 13 - timer interrupt comes from an architecturally mandated real-time timer that is 14 - controlled via Supervisor Binary Interface (SBI) calls and CSR reads. External 15 - interrupts connect all other device interrupts to the HLIC, which are routed 16 - via the platform-level interrupt controller (PLIC). 17 - 18 - All RISC-V systems that conform to the supervisor ISA specification are 19 - required to have a HLIC with these three interrupt sources present. Since the 20 - interrupt map is defined by the ISA it's not listed in the HLIC's device tree 21 - entry, though external interrupt controllers (like the PLIC, for example) will 22 - need to define how their interrupts map to the relevant HLICs. This means 23 - a PLIC interrupt property will typically list the HLICs for all present HARTs 24 - in the system. 25 - 26 - Required properties: 27 - - compatible : "riscv,cpu-intc" 28 - - #interrupt-cells : should be <1>. The interrupt sources are defined by the 29 - RISC-V supervisor ISA manual, with only the following three interrupts being 30 - defined for supervisor mode: 31 - - Source 1 is the supervisor software interrupt, which can be sent by an SBI 32 - call and is reserved for use by software. 33 - - Source 5 is the supervisor timer interrupt, which can be configured by 34 - SBI calls and implements a one-shot timer. 35 - - Source 9 is the supervisor external interrupt, which chains to all other 36 - device interrupts. 37 - - interrupt-controller : Identifies the node as an interrupt controller 38 - 39 - Furthermore, this interrupt-controller MUST be embedded inside the cpu 40 - definition of the hart whose CSRs control these local interrupts. 41 - 42 - An example device tree entry for a HLIC is show below. 43 - 44 - cpu1: cpu@1 { 45 - compatible = "riscv"; 46 - ... 47 - cpu1-intc: interrupt-controller { 48 - #interrupt-cells = <1>; 49 - compatible = "sifive,fu540-c000-cpu-intc", "riscv,cpu-intc"; 50 - interrupt-controller; 51 - }; 52 - };

+73

Documentation/devicetree/bindings/interrupt-controller/riscv,cpu-intc.yaml

··· 1 + # SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause 2 + %YAML 1.2 3 + --- 4 + $id: http://devicetree.org/schemas/interrupt-controller/riscv,cpu-intc.yaml# 5 + $schema: http://devicetree.org/meta-schemas/core.yaml# 6 + 7 + title: RISC-V Hart-Level Interrupt Controller (HLIC) 8 + 9 + description: 10 + RISC-V cores include Control Status Registers (CSRs) which are local to 11 + each CPU core (HART in RISC-V terminology) and can be read or written by 12 + software. Some of these CSRs are used to control local interrupts connected 13 + to the core. Every interrupt is ultimately routed through a hart's HLIC 14 + before it interrupts that hart. 15 + 16 + The RISC-V supervisor ISA manual specifies three interrupt sources that are 17 + attached to every HLIC namely software interrupts, the timer interrupt, and 18 + external interrupts. Software interrupts are used to send IPIs between 19 + cores. The timer interrupt comes from an architecturally mandated real- 20 + time timer that is controlled via Supervisor Binary Interface (SBI) calls 21 + and CSR reads. External interrupts connect all other device interrupts to 22 + the HLIC, which are routed via the platform-level interrupt controller 23 + (PLIC). 24 + 25 + All RISC-V systems that conform to the supervisor ISA specification are 26 + required to have a HLIC with these three interrupt sources present. Since 27 + the interrupt map is defined by the ISA it's not listed in the HLIC's device 28 + tree entry, though external interrupt controllers (like the PLIC, for 29 + example) will need to define how their interrupts map to the relevant HLICs. 30 + This means a PLIC interrupt property will typically list the HLICs for all 31 + present HARTs in the system. 32 + 33 + maintainers: 34 + - Palmer Dabbelt <palmer@dabbelt.com> 35 + - Paul Walmsley <paul.walmsley@sifive.com> 36 + 37 + properties: 38 + compatible: 39 + oneOf: 40 + - items: 41 + - const: andestech,cpu-intc 42 + - const: riscv,cpu-intc 43 + - const: riscv,cpu-intc 44 + 45 + interrupt-controller: true 46 + 47 + '#interrupt-cells': 48 + const: 1 49 + description: | 50 + The interrupt sources are defined by the RISC-V supervisor ISA manual, 51 + with only the following three interrupts being defined for 52 + supervisor mode: 53 + - Source 1 is the supervisor software interrupt, which can be sent by 54 + an SBI call and is reserved for use by software. 55 + - Source 5 is the supervisor timer interrupt, which can be configured 56 + by SBI calls and implements a one-shot timer. 57 + - Source 9 is the supervisor external interrupt, which chains to all 58 + other device interrupts. 59 + 60 + required: 61 + - compatible 62 + - '#interrupt-cells' 63 + - interrupt-controller 64 + 65 + additionalProperties: false 66 + 67 + examples: 68 + - | 69 + interrupt-controller { 70 + #interrupt-cells = <1>; 71 + compatible = "riscv,cpu-intc"; 72 + interrupt-controller; 73 + };

+1 -20

Documentation/devicetree/bindings/riscv/cpus.yaml

··· 103 103 104 104 interrupt-controller: 105 105 type: object 106 - additionalProperties: false 107 - description: Describes the CPU's local interrupt controller 108 - 109 - properties: 110 - '#interrupt-cells': 111 - const: 1 112 - 113 - compatible: 114 - oneOf: 115 - - items: 116 - - const: andestech,cpu-intc 117 - - const: riscv,cpu-intc 118 - - const: riscv,cpu-intc 119 - 120 - interrupt-controller: true 121 - 122 - required: 123 - - '#interrupt-cells' 124 - - compatible 125 - - interrupt-controller 106 + $ref: /schemas/interrupt-controller/riscv,cpu-intc.yaml# 126 107 127 108 cpu-idle-states: 128 109 $ref: /schemas/types.yaml#/definitions/phandle-array

+132

Documentation/devicetree/bindings/riscv/extensions.yaml

··· 177 177 is supported as ratified at commit 5059e0ca641c ("update to 178 178 ratified") of the riscv-zacas. 179 179 180 + - const: zawrs 181 + description: | 182 + The Zawrs extension for entering a low-power state or for trapping 183 + to a hypervisor while waiting on a store to a memory location, as 184 + ratified in commit 98918c844281 ("Merge pull request #1217 from 185 + riscv/zawrs") of riscv-isa-manual. 186 + 180 187 - const: zba 181 188 description: | 182 189 The standard Zba bit-manipulation extension for address generation ··· 226 219 The standard Zbs bit-manipulation extension for single-bit 227 220 instructions as ratified at commit 6d33919 ("Merge pull request #158 228 221 from hirooih/clmul-fix-loop-end-condition") of riscv-bitmanip. 222 + 223 + - const: zca 224 + description: | 225 + The Zca extension part of Zc* standard extensions for code size 226 + reduction, as ratified in commit 8be3419c1c0 ("Zcf doesn't exist on 227 + RV64 as it contains no instructions") of riscv-code-size-reduction, 228 + merged in the riscv-isa-manual by commit dbc79cf28a2 ("Initial seed 229 + of zc.adoc to src tree."). 230 + 231 + - const: zcb 232 + description: | 233 + The Zcb extension part of Zc* standard extensions for code size 234 + reduction, as ratified in commit 8be3419c1c0 ("Zcf doesn't exist on 235 + RV64 as it contains no instructions") of riscv-code-size-reduction, 236 + merged in the riscv-isa-manual by commit dbc79cf28a2 ("Initial seed 237 + of zc.adoc to src tree."). 238 + 239 + - const: zcd 240 + description: | 241 + The Zcd extension part of Zc* standard extensions for code size 242 + reduction, as ratified in commit 8be3419c1c0 ("Zcf doesn't exist on 243 + RV64 as it contains no instructions") of riscv-code-size-reduction, 244 + merged in the riscv-isa-manual by commit dbc79cf28a2 ("Initial seed 245 + of zc.adoc to src tree."). 246 + 247 + - const: zcf 248 + description: | 249 + The Zcf extension part of Zc* standard extensions for code size 250 + reduction, as ratified in commit 8be3419c1c0 ("Zcf doesn't exist on 251 + RV64 as it contains no instructions") of riscv-code-size-reduction, 252 + merged in the riscv-isa-manual by commit dbc79cf28a2 ("Initial seed 253 + of zc.adoc to src tree."). 254 + 255 + - const: zcmop 256 + description: 257 + The standard Zcmop extension version 1.0, as ratified in commit 258 + c732a4f39a4 ("Zcmop is ratified/1.0") of the riscv-isa-manual. 229 259 230 260 - const: zfa 231 261 description: ··· 407 363 ratified in the 20191213 version of the unprivileged ISA 408 364 specification. 409 365 366 + - const: zimop 367 + description: 368 + The standard Zimop extension version 1.0, as ratified in commit 369 + 58220614a5f ("Zimop is ratified/1.0") of the riscv-isa-manual. 370 + 410 371 - const: ztso 411 372 description: 412 373 The standard Ztso extension for total store ordering, as ratified ··· 429 380 The standard Zvbc extension for vectored carryless multiplication 430 381 instructions, as ratified in commit 56ed795 ("Update 431 382 riscv-crypto-spec-vector.adoc") of riscv-crypto. 383 + 384 + - const: zve32f 385 + description: 386 + The standard Zve32f extension for embedded processors, as ratified 387 + in commit 6f702a2 ("Vector extensions are now ratified") of 388 + riscv-v-spec. 389 + 390 + - const: zve32x 391 + description: 392 + The standard Zve32x extension for embedded processors, as ratified 393 + in commit 6f702a2 ("Vector extensions are now ratified") of 394 + riscv-v-spec. 395 + 396 + - const: zve64d 397 + description: 398 + The standard Zve64d extension for embedded processors, as ratified 399 + in commit 6f702a2 ("Vector extensions are now ratified") of 400 + riscv-v-spec. 401 + 402 + - const: zve64f 403 + description: 404 + The standard Zve64f extension for embedded processors, as ratified 405 + in commit 6f702a2 ("Vector extensions are now ratified") of 406 + riscv-v-spec. 407 + 408 + - const: zve64x 409 + description: 410 + The standard Zve64x extension for embedded processors, as ratified 411 + in commit 6f702a2 ("Vector extensions are now ratified") of 412 + riscv-v-spec. 432 413 433 414 - const: zvfh 434 415 description: ··· 562 483 and privilege mode filtering. For more details, see Counter Related 563 484 Registers in the AX45MP datasheet. 564 485 https://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf 486 + 487 + allOf: 488 + # Zcb depends on Zca 489 + - if: 490 + contains: 491 + const: zcb 492 + then: 493 + contains: 494 + const: zca 495 + # Zcd depends on Zca and D 496 + - if: 497 + contains: 498 + const: zcd 499 + then: 500 + allOf: 501 + - contains: 502 + const: zca 503 + - contains: 504 + const: d 505 + # Zcf depends on Zca and F 506 + - if: 507 + contains: 508 + const: zcf 509 + then: 510 + allOf: 511 + - contains: 512 + const: zca 513 + - contains: 514 + const: f 515 + # Zcmop depends on Zca 516 + - if: 517 + contains: 518 + const: zcmop 519 + then: 520 + contains: 521 + const: zca 522 + 523 + allOf: 524 + # Zcf extension does not exist on rv64 525 + - if: 526 + properties: 527 + riscv,isa-extensions: 528 + contains: 529 + const: zcf 530 + riscv,isa-base: 531 + contains: 532 + const: rv64i 533 + then: 534 + properties: 535 + riscv,isa-extensions: 536 + not: 537 + contains: 538 + const: zcf 565 539 566 540 additionalProperties: true 567 541 ...

+52 -7

arch/riscv/Kconfig

··· 16 16 select ACPI_REDUCED_HARDWARE_ONLY if ACPI 17 17 select ARCH_DMA_DEFAULT_COHERENT 18 18 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION 19 + select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP 20 + select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG 19 21 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 20 22 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE 21 23 select ARCH_HAS_BINFMT_FLAT ··· 37 35 select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE 38 36 select ARCH_HAS_PMEM_API 39 37 select ARCH_HAS_PREPARE_SYNC_CORE_CMD 38 + select ARCH_HAS_PTE_DEVMAP if 64BIT && MMU 40 39 select ARCH_HAS_PTE_SPECIAL 41 40 select ARCH_HAS_SET_DIRECT_MAP if MMU 42 41 select ARCH_HAS_SET_MEMORY if MMU ··· 49 46 select ARCH_HAS_UBSAN 50 47 select ARCH_HAS_VDSO_DATA 51 48 select ARCH_KEEP_MEMBLOCK if ACPI 49 + select ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE if 64BIT && MMU 52 50 select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX 53 51 select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT 54 52 select ARCH_STACKWALK ··· 73 69 select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT 74 70 select ARCH_WANT_HUGE_PMD_SHARE if 64BIT 75 71 select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL 72 + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP 76 73 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP 77 74 select ARCH_WANTS_NO_INSTR 78 75 select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE ··· 600 595 preemption. Enabling this config will result in higher memory 601 596 consumption due to the allocation of per-task's kernel Vector context. 602 597 598 + config RISCV_ISA_ZAWRS 599 + bool "Zawrs extension support for more efficient busy waiting" 600 + depends on RISCV_ALTERNATIVE 601 + default y 602 + help 603 + The Zawrs extension defines instructions to be used in polling loops 604 + which allow a hart to enter a low-power state or to trap to the 605 + hypervisor while waiting on a store to a memory location. Enable the 606 + use of these instructions in the kernel when the Zawrs extension is 607 + detected at boot. 608 + 609 + If you don't know what to do here, say Y. 610 + 603 611 config TOOLCHAIN_HAS_ZBB 604 612 bool 605 613 default y ··· 655 637 656 638 If you don't know what to do here, say Y. 657 639 640 + config TOOLCHAIN_HAS_ZBC 641 + bool 642 + default y 643 + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zbc) 644 + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zbc) 645 + depends on LLD_VERSION >= 150000 || LD_VERSION >= 23900 646 + depends on AS_HAS_OPTION_ARCH 647 + 648 + config RISCV_ISA_ZBC 649 + bool "Zbc extension support for carry-less multiplication instructions" 650 + depends on TOOLCHAIN_HAS_ZBC 651 + depends on MMU 652 + depends on RISCV_ALTERNATIVE 653 + default y 654 + help 655 + Adds support to dynamically detect the presence of the Zbc 656 + extension (carry-less multiplication) and enable its usage. 657 + 658 + The Zbc extension could accelerate CRC (cyclic redundancy check) 659 + calculations. 660 + 661 + If you don't know what to do here, say Y. 662 + 658 663 config RISCV_ISA_ZICBOM 659 664 bool "Zicbom extension support for non-coherent DMA operation" 660 665 depends on MMU ··· 706 665 The Zicboz extension is used for faster zeroing of memory. 707 666 708 667 If you don't know what to do here, say Y. 709 - 710 - config TOOLCHAIN_HAS_ZIHINTPAUSE 711 - bool 712 - default y 713 - depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zihintpause) 714 - depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zihintpause) 715 - depends on LLD_VERSION >= 150000 || LD_VERSION >= 23600 716 668 717 669 config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI 718 670 def_bool y ··· 1012 978 clock, and platform reset). A UEFI stub is also provided to 1013 979 allow the kernel to be booted as an EFI application. This 1014 980 is only useful on systems that have UEFI firmware. 981 + 982 + config DMI 983 + bool "Enable support for SMBIOS (DMI) tables" 984 + depends on EFI 985 + default y 986 + help 987 + This enables SMBIOS/DMI feature for systems. 988 + 989 + This option is only useful on systems that have UEFI firmware. 990 + However, even with this option, the resultant kernel should 991 + continue to boot on existing non-UEFI platforms. 1015 992 1016 993 config CC_HAVE_STACKPROTECTOR_TLS 1017 994 def_bool $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=tp -mstack-protector-guard-offset=0)

-3

arch/riscv/Makefile

··· 82 82 riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei 83 83 endif 84 84 85 - # Check if the toolchain supports Zihintpause extension 86 - riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause 87 - 88 85 # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by 89 86 # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) 90 87 KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')

+4 -4

arch/riscv/include/asm/arch_hweight.h

··· 26 26 27 27 asm (".option push\n" 28 28 ".option arch,+zbb\n" 29 - CPOPW "%0, %0\n" 29 + CPOPW "%0, %1\n" 30 30 ".option pop\n" 31 - : "+r" (w) : :); 31 + : "=r" (w) : "r" (w) :); 32 32 33 33 return w; 34 34 ··· 57 57 58 58 asm (".option push\n" 59 59 ".option arch,+zbb\n" 60 - "cpop %0, %0\n" 60 + "cpop %0, %1\n" 61 61 ".option pop\n" 62 - : "+r" (w) : :); 62 + : "=r" (w) : "r" (w) :); 63 63 64 64 return w; 65 65

+30 -15

arch/riscv/include/asm/barrier.h

··· 11 11 #define _ASM_RISCV_BARRIER_H 12 12 13 13 #ifndef __ASSEMBLY__ 14 + #include <asm/cmpxchg.h> 14 15 #include <asm/fence.h> 15 16 16 17 #define nop() __asm__ __volatile__ ("nop") ··· 28 27 #define __smp_mb() RISCV_FENCE(rw, rw) 29 28 #define __smp_rmb() RISCV_FENCE(r, r) 30 29 #define __smp_wmb() RISCV_FENCE(w, w) 31 - 32 - #define __smp_store_release(p, v) \ 33 - do { \ 34 - compiletime_assert_atomic_type(*p); \ 35 - RISCV_FENCE(rw, w); \ 36 - WRITE_ONCE(*p, v); \ 37 - } while (0) 38 - 39 - #define __smp_load_acquire(p) \ 40 - ({ \ 41 - typeof(*p) ___p1 = READ_ONCE(*p); \ 42 - compiletime_assert_atomic_type(*p); \ 43 - RISCV_FENCE(r, rw); \ 44 - ___p1; \ 45 - }) 46 30 47 31 /* 48 32 * This is a very specific barrier: it's currently only used in two places in ··· 55 69 * the new hart. 56 70 */ 57 71 #define smp_mb__after_spinlock() RISCV_FENCE(iorw, iorw) 72 + 73 + #define __smp_store_release(p, v) \ 74 + do { \ 75 + compiletime_assert_atomic_type(*p); \ 76 + RISCV_FENCE(rw, w); \ 77 + WRITE_ONCE(*p, v); \ 78 + } while (0) 79 + 80 + #define __smp_load_acquire(p) \ 81 + ({ \ 82 + typeof(*p) ___p1 = READ_ONCE(*p); \ 83 + compiletime_assert_atomic_type(*p); \ 84 + RISCV_FENCE(r, rw); \ 85 + ___p1; \ 86 + }) 87 + 88 + #ifdef CONFIG_RISCV_ISA_ZAWRS 89 + #define smp_cond_load_relaxed(ptr, cond_expr) ({ \ 90 + typeof(ptr) __PTR = (ptr); \ 91 + __unqual_scalar_typeof(*ptr) VAL; \ 92 + for (;;) { \ 93 + VAL = READ_ONCE(*__PTR); \ 94 + if (cond_expr) \ 95 + break; \ 96 + __cmpwait_relaxed(ptr, VAL); \ 97 + } \ 98 + (typeof(*ptr))VAL; \ 99 + }) 100 + #endif 58 101 59 102 #include <asm-generic/barrier.h> 60 103

+58

arch/riscv/include/asm/cmpxchg.h

··· 8 8 9 9 #include <linux/bug.h> 10 10 11 + #include <asm/alternative-macros.h> 11 12 #include <asm/fence.h> 13 + #include <asm/hwcap.h> 14 + #include <asm/insn-def.h> 12 15 13 16 #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ 14 17 ({ \ ··· 225 222 BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ 226 223 arch_cmpxchg_release((ptr), (o), (n)); \ 227 224 }) 225 + 226 + #ifdef CONFIG_RISCV_ISA_ZAWRS 227 + /* 228 + * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to 229 + * @val we expect it to still terminate within a "reasonable" amount of time 230 + * for an implementation-specific other reason, a pending, locally-enabled 231 + * interrupt, or because it has been configured to raise an illegal 232 + * instruction exception. 233 + */ 234 + static __always_inline void __cmpwait(volatile void *ptr, 235 + unsigned long val, 236 + int size) 237 + { 238 + unsigned long tmp; 239 + 240 + asm goto(ALTERNATIVE("j %l[no_zawrs]", "nop", 241 + 0, RISCV_ISA_EXT_ZAWRS, 1) 242 + : : : : no_zawrs); 243 + 244 + switch (size) { 245 + case 4: 246 + asm volatile( 247 + " lr.w %0, %1\n" 248 + " xor %0, %0, %2\n" 249 + " bnez %0, 1f\n" 250 + ZAWRS_WRS_NTO "\n" 251 + "1:" 252 + : "=&r" (tmp), "+A" (*(u32 *)ptr) 253 + : "r" (val)); 254 + break; 255 + #if __riscv_xlen == 64 256 + case 8: 257 + asm volatile( 258 + " lr.d %0, %1\n" 259 + " xor %0, %0, %2\n" 260 + " bnez %0, 1f\n" 261 + ZAWRS_WRS_NTO "\n" 262 + "1:" 263 + : "=&r" (tmp), "+A" (*(u64 *)ptr) 264 + : "r" (val)); 265 + break; 266 + #endif 267 + default: 268 + BUILD_BUG(); 269 + } 270 + 271 + return; 272 + 273 + no_zawrs: 274 + asm volatile(RISCV_PAUSE : : : "memory"); 275 + } 276 + 277 + #define __cmpwait_relaxed(ptr, val) \ 278 + __cmpwait((ptr), (unsigned long)(val), sizeof(*(ptr))) 279 + #endif 228 280 229 281 #endif /* _ASM_RISCV_CMPXCHG_H */

+1

arch/riscv/include/asm/cpufeature.h

··· 70 70 const char *property; 71 71 const unsigned int *subset_ext_ids; 72 72 const unsigned int subset_ext_size; 73 + int (*validate)(const struct riscv_isa_ext_data *data, const unsigned long *isa_bitmap); 73 74 }; 74 75 75 76 extern const struct riscv_isa_ext_data riscv_isa_ext[];

+24

arch/riscv/include/asm/dmi.h

··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + /* 3 + * Copyright (C) 2024 Intel Corporation 4 + * 5 + * based on arch/arm64/include/asm/dmi.h 6 + * 7 + * This file is subject to the terms and conditions of the GNU General Public 8 + * License. See the file "COPYING" in the main directory of this archive 9 + * for more details. 10 + */ 11 + 12 + #ifndef __ASM_DMI_H 13 + #define __ASM_DMI_H 14 + 15 + #include <linux/io.h> 16 + #include <linux/slab.h> 17 + 18 + #define dmi_early_remap(x, l) memremap(x, l, MEMREMAP_WB) 19 + #define dmi_early_unmap(x, l) memunmap(x) 20 + #define dmi_remap(x, l) memremap(x, l, MEMREMAP_WB) 21 + #define dmi_unmap(x) memunmap(x) 22 + #define dmi_alloc(l) kzalloc(l, GFP_KERNEL) 23 + 24 + #endif

+12

arch/riscv/include/asm/hwcap.h

··· 81 81 #define RISCV_ISA_EXT_ZTSO 72 82 82 #define RISCV_ISA_EXT_ZACAS 73 83 83 #define RISCV_ISA_EXT_XANDESPMU 74 84 + #define RISCV_ISA_EXT_ZVE32X 75 85 + #define RISCV_ISA_EXT_ZVE32F 76 86 + #define RISCV_ISA_EXT_ZVE64X 77 87 + #define RISCV_ISA_EXT_ZVE64F 78 88 + #define RISCV_ISA_EXT_ZVE64D 79 89 + #define RISCV_ISA_EXT_ZIMOP 80 90 + #define RISCV_ISA_EXT_ZCA 81 91 + #define RISCV_ISA_EXT_ZCB 82 92 + #define RISCV_ISA_EXT_ZCD 83 93 + #define RISCV_ISA_EXT_ZCF 84 94 + #define RISCV_ISA_EXT_ZCMOP 85 95 + #define RISCV_ISA_EXT_ZAWRS 86 84 96 85 97 #define RISCV_ISA_EXT_XLINUXENVCFG 127 86 98

+1 -1

arch/riscv/include/asm/hwprobe.h

··· 8 8 9 9 #include <uapi/asm/hwprobe.h> 10 10 11 - #define RISCV_HWPROBE_MAX_KEY 6 11 + #define RISCV_HWPROBE_MAX_KEY 7 12 12 13 13 static inline bool riscv_hwprobe_key_is_valid(__s64 key) 14 14 {

+4

arch/riscv/include/asm/insn-def.h

··· 196 196 INSN_I(OPCODE_MISC_MEM, FUNC3(2), __RD(0), \ 197 197 RS1(base), SIMM12(4)) 198 198 199 + #define RISCV_PAUSE ".4byte 0x100000f" 200 + #define ZAWRS_WRS_NTO ".4byte 0x00d00073" 201 + #define ZAWRS_WRS_STO ".4byte 0x01d00073" 202 + 199 203 #endif /* __ASM_INSN_DEF_H */

+3 -1

arch/riscv/include/asm/jump_label.h

··· 12 12 #include <linux/types.h> 13 13 #include <asm/asm.h> 14 14 15 + #define HAVE_JUMP_LABEL_BATCH 16 + 15 17 #define JUMP_LABEL_NOP_SIZE 4 16 18 17 19 static __always_inline bool arch_static_branch(struct static_key * const key, ··· 46 44 " .option push \n\t" 47 45 " .option norelax \n\t" 48 46 " .option norvc \n\t" 49 - "1: jal zero, %l[label] \n\t" 47 + "1: j %l[label] \n\t" 50 48 " .option pop \n\t" 51 49 " .pushsection __jump_table, \"aw\" \n\t" 52 50 " .align " RISCV_LGPTR " \n\t"

+2 -2

arch/riscv/include/asm/kasan.h

··· 6 6 7 7 #ifndef __ASSEMBLY__ 8 8 9 - #ifdef CONFIG_KASAN 10 - 11 9 /* 12 10 * The following comment was copied from arm64: 13 11 * KASAN_SHADOW_START: beginning of the kernel virtual addresses. ··· 32 34 */ 33 35 #define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK) 34 36 #define KASAN_SHADOW_END MODULES_LOWEST_VADDR 37 + 38 + #ifdef CONFIG_KASAN 35 39 #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) 36 40 37 41 void kasan_init(void);

+1

arch/riscv/include/asm/kvm_host.h

··· 80 80 struct kvm_vcpu_stat_generic generic; 81 81 u64 ecall_exit_stat; 82 82 u64 wfi_exit_stat; 83 + u64 wrs_exit_stat; 83 84 u64 mmio_exit_user; 84 85 u64 mmio_exit_kernel; 85 86 u64 csr_exit_user;

+2 -2

arch/riscv/include/asm/mmu.h

··· 31 31 #define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) 32 32 #define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) 33 33 34 - void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, 35 - phys_addr_t sz, pgprot_t prot); 34 + void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, 35 + pgprot_t prot); 36 36 #endif /* __ASSEMBLY__ */ 37 37 38 38 #endif /* _ASM_RISCV_MMU_H */

+5

arch/riscv/include/asm/page.h

··· 188 188 189 189 unsigned long kaslr_offset(void); 190 190 191 + static __always_inline void *pfn_to_kaddr(unsigned long pfn) 192 + { 193 + return __va(pfn << PAGE_SHIFT); 194 + } 195 + 191 196 #endif /* __ASSEMBLY__ */ 192 197 193 198 #define virt_addr_valid(vaddr) ({ \

+1 -1

arch/riscv/include/asm/patch.h

··· 9 9 int patch_insn_write(void *addr, const void *insn, size_t len); 10 10 int patch_text_nosync(void *addr, const void *insns, size_t len); 11 11 int patch_text_set_nosync(void *addr, u8 c, size_t len); 12 - int patch_text(void *addr, u32 *insns, int ninsns); 12 + int patch_text(void *addr, u32 *insns, size_t len); 13 13 14 14 extern int riscv_patch_in_stop_machine; 15 15

+20

arch/riscv/include/asm/pgtable-64.h

··· 398 398 #define p4d_offset p4d_offset 399 399 p4d_t *p4d_offset(pgd_t *pgd, unsigned long address); 400 400 401 + #ifdef CONFIG_TRANSPARENT_HUGEPAGE 402 + static inline int pte_devmap(pte_t pte); 403 + static inline pte_t pmd_pte(pmd_t pmd); 404 + 405 + static inline int pmd_devmap(pmd_t pmd) 406 + { 407 + return pte_devmap(pmd_pte(pmd)); 408 + } 409 + 410 + static inline int pud_devmap(pud_t pud) 411 + { 412 + return 0; 413 + } 414 + 415 + static inline int pgd_devmap(pgd_t pgd) 416 + { 417 + return 0; 418 + } 419 + #endif 420 + 401 421 #endif /* _ASM_RISCV_PGTABLE_64_H */

+1

arch/riscv/include/asm/pgtable-bits.h

··· 19 19 #define _PAGE_SOFT (3 << 8) /* Reserved for software */ 20 20 21 21 #define _PAGE_SPECIAL (1 << 8) /* RSW: 0x1 */ 22 + #define _PAGE_DEVMAP (1 << 9) /* RSW, devmap */ 22 23 #define _PAGE_TABLE _PAGE_PRESENT 23 24 24 25 /*

+31 -1

arch/riscv/include/asm/pgtable.h

··· 165 165 #endif 166 166 }; 167 167 168 - extern struct pt_alloc_ops pt_ops __initdata; 168 + extern struct pt_alloc_ops pt_ops __meminitdata; 169 169 170 170 #ifdef CONFIG_MMU 171 171 /* Number of PGD entries that a user-mode program can use */ ··· 350 350 return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); 351 351 } 352 352 353 + #define pte_accessible pte_accessible 354 + static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a) 355 + { 356 + if (pte_val(a) & _PAGE_PRESENT) 357 + return true; 358 + 359 + if ((pte_val(a) & _PAGE_PROT_NONE) && 360 + atomic_read(&mm->tlb_flush_pending)) 361 + return true; 362 + 363 + return false; 364 + } 365 + 353 366 static inline int pte_none(pte_t pte) 354 367 { 355 368 return (pte_val(pte) == 0); ··· 403 390 return pte_val(pte) & _PAGE_SPECIAL; 404 391 } 405 392 393 + #ifdef CONFIG_ARCH_HAS_PTE_DEVMAP 394 + static inline int pte_devmap(pte_t pte) 395 + { 396 + return pte_val(pte) & _PAGE_DEVMAP; 397 + } 398 + #endif 399 + 406 400 /* static inline pte_t pte_rdprotect(pte_t pte) */ 407 401 408 402 static inline pte_t pte_wrprotect(pte_t pte) ··· 449 429 static inline pte_t pte_mkspecial(pte_t pte) 450 430 { 451 431 return __pte(pte_val(pte) | _PAGE_SPECIAL); 432 + } 433 + 434 + static inline pte_t pte_mkdevmap(pte_t pte) 435 + { 436 + return __pte(pte_val(pte) | _PAGE_DEVMAP); 452 437 } 453 438 454 439 static inline pte_t pte_mkhuge(pte_t pte) ··· 744 719 static inline pmd_t pmd_mkdirty(pmd_t pmd) 745 720 { 746 721 return pte_pmd(pte_mkdirty(pmd_pte(pmd))); 722 + } 723 + 724 + static inline pmd_t pmd_mkdevmap(pmd_t pmd) 725 + { 726 + return pte_pmd(pte_mkdevmap(pmd_pte(pmd))); 747 727 } 748 728 749 729 static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,

+6

arch/riscv/include/asm/processor.h

··· 57 57 58 58 #define STACK_TOP DEFAULT_MAP_WINDOW 59 59 60 + #ifdef CONFIG_MMU 61 + #define user_max_virt_addr() arch_get_mmap_end(ULONG_MAX, 0, 0) 62 + #else 63 + #define user_max_virt_addr() 0 64 + #endif /* CONFIG_MMU */ 65 + 60 66 /* 61 67 * This decides where the kernel will search for a free chunk of vm 62 68 * space during mmap's.

+6 -4

arch/riscv/include/asm/sbi.h

··· 304 304 }; 305 305 306 306 void sbi_init(void); 307 - struct sbiret sbi_ecall(int ext, int fid, unsigned long arg0, 308 - unsigned long arg1, unsigned long arg2, 309 - unsigned long arg3, unsigned long arg4, 310 - unsigned long arg5); 307 + struct sbiret __sbi_ecall(unsigned long arg0, unsigned long arg1, 308 + unsigned long arg2, unsigned long arg3, 309 + unsigned long arg4, unsigned long arg5, 310 + int fid, int ext); 311 + #define sbi_ecall(e, f, a0, a1, a2, a3, a4, a5) \ 312 + __sbi_ecall(a0, a1, a2, a3, a4, a5, f, e) 311 313 312 314 #ifdef CONFIG_RISCV_SBI_V01 313 315 void sbi_console_putchar(int ch);

+54

arch/riscv/include/asm/trace.h

··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + #undef TRACE_SYSTEM 3 + #define TRACE_SYSTEM riscv 4 + 5 + #if !defined(_TRACE_RISCV_H) || defined(TRACE_HEADER_MULTI_READ) 6 + #define _TRACE_RISCV_H 7 + 8 + #include <linux/tracepoint.h> 9 + 10 + TRACE_EVENT_CONDITION(sbi_call, 11 + TP_PROTO(int ext, int fid), 12 + TP_ARGS(ext, fid), 13 + TP_CONDITION(ext != SBI_EXT_HSM), 14 + 15 + TP_STRUCT__entry( 16 + __field(int, ext) 17 + __field(int, fid) 18 + ), 19 + 20 + TP_fast_assign( 21 + __entry->ext = ext; 22 + __entry->fid = fid; 23 + ), 24 + 25 + TP_printk("ext=0x%x fid=%d", __entry->ext, __entry->fid) 26 + ); 27 + 28 + TRACE_EVENT_CONDITION(sbi_return, 29 + TP_PROTO(int ext, long error, long value), 30 + TP_ARGS(ext, error, value), 31 + TP_CONDITION(ext != SBI_EXT_HSM), 32 + 33 + TP_STRUCT__entry( 34 + __field(long, error) 35 + __field(long, value) 36 + ), 37 + 38 + TP_fast_assign( 39 + __entry->error = error; 40 + __entry->value = value; 41 + ), 42 + 43 + TP_printk("error=%ld value=0x%lx", __entry->error, __entry->value) 44 + ); 45 + 46 + #endif /* _TRACE_RISCV_H */ 47 + 48 + #undef TRACE_INCLUDE_PATH 49 + #undef TRACE_INCLUDE_FILE 50 + 51 + #define TRACE_INCLUDE_PATH asm 52 + #define TRACE_INCLUDE_FILE trace 53 + 54 + #include <trace/define_trace.h>

+2 -6

arch/riscv/include/asm/vdso/processor.h

··· 5 5 #ifndef __ASSEMBLY__ 6 6 7 7 #include <asm/barrier.h> 8 + #include <asm/insn-def.h> 8 9 9 10 static inline void cpu_relax(void) 10 11 { ··· 15 14 __asm__ __volatile__ ("div %0, %0, zero" : "=r" (dummy)); 16 15 #endif 17 16 18 - #ifdef CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE 19 17 /* 20 18 * Reduce instruction retirement. 21 19 * This assumes the PC changes. 22 20 */ 23 - __asm__ __volatile__ ("pause"); 24 - #else 25 - /* Encoding of the pause instruction */ 26 - __asm__ __volatile__ (".4byte 0x100000F"); 27 - #endif 21 + __asm__ __volatile__ (RISCV_PAUSE); 28 22 barrier(); 29 23 } 30 24

+5 -5

arch/riscv/include/asm/vector.h

··· 37 37 38 38 static __always_inline bool has_vector(void) 39 39 { 40 - return riscv_has_extension_unlikely(RISCV_ISA_EXT_v); 40 + return riscv_has_extension_unlikely(RISCV_ISA_EXT_ZVE32X); 41 41 } 42 42 43 43 static inline void __riscv_v_vstate_clean(struct pt_regs *regs) ··· 91 91 { 92 92 asm volatile ( 93 93 ".option push\n\t" 94 - ".option arch, +v\n\t" 94 + ".option arch, +zve32x\n\t" 95 95 "vsetvl x0, %2, %1\n\t" 96 96 ".option pop\n\t" 97 97 "csrw " __stringify(CSR_VSTART) ", %0\n\t" ··· 109 109 __vstate_csr_save(save_to); 110 110 asm volatile ( 111 111 ".option push\n\t" 112 - ".option arch, +v\n\t" 112 + ".option arch, +zve32x\n\t" 113 113 "vsetvli %0, x0, e8, m8, ta, ma\n\t" 114 114 "vse8.v v0, (%1)\n\t" 115 115 "add %1, %1, %0\n\t" ··· 131 131 riscv_v_enable(); 132 132 asm volatile ( 133 133 ".option push\n\t" 134 - ".option arch, +v\n\t" 134 + ".option arch, +zve32x\n\t" 135 135 "vsetvli %0, x0, e8, m8, ta, ma\n\t" 136 136 "vle8.v v0, (%1)\n\t" 137 137 "add %1, %1, %0\n\t" ··· 153 153 riscv_v_enable(); 154 154 asm volatile ( 155 155 ".option push\n\t" 156 - ".option arch, +v\n\t" 156 + ".option arch, +zve32x\n\t" 157 157 "vsetvli %0, x0, e8, m8, ta, ma\n\t" 158 158 "vmv.v.i v0, -1\n\t" 159 159 "vmv.v.i v8, -1\n\t"

+13

arch/riscv/include/uapi/asm/hwprobe.h

··· 60 60 #define RISCV_HWPROBE_EXT_ZACAS (1ULL << 34) 61 61 #define RISCV_HWPROBE_EXT_ZICOND (1ULL << 35) 62 62 #define RISCV_HWPROBE_EXT_ZIHINTPAUSE (1ULL << 36) 63 + #define RISCV_HWPROBE_EXT_ZVE32X (1ULL << 37) 64 + #define RISCV_HWPROBE_EXT_ZVE32F (1ULL << 38) 65 + #define RISCV_HWPROBE_EXT_ZVE64X (1ULL << 39) 66 + #define RISCV_HWPROBE_EXT_ZVE64F (1ULL << 40) 67 + #define RISCV_HWPROBE_EXT_ZVE64D (1ULL << 41) 68 + #define RISCV_HWPROBE_EXT_ZIMOP (1ULL << 42) 69 + #define RISCV_HWPROBE_EXT_ZCA (1ULL << 43) 70 + #define RISCV_HWPROBE_EXT_ZCB (1ULL << 44) 71 + #define RISCV_HWPROBE_EXT_ZCD (1ULL << 45) 72 + #define RISCV_HWPROBE_EXT_ZCF (1ULL << 46) 73 + #define RISCV_HWPROBE_EXT_ZCMOP (1ULL << 47) 74 + #define RISCV_HWPROBE_EXT_ZAWRS (1ULL << 48) 63 75 #define RISCV_HWPROBE_KEY_CPUPERF_0 5 64 76 #define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) 65 77 #define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) ··· 80 68 #define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0) 81 69 #define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) 82 70 #define RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE 6 71 + #define RISCV_HWPROBE_KEY_HIGHEST_VIRT_ADDRESS 7 83 72 /* Increase RISCV_HWPROBE_MAX_KEY when adding items. */ 84 73 85 74 /* Flags */

+7

arch/riscv/include/uapi/asm/kvm.h

··· 168 168 KVM_RISCV_ISA_EXT_ZTSO, 169 169 KVM_RISCV_ISA_EXT_ZACAS, 170 170 KVM_RISCV_ISA_EXT_SSCOFPMF, 171 + KVM_RISCV_ISA_EXT_ZIMOP, 172 + KVM_RISCV_ISA_EXT_ZCA, 173 + KVM_RISCV_ISA_EXT_ZCB, 174 + KVM_RISCV_ISA_EXT_ZCD, 175 + KVM_RISCV_ISA_EXT_ZCF, 176 + KVM_RISCV_ISA_EXT_ZCMOP, 177 + KVM_RISCV_ISA_EXT_ZAWRS, 171 178 KVM_RISCV_ISA_EXT_MAX, 172 179 }; 173 180

+242 -92

arch/riscv/kernel/cpufeature.c

··· 72 72 } 73 73 EXPORT_SYMBOL_GPL(__riscv_isa_extension_available); 74 74 75 - static bool riscv_isa_extension_check(int id) 75 + static int riscv_ext_zicbom_validate(const struct riscv_isa_ext_data *data, 76 + const unsigned long *isa_bitmap) 76 77 { 77 - switch (id) { 78 - case RISCV_ISA_EXT_ZICBOM: 79 - if (!riscv_cbom_block_size) { 80 - pr_err("Zicbom detected in ISA string, disabling as no cbom-block-size found\n"); 81 - return false; 82 - } else if (!is_power_of_2(riscv_cbom_block_size)) { 83 - pr_err("Zicbom disabled as cbom-block-size present, but is not a power-of-2\n"); 84 - return false; 85 - } 86 - return true; 87 - case RISCV_ISA_EXT_ZICBOZ: 88 - if (!riscv_cboz_block_size) { 89 - pr_err("Zicboz detected in ISA string, disabling as no cboz-block-size found\n"); 90 - return false; 91 - } else if (!is_power_of_2(riscv_cboz_block_size)) { 92 - pr_err("Zicboz disabled as cboz-block-size present, but is not a power-of-2\n"); 93 - return false; 94 - } 95 - return true; 96 - case RISCV_ISA_EXT_INVALID: 97 - return false; 78 + if (!riscv_cbom_block_size) { 79 + pr_err("Zicbom detected in ISA string, disabling as no cbom-block-size found\n"); 80 + return -EINVAL; 98 81 } 99 - 100 - return true; 82 + if (!is_power_of_2(riscv_cbom_block_size)) { 83 + pr_err("Zicbom disabled as cbom-block-size present, but is not a power-of-2\n"); 84 + return -EINVAL; 85 + } 86 + return 0; 101 87 } 102 88 103 - #define _RISCV_ISA_EXT_DATA(_name, _id, _subset_exts, _subset_exts_size) { \ 104 - .name = #_name, \ 105 - .property = #_name, \ 106 - .id = _id, \ 107 - .subset_ext_ids = _subset_exts, \ 108 - .subset_ext_size = _subset_exts_size \ 89 + static int riscv_ext_zicboz_validate(const struct riscv_isa_ext_data *data, 90 + const unsigned long *isa_bitmap) 91 + { 92 + if (!riscv_cboz_block_size) { 93 + pr_err("Zicboz detected in ISA string, disabling as no cboz-block-size found\n"); 94 + return -EINVAL; 95 + } 96 + if (!is_power_of_2(riscv_cboz_block_size)) { 97 + pr_err("Zicboz disabled as cboz-block-size present, but is not a power-of-2\n"); 98 + return -EINVAL; 99 + } 100 + return 0; 109 101 } 110 102 111 - #define __RISCV_ISA_EXT_DATA(_name, _id) _RISCV_ISA_EXT_DATA(_name, _id, NULL, 0) 103 + #define _RISCV_ISA_EXT_DATA(_name, _id, _subset_exts, _subset_exts_size, _validate) { \ 104 + .name = #_name, \ 105 + .property = #_name, \ 106 + .id = _id, \ 107 + .subset_ext_ids = _subset_exts, \ 108 + .subset_ext_size = _subset_exts_size, \ 109 + .validate = _validate \ 110 + } 111 + 112 + #define __RISCV_ISA_EXT_DATA(_name, _id) _RISCV_ISA_EXT_DATA(_name, _id, NULL, 0, NULL) 113 + 114 + #define __RISCV_ISA_EXT_DATA_VALIDATE(_name, _id, _validate) \ 115 + _RISCV_ISA_EXT_DATA(_name, _id, NULL, 0, _validate) 112 116 113 117 /* Used to declare pure "lasso" extension (Zk for instance) */ 114 118 #define __RISCV_ISA_EXT_BUNDLE(_name, _bundled_exts) \ 115 - _RISCV_ISA_EXT_DATA(_name, RISCV_ISA_EXT_INVALID, _bundled_exts, ARRAY_SIZE(_bundled_exts)) 119 + _RISCV_ISA_EXT_DATA(_name, RISCV_ISA_EXT_INVALID, _bundled_exts, \ 120 + ARRAY_SIZE(_bundled_exts), NULL) 116 121 117 122 /* Used to declare extensions that are a superset of other extensions (Zvbb for instance) */ 118 123 #define __RISCV_ISA_EXT_SUPERSET(_name, _id, _sub_exts) \ 119 - _RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts)) 124 + _RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), NULL) 125 + #define __RISCV_ISA_EXT_SUPERSET_VALIDATE(_name, _id, _sub_exts, _validate) \ 126 + _RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), _validate) 127 + 128 + static int riscv_ext_zca_depends(const struct riscv_isa_ext_data *data, 129 + const unsigned long *isa_bitmap) 130 + { 131 + if (__riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_ZCA)) 132 + return 0; 133 + 134 + return -EPROBE_DEFER; 135 + } 136 + static int riscv_ext_zcd_validate(const struct riscv_isa_ext_data *data, 137 + const unsigned long *isa_bitmap) 138 + { 139 + if (__riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_ZCA) && 140 + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_d)) 141 + return 0; 142 + 143 + return -EPROBE_DEFER; 144 + } 145 + 146 + static int riscv_ext_zcf_validate(const struct riscv_isa_ext_data *data, 147 + const unsigned long *isa_bitmap) 148 + { 149 + if (IS_ENABLED(CONFIG_64BIT)) 150 + return -EINVAL; 151 + 152 + if (__riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_ZCA) && 153 + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_f)) 154 + return 0; 155 + 156 + return -EPROBE_DEFER; 157 + } 120 158 121 159 static const unsigned int riscv_zk_bundled_exts[] = { 122 160 RISCV_ISA_EXT_ZBKB, ··· 226 188 RISCV_ISA_EXT_ZVKB 227 189 }; 228 190 191 + #define RISCV_ISA_EXT_ZVE64F_IMPLY_LIST \ 192 + RISCV_ISA_EXT_ZVE64X, \ 193 + RISCV_ISA_EXT_ZVE32F, \ 194 + RISCV_ISA_EXT_ZVE32X 195 + 196 + #define RISCV_ISA_EXT_ZVE64D_IMPLY_LIST \ 197 + RISCV_ISA_EXT_ZVE64F, \ 198 + RISCV_ISA_EXT_ZVE64F_IMPLY_LIST 199 + 200 + #define RISCV_ISA_EXT_V_IMPLY_LIST \ 201 + RISCV_ISA_EXT_ZVE64D, \ 202 + RISCV_ISA_EXT_ZVE64D_IMPLY_LIST 203 + 204 + static const unsigned int riscv_zve32f_exts[] = { 205 + RISCV_ISA_EXT_ZVE32X 206 + }; 207 + 208 + static const unsigned int riscv_zve64f_exts[] = { 209 + RISCV_ISA_EXT_ZVE64F_IMPLY_LIST 210 + }; 211 + 212 + static const unsigned int riscv_zve64d_exts[] = { 213 + RISCV_ISA_EXT_ZVE64D_IMPLY_LIST 214 + }; 215 + 216 + static const unsigned int riscv_v_exts[] = { 217 + RISCV_ISA_EXT_V_IMPLY_LIST 218 + }; 219 + 220 + static const unsigned int riscv_zve64x_exts[] = { 221 + RISCV_ISA_EXT_ZVE32X, 222 + RISCV_ISA_EXT_ZVE64X 223 + }; 224 + 229 225 /* 230 226 * While the [ms]envcfg CSRs were not defined until version 1.12 of the RISC-V 231 227 * privileged ISA, the existence of the CSRs is implied by any extension which ··· 268 196 */ 269 197 static const unsigned int riscv_xlinuxenvcfg_exts[] = { 270 198 RISCV_ISA_EXT_XLINUXENVCFG 199 + }; 200 + 201 + /* 202 + * Zc* spec states that: 203 + * - C always implies Zca 204 + * - C+F implies Zcf (RV32 only) 205 + * - C+D implies Zcd 206 + * 207 + * These extensions will be enabled and then validated depending on the 208 + * availability of F/D RV32. 209 + */ 210 + static const unsigned int riscv_c_exts[] = { 211 + RISCV_ISA_EXT_ZCA, 212 + RISCV_ISA_EXT_ZCF, 213 + RISCV_ISA_EXT_ZCD, 271 214 }; 272 215 273 216 /* ··· 331 244 __RISCV_ISA_EXT_DATA(f, RISCV_ISA_EXT_f), 332 245 __RISCV_ISA_EXT_DATA(d, RISCV_ISA_EXT_d), 333 246 __RISCV_ISA_EXT_DATA(q, RISCV_ISA_EXT_q), 334 - __RISCV_ISA_EXT_DATA(c, RISCV_ISA_EXT_c), 335 - __RISCV_ISA_EXT_DATA(v, RISCV_ISA_EXT_v), 247 + __RISCV_ISA_EXT_SUPERSET(c, RISCV_ISA_EXT_c, riscv_c_exts), 248 + __RISCV_ISA_EXT_SUPERSET(v, RISCV_ISA_EXT_v, riscv_v_exts), 336 249 __RISCV_ISA_EXT_DATA(h, RISCV_ISA_EXT_h), 337 - __RISCV_ISA_EXT_SUPERSET(zicbom, RISCV_ISA_EXT_ZICBOM, riscv_xlinuxenvcfg_exts), 338 - __RISCV_ISA_EXT_SUPERSET(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts), 250 + __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicbom, RISCV_ISA_EXT_ZICBOM, riscv_xlinuxenvcfg_exts, 251 + riscv_ext_zicbom_validate), 252 + __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, 253 + riscv_ext_zicboz_validate), 339 254 __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), 340 255 __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), 341 256 __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), ··· 345 256 __RISCV_ISA_EXT_DATA(zihintntl, RISCV_ISA_EXT_ZIHINTNTL), 346 257 __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), 347 258 __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), 259 + __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), 348 260 __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), 261 + __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), 349 262 __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), 350 263 __RISCV_ISA_EXT_DATA(zfh, RISCV_ISA_EXT_ZFH), 351 264 __RISCV_ISA_EXT_DATA(zfhmin, RISCV_ISA_EXT_ZFHMIN), 265 + __RISCV_ISA_EXT_DATA(zca, RISCV_ISA_EXT_ZCA), 266 + __RISCV_ISA_EXT_DATA_VALIDATE(zcb, RISCV_ISA_EXT_ZCB, riscv_ext_zca_depends), 267 + __RISCV_ISA_EXT_DATA_VALIDATE(zcd, RISCV_ISA_EXT_ZCD, riscv_ext_zcd_validate), 268 + __RISCV_ISA_EXT_DATA_VALIDATE(zcf, RISCV_ISA_EXT_ZCF, riscv_ext_zcf_validate), 269 + __RISCV_ISA_EXT_DATA_VALIDATE(zcmop, RISCV_ISA_EXT_ZCMOP, riscv_ext_zca_depends), 352 270 __RISCV_ISA_EXT_DATA(zba, RISCV_ISA_EXT_ZBA), 353 271 __RISCV_ISA_EXT_DATA(zbb, RISCV_ISA_EXT_ZBB), 354 272 __RISCV_ISA_EXT_DATA(zbc, RISCV_ISA_EXT_ZBC), ··· 376 280 __RISCV_ISA_EXT_DATA(ztso, RISCV_ISA_EXT_ZTSO), 377 281 __RISCV_ISA_EXT_SUPERSET(zvbb, RISCV_ISA_EXT_ZVBB, riscv_zvbb_exts), 378 282 __RISCV_ISA_EXT_DATA(zvbc, RISCV_ISA_EXT_ZVBC), 283 + __RISCV_ISA_EXT_SUPERSET(zve32f, RISCV_ISA_EXT_ZVE32F, riscv_zve32f_exts), 284 + __RISCV_ISA_EXT_DATA(zve32x, RISCV_ISA_EXT_ZVE32X), 285 + __RISCV_ISA_EXT_SUPERSET(zve64d, RISCV_ISA_EXT_ZVE64D, riscv_zve64d_exts), 286 + __RISCV_ISA_EXT_SUPERSET(zve64f, RISCV_ISA_EXT_ZVE64F, riscv_zve64f_exts), 287 + __RISCV_ISA_EXT_SUPERSET(zve64x, RISCV_ISA_EXT_ZVE64X, riscv_zve64x_exts), 379 288 __RISCV_ISA_EXT_DATA(zvfh, RISCV_ISA_EXT_ZVFH), 380 289 __RISCV_ISA_EXT_DATA(zvfhmin, RISCV_ISA_EXT_ZVFHMIN), 381 290 __RISCV_ISA_EXT_DATA(zvkb, RISCV_ISA_EXT_ZVKB), ··· 410 309 411 310 const size_t riscv_isa_ext_count = ARRAY_SIZE(riscv_isa_ext); 412 311 413 - static void __init match_isa_ext(const struct riscv_isa_ext_data *ext, const char *name, 414 - const char *name_end, struct riscv_isainfo *isainfo) 312 + static void riscv_isa_set_ext(const struct riscv_isa_ext_data *ext, unsigned long *bitmap) 415 313 { 416 - if ((name_end - name == strlen(ext->name)) && 417 - !strncasecmp(name, ext->name, name_end - name)) { 418 - /* 419 - * If this is a bundle, enable all the ISA extensions that 420 - * comprise the bundle. 421 - */ 422 - if (ext->subset_ext_size) { 423 - for (int i = 0; i < ext->subset_ext_size; i++) { 424 - if (riscv_isa_extension_check(ext->subset_ext_ids[i])) 425 - set_bit(ext->subset_ext_ids[i], isainfo->isa); 426 - } 427 - } 314 + if (ext->id != RISCV_ISA_EXT_INVALID) 315 + set_bit(ext->id, bitmap); 428 316 429 - /* 430 - * This is valid even for bundle extensions which uses the RISCV_ISA_EXT_INVALID id 431 - * (rejected by riscv_isa_extension_check()). 432 - */ 433 - if (riscv_isa_extension_check(ext->id)) 434 - set_bit(ext->id, isainfo->isa); 317 + for (int i = 0; i < ext->subset_ext_size; i++) { 318 + if (ext->subset_ext_ids[i] != RISCV_ISA_EXT_INVALID) 319 + set_bit(ext->subset_ext_ids[i], bitmap); 435 320 } 436 321 } 437 322 438 - static void __init riscv_parse_isa_string(unsigned long *this_hwcap, struct riscv_isainfo *isainfo, 439 - unsigned long *isa2hwcap, const char *isa) 323 + static const struct riscv_isa_ext_data *riscv_get_isa_ext_data(unsigned int ext_id) 324 + { 325 + for (int i = 0; i < riscv_isa_ext_count; i++) { 326 + if (riscv_isa_ext[i].id == ext_id) 327 + return &riscv_isa_ext[i]; 328 + } 329 + 330 + return NULL; 331 + } 332 + 333 + /* 334 + * "Resolve" a source ISA bitmap into one that matches kernel configuration as 335 + * well as correct extension dependencies. Some extensions depends on specific 336 + * kernel configuration to be usable (V needs CONFIG_RISCV_ISA_V for instance) 337 + * and this function will actually validate all the extensions provided in 338 + * source_isa into the resolved_isa based on extensions validate() callbacks. 339 + */ 340 + static void __init riscv_resolve_isa(unsigned long *source_isa, 341 + unsigned long *resolved_isa, unsigned long *this_hwcap, 342 + unsigned long *isa2hwcap) 343 + { 344 + bool loop; 345 + const struct riscv_isa_ext_data *ext; 346 + DECLARE_BITMAP(prev_resolved_isa, RISCV_ISA_EXT_MAX); 347 + int max_loop_count = riscv_isa_ext_count, ret; 348 + unsigned int bit; 349 + 350 + do { 351 + loop = false; 352 + if (max_loop_count-- < 0) { 353 + pr_err("Failed to reach a stable ISA state\n"); 354 + return; 355 + } 356 + bitmap_copy(prev_resolved_isa, resolved_isa, RISCV_ISA_EXT_MAX); 357 + for_each_set_bit(bit, source_isa, RISCV_ISA_EXT_MAX) { 358 + ext = riscv_get_isa_ext_data(bit); 359 + if (!ext) 360 + continue; 361 + 362 + if (ext->validate) { 363 + ret = ext->validate(ext, resolved_isa); 364 + if (ret == -EPROBE_DEFER) { 365 + loop = true; 366 + continue; 367 + } else if (ret) { 368 + /* Disable the extension entirely */ 369 + clear_bit(ext->id, source_isa); 370 + continue; 371 + } 372 + } 373 + 374 + set_bit(ext->id, resolved_isa); 375 + /* No need to keep it in source isa now that it is enabled */ 376 + clear_bit(ext->id, source_isa); 377 + 378 + /* Single letter extensions get set in hwcap */ 379 + if (ext->id < RISCV_ISA_EXT_BASE) 380 + *this_hwcap |= isa2hwcap[ext->id]; 381 + } 382 + } while (loop && memcmp(prev_resolved_isa, resolved_isa, sizeof(prev_resolved_isa))); 383 + } 384 + 385 + static void __init match_isa_ext(const char *name, const char *name_end, unsigned long *bitmap) 386 + { 387 + for (int i = 0; i < riscv_isa_ext_count; i++) { 388 + const struct riscv_isa_ext_data *ext = &riscv_isa_ext[i]; 389 + 390 + if ((name_end - name == strlen(ext->name)) && 391 + !strncasecmp(name, ext->name, name_end - name)) { 392 + riscv_isa_set_ext(ext, bitmap); 393 + break; 394 + } 395 + } 396 + } 397 + 398 + static void __init riscv_parse_isa_string(const char *isa, unsigned long *bitmap) 440 399 { 441 400 /* 442 401 * For all possible cpus, we have already validated in ··· 509 348 while (*isa) { 510 349 const char *ext = isa++; 511 350 const char *ext_end = isa; 512 - bool ext_long = false, ext_err = false; 351 + bool ext_err = false; 513 352 514 353 switch (*ext) { 515 354 case 's': ··· 549 388 * character itself while eliminating the extensions version number. 550 389 * A simple re-increment solves this problem. 551 390 */ 552 - ext_long = true; 553 391 for (; *isa && *isa != '_'; ++isa) 554 392 if (unlikely(!isalnum(*isa))) 555 393 ext_err = true; ··· 628 468 629 469 if (unlikely(ext_err)) 630 470 continue; 631 - if (!ext_long) { 632 - int nr = tolower(*ext) - 'a'; 633 471 634 - if (riscv_isa_extension_check(nr)) { 635 - *this_hwcap |= isa2hwcap[nr]; 636 - set_bit(nr, isainfo->isa); 637 - } 638 - } else { 639 - for (int i = 0; i < riscv_isa_ext_count; i++) 640 - match_isa_ext(&riscv_isa_ext[i], ext, ext_end, isainfo); 641 - } 472 + match_isa_ext(ext, ext_end, bitmap); 642 473 } 643 474 } 644 475 ··· 656 505 for_each_possible_cpu(cpu) { 657 506 struct riscv_isainfo *isainfo = &hart_isa[cpu]; 658 507 unsigned long this_hwcap = 0; 508 + DECLARE_BITMAP(source_isa, RISCV_ISA_EXT_MAX) = { 0 }; 659 509 660 510 if (acpi_disabled) { 661 511 node = of_cpu_device_node_get(cpu); ··· 679 527 } 680 528 } 681 529 682 - riscv_parse_isa_string(&this_hwcap, isainfo, isa2hwcap, isa); 530 + riscv_parse_isa_string(isa, source_isa); 683 531 684 532 /* 685 533 * These ones were as they were part of the base ISA when the ··· 687 535 * unconditionally where `i` is in riscv,isa on DT systems. 688 536 */ 689 537 if (acpi_disabled) { 690 - set_bit(RISCV_ISA_EXT_ZICSR, isainfo->isa); 691 - set_bit(RISCV_ISA_EXT_ZIFENCEI, isainfo->isa); 692 - set_bit(RISCV_ISA_EXT_ZICNTR, isainfo->isa); 693 - set_bit(RISCV_ISA_EXT_ZIHPM, isainfo->isa); 538 + set_bit(RISCV_ISA_EXT_ZICSR, source_isa); 539 + set_bit(RISCV_ISA_EXT_ZIFENCEI, source_isa); 540 + set_bit(RISCV_ISA_EXT_ZICNTR, source_isa); 541 + set_bit(RISCV_ISA_EXT_ZIHPM, source_isa); 694 542 } 695 543 696 544 /* ··· 703 551 */ 704 552 if (acpi_disabled && boot_vendorid == THEAD_VENDOR_ID && boot_archid == 0x0) { 705 553 this_hwcap &= ~isa2hwcap[RISCV_ISA_EXT_v]; 706 - clear_bit(RISCV_ISA_EXT_v, isainfo->isa); 554 + clear_bit(RISCV_ISA_EXT_v, source_isa); 707 555 } 556 + 557 + riscv_resolve_isa(source_isa, isainfo->isa, &this_hwcap, isa2hwcap); 708 558 709 559 /* 710 560 * All "okay" hart should have same isa. Set HWCAP based on ··· 736 582 unsigned long this_hwcap = 0; 737 583 struct device_node *cpu_node; 738 584 struct riscv_isainfo *isainfo = &hart_isa[cpu]; 585 + DECLARE_BITMAP(source_isa, RISCV_ISA_EXT_MAX) = { 0 }; 739 586 740 587 cpu_node = of_cpu_device_node_get(cpu); 741 588 if (!cpu_node) { ··· 756 601 ext->property) < 0) 757 602 continue; 758 603 759 - if (ext->subset_ext_size) { 760 - for (int j = 0; j < ext->subset_ext_size; j++) { 761 - if (riscv_isa_extension_check(ext->subset_ext_ids[j])) 762 - set_bit(ext->subset_ext_ids[j], isainfo->isa); 763 - } 764 - } 765 - 766 - if (riscv_isa_extension_check(ext->id)) { 767 - set_bit(ext->id, isainfo->isa); 768 - 769 - /* Only single letter extensions get set in hwcap */ 770 - if (strnlen(riscv_isa_ext[i].name, 2) == 1) 771 - this_hwcap |= isa2hwcap[riscv_isa_ext[i].id]; 772 - } 604 + riscv_isa_set_ext(ext, source_isa); 773 605 } 606 + 607 + riscv_resolve_isa(source_isa, isainfo->isa, &this_hwcap, isa2hwcap); 774 608 775 609 of_node_put(cpu_node); 776 610 ··· 830 686 elf_hwcap &= ~COMPAT_HWCAP_ISA_F; 831 687 } 832 688 833 - if (elf_hwcap & COMPAT_HWCAP_ISA_V) { 689 + if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_ZVE32X)) { 690 + /* 691 + * This cannot fail when called on the boot hart 692 + */ 834 693 riscv_v_setup_vsize(); 694 + } 695 + 696 + if (elf_hwcap & COMPAT_HWCAP_ISA_V) { 835 697 /* 836 698 * ISA string in device tree might have 'v' flag, but 837 699 * CONFIG_RISCV_ISA_V is disabled in kernel.

+15 -7

arch/riscv/kernel/head.S

··· 165 165 #endif 166 166 call .Lsetup_trap_vector 167 167 scs_load_current 168 - tail smp_callin 168 + call smp_callin 169 169 #endif /* CONFIG_SMP */ 170 + 171 + .align 2 172 + .Lsecondary_park: 173 + /* 174 + * Park this hart if we: 175 + * - have too many harts on CONFIG_RISCV_BOOT_SPINWAIT 176 + * - receive an early trap, before setup_trap_vector finished 177 + * - fail in smp_callin(), as a successful one wouldn't return 178 + */ 179 + wfi 180 + j .Lsecondary_park 170 181 171 182 .align 2 172 183 .Lsetup_trap_vector: ··· 191 180 */ 192 181 csrw CSR_SCRATCH, zero 193 182 ret 194 - 195 - .align 2 196 - .Lsecondary_park: 197 - /* We lack SMP support or have too many harts, so park this hart */ 198 - wfi 199 - j .Lsecondary_park 200 183 201 184 SYM_CODE_END(_start) 202 185 ··· 305 300 #else 306 301 mv a0, a1 307 302 #endif /* CONFIG_BUILTIN_DTB */ 303 + /* Set trap vector to spin forever to help debug */ 304 + la a3, .Lsecondary_park 305 + csrw CSR_TVEC, a3 308 306 call setup_vm 309 307 #ifdef CONFIG_MMU 310 308 la a0, early_pg_dir

+12 -4

arch/riscv/kernel/jump_label.c

··· 9 9 #include <linux/memory.h> 10 10 #include <linux/mutex.h> 11 11 #include <asm/bug.h> 12 + #include <asm/cacheflush.h> 12 13 #include <asm/patch.h> 13 14 14 15 #define RISCV_INSN_NOP 0x00000013U 15 16 #define RISCV_INSN_JAL 0x0000006fU 16 17 17 - void arch_jump_label_transform(struct jump_entry *entry, 18 - enum jump_label_type type) 18 + bool arch_jump_label_transform_queue(struct jump_entry *entry, 19 + enum jump_label_type type) 19 20 { 20 21 void *addr = (void *)jump_entry_code(entry); 21 22 u32 insn; ··· 25 24 long offset = jump_entry_target(entry) - jump_entry_code(entry); 26 25 27 26 if (WARN_ON(offset & 1 || offset < -524288 || offset >= 524288)) 28 - return; 27 + return true; 29 28 30 29 insn = RISCV_INSN_JAL | 31 30 (((u32)offset & GENMASK(19, 12)) << (12 - 12)) | ··· 37 36 } 38 37 39 38 mutex_lock(&text_mutex); 40 - patch_text_nosync(addr, &insn, sizeof(insn)); 39 + patch_insn_write(addr, &insn, sizeof(insn)); 41 40 mutex_unlock(&text_mutex); 41 + 42 + return true; 43 + } 44 + 45 + void arch_jump_label_transform_apply(void) 46 + { 47 + flush_icache_all(); 42 48 }

+29 -30

arch/riscv/kernel/patch.c

··· 19 19 struct patch_insn { 20 20 void *addr; 21 21 u32 *insns; 22 - int ninsns; 22 + size_t len; 23 23 atomic_t cpu_count; 24 24 }; 25 25 ··· 54 54 BUG_ON(!page); 55 55 56 56 return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + 57 - (uintaddr & ~PAGE_MASK)); 57 + offset_in_page(addr)); 58 58 } 59 59 60 60 static void patch_unmap(int fixmap) ··· 65 65 66 66 static int __patch_insn_set(void *addr, u8 c, size_t len) 67 67 { 68 + bool across_pages = (offset_in_page(addr) + len) > PAGE_SIZE; 68 69 void *waddr = addr; 69 - bool across_pages = (((uintptr_t)addr & ~PAGE_MASK) + len) > PAGE_SIZE; 70 70 71 71 /* 72 72 * Only two pages can be mapped at a time for writing. ··· 110 110 111 111 static int __patch_insn_write(void *addr, const void *insn, size_t len) 112 112 { 113 + bool across_pages = (offset_in_page(addr) + len) > PAGE_SIZE; 113 114 void *waddr = addr; 114 - bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; 115 115 int ret; 116 116 117 117 /* ··· 179 179 180 180 static int patch_insn_set(void *addr, u8 c, size_t len) 181 181 { 182 - size_t patched = 0; 183 182 size_t size; 184 - int ret = 0; 183 + int ret; 185 184 186 185 /* 187 186 * __patch_insn_set() can only work on 2 pages at a time so call it in a 188 187 * loop with len <= 2 * PAGE_SIZE. 189 188 */ 190 - while (patched < len && !ret) { 191 - size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); 192 - ret = __patch_insn_set(addr + patched, c, size); 189 + while (len) { 190 + size = min(len, PAGE_SIZE * 2 - offset_in_page(addr)); 191 + ret = __patch_insn_set(addr, c, size); 192 + if (ret) 193 + return ret; 193 194 194 - patched += size; 195 + addr += size; 196 + len -= size; 195 197 } 196 198 197 - return ret; 199 + return 0; 198 200 } 199 201 NOKPROBE_SYMBOL(patch_insn_set); 200 202 201 203 int patch_text_set_nosync(void *addr, u8 c, size_t len) 202 204 { 203 - u32 *tp = addr; 204 205 int ret; 205 206 206 - ret = patch_insn_set(tp, c, len); 207 + ret = patch_insn_set(addr, c, len); 207 208 208 209 return ret; 209 210 } ··· 212 211 213 212 int patch_insn_write(void *addr, const void *insn, size_t len) 214 213 { 215 - size_t patched = 0; 216 214 size_t size; 217 - int ret = 0; 215 + int ret; 218 216 219 217 /* 220 218 * Copy the instructions to the destination address, two pages at a time 221 219 * because __patch_insn_write() can only handle len <= 2 * PAGE_SIZE. 222 220 */ 223 - while (patched < len && !ret) { 224 - size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); 225 - ret = __patch_insn_write(addr + patched, insn + patched, size); 221 + while (len) { 222 + size = min(len, PAGE_SIZE * 2 - offset_in_page(addr)); 223 + ret = __patch_insn_write(addr, insn, size); 224 + if (ret) 225 + return ret; 226 226 227 - patched += size; 227 + addr += size; 228 + insn += size; 229 + len -= size; 228 230 } 229 231 230 - return ret; 232 + return 0; 231 233 } 232 234 NOKPROBE_SYMBOL(patch_insn_write); 233 235 234 236 int patch_text_nosync(void *addr, const void *insns, size_t len) 235 237 { 236 - u32 *tp = addr; 237 238 int ret; 238 239 239 - ret = patch_insn_write(tp, insns, len); 240 + ret = patch_insn_write(addr, insns, len); 240 241 241 242 return ret; 242 243 } ··· 247 244 static int patch_text_cb(void *data) 248 245 { 249 246 struct patch_insn *patch = data; 250 - unsigned long len; 251 - int i, ret = 0; 247 + int ret = 0; 252 248 253 249 if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) { 254 - for (i = 0; ret == 0 && i < patch->ninsns; i++) { 255 - len = GET_INSN_LENGTH(patch->insns[i]); 256 - ret = patch_insn_write(patch->addr + i * len, &patch->insns[i], len); 257 - } 250 + ret = patch_insn_write(patch->addr, patch->insns, patch->len); 258 251 /* 259 252 * Make sure the patching store is effective *before* we 260 253 * increment the counter which releases all waiting CPUs ··· 270 271 } 271 272 NOKPROBE_SYMBOL(patch_text_cb); 272 273 273 - int patch_text(void *addr, u32 *insns, int ninsns) 274 + int patch_text(void *addr, u32 *insns, size_t len) 274 275 { 275 276 int ret; 276 277 struct patch_insn patch = { 277 278 .addr = addr, 278 279 .insns = insns, 279 - .ninsns = ninsns, 280 + .len = len, 280 281 .cpu_count = ATOMIC_INIT(0), 281 282 }; 282 283

+10 -9

arch/riscv/kernel/probes/kprobes.c

··· 24 24 25 25 static void __kprobes arch_prepare_ss_slot(struct kprobe *p) 26 26 { 27 + size_t len = GET_INSN_LENGTH(p->opcode); 27 28 u32 insn = __BUG_INSN_32; 28 - unsigned long offset = GET_INSN_LENGTH(p->opcode); 29 29 30 - p->ainsn.api.restore = (unsigned long)p->addr + offset; 30 + p->ainsn.api.restore = (unsigned long)p->addr + len; 31 31 32 - patch_text(p->ainsn.api.insn, &p->opcode, 1); 33 - patch_text((void *)((unsigned long)(p->ainsn.api.insn) + offset), 34 - &insn, 1); 32 + patch_text_nosync(p->ainsn.api.insn, &p->opcode, len); 33 + patch_text_nosync(p->ainsn.api.insn + len, &insn, GET_INSN_LENGTH(insn)); 35 34 } 36 35 37 36 static void __kprobes arch_prepare_simulate(struct kprobe *p) ··· 107 108 /* install breakpoint in text */ 108 109 void __kprobes arch_arm_kprobe(struct kprobe *p) 109 110 { 110 - u32 insn = (p->opcode & __INSN_LENGTH_MASK) == __INSN_LENGTH_32 ? 111 - __BUG_INSN_32 : __BUG_INSN_16; 111 + size_t len = GET_INSN_LENGTH(p->opcode); 112 + u32 insn = len == 4 ? __BUG_INSN_32 : __BUG_INSN_16; 112 113 113 - patch_text(p->addr, &insn, 1); 114 + patch_text(p->addr, &insn, len); 114 115 } 115 116 116 117 /* remove breakpoint from text */ 117 118 void __kprobes arch_disarm_kprobe(struct kprobe *p) 118 119 { 119 - patch_text(p->addr, &p->opcode, 1); 120 + size_t len = GET_INSN_LENGTH(p->opcode); 121 + 122 + patch_text(p->addr, &p->opcode, len); 120 123 } 121 124 122 125 void __kprobes arch_remove_kprobe(struct kprobe *p)

+12 -5

arch/riscv/kernel/sbi.c

··· 14 14 #include <asm/smp.h> 15 15 #include <asm/tlbflush.h> 16 16 17 + #define CREATE_TRACE_POINTS 18 + #include <asm/trace.h> 19 + 17 20 /* default SBI version is 0.1 */ 18 21 unsigned long sbi_spec_version __ro_after_init = SBI_SPEC_VERSION_DEFAULT; 19 22 EXPORT_SYMBOL(sbi_spec_version); ··· 27 24 unsigned long start, unsigned long size, 28 25 unsigned long arg4, unsigned long arg5) __ro_after_init; 29 26 30 - struct sbiret sbi_ecall(int ext, int fid, unsigned long arg0, 31 - unsigned long arg1, unsigned long arg2, 32 - unsigned long arg3, unsigned long arg4, 33 - unsigned long arg5) 27 + struct sbiret __sbi_ecall(unsigned long arg0, unsigned long arg1, 28 + unsigned long arg2, unsigned long arg3, 29 + unsigned long arg4, unsigned long arg5, 30 + int fid, int ext) 34 31 { 35 32 struct sbiret ret; 33 + 34 + trace_sbi_call(ext, fid); 36 35 37 36 register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); 38 37 register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); ··· 51 46 ret.error = a0; 52 47 ret.value = a1; 53 48 49 + trace_sbi_return(ext, ret.error, ret.value); 50 + 54 51 return ret; 55 52 } 56 - EXPORT_SYMBOL(sbi_ecall); 53 + EXPORT_SYMBOL(__sbi_ecall); 57 54 58 55 int sbi_err_map_linux_errno(int err) 59 56 {

+9 -5

arch/riscv/kernel/smpboot.c

··· 214 214 struct mm_struct *mm = &init_mm; 215 215 unsigned int curr_cpuid = smp_processor_id(); 216 216 217 + if (has_vector()) { 218 + /* 219 + * Return as early as possible so the hart with a mismatching 220 + * vlen won't boot. 221 + */ 222 + if (riscv_v_setup_vsize()) 223 + return; 224 + } 225 + 217 226 /* All kernel threads share the same mm context. */ 218 227 mmgrab(mm); 219 228 current->active_mm = mm; ··· 234 225 235 226 numa_add_cpu(curr_cpuid); 236 227 set_cpu_online(curr_cpuid, true); 237 - 238 - if (has_vector()) { 239 - if (riscv_v_setup_vsize()) 240 - elf_hwcap &= ~COMPAT_HWCAP_ISA_V; 241 - } 242 228 243 229 riscv_user_isa_enable(); 244 230

+21 -1

arch/riscv/kernel/sys_hwprobe.c

··· 8 8 #include <asm/cacheflush.h> 9 9 #include <asm/cpufeature.h> 10 10 #include <asm/hwprobe.h> 11 + #include <asm/processor.h> 11 12 #include <asm/sbi.h> 12 13 #include <asm/switch_to.h> 13 14 #include <asm/uaccess.h> ··· 70 69 if (riscv_isa_extension_available(NULL, c)) 71 70 pair->value |= RISCV_HWPROBE_IMA_C; 72 71 73 - if (has_vector()) 72 + if (has_vector() && riscv_isa_extension_available(NULL, v)) 74 73 pair->value |= RISCV_HWPROBE_IMA_V; 75 74 76 75 /* ··· 113 112 EXT_KEY(ZACAS); 114 113 EXT_KEY(ZICOND); 115 114 EXT_KEY(ZIHINTPAUSE); 115 + EXT_KEY(ZIMOP); 116 + EXT_KEY(ZCA); 117 + EXT_KEY(ZCB); 118 + EXT_KEY(ZCMOP); 119 + EXT_KEY(ZAWRS); 116 120 121 + /* 122 + * All the following extensions must depend on the kernel 123 + * support of V. 124 + */ 117 125 if (has_vector()) { 126 + EXT_KEY(ZVE32X); 127 + EXT_KEY(ZVE32F); 128 + EXT_KEY(ZVE64X); 129 + EXT_KEY(ZVE64F); 130 + EXT_KEY(ZVE64D); 118 131 EXT_KEY(ZVBB); 119 132 EXT_KEY(ZVBC); 120 133 EXT_KEY(ZVKB); ··· 147 132 EXT_KEY(ZFH); 148 133 EXT_KEY(ZFHMIN); 149 134 EXT_KEY(ZFA); 135 + EXT_KEY(ZCD); 136 + EXT_KEY(ZCF); 150 137 } 151 138 #undef EXT_KEY 152 139 } ··· 232 215 pair->value = 0; 233 216 if (hwprobe_ext0_has(cpus, RISCV_HWPROBE_EXT_ZICBOZ)) 234 217 pair->value = riscv_cboz_block_size; 218 + break; 219 + case RISCV_HWPROBE_KEY_HIGHEST_VIRT_ADDRESS: 220 + pair->value = user_max_virt_addr(); 235 221 break; 236 222 237 223 /*

+4 -1

arch/riscv/kernel/vector.c

··· 173 173 u32 __user *epc = (u32 __user *)regs->epc; 174 174 u32 insn = (u32)regs->badaddr; 175 175 176 + if (!has_vector()) 177 + return false; 178 + 176 179 /* Do not handle if V is not supported, or disabled */ 177 - if (!(ELF_HWCAP & COMPAT_HWCAP_ISA_V)) 180 + if (!riscv_v_vstate_ctrl_user_allowed()) 178 181 return false; 179 182 180 183 /* If V has been enabled then it is not the first-use trap */

+1

arch/riscv/kvm/vcpu.c

··· 25 25 KVM_GENERIC_VCPU_STATS(), 26 26 STATS_DESC_COUNTER(VCPU, ecall_exit_stat), 27 27 STATS_DESC_COUNTER(VCPU, wfi_exit_stat), 28 + STATS_DESC_COUNTER(VCPU, wrs_exit_stat), 28 29 STATS_DESC_COUNTER(VCPU, mmio_exit_user), 29 30 STATS_DESC_COUNTER(VCPU, mmio_exit_kernel), 30 31 STATS_DESC_COUNTER(VCPU, csr_exit_user),

+15

arch/riscv/kvm/vcpu_insn.c

··· 16 16 #define INSN_MASK_WFI 0xffffffff 17 17 #define INSN_MATCH_WFI 0x10500073 18 18 19 + #define INSN_MASK_WRS 0xffffffff 20 + #define INSN_MATCH_WRS 0x00d00073 21 + 19 22 #define INSN_MATCH_CSRRW 0x1073 20 23 #define INSN_MASK_CSRRW 0x707f 21 24 #define INSN_MATCH_CSRRS 0x2073 ··· 206 203 return KVM_INSN_CONTINUE_NEXT_SEPC; 207 204 } 208 205 206 + static int wrs_insn(struct kvm_vcpu *vcpu, struct kvm_run *run, ulong insn) 207 + { 208 + vcpu->stat.wrs_exit_stat++; 209 + kvm_vcpu_on_spin(vcpu, vcpu->arch.guest_context.sstatus & SR_SPP); 210 + return KVM_INSN_CONTINUE_NEXT_SEPC; 211 + } 212 + 209 213 struct csr_func { 210 214 unsigned int base; 211 215 unsigned int count; ··· 387 377 .mask = INSN_MASK_WFI, 388 378 .match = INSN_MATCH_WFI, 389 379 .func = wfi_insn, 380 + }, 381 + { 382 + .mask = INSN_MASK_WRS, 383 + .match = INSN_MATCH_WRS, 384 + .func = wrs_insn, 390 385 }, 391 386 }; 392 387

+14

arch/riscv/kvm/vcpu_onereg.c

··· 42 42 KVM_ISA_EXT_ARR(SVNAPOT), 43 43 KVM_ISA_EXT_ARR(SVPBMT), 44 44 KVM_ISA_EXT_ARR(ZACAS), 45 + KVM_ISA_EXT_ARR(ZAWRS), 45 46 KVM_ISA_EXT_ARR(ZBA), 46 47 KVM_ISA_EXT_ARR(ZBB), 47 48 KVM_ISA_EXT_ARR(ZBC), ··· 50 49 KVM_ISA_EXT_ARR(ZBKC), 51 50 KVM_ISA_EXT_ARR(ZBKX), 52 51 KVM_ISA_EXT_ARR(ZBS), 52 + KVM_ISA_EXT_ARR(ZCA), 53 + KVM_ISA_EXT_ARR(ZCB), 54 + KVM_ISA_EXT_ARR(ZCD), 55 + KVM_ISA_EXT_ARR(ZCF), 56 + KVM_ISA_EXT_ARR(ZCMOP), 53 57 KVM_ISA_EXT_ARR(ZFA), 54 58 KVM_ISA_EXT_ARR(ZFH), 55 59 KVM_ISA_EXT_ARR(ZFHMIN), ··· 67 61 KVM_ISA_EXT_ARR(ZIHINTNTL), 68 62 KVM_ISA_EXT_ARR(ZIHINTPAUSE), 69 63 KVM_ISA_EXT_ARR(ZIHPM), 64 + KVM_ISA_EXT_ARR(ZIMOP), 70 65 KVM_ISA_EXT_ARR(ZKND), 71 66 KVM_ISA_EXT_ARR(ZKNE), 72 67 KVM_ISA_EXT_ARR(ZKNH), ··· 133 126 case KVM_RISCV_ISA_EXT_SVINVAL: 134 127 case KVM_RISCV_ISA_EXT_SVNAPOT: 135 128 case KVM_RISCV_ISA_EXT_ZACAS: 129 + case KVM_RISCV_ISA_EXT_ZAWRS: 136 130 case KVM_RISCV_ISA_EXT_ZBA: 137 131 case KVM_RISCV_ISA_EXT_ZBB: 138 132 case KVM_RISCV_ISA_EXT_ZBC: ··· 141 133 case KVM_RISCV_ISA_EXT_ZBKC: 142 134 case KVM_RISCV_ISA_EXT_ZBKX: 143 135 case KVM_RISCV_ISA_EXT_ZBS: 136 + case KVM_RISCV_ISA_EXT_ZCA: 137 + case KVM_RISCV_ISA_EXT_ZCB: 138 + case KVM_RISCV_ISA_EXT_ZCD: 139 + case KVM_RISCV_ISA_EXT_ZCF: 140 + case KVM_RISCV_ISA_EXT_ZCMOP: 144 141 case KVM_RISCV_ISA_EXT_ZFA: 145 142 case KVM_RISCV_ISA_EXT_ZFH: 146 143 case KVM_RISCV_ISA_EXT_ZFHMIN: ··· 156 143 case KVM_RISCV_ISA_EXT_ZIHINTNTL: 157 144 case KVM_RISCV_ISA_EXT_ZIHINTPAUSE: 158 145 case KVM_RISCV_ISA_EXT_ZIHPM: 146 + case KVM_RISCV_ISA_EXT_ZIMOP: 159 147 case KVM_RISCV_ISA_EXT_ZKND: 160 148 case KVM_RISCV_ISA_EXT_ZKNE: 161 149 case KVM_RISCV_ISA_EXT_ZKNH:

+1

arch/riscv/lib/Makefile

··· 13 13 lib-$(CONFIG_MMU) += uaccess.o 14 14 lib-$(CONFIG_64BIT) += tishift.o 15 15 lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o 16 + lib-$(CONFIG_RISCV_ISA_ZBC) += crc32.o 16 17 17 18 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o 18 19 lib-$(CONFIG_RISCV_ISA_V) += xor.o

+294

arch/riscv/lib/crc32.c

··· 1 + // SPDX-License-Identifier: GPL-2.0-only 2 + /* 3 + * Accelerated CRC32 implementation with Zbc extension. 4 + * 5 + * Copyright (C) 2024 Intel Corporation 6 + */ 7 + 8 + #include <asm/hwcap.h> 9 + #include <asm/alternative-macros.h> 10 + #include <asm/byteorder.h> 11 + 12 + #include <linux/types.h> 13 + #include <linux/minmax.h> 14 + #include <linux/crc32poly.h> 15 + #include <linux/crc32.h> 16 + #include <linux/byteorder/generic.h> 17 + 18 + /* 19 + * Refer to https://www.corsix.org/content/barrett-reduction-polynomials for 20 + * better understanding of how this math works. 21 + * 22 + * let "+" denotes polynomial add (XOR) 23 + * let "-" denotes polynomial sub (XOR) 24 + * let "*" denotes polynomial multiplication 25 + * let "/" denotes polynomial floor division 26 + * let "S" denotes source data, XLEN bit wide 27 + * let "P" denotes CRC32 polynomial 28 + * let "T" denotes 2^(XLEN+32) 29 + * let "QT" denotes quotient of T/P, with the bit for 2^XLEN being implicit 30 + * 31 + * crc32(S, P) 32 + * => S * (2^32) - S * (2^32) / P * P 33 + * => lowest 32 bits of: S * (2^32) / P * P 34 + * => lowest 32 bits of: S * (2^32) * (T / P) / T * P 35 + * => lowest 32 bits of: S * (2^32) * quotient / T * P 36 + * => lowest 32 bits of: S * quotient / 2^XLEN * P 37 + * => lowest 32 bits of: (clmul_high_part(S, QT) + S) * P 38 + * => clmul_low_part(clmul_high_part(S, QT) + S, P) 39 + * 40 + * In terms of below implementations, the BE case is more intuitive, since the 41 + * higher order bit sits at more significant position. 42 + */ 43 + 44 + #if __riscv_xlen == 64 45 + /* Slide by XLEN bits per iteration */ 46 + # define STEP_ORDER 3 47 + 48 + /* Each below polynomial quotient has an implicit bit for 2^XLEN */ 49 + 50 + /* Polynomial quotient of (2^(XLEN+32))/CRC32_POLY, in LE format */ 51 + # define CRC32_POLY_QT_LE 0x5a72d812fb808b20 52 + 53 + /* Polynomial quotient of (2^(XLEN+32))/CRC32C_POLY, in LE format */ 54 + # define CRC32C_POLY_QT_LE 0xa434f61c6f5389f8 55 + 56 + /* Polynomial quotient of (2^(XLEN+32))/CRC32_POLY, in BE format, it should be 57 + * the same as the bit-reversed version of CRC32_POLY_QT_LE 58 + */ 59 + # define CRC32_POLY_QT_BE 0x04d101df481b4e5a 60 + 61 + static inline u64 crc32_le_prep(u32 crc, unsigned long const *ptr) 62 + { 63 + return (u64)crc ^ (__force u64)__cpu_to_le64(*ptr); 64 + } 65 + 66 + static inline u32 crc32_le_zbc(unsigned long s, u32 poly, unsigned long poly_qt) 67 + { 68 + u32 crc; 69 + 70 + /* We don't have a "clmulrh" insn, so use clmul + slli instead. */ 71 + asm volatile (".option push\n" 72 + ".option arch,+zbc\n" 73 + "clmul %0, %1, %2\n" 74 + "slli %0, %0, 1\n" 75 + "xor %0, %0, %1\n" 76 + "clmulr %0, %0, %3\n" 77 + "srli %0, %0, 32\n" 78 + ".option pop\n" 79 + : "=&r" (crc) 80 + : "r" (s), 81 + "r" (poly_qt), 82 + "r" ((u64)poly << 32) 83 + :); 84 + return crc; 85 + } 86 + 87 + static inline u64 crc32_be_prep(u32 crc, unsigned long const *ptr) 88 + { 89 + return ((u64)crc << 32) ^ (__force u64)__cpu_to_be64(*ptr); 90 + } 91 + 92 + #elif __riscv_xlen == 32 93 + # define STEP_ORDER 2 94 + /* Each quotient should match the upper half of its analog in RV64 */ 95 + # define CRC32_POLY_QT_LE 0xfb808b20 96 + # define CRC32C_POLY_QT_LE 0x6f5389f8 97 + # define CRC32_POLY_QT_BE 0x04d101df 98 + 99 + static inline u32 crc32_le_prep(u32 crc, unsigned long const *ptr) 100 + { 101 + return crc ^ (__force u32)__cpu_to_le32(*ptr); 102 + } 103 + 104 + static inline u32 crc32_le_zbc(unsigned long s, u32 poly, unsigned long poly_qt) 105 + { 106 + u32 crc; 107 + 108 + /* We don't have a "clmulrh" insn, so use clmul + slli instead. */ 109 + asm volatile (".option push\n" 110 + ".option arch,+zbc\n" 111 + "clmul %0, %1, %2\n" 112 + "slli %0, %0, 1\n" 113 + "xor %0, %0, %1\n" 114 + "clmulr %0, %0, %3\n" 115 + ".option pop\n" 116 + : "=&r" (crc) 117 + : "r" (s), 118 + "r" (poly_qt), 119 + "r" (poly) 120 + :); 121 + return crc; 122 + } 123 + 124 + static inline u32 crc32_be_prep(u32 crc, unsigned long const *ptr) 125 + { 126 + return crc ^ (__force u32)__cpu_to_be32(*ptr); 127 + } 128 + 129 + #else 130 + # error "Unexpected __riscv_xlen" 131 + #endif 132 + 133 + static inline u32 crc32_be_zbc(unsigned long s) 134 + { 135 + u32 crc; 136 + 137 + asm volatile (".option push\n" 138 + ".option arch,+zbc\n" 139 + "clmulh %0, %1, %2\n" 140 + "xor %0, %0, %1\n" 141 + "clmul %0, %0, %3\n" 142 + ".option pop\n" 143 + : "=&r" (crc) 144 + : "r" (s), 145 + "r" (CRC32_POLY_QT_BE), 146 + "r" (CRC32_POLY_BE) 147 + :); 148 + return crc; 149 + } 150 + 151 + #define STEP (1 << STEP_ORDER) 152 + #define OFFSET_MASK (STEP - 1) 153 + 154 + typedef u32 (*fallback)(u32 crc, unsigned char const *p, size_t len); 155 + 156 + static inline u32 crc32_le_unaligned(u32 crc, unsigned char const *p, 157 + size_t len, u32 poly, 158 + unsigned long poly_qt) 159 + { 160 + size_t bits = len * 8; 161 + unsigned long s = 0; 162 + u32 crc_low = 0; 163 + 164 + for (int i = 0; i < len; i++) 165 + s = ((unsigned long)*p++ << (__riscv_xlen - 8)) | (s >> 8); 166 + 167 + s ^= (unsigned long)crc << (__riscv_xlen - bits); 168 + if (__riscv_xlen == 32 || len < sizeof(u32)) 169 + crc_low = crc >> bits; 170 + 171 + crc = crc32_le_zbc(s, poly, poly_qt); 172 + crc ^= crc_low; 173 + 174 + return crc; 175 + } 176 + 177 + static inline u32 __pure crc32_le_generic(u32 crc, unsigned char const *p, 178 + size_t len, u32 poly, 179 + unsigned long poly_qt, 180 + fallback crc_fb) 181 + { 182 + size_t offset, head_len, tail_len; 183 + unsigned long const *p_ul; 184 + unsigned long s; 185 + 186 + asm goto(ALTERNATIVE("j %l[legacy]", "nop", 0, 187 + RISCV_ISA_EXT_ZBC, 1) 188 + : : : : legacy); 189 + 190 + /* Handle the unaligned head. */ 191 + offset = (unsigned long)p & OFFSET_MASK; 192 + if (offset && len) { 193 + head_len = min(STEP - offset, len); 194 + crc = crc32_le_unaligned(crc, p, head_len, poly, poly_qt); 195 + p += head_len; 196 + len -= head_len; 197 + } 198 + 199 + tail_len = len & OFFSET_MASK; 200 + len = len >> STEP_ORDER; 201 + p_ul = (unsigned long const *)p; 202 + 203 + for (int i = 0; i < len; i++) { 204 + s = crc32_le_prep(crc, p_ul); 205 + crc = crc32_le_zbc(s, poly, poly_qt); 206 + p_ul++; 207 + } 208 + 209 + /* Handle the tail bytes. */ 210 + p = (unsigned char const *)p_ul; 211 + if (tail_len) 212 + crc = crc32_le_unaligned(crc, p, tail_len, poly, poly_qt); 213 + 214 + return crc; 215 + 216 + legacy: 217 + return crc_fb(crc, p, len); 218 + } 219 + 220 + u32 __pure crc32_le(u32 crc, unsigned char const *p, size_t len) 221 + { 222 + return crc32_le_generic(crc, p, len, CRC32_POLY_LE, CRC32_POLY_QT_LE, 223 + crc32_le_base); 224 + } 225 + 226 + u32 __pure __crc32c_le(u32 crc, unsigned char const *p, size_t len) 227 + { 228 + return crc32_le_generic(crc, p, len, CRC32C_POLY_LE, 229 + CRC32C_POLY_QT_LE, __crc32c_le_base); 230 + } 231 + 232 + static inline u32 crc32_be_unaligned(u32 crc, unsigned char const *p, 233 + size_t len) 234 + { 235 + size_t bits = len * 8; 236 + unsigned long s = 0; 237 + u32 crc_low = 0; 238 + 239 + s = 0; 240 + for (int i = 0; i < len; i++) 241 + s = *p++ | (s << 8); 242 + 243 + if (__riscv_xlen == 32 || len < sizeof(u32)) { 244 + s ^= crc >> (32 - bits); 245 + crc_low = crc << bits; 246 + } else { 247 + s ^= (unsigned long)crc << (bits - 32); 248 + } 249 + 250 + crc = crc32_be_zbc(s); 251 + crc ^= crc_low; 252 + 253 + return crc; 254 + } 255 + 256 + u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len) 257 + { 258 + size_t offset, head_len, tail_len; 259 + unsigned long const *p_ul; 260 + unsigned long s; 261 + 262 + asm goto(ALTERNATIVE("j %l[legacy]", "nop", 0, 263 + RISCV_ISA_EXT_ZBC, 1) 264 + : : : : legacy); 265 + 266 + /* Handle the unaligned head. */ 267 + offset = (unsigned long)p & OFFSET_MASK; 268 + if (offset && len) { 269 + head_len = min(STEP - offset, len); 270 + crc = crc32_be_unaligned(crc, p, head_len); 271 + p += head_len; 272 + len -= head_len; 273 + } 274 + 275 + tail_len = len & OFFSET_MASK; 276 + len = len >> STEP_ORDER; 277 + p_ul = (unsigned long const *)p; 278 + 279 + for (int i = 0; i < len; i++) { 280 + s = crc32_be_prep(crc, p_ul); 281 + crc = crc32_be_zbc(s); 282 + p_ul++; 283 + } 284 + 285 + /* Handle the tail bytes. */ 286 + p = (unsigned char const *)p_ul; 287 + if (tail_len) 288 + crc = crc32_be_unaligned(crc, p, tail_len); 289 + 290 + return crc; 291 + 292 + legacy: 293 + return crc32_be_base(crc, p, len); 294 + }

+1 -1

arch/riscv/lib/uaccess.S

··· 14 14 15 15 SYM_FUNC_START(__asm_copy_to_user) 16 16 #ifdef CONFIG_RISCV_ISA_V 17 - ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V) 17 + ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_ZVE32X, CONFIG_RISCV_ISA_V) 18 18 REG_L t0, riscv_v_usercopy_threshold 19 19 bltu a2, t0, fallback_scalar_usercopy 20 20 tail enter_vector_usercopy

+307 -40

arch/riscv/mm/init.c

··· 28 28 29 29 #include <asm/fixmap.h> 30 30 #include <asm/io.h> 31 + #include <asm/kasan.h> 31 32 #include <asm/numa.h> 32 33 #include <asm/pgtable.h> 33 34 #include <asm/sections.h> ··· 297 296 } 298 297 299 298 #ifdef CONFIG_MMU 300 - struct pt_alloc_ops pt_ops __initdata; 299 + struct pt_alloc_ops pt_ops __meminitdata; 301 300 302 301 pgd_t swapper_pg_dir[PTRS_PER_PGD] __page_aligned_bss; 303 302 pgd_t trampoline_pg_dir[PTRS_PER_PGD] __page_aligned_bss; ··· 359 358 return (pte_t *)set_fixmap_offset(FIX_PTE, pa); 360 359 } 361 360 362 - static inline pte_t *__init get_pte_virt_late(phys_addr_t pa) 361 + static inline pte_t *__meminit get_pte_virt_late(phys_addr_t pa) 363 362 { 364 363 return (pte_t *) __va(pa); 365 364 } ··· 378 377 return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); 379 378 } 380 379 381 - static phys_addr_t __init alloc_pte_late(uintptr_t va) 380 + static phys_addr_t __meminit alloc_pte_late(uintptr_t va) 382 381 { 383 382 struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); 384 383 ··· 386 385 return __pa((pte_t *)ptdesc_address(ptdesc)); 387 386 } 388 387 389 - static void __init create_pte_mapping(pte_t *ptep, 390 - uintptr_t va, phys_addr_t pa, 391 - phys_addr_t sz, pgprot_t prot) 388 + static void __meminit create_pte_mapping(pte_t *ptep, uintptr_t va, phys_addr_t pa, phys_addr_t sz, 389 + pgprot_t prot) 392 390 { 393 391 uintptr_t pte_idx = pte_index(va); 394 392 ··· 441 441 return (pmd_t *)set_fixmap_offset(FIX_PMD, pa); 442 442 } 443 443 444 - static pmd_t *__init get_pmd_virt_late(phys_addr_t pa) 444 + static pmd_t *__meminit get_pmd_virt_late(phys_addr_t pa) 445 445 { 446 446 return (pmd_t *) __va(pa); 447 447 } ··· 458 458 return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); 459 459 } 460 460 461 - static phys_addr_t __init alloc_pmd_late(uintptr_t va) 461 + static phys_addr_t __meminit alloc_pmd_late(uintptr_t va) 462 462 { 463 463 struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); 464 464 ··· 466 466 return __pa((pmd_t *)ptdesc_address(ptdesc)); 467 467 } 468 468 469 - static void __init create_pmd_mapping(pmd_t *pmdp, 470 - uintptr_t va, phys_addr_t pa, 471 - phys_addr_t sz, pgprot_t prot) 469 + static void __meminit create_pmd_mapping(pmd_t *pmdp, 470 + uintptr_t va, phys_addr_t pa, 471 + phys_addr_t sz, pgprot_t prot) 472 472 { 473 473 pte_t *ptep; 474 474 phys_addr_t pte_phys; ··· 504 504 return (pud_t *)set_fixmap_offset(FIX_PUD, pa); 505 505 } 506 506 507 - static pud_t *__init get_pud_virt_late(phys_addr_t pa) 507 + static pud_t *__meminit get_pud_virt_late(phys_addr_t pa) 508 508 { 509 509 return (pud_t *)__va(pa); 510 510 } ··· 522 522 return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); 523 523 } 524 524 525 - static phys_addr_t alloc_pud_late(uintptr_t va) 525 + static phys_addr_t __meminit alloc_pud_late(uintptr_t va) 526 526 { 527 527 unsigned long vaddr; 528 528 ··· 542 542 return (p4d_t *)set_fixmap_offset(FIX_P4D, pa); 543 543 } 544 544 545 - static p4d_t *__init get_p4d_virt_late(phys_addr_t pa) 545 + static p4d_t *__meminit get_p4d_virt_late(phys_addr_t pa) 546 546 { 547 547 return (p4d_t *)__va(pa); 548 548 } ··· 560 560 return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); 561 561 } 562 562 563 - static phys_addr_t alloc_p4d_late(uintptr_t va) 563 + static phys_addr_t __meminit alloc_p4d_late(uintptr_t va) 564 564 { 565 565 unsigned long vaddr; 566 566 ··· 569 569 return __pa(vaddr); 570 570 } 571 571 572 - static void __init create_pud_mapping(pud_t *pudp, 573 - uintptr_t va, phys_addr_t pa, 574 - phys_addr_t sz, pgprot_t prot) 572 + static void __meminit create_pud_mapping(pud_t *pudp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, 573 + pgprot_t prot) 575 574 { 576 575 pmd_t *nextp; 577 576 phys_addr_t next_phys; ··· 595 596 create_pmd_mapping(nextp, va, pa, sz, prot); 596 597 } 597 598 598 - static void __init create_p4d_mapping(p4d_t *p4dp, 599 - uintptr_t va, phys_addr_t pa, 600 - phys_addr_t sz, pgprot_t prot) 599 + static void __meminit create_p4d_mapping(p4d_t *p4dp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, 600 + pgprot_t prot) 601 601 { 602 602 pud_t *nextp; 603 603 phys_addr_t next_phys; ··· 652 654 #define create_pmd_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0) 653 655 #endif /* __PAGETABLE_PMD_FOLDED */ 654 656 655 - void __init create_pgd_mapping(pgd_t *pgdp, 656 - uintptr_t va, phys_addr_t pa, 657 - phys_addr_t sz, pgprot_t prot) 657 + void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, 658 + pgprot_t prot) 658 659 { 659 660 pgd_next_t *nextp; 660 661 phys_addr_t next_phys; ··· 678 681 create_pgd_next_mapping(nextp, va, pa, sz, prot); 679 682 } 680 683 681 - static uintptr_t __init best_map_size(phys_addr_t pa, uintptr_t va, 682 - phys_addr_t size) 684 + static uintptr_t __meminit best_map_size(phys_addr_t pa, uintptr_t va, phys_addr_t size) 683 685 { 684 686 if (debug_pagealloc_enabled()) 685 687 return PAGE_SIZE; ··· 714 718 #endif 715 719 716 720 #ifdef CONFIG_STRICT_KERNEL_RWX 717 - static __init pgprot_t pgprot_from_va(uintptr_t va) 721 + static __meminit pgprot_t pgprot_from_va(uintptr_t va) 718 722 { 719 723 if (is_va_kernel_text(va)) 720 724 return PAGE_KERNEL_READ_EXEC; ··· 739 743 set_memory_ro); 740 744 } 741 745 #else 742 - static __init pgprot_t pgprot_from_va(uintptr_t va) 746 + static __meminit pgprot_t pgprot_from_va(uintptr_t va) 743 747 { 744 748 if (IS_ENABLED(CONFIG_64BIT) && !is_kernel_mapping(va)) 745 749 return PAGE_KERNEL; ··· 1231 1235 pt_ops_set_fixmap(); 1232 1236 } 1233 1237 1234 - static void __init create_linear_mapping_range(phys_addr_t start, 1235 - phys_addr_t end, 1236 - uintptr_t fixed_map_size) 1238 + static void __meminit create_linear_mapping_range(phys_addr_t start, phys_addr_t end, 1239 + uintptr_t fixed_map_size, const pgprot_t *pgprot) 1237 1240 { 1238 1241 phys_addr_t pa; 1239 1242 uintptr_t va, map_size; ··· 1243 1248 best_map_size(pa, va, end - pa); 1244 1249 1245 1250 create_pgd_mapping(swapper_pg_dir, va, pa, map_size, 1246 - pgprot_from_va(va)); 1251 + pgprot ? *pgprot : pgprot_from_va(va)); 1247 1252 } 1248 1253 } 1249 1254 ··· 1287 1292 if (end >= __pa(PAGE_OFFSET) + memory_limit) 1288 1293 end = __pa(PAGE_OFFSET) + memory_limit; 1289 1294 1290 - create_linear_mapping_range(start, end, 0); 1295 + create_linear_mapping_range(start, end, 0, NULL); 1291 1296 } 1292 1297 1293 1298 #ifdef CONFIG_STRICT_KERNEL_RWX 1294 - create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0); 1295 - create_linear_mapping_range(krodata_start, 1296 - krodata_start + krodata_size, 0); 1299 + create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0, NULL); 1300 + create_linear_mapping_range(krodata_start, krodata_start + krodata_size, 0, NULL); 1297 1301 1298 1302 memblock_clear_nomap(ktext_start, ktext_size); 1299 1303 memblock_clear_nomap(krodata_start, krodata_size); 1300 1304 #endif 1301 1305 1302 1306 #ifdef CONFIG_KFENCE 1303 - create_linear_mapping_range(kfence_pool, 1304 - kfence_pool + KFENCE_POOL_SIZE, 1305 - PAGE_SIZE); 1307 + create_linear_mapping_range(kfence_pool, kfence_pool + KFENCE_POOL_SIZE, PAGE_SIZE, NULL); 1306 1308 1307 1309 memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE); 1308 1310 #endif ··· 1431 1439 * memory hotplug, we are not able to update all the page tables with 1432 1440 * the new PMDs. 1433 1441 */ 1434 - return vmemmap_populate_hugepages(start, end, node, NULL); 1442 + return vmemmap_populate_hugepages(start, end, node, altmap); 1435 1443 } 1436 1444 #endif 1437 1445 ··· 1485 1493 panic("Failed to pre-allocate %s pages for %s area\n", lvl, area); 1486 1494 } 1487 1495 1496 + #define PAGE_END KASAN_SHADOW_START 1497 + 1488 1498 void __init pgtable_cache_init(void) 1489 1499 { 1490 1500 preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc"); 1491 1501 if (IS_ENABLED(CONFIG_MODULES)) 1492 1502 preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules"); 1503 + if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG)) { 1504 + preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap"); 1505 + preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map"); 1506 + if (IS_ENABLED(CONFIG_KASAN)) 1507 + preallocate_pgd_pages_range(KASAN_SHADOW_START, KASAN_SHADOW_END, "kasan"); 1508 + } 1493 1509 } 1494 1510 #endif 1495 1511 ··· 1534 1534 } 1535 1535 #endif /* CONFIG_MMU */ 1536 1536 #endif /* CONFIG_EXECMEM */ 1537 + 1538 + #ifdef CONFIG_MEMORY_HOTPLUG 1539 + static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) 1540 + { 1541 + struct page *page = pmd_page(*pmd); 1542 + struct ptdesc *ptdesc = page_ptdesc(page); 1543 + pte_t *pte; 1544 + int i; 1545 + 1546 + for (i = 0; i < PTRS_PER_PTE; i++) { 1547 + pte = pte_start + i; 1548 + if (!pte_none(*pte)) 1549 + return; 1550 + } 1551 + 1552 + pagetable_pte_dtor(ptdesc); 1553 + if (PageReserved(page)) 1554 + free_reserved_page(page); 1555 + else 1556 + pagetable_free(ptdesc); 1557 + pmd_clear(pmd); 1558 + } 1559 + 1560 + static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud) 1561 + { 1562 + struct page *page = pud_page(*pud); 1563 + struct ptdesc *ptdesc = page_ptdesc(page); 1564 + pmd_t *pmd; 1565 + int i; 1566 + 1567 + for (i = 0; i < PTRS_PER_PMD; i++) { 1568 + pmd = pmd_start + i; 1569 + if (!pmd_none(*pmd)) 1570 + return; 1571 + } 1572 + 1573 + pagetable_pmd_dtor(ptdesc); 1574 + if (PageReserved(page)) 1575 + free_reserved_page(page); 1576 + else 1577 + pagetable_free(ptdesc); 1578 + pud_clear(pud); 1579 + } 1580 + 1581 + static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d) 1582 + { 1583 + struct page *page = p4d_page(*p4d); 1584 + pud_t *pud; 1585 + int i; 1586 + 1587 + for (i = 0; i < PTRS_PER_PUD; i++) { 1588 + pud = pud_start + i; 1589 + if (!pud_none(*pud)) 1590 + return; 1591 + } 1592 + 1593 + if (PageReserved(page)) 1594 + free_reserved_page(page); 1595 + else 1596 + free_pages((unsigned long)page_address(page), 0); 1597 + p4d_clear(p4d); 1598 + } 1599 + 1600 + static void __meminit free_vmemmap_storage(struct page *page, size_t size, 1601 + struct vmem_altmap *altmap) 1602 + { 1603 + int order = get_order(size); 1604 + 1605 + if (altmap) { 1606 + vmem_altmap_free(altmap, size >> PAGE_SHIFT); 1607 + return; 1608 + } 1609 + 1610 + if (PageReserved(page)) { 1611 + unsigned int nr_pages = 1 << order; 1612 + 1613 + while (nr_pages--) 1614 + free_reserved_page(page++); 1615 + return; 1616 + } 1617 + 1618 + free_pages((unsigned long)page_address(page), order); 1619 + } 1620 + 1621 + static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, unsigned long end, 1622 + bool is_vmemmap, struct vmem_altmap *altmap) 1623 + { 1624 + unsigned long next; 1625 + pte_t *ptep, pte; 1626 + 1627 + for (; addr < end; addr = next) { 1628 + next = (addr + PAGE_SIZE) & PAGE_MASK; 1629 + if (next > end) 1630 + next = end; 1631 + 1632 + ptep = pte_base + pte_index(addr); 1633 + pte = ptep_get(ptep); 1634 + if (!pte_present(*ptep)) 1635 + continue; 1636 + 1637 + pte_clear(&init_mm, addr, ptep); 1638 + if (is_vmemmap) 1639 + free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap); 1640 + } 1641 + } 1642 + 1643 + static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long addr, unsigned long end, 1644 + bool is_vmemmap, struct vmem_altmap *altmap) 1645 + { 1646 + unsigned long next; 1647 + pte_t *pte_base; 1648 + pmd_t *pmdp, pmd; 1649 + 1650 + for (; addr < end; addr = next) { 1651 + next = pmd_addr_end(addr, end); 1652 + pmdp = pmd_base + pmd_index(addr); 1653 + pmd = pmdp_get(pmdp); 1654 + if (!pmd_present(pmd)) 1655 + continue; 1656 + 1657 + if (pmd_leaf(pmd)) { 1658 + pmd_clear(pmdp); 1659 + if (is_vmemmap) 1660 + free_vmemmap_storage(pmd_page(pmd), PMD_SIZE, altmap); 1661 + continue; 1662 + } 1663 + 1664 + pte_base = (pte_t *)pmd_page_vaddr(*pmdp); 1665 + remove_pte_mapping(pte_base, addr, next, is_vmemmap, altmap); 1666 + free_pte_table(pte_base, pmdp); 1667 + } 1668 + } 1669 + 1670 + static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unsigned long end, 1671 + bool is_vmemmap, struct vmem_altmap *altmap) 1672 + { 1673 + unsigned long next; 1674 + pud_t *pudp, pud; 1675 + pmd_t *pmd_base; 1676 + 1677 + for (; addr < end; addr = next) { 1678 + next = pud_addr_end(addr, end); 1679 + pudp = pud_base + pud_index(addr); 1680 + pud = pudp_get(pudp); 1681 + if (!pud_present(pud)) 1682 + continue; 1683 + 1684 + if (pud_leaf(pud)) { 1685 + if (pgtable_l4_enabled) { 1686 + pud_clear(pudp); 1687 + if (is_vmemmap) 1688 + free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap); 1689 + } 1690 + continue; 1691 + } 1692 + 1693 + pmd_base = pmd_offset(pudp, 0); 1694 + remove_pmd_mapping(pmd_base, addr, next, is_vmemmap, altmap); 1695 + 1696 + if (pgtable_l4_enabled) 1697 + free_pmd_table(pmd_base, pudp); 1698 + } 1699 + } 1700 + 1701 + static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long addr, unsigned long end, 1702 + bool is_vmemmap, struct vmem_altmap *altmap) 1703 + { 1704 + unsigned long next; 1705 + p4d_t *p4dp, p4d; 1706 + pud_t *pud_base; 1707 + 1708 + for (; addr < end; addr = next) { 1709 + next = p4d_addr_end(addr, end); 1710 + p4dp = p4d_base + p4d_index(addr); 1711 + p4d = p4dp_get(p4dp); 1712 + if (!p4d_present(p4d)) 1713 + continue; 1714 + 1715 + if (p4d_leaf(p4d)) { 1716 + if (pgtable_l5_enabled) { 1717 + p4d_clear(p4dp); 1718 + if (is_vmemmap) 1719 + free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap); 1720 + } 1721 + continue; 1722 + } 1723 + 1724 + pud_base = pud_offset(p4dp, 0); 1725 + remove_pud_mapping(pud_base, addr, next, is_vmemmap, altmap); 1726 + 1727 + if (pgtable_l5_enabled) 1728 + free_pud_table(pud_base, p4dp); 1729 + } 1730 + } 1731 + 1732 + static void __meminit remove_pgd_mapping(unsigned long va, unsigned long end, bool is_vmemmap, 1733 + struct vmem_altmap *altmap) 1734 + { 1735 + unsigned long addr, next; 1736 + p4d_t *p4d_base; 1737 + pgd_t *pgd; 1738 + 1739 + for (addr = va; addr < end; addr = next) { 1740 + next = pgd_addr_end(addr, end); 1741 + pgd = pgd_offset_k(addr); 1742 + 1743 + if (!pgd_present(*pgd)) 1744 + continue; 1745 + 1746 + if (pgd_leaf(*pgd)) 1747 + continue; 1748 + 1749 + p4d_base = p4d_offset(pgd, 0); 1750 + remove_p4d_mapping(p4d_base, addr, next, is_vmemmap, altmap); 1751 + } 1752 + 1753 + flush_tlb_all(); 1754 + } 1755 + 1756 + static void __meminit remove_linear_mapping(phys_addr_t start, u64 size) 1757 + { 1758 + unsigned long va = (unsigned long)__va(start); 1759 + unsigned long end = (unsigned long)__va(start + size); 1760 + 1761 + remove_pgd_mapping(va, end, false, NULL); 1762 + } 1763 + 1764 + struct range arch_get_mappable_range(void) 1765 + { 1766 + struct range mhp_range; 1767 + 1768 + mhp_range.start = __pa(PAGE_OFFSET); 1769 + mhp_range.end = __pa(PAGE_END - 1); 1770 + return mhp_range; 1771 + } 1772 + 1773 + int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) 1774 + { 1775 + int ret = 0; 1776 + 1777 + create_linear_mapping_range(start, start + size, 0, &params->pgprot); 1778 + ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, params); 1779 + if (ret) { 1780 + remove_linear_mapping(start, size); 1781 + goto out; 1782 + } 1783 + 1784 + max_pfn = PFN_UP(start + size); 1785 + max_low_pfn = max_pfn; 1786 + 1787 + out: 1788 + flush_tlb_all(); 1789 + return ret; 1790 + } 1791 + 1792 + void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) 1793 + { 1794 + __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap); 1795 + remove_linear_mapping(start, size); 1796 + flush_tlb_all(); 1797 + } 1798 + 1799 + void __ref vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap) 1800 + { 1801 + remove_pgd_mapping(start, end, true, altmap); 1802 + } 1803 + #endif /* CONFIG_MEMORY_HOTPLUG */

+3

arch/riscv/mm/ptdump.c

··· 6 6 #include <linux/efi.h> 7 7 #include <linux/init.h> 8 8 #include <linux/debugfs.h> 9 + #include <linux/memory_hotplug.h> 9 10 #include <linux/seq_file.h> 10 11 #include <linux/ptdump.h> 11 12 ··· 371 370 372 371 static int ptdump_show(struct seq_file *m, void *v) 373 372 { 373 + get_online_mems(); 374 374 ptdump_walk(m, m->private); 375 + put_online_mems(); 375 376 376 377 return 0; 377 378 }

+4 -3

arch/riscv/net/bpf_jit_comp64.c

··· 17 17 18 18 #define RV_MAX_REG_ARGS 8 19 19 #define RV_FENTRY_NINSNS 2 20 + #define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4) 20 21 /* imm that allows emit_imm to emit max count insns */ 21 22 #define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF 22 23 ··· 677 676 if (ret) 678 677 return ret; 679 678 680 - if (memcmp(ip, old_insns, RV_FENTRY_NINSNS * 4)) 679 + if (memcmp(ip, old_insns, RV_FENTRY_NBYTES)) 681 680 return -EFAULT; 682 681 683 682 ret = gen_jump_or_nops(new_addr, ip, new_insns, is_call); ··· 686 685 687 686 cpus_read_lock(); 688 687 mutex_lock(&text_mutex); 689 - if (memcmp(ip, new_insns, RV_FENTRY_NINSNS * 4)) 690 - ret = patch_text(ip, new_insns, RV_FENTRY_NINSNS); 688 + if (memcmp(ip, new_insns, RV_FENTRY_NBYTES)) 689 + ret = patch_text(ip, new_insns, RV_FENTRY_NBYTES); 691 690 mutex_unlock(&text_mutex); 692 691 cpus_read_unlock(); 693 692

+13

drivers/firmware/efi/riscv-runtime.c

··· 152 152 { 153 153 efi_virtmap_unload(); 154 154 } 155 + 156 + static int __init riscv_dmi_init(void) 157 + { 158 + /* 159 + * On riscv, DMI depends on UEFI, and dmi_setup() needs to 160 + * be called early because dmi_id_init(), which is an arch_initcall 161 + * itself, depends on dmi_scan_machine() having been called already. 162 + */ 163 + dmi_setup(); 164 + 165 + return 0; 166 + } 167 + core_initcall(riscv_dmi_init);

+1 -1

drivers/virtio/Kconfig

··· 122 122 123 123 config VIRTIO_MEM 124 124 tristate "Virtio mem driver" 125 - depends on X86_64 || ARM64 125 + depends on X86_64 || ARM64 || RISCV 126 126 depends on VIRTIO 127 127 depends on MEMORY_HOTPLUG 128 128 depends on MEMORY_HOTREMOVE

+3

include/linux/crc32.h

··· 9 9 #include <linux/bitrev.h> 10 10 11 11 u32 __pure crc32_le(u32 crc, unsigned char const *p, size_t len); 12 + u32 __pure crc32_le_base(u32 crc, unsigned char const *p, size_t len); 12 13 u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len); 14 + u32 __pure crc32_be_base(u32 crc, unsigned char const *p, size_t len); 13 15 14 16 /** 15 17 * crc32_le_combine - Combine two crc32 check values into one. For two ··· 39 37 } 40 38 41 39 u32 __pure __crc32c_le(u32 crc, unsigned char const *p, size_t len); 40 + u32 __pure __crc32c_le_base(u32 crc, unsigned char const *p, size_t len); 42 41 43 42 /** 44 43 * __crc32c_le_combine - Combine two crc32c check values into one. For two

+28

tools/testing/selftests/kvm/riscv/get-reg-list.c

··· 49 49 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SVNAPOT: 50 50 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SVPBMT: 51 51 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZACAS: 52 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZAWRS: 52 53 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBA: 53 54 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBB: 54 55 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBC: ··· 57 56 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBKC: 58 57 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBKX: 59 58 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZBS: 59 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZCA: 60 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZCB: 61 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZCD: 62 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZCF: 63 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZCMOP: 60 64 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZFA: 61 65 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZFH: 62 66 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZFHMIN: ··· 74 68 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZIHINTNTL: 75 69 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZIHINTPAUSE: 76 70 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZIHPM: 71 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZIMOP: 77 72 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZKND: 78 73 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZKNE: 79 74 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZKNH: ··· 422 415 KVM_ISA_EXT_ARR(SVNAPOT), 423 416 KVM_ISA_EXT_ARR(SVPBMT), 424 417 KVM_ISA_EXT_ARR(ZACAS), 418 + KVM_ISA_EXT_ARR(ZAWRS), 425 419 KVM_ISA_EXT_ARR(ZBA), 426 420 KVM_ISA_EXT_ARR(ZBB), 427 421 KVM_ISA_EXT_ARR(ZBC), ··· 430 422 KVM_ISA_EXT_ARR(ZBKC), 431 423 KVM_ISA_EXT_ARR(ZBKX), 432 424 KVM_ISA_EXT_ARR(ZBS), 425 + KVM_ISA_EXT_ARR(ZCA), 426 + KVM_ISA_EXT_ARR(ZCB), 427 + KVM_ISA_EXT_ARR(ZCD), 428 + KVM_ISA_EXT_ARR(ZCF), 429 + KVM_ISA_EXT_ARR(ZCMOP), 433 430 KVM_ISA_EXT_ARR(ZFA), 434 431 KVM_ISA_EXT_ARR(ZFH), 435 432 KVM_ISA_EXT_ARR(ZFHMIN), ··· 447 434 KVM_ISA_EXT_ARR(ZIHINTNTL), 448 435 KVM_ISA_EXT_ARR(ZIHINTPAUSE), 449 436 KVM_ISA_EXT_ARR(ZIHPM), 437 + KVM_ISA_EXT_ARR(ZIMOP), 450 438 KVM_ISA_EXT_ARR(ZKND), 451 439 KVM_ISA_EXT_ARR(ZKNE), 452 440 KVM_ISA_EXT_ARR(ZKNH), ··· 953 939 KVM_ISA_EXT_SIMPLE_CONFIG(svnapot, SVNAPOT); 954 940 KVM_ISA_EXT_SIMPLE_CONFIG(svpbmt, SVPBMT); 955 941 KVM_ISA_EXT_SIMPLE_CONFIG(zacas, ZACAS); 942 + KVM_ISA_EXT_SIMPLE_CONFIG(zawrs, ZAWRS); 956 943 KVM_ISA_EXT_SIMPLE_CONFIG(zba, ZBA); 957 944 KVM_ISA_EXT_SIMPLE_CONFIG(zbb, ZBB); 958 945 KVM_ISA_EXT_SIMPLE_CONFIG(zbc, ZBC); ··· 961 946 KVM_ISA_EXT_SIMPLE_CONFIG(zbkc, ZBKC); 962 947 KVM_ISA_EXT_SIMPLE_CONFIG(zbkx, ZBKX); 963 948 KVM_ISA_EXT_SIMPLE_CONFIG(zbs, ZBS); 949 + KVM_ISA_EXT_SIMPLE_CONFIG(zca, ZCA), 950 + KVM_ISA_EXT_SIMPLE_CONFIG(zcb, ZCB), 951 + KVM_ISA_EXT_SIMPLE_CONFIG(zcd, ZCD), 952 + KVM_ISA_EXT_SIMPLE_CONFIG(zcf, ZCF), 953 + KVM_ISA_EXT_SIMPLE_CONFIG(zcmop, ZCMOP); 964 954 KVM_ISA_EXT_SIMPLE_CONFIG(zfa, ZFA); 965 955 KVM_ISA_EXT_SIMPLE_CONFIG(zfh, ZFH); 966 956 KVM_ISA_EXT_SIMPLE_CONFIG(zfhmin, ZFHMIN); ··· 978 958 KVM_ISA_EXT_SIMPLE_CONFIG(zihintntl, ZIHINTNTL); 979 959 KVM_ISA_EXT_SIMPLE_CONFIG(zihintpause, ZIHINTPAUSE); 980 960 KVM_ISA_EXT_SIMPLE_CONFIG(zihpm, ZIHPM); 961 + KVM_ISA_EXT_SIMPLE_CONFIG(zimop, ZIMOP); 981 962 KVM_ISA_EXT_SIMPLE_CONFIG(zknd, ZKND); 982 963 KVM_ISA_EXT_SIMPLE_CONFIG(zkne, ZKNE); 983 964 KVM_ISA_EXT_SIMPLE_CONFIG(zknh, ZKNH); ··· 1016 995 &config_svnapot, 1017 996 &config_svpbmt, 1018 997 &config_zacas, 998 + &config_zawrs, 1019 999 &config_zba, 1020 1000 &config_zbb, 1021 1001 &config_zbc, ··· 1024 1002 &config_zbkc, 1025 1003 &config_zbkx, 1026 1004 &config_zbs, 1005 + &config_zca, 1006 + &config_zcb, 1007 + &config_zcd, 1008 + &config_zcf, 1009 + &config_zcmop, 1027 1010 &config_zfa, 1028 1011 &config_zfh, 1029 1012 &config_zfhmin, ··· 1041 1014 &config_zihintntl, 1042 1015 &config_zihintpause, 1043 1016 &config_zihpm, 1017 + &config_zimop, 1044 1018 &config_zknd, 1045 1019 &config_zkne, 1046 1020 &config_zknh,

+3 -3

tools/testing/selftests/riscv/vector/vstate_prctl.c

··· 88 88 return -2; 89 89 } 90 90 91 - if (!(pair.value & RISCV_HWPROBE_IMA_V)) { 91 + if (!(pair.value & RISCV_HWPROBE_EXT_ZVE32X)) { 92 92 rc = prctl(PR_RISCV_V_GET_CONTROL); 93 93 if (rc != -1 || errno != EINVAL) { 94 - ksft_test_result_fail("GET_CONTROL should fail on kernel/hw without V\n"); 94 + ksft_test_result_fail("GET_CONTROL should fail on kernel/hw without ZVE32X\n"); 95 95 return -3; 96 96 } 97 97 98 98 rc = prctl(PR_RISCV_V_SET_CONTROL, PR_RISCV_V_VSTATE_CTRL_ON); 99 99 if (rc != -1 || errno != EINVAL) { 100 - ksft_test_result_fail("GET_CONTROL should fail on kernel/hw without V\n"); 100 + ksft_test_result_fail("SET_CONTROL should fail on kernel/hw without ZVE32X\n"); 101 101 return -4; 102 102 } 103 103

Configure Feed

Configure Feed