Merge tag 'riscv-for-linus-6.13-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

+19

Documentation/arch/riscv/hwprobe.rst

··· 239 239 ratified in commit 98918c844281 ("Merge pull request #1217 from 240 240 riscv/zawrs") of riscv-isa-manual. 241 241 242 + * :c:macro:`RISCV_HWPROBE_EXT_SUPM`: The Supm extension is supported as 243 + defined in version 1.0 of the RISC-V Pointer Masking extensions. 244 + 242 245 * :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: Deprecated. Returns similar values to 243 246 :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was 244 247 mistakenly classified as a bitmask rather than a value. ··· 277 274 represent the highest userspace virtual address usable. 278 275 279 276 * :c:macro:`RISCV_HWPROBE_KEY_TIME_CSR_FREQ`: Frequency (in Hz) of `time CSR`. 277 + 278 + * :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF`: An enum value describing the 279 + performance of misaligned vector accesses on the selected set of processors. 280 + 281 + * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN`: The performance of misaligned 282 + vector accesses is unknown. 283 + 284 + * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW`: 32-bit misaligned accesses using vector 285 + registers are slower than the equivalent quantity of byte accesses via vector registers. 286 + Misaligned accesses may be supported directly in hardware, or trapped and emulated by software. 287 + 288 + * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_FAST`: 32-bit misaligned accesses using vector 289 + registers are faster than the equivalent quantity of byte accesses via vector registers. 290 + 291 + * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED`: Misaligned vector accesses are 292 + not supported at all and will generate a misaligned address fault.

+16

Documentation/arch/riscv/uabi.rst

··· 68 68 Misaligned scalar accesses are supported in userspace, but they may perform 69 69 poorly. Misaligned vector accesses are only supported if the Zicclsm extension 70 70 is supported. 71 + 72 + Pointer masking 73 + --------------- 74 + 75 + Support for pointer masking in userspace (the Supm extension) is provided via 76 + the ``PR_SET_TAGGED_ADDR_CTRL`` and ``PR_GET_TAGGED_ADDR_CTRL`` ``prctl()`` 77 + operations. Pointer masking is disabled by default. To enable it, userspace 78 + must call ``PR_SET_TAGGED_ADDR_CTRL`` with the ``PR_PMLEN`` field set to the 79 + number of mask/tag bits needed by the application. ``PR_PMLEN`` is interpreted 80 + as a lower bound; if the kernel is unable to satisfy the request, the 81 + ``PR_SET_TAGGED_ADDR_CTRL`` operation will fail. The actual number of tag bits 82 + is returned in ``PR_PMLEN`` by the ``PR_GET_TAGGED_ADDR_CTRL`` operation. 83 + 84 + Additionally, when pointer masking is enabled (``PR_PMLEN`` is greater than 0), 85 + a tagged address ABI is supported, with the same interface and behavior as 86 + documented for AArch64 (Documentation/arch/arm64/tagged-address-abi.rst).

+30

Documentation/devicetree/bindings/riscv/extensions.yaml

··· 128 128 changes to interrupts as frozen at commit ccbddab ("Merge pull 129 129 request #42 from riscv/jhauser-2023-RC4") of riscv-aia. 130 130 131 + - const: smmpm 132 + description: | 133 + The standard Smmpm extension for M-mode pointer masking as 134 + ratified at commit d70011dde6c2 ("Update to ratified state") 135 + of riscv-j-extension. 136 + 137 + - const: smnpm 138 + description: | 139 + The standard Smnpm extension for next-mode pointer masking as 140 + ratified at commit d70011dde6c2 ("Update to ratified state") 141 + of riscv-j-extension. 142 + 131 143 - const: smstateen 132 144 description: | 133 145 The standard Smstateen extension for controlling access to CSRs ··· 158 146 The standard Sscofpmf supervisor-level extension for count overflow 159 147 and mode-based filtering as ratified at commit 01d1df0 ("Add ability 160 148 to manually trigger workflow. (#2)") of riscv-count-overflow. 149 + 150 + - const: ssnpm 151 + description: | 152 + The standard Ssnpm extension for next-mode pointer masking as 153 + ratified at commit d70011dde6c2 ("Update to ratified state") 154 + of riscv-j-extension. 161 155 162 156 - const: sstc 163 157 description: | ··· 195 177 address-translation cache behaviour with respect to invalid entries 196 178 as ratified at commit 4a69197e5617 ("Update to ratified state") of 197 179 riscv-svvptc. 180 + 181 + - const: zabha 182 + description: | 183 + The Zabha extension for Byte and Halfword Atomic Memory Operations 184 + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of 185 + riscv-zabha. 198 186 199 187 - const: zacas 200 188 description: | ··· 313 289 16-bit half-precision binary floating-point instructions, as ratified 314 290 in commit 64074bc ("Update version numbers for Zfh/Zfinx") of 315 291 riscv-isa-manual. 292 + 293 + - const: ziccrse 294 + description: 295 + The standard Ziccrse extension which provides forward progress 296 + guarantee on LR/SC sequences, as ratified in commit b1d806605f87 297 + ("Updated to ratified state.") of the riscv profiles specification. 316 298 317 299 - const: zk 318 300 description:

+1 -1

Documentation/features/locking/queued-spinlocks/arch-support.txt

··· 20 20 | openrisc: | ok | 21 21 | parisc: | TODO | 22 22 | powerpc: | ok | 23 - | riscv: | TODO | 23 + | riscv: | ok | 24 24 | s390: | TODO | 25 25 | sh: | TODO | 26 26 | sparc: | ok |

+136 -2

arch/riscv/Kconfig

··· 83 83 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP 84 84 select ARCH_WANTS_NO_INSTR 85 85 select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE 86 + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS 86 87 select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU 87 88 select BUILDTIME_TABLE_SORT if MMU 88 89 select CLINT_TIMER if RISCV_M_MODE ··· 117 116 select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO 118 117 select HARDIRQS_SW_RESEND 119 118 select HAS_IOPORT if MMU 119 + select HAVE_ALIGNED_STRUCT_PAGE 120 120 select HAVE_ARCH_AUDITSYSCALL 121 121 select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP 122 122 select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT ··· 509 507 Specify the maximum number of NUMA Nodes available on the target 510 508 system. Increases memory reserved to accommodate various tables. 511 509 510 + choice 511 + prompt "RISC-V spinlock type" 512 + default RISCV_COMBO_SPINLOCKS 513 + 514 + config RISCV_TICKET_SPINLOCKS 515 + bool "Using ticket spinlock" 516 + 517 + config RISCV_QUEUED_SPINLOCKS 518 + bool "Using queued spinlock" 519 + depends on SMP && MMU && NONPORTABLE 520 + select ARCH_USE_QUEUED_SPINLOCKS 521 + help 522 + The queued spinlock implementation requires the forward progress 523 + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or 524 + LR/SC with Ziccrse provide such guarantee. 525 + 526 + Select this if and only if Zabha or Ziccrse is available on your 527 + platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms 528 + without one of those extensions. 529 + 530 + If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks 531 + when supported and otherwise ticket spinlocks. 532 + 533 + config RISCV_COMBO_SPINLOCKS 534 + bool "Using combo spinlock" 535 + depends on SMP && MMU 536 + select ARCH_USE_QUEUED_SPINLOCKS 537 + help 538 + Embed both queued spinlock and ticket lock so that the spinlock 539 + implementation can be chosen at runtime. 540 + 541 + endchoice 542 + 512 543 config RISCV_ALTERNATIVE 513 544 bool 514 545 depends on !XIP_KERNEL ··· 566 531 Linux binary. 567 532 568 533 If you don't know what to do here, say Y. 534 + 535 + config RISCV_ISA_SUPM 536 + bool "Supm extension for userspace pointer masking" 537 + depends on 64BIT 538 + default y 539 + help 540 + Add support for pointer masking in userspace (Supm) when the 541 + underlying hardware extension (Smnpm or Ssnpm) is detected at boot. 542 + 543 + If this option is disabled, userspace will be unable to use 544 + the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API. 569 545 570 546 config RISCV_ISA_SVNAPOT 571 547 bool "Svnapot extension support for supervisor mode NAPOT pages" ··· 678 632 hypervisor while waiting on a store to a memory location. Enable the 679 633 use of these instructions in the kernel when the Zawrs extension is 680 634 detected at boot. 635 + 636 + config TOOLCHAIN_HAS_ZABHA 637 + bool 638 + default y 639 + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) 640 + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) 641 + depends on AS_HAS_OPTION_ARCH 642 + 643 + config RISCV_ISA_ZABHA 644 + bool "Zabha extension support for atomic byte/halfword operations" 645 + depends on TOOLCHAIN_HAS_ZABHA 646 + depends on RISCV_ALTERNATIVE 647 + default y 648 + help 649 + Enable the use of the Zabha ISA-extension to implement kernel 650 + byte/halfword atomic memory operations when it is detected at boot. 651 + 652 + If you don't know what to do here, say Y. 653 + 654 + config TOOLCHAIN_HAS_ZACAS 655 + bool 656 + default y 657 + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) 658 + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) 659 + depends on AS_HAS_OPTION_ARCH 660 + 661 + config RISCV_ISA_ZACAS 662 + bool "Zacas extension support for atomic CAS" 663 + depends on TOOLCHAIN_HAS_ZACAS 664 + depends on RISCV_ALTERNATIVE 665 + default y 666 + help 667 + Enable the use of the Zacas ISA-extension to implement kernel atomic 668 + cmpxchg operations when it is detected at boot. 681 669 682 670 If you don't know what to do here, say Y. 683 671 ··· 866 786 867 787 config RISCV_MISALIGNED 868 788 bool 789 + help 790 + Embed support for detecting and emulating misaligned 791 + scalar or vector loads and stores. 792 + 793 + config RISCV_SCALAR_MISALIGNED 794 + bool 795 + select RISCV_MISALIGNED 869 796 select SYSCTL_ARCH_UNALIGN_ALLOW 870 797 help 871 798 Embed support for emulating misaligned loads and stores. 799 + 800 + config RISCV_VECTOR_MISALIGNED 801 + bool 802 + select RISCV_MISALIGNED 803 + depends on RISCV_ISA_V 804 + help 805 + Enable detecting support for vector misaligned loads and stores. 872 806 873 807 choice 874 808 prompt "Unaligned Accesses Support" ··· 895 801 896 802 config RISCV_PROBE_UNALIGNED_ACCESS 897 803 bool "Probe for hardware unaligned access support" 898 - select RISCV_MISALIGNED 804 + select RISCV_SCALAR_MISALIGNED 899 805 help 900 806 During boot, the kernel will run a series of tests to determine the 901 807 speed of unaligned accesses. This probing will dynamically determine ··· 906 812 907 813 config RISCV_EMULATED_UNALIGNED_ACCESS 908 814 bool "Emulate unaligned access where system support is missing" 909 - select RISCV_MISALIGNED 815 + select RISCV_SCALAR_MISALIGNED 910 816 help 911 817 If unaligned memory accesses trap into the kernel as they are not 912 818 supported by the system, the kernel will emulate the unaligned ··· 928 834 select HAVE_EFFICIENT_UNALIGNED_ACCESS 929 835 help 930 836 Assume that the system supports fast unaligned memory accesses. When 837 + enabled, this option improves the performance of the kernel on such 838 + systems. However, the kernel and userspace programs will run much more 839 + slowly, or will not be able to run at all, on systems that do not 840 + support efficient unaligned memory accesses. 841 + 842 + endchoice 843 + 844 + choice 845 + prompt "Vector unaligned Accesses Support" 846 + depends on RISCV_ISA_V 847 + default RISCV_PROBE_VECTOR_UNALIGNED_ACCESS 848 + help 849 + This determines the level of support for vector unaligned accesses. This 850 + information is used by the kernel to perform optimizations. It is also 851 + exposed to user space via the hwprobe syscall. The hardware will be 852 + probed at boot by default. 853 + 854 + config RISCV_PROBE_VECTOR_UNALIGNED_ACCESS 855 + bool "Probe speed of vector unaligned accesses" 856 + select RISCV_VECTOR_MISALIGNED 857 + depends on RISCV_ISA_V 858 + help 859 + During boot, the kernel will run a series of tests to determine the 860 + speed of vector unaligned accesses if they are supported. This probing 861 + will dynamically determine the speed of vector unaligned accesses on 862 + the underlying system if they are supported. 863 + 864 + config RISCV_SLOW_VECTOR_UNALIGNED_ACCESS 865 + bool "Assume the system supports slow vector unaligned memory accesses" 866 + depends on NONPORTABLE 867 + help 868 + Assume that the system supports slow vector unaligned memory accesses. The 869 + kernel and userspace programs may not be able to run at all on systems 870 + that do not support unaligned memory accesses. 871 + 872 + config RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS 873 + bool "Assume the system supports fast vector unaligned memory accesses" 874 + depends on NONPORTABLE 875 + help 876 + Assume that the system supports fast vector unaligned memory accesses. When 931 877 enabled, this option improves the performance of the kernel on such 932 878 systems. However, the kernel and userspace programs will run much more 933 879 slowly, or will not be able to run at all, on systems that do not

+6

arch/riscv/Makefile

··· 82 82 riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei 83 83 endif 84 84 85 + # Check if the toolchain supports Zacas 86 + riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas 87 + 88 + # Check if the toolchain supports Zabha 89 + riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha 90 + 85 91 # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by 86 92 # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) 87 93 KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')

+1

arch/riscv/configs/defconfig

··· 167 167 CONFIG_PINCTRL_SOPHGO_CV1812H=y 168 168 CONFIG_PINCTRL_SOPHGO_SG2000=y 169 169 CONFIG_PINCTRL_SOPHGO_SG2002=y 170 + CONFIG_GPIO_DWAPB=y 170 171 CONFIG_GPIO_SIFIVE=y 171 172 CONFIG_POWER_RESET_GPIO_RESTART=y 172 173 CONFIG_SENSORS_SFCTEMP=m

+3 -1

arch/riscv/include/asm/Kbuild

··· 6 6 generic-y += flat.h 7 7 generic-y += kvm_para.h 8 8 generic-y += mmzone.h 9 + generic-y += mcs_spinlock.h 9 10 generic-y += parport.h 10 - generic-y += spinlock.h 11 11 generic-y += spinlock_types.h 12 + generic-y += ticket_spinlock.h 12 13 generic-y += qrwlock.h 13 14 generic-y += qrwlock_types.h 15 + generic-y += qspinlock.h 14 16 generic-y += user.h 15 17 generic-y += vmlinux.lds.h

+207 -77

arch/riscv/include/asm/cmpxchg.h

··· 12 12 #include <asm/fence.h> 13 13 #include <asm/hwcap.h> 14 14 #include <asm/insn-def.h> 15 + #include <asm/cpufeature-macros.h> 15 16 16 - #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ 17 - ({ \ 18 - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ 19 - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ 20 - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ 21 - << __s; \ 22 - ulong __newx = (ulong)(n) << __s; \ 23 - ulong __retx; \ 24 - ulong __rc; \ 25 - \ 26 - __asm__ __volatile__ ( \ 27 - prepend \ 28 - "0: lr.w %0, %2\n" \ 29 - " and %1, %0, %z4\n" \ 30 - " or %1, %1, %z3\n" \ 31 - " sc.w" sc_sfx " %1, %1, %2\n" \ 32 - " bnez %1, 0b\n" \ 33 - append \ 34 - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ 35 - : "rJ" (__newx), "rJ" (~__mask) \ 36 - : "memory"); \ 37 - \ 38 - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ 17 + #define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ 18 + swap_append, r, p, n) \ 19 + ({ \ 20 + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ 21 + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ 22 + __asm__ __volatile__ ( \ 23 + prepend \ 24 + " amoswap" swap_sfx " %0, %z2, %1\n" \ 25 + swap_append \ 26 + : "=&r" (r), "+A" (*(p)) \ 27 + : "rJ" (n) \ 28 + : "memory"); \ 29 + } else { \ 30 + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ 31 + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ 32 + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ 33 + << __s; \ 34 + ulong __newx = (ulong)(n) << __s; \ 35 + ulong __retx; \ 36 + ulong __rc; \ 37 + \ 38 + __asm__ __volatile__ ( \ 39 + prepend \ 40 + "0: lr.w %0, %2\n" \ 41 + " and %1, %0, %z4\n" \ 42 + " or %1, %1, %z3\n" \ 43 + " sc.w" sc_sfx " %1, %1, %2\n" \ 44 + " bnez %1, 0b\n" \ 45 + sc_append \ 46 + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ 47 + : "rJ" (__newx), "rJ" (~__mask) \ 48 + : "memory"); \ 49 + \ 50 + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ 51 + } \ 39 52 }) 40 53 41 54 #define __arch_xchg(sfx, prepend, append, r, p, n) \ ··· 71 58 \ 72 59 switch (sizeof(*__ptr)) { \ 73 60 case 1: \ 61 + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ 62 + prepend, sc_append, swap_append, \ 63 + __ret, __ptr, __new); \ 64 + break; \ 74 65 case 2: \ 75 - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ 66 + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ 67 + prepend, sc_append, swap_append, \ 76 68 __ret, __ptr, __new); \ 77 69 break; \ 78 70 case 4: \ ··· 124 106 * store NEW in MEM. Return the initial value in MEM. Success is 125 107 * indicated by comparing RETURN with OLD. 126 108 */ 127 - 128 - #define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ 129 - ({ \ 130 - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ 131 - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ 132 - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ 133 - << __s; \ 134 - ulong __newx = (ulong)(n) << __s; \ 135 - ulong __oldx = (ulong)(o) << __s; \ 136 - ulong __retx; \ 137 - ulong __rc; \ 138 - \ 139 - __asm__ __volatile__ ( \ 140 - prepend \ 141 - "0: lr.w %0, %2\n" \ 142 - " and %1, %0, %z5\n" \ 143 - " bne %1, %z3, 1f\n" \ 144 - " and %1, %0, %z6\n" \ 145 - " or %1, %1, %z4\n" \ 146 - " sc.w" sc_sfx " %1, %1, %2\n" \ 147 - " bnez %1, 0b\n" \ 148 - append \ 149 - "1:\n" \ 150 - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ 151 - : "rJ" ((long)__oldx), "rJ" (__newx), \ 152 - "rJ" (__mask), "rJ" (~__mask) \ 153 - : "memory"); \ 154 - \ 155 - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ 109 + #define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ 110 + sc_prepend, sc_append, \ 111 + cas_prepend, cas_append, \ 112 + r, p, o, n) \ 113 + ({ \ 114 + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ 115 + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ 116 + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ 117 + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ 118 + r = o; \ 119 + \ 120 + __asm__ __volatile__ ( \ 121 + cas_prepend \ 122 + " amocas" cas_sfx " %0, %z2, %1\n" \ 123 + cas_append \ 124 + : "+&r" (r), "+A" (*(p)) \ 125 + : "rJ" (n) \ 126 + : "memory"); \ 127 + } else { \ 128 + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ 129 + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ 130 + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ 131 + << __s; \ 132 + ulong __newx = (ulong)(n) << __s; \ 133 + ulong __oldx = (ulong)(o) << __s; \ 134 + ulong __retx; \ 135 + ulong __rc; \ 136 + \ 137 + __asm__ __volatile__ ( \ 138 + sc_prepend \ 139 + "0: lr.w %0, %2\n" \ 140 + " and %1, %0, %z5\n" \ 141 + " bne %1, %z3, 1f\n" \ 142 + " and %1, %0, %z6\n" \ 143 + " or %1, %1, %z4\n" \ 144 + " sc.w" sc_sfx " %1, %1, %2\n" \ 145 + " bnez %1, 0b\n" \ 146 + sc_append \ 147 + "1:\n" \ 148 + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ 149 + : "rJ" ((long)__oldx), "rJ" (__newx), \ 150 + "rJ" (__mask), "rJ" (~__mask) \ 151 + : "memory"); \ 152 + \ 153 + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ 154 + } \ 156 155 }) 157 156 158 - #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ 157 + #define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ 158 + sc_prepend, sc_append, \ 159 + cas_prepend, cas_append, \ 160 + r, p, co, o, n) \ 159 161 ({ \ 160 - register unsigned int __rc; \ 162 + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ 163 + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ 164 + r = o; \ 161 165 \ 162 - __asm__ __volatile__ ( \ 163 - prepend \ 164 - "0: lr" lr_sfx " %0, %2\n" \ 165 - " bne %0, %z3, 1f\n" \ 166 - " sc" sc_sfx " %1, %z4, %2\n" \ 167 - " bnez %1, 0b\n" \ 168 - append \ 169 - "1:\n" \ 170 - : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ 171 - : "rJ" (co o), "rJ" (n) \ 172 - : "memory"); \ 166 + __asm__ __volatile__ ( \ 167 + cas_prepend \ 168 + " amocas" cas_sfx " %0, %z2, %1\n" \ 169 + cas_append \ 170 + : "+&r" (r), "+A" (*(p)) \ 171 + : "rJ" (n) \ 172 + : "memory"); \ 173 + } else { \ 174 + register unsigned int __rc; \ 175 + \ 176 + __asm__ __volatile__ ( \ 177 + sc_prepend \ 178 + "0: lr" lr_sfx " %0, %2\n" \ 179 + " bne %0, %z3, 1f\n" \ 180 + " sc" sc_sfx " %1, %z4, %2\n" \ 181 + " bnez %1, 0b\n" \ 182 + sc_append \ 183 + "1:\n" \ 184 + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ 185 + : "rJ" (co o), "rJ" (n) \ 186 + : "memory"); \ 187 + } \ 173 188 }) 174 189 175 - #define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ 190 + #define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ 191 + sc_prepend, sc_append, \ 192 + cas_prepend, cas_append) \ 176 193 ({ \ 177 194 __typeof__(ptr) __ptr = (ptr); \ 178 195 __typeof__(*(__ptr)) __old = (old); \ ··· 216 163 \ 217 164 switch (sizeof(*__ptr)) { \ 218 165 case 1: \ 166 + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ 167 + sc_prepend, sc_append, \ 168 + cas_prepend, cas_append, \ 169 + __ret, __ptr, __old, __new); \ 170 + break; \ 219 171 case 2: \ 220 - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ 221 - __ret, __ptr, __old, __new); \ 172 + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ 173 + sc_prepend, sc_append, \ 174 + cas_prepend, cas_append, \ 175 + __ret, __ptr, __old, __new); \ 222 176 break; \ 223 177 case 4: \ 224 - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ 225 - __ret, __ptr, (long), __old, __new); \ 178 + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ 179 + sc_prepend, sc_append, \ 180 + cas_prepend, cas_append, \ 181 + __ret, __ptr, (long), __old, __new); \ 226 182 break; \ 227 183 case 8: \ 228 - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ 229 - __ret, __ptr, /**/, __old, __new); \ 184 + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ 185 + sc_prepend, sc_append, \ 186 + cas_prepend, cas_append, \ 187 + __ret, __ptr, /**/, __old, __new); \ 230 188 break; \ 231 189 default: \ 232 190 BUILD_BUG(); \ ··· 245 181 (__typeof__(*(__ptr)))__ret; \ 246 182 }) 247 183 184 + /* 185 + * These macros are here to improve the readability of the arch_cmpxchg_XXX() 186 + * macros. 187 + */ 188 + #define SC_SFX(x) x 189 + #define CAS_SFX(x) x 190 + #define SC_PREPEND(x) x 191 + #define SC_APPEND(x) x 192 + #define CAS_PREPEND(x) x 193 + #define CAS_APPEND(x) x 194 + 248 195 #define arch_cmpxchg_relaxed(ptr, o, n) \ 249 - _arch_cmpxchg((ptr), (o), (n), "", "", "") 196 + _arch_cmpxchg((ptr), (o), (n), \ 197 + SC_SFX(""), CAS_SFX(""), \ 198 + SC_PREPEND(""), SC_APPEND(""), \ 199 + CAS_PREPEND(""), CAS_APPEND("")) 250 200 251 201 #define arch_cmpxchg_acquire(ptr, o, n) \ 252 - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) 202 + _arch_cmpxchg((ptr), (o), (n), \ 203 + SC_SFX(""), CAS_SFX(""), \ 204 + SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \ 205 + CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER)) 253 206 254 207 #define arch_cmpxchg_release(ptr, o, n) \ 255 - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") 208 + _arch_cmpxchg((ptr), (o), (n), \ 209 + SC_SFX(""), CAS_SFX(""), \ 210 + SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \ 211 + CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND("")) 256 212 257 213 #define arch_cmpxchg(ptr, o, n) \ 258 - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") 214 + _arch_cmpxchg((ptr), (o), (n), \ 215 + SC_SFX(".rl"), CAS_SFX(".aqrl"), \ 216 + SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \ 217 + CAS_PREPEND(""), CAS_APPEND("")) 259 218 260 219 #define arch_cmpxchg_local(ptr, o, n) \ 261 220 arch_cmpxchg_relaxed((ptr), (o), (n)) ··· 313 226 arch_cmpxchg_release((ptr), (o), (n)); \ 314 227 }) 315 228 229 + #if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) 230 + 231 + #define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) 232 + 233 + union __u128_halves { 234 + u128 full; 235 + struct { 236 + u64 low, high; 237 + }; 238 + }; 239 + 240 + #define __arch_cmpxchg128(p, o, n, cas_sfx) \ 241 + ({ \ 242 + __typeof__(*(p)) __o = (o); \ 243 + union __u128_halves __hn = { .full = (n) }; \ 244 + union __u128_halves __ho = { .full = (__o) }; \ 245 + register unsigned long t1 asm ("t1") = __hn.low; \ 246 + register unsigned long t2 asm ("t2") = __hn.high; \ 247 + register unsigned long t3 asm ("t3") = __ho.low; \ 248 + register unsigned long t4 asm ("t4") = __ho.high; \ 249 + \ 250 + __asm__ __volatile__ ( \ 251 + " amocas.q" cas_sfx " %0, %z3, %2" \ 252 + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ 253 + : "rJ" (t1), "rJ" (t2) \ 254 + : "memory"); \ 255 + \ 256 + ((u128)t4 << 64) | t3; \ 257 + }) 258 + 259 + #define arch_cmpxchg128(ptr, o, n) \ 260 + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") 261 + 262 + #define arch_cmpxchg128_local(ptr, o, n) \ 263 + __arch_cmpxchg128((ptr), (o), (n), "") 264 + 265 + #endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ 266 + 316 267 #ifdef CONFIG_RISCV_ISA_ZAWRS 317 268 /* 318 269 * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to ··· 370 245 : : : : no_zawrs); 371 246 372 247 switch (size) { 248 + case 1: 249 + fallthrough; 250 + case 2: 251 + /* RISC-V doesn't have lr instructions on byte and half-word. */ 252 + goto no_zawrs; 373 253 case 4: 374 254 asm volatile( 375 255 " lr.w %0, %1\n"

-1

arch/riscv/include/asm/compat.h

··· 9 9 */ 10 10 #include <linux/types.h> 11 11 #include <linux/sched.h> 12 - #include <linux/sched/task_stack.h> 13 12 #include <asm-generic/compat.h> 14 13 15 14 static inline int is_compat_task(void)

+66

arch/riscv/include/asm/cpufeature-macros.h

··· 1 + /* SPDX-License-Identifier: GPL-2.0-only */ 2 + /* 3 + * Copyright 2022-2024 Rivos, Inc 4 + */ 5 + 6 + #ifndef _ASM_CPUFEATURE_MACROS_H 7 + #define _ASM_CPUFEATURE_MACROS_H 8 + 9 + #include <asm/hwcap.h> 10 + #include <asm/alternative-macros.h> 11 + 12 + #define STANDARD_EXT 0 13 + 14 + bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); 15 + #define riscv_isa_extension_available(isa_bitmap, ext) \ 16 + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) 17 + 18 + static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, 19 + const unsigned long ext) 20 + { 21 + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) 22 + : 23 + : [vendor] "i" (vendor), [ext] "i" (ext) 24 + : 25 + : l_no); 26 + 27 + return true; 28 + l_no: 29 + return false; 30 + } 31 + 32 + static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, 33 + const unsigned long ext) 34 + { 35 + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) 36 + : 37 + : [vendor] "i" (vendor), [ext] "i" (ext) 38 + : 39 + : l_yes); 40 + 41 + return false; 42 + l_yes: 43 + return true; 44 + } 45 + 46 + static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) 47 + { 48 + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); 49 + 50 + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) 51 + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); 52 + 53 + return __riscv_isa_extension_available(NULL, ext); 54 + } 55 + 56 + static __always_inline bool riscv_has_extension_likely(const unsigned long ext) 57 + { 58 + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); 59 + 60 + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) 61 + return __riscv_has_extension_likely(STANDARD_EXT, ext); 62 + 63 + return __riscv_isa_extension_available(NULL, ext); 64 + } 65 + 66 + #endif /* _ASM_CPUFEATURE_MACROS_H */

+14 -59

arch/riscv/include/asm/cpufeature.h

··· 8 8 9 9 #include <linux/bitmap.h> 10 10 #include <linux/jump_label.h> 11 + #include <linux/workqueue.h> 12 + #include <linux/kconfig.h> 13 + #include <linux/percpu-defs.h> 14 + #include <linux/threads.h> 11 15 #include <asm/hwcap.h> 12 - #include <asm/alternative-macros.h> 13 - #include <asm/errno.h> 16 + #include <asm/cpufeature-macros.h> 14 17 15 18 /* 16 19 * These are probed via a device_initcall(), via either the SBI or directly ··· 34 31 /* Per-cpu ISA extensions. */ 35 32 extern struct riscv_isainfo hart_isa[NR_CPUS]; 36 33 37 - void riscv_user_isa_enable(void); 34 + void __init riscv_user_isa_enable(void); 38 35 39 36 #define _RISCV_ISA_EXT_DATA(_name, _id, _subset_exts, _subset_exts_size, _validate) { \ 40 37 .name = #_name, \ ··· 61 58 #define __RISCV_ISA_EXT_SUPERSET_VALIDATE(_name, _id, _sub_exts, _validate) \ 62 59 _RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), _validate) 63 60 64 - #if defined(CONFIG_RISCV_MISALIGNED) 65 61 bool check_unaligned_access_emulated_all_cpus(void); 62 + #if defined(CONFIG_RISCV_SCALAR_MISALIGNED) 63 + void check_unaligned_access_emulated(struct work_struct *work __always_unused); 66 64 void unaligned_emulation_finish(void); 67 65 bool unaligned_ctl_available(void); 68 66 DECLARE_PER_CPU(long, misaligned_access_speed); ··· 72 68 { 73 69 return false; 74 70 } 71 + #endif 72 + 73 + bool check_vector_unaligned_access_emulated_all_cpus(void); 74 + #if defined(CONFIG_RISCV_VECTOR_MISALIGNED) 75 + void check_vector_unaligned_access_emulated(struct work_struct *work __always_unused); 76 + DECLARE_PER_CPU(long, vector_misaligned_access); 75 77 #endif 76 78 77 79 #if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) ··· 113 103 extern bool riscv_isa_fallback; 114 104 115 105 unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); 116 - 117 - #define STANDARD_EXT 0 118 - 119 - bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); 120 - #define riscv_isa_extension_available(isa_bitmap, ext) \ 121 - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) 122 - 123 - static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, 124 - const unsigned long ext) 125 - { 126 - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) 127 - : 128 - : [vendor] "i" (vendor), [ext] "i" (ext) 129 - : 130 - : l_no); 131 - 132 - return true; 133 - l_no: 134 - return false; 135 - } 136 - 137 - static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, 138 - const unsigned long ext) 139 - { 140 - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) 141 - : 142 - : [vendor] "i" (vendor), [ext] "i" (ext) 143 - : 144 - : l_yes); 145 - 146 - return false; 147 - l_yes: 148 - return true; 149 - } 150 - 151 - static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) 152 - { 153 - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); 154 - 155 - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) 156 - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); 157 - 158 - return __riscv_isa_extension_available(NULL, ext); 159 - } 160 - 161 - static __always_inline bool riscv_has_extension_likely(const unsigned long ext) 162 - { 163 - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); 164 - 165 - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) 166 - return __riscv_has_extension_likely(STANDARD_EXT, ext); 167 - 168 - return __riscv_isa_extension_available(NULL, ext); 169 - } 170 - 171 106 static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext) 172 107 { 173 108 compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");

+16

arch/riscv/include/asm/csr.h

··· 119 119 120 120 /* HSTATUS flags */ 121 121 #ifdef CONFIG_64BIT 122 + #define HSTATUS_HUPMM _AC(0x3000000000000, UL) 123 + #define HSTATUS_HUPMM_PMLEN_0 _AC(0x0000000000000, UL) 124 + #define HSTATUS_HUPMM_PMLEN_7 _AC(0x2000000000000, UL) 125 + #define HSTATUS_HUPMM_PMLEN_16 _AC(0x3000000000000, UL) 122 126 #define HSTATUS_VSXL _AC(0x300000000, UL) 123 127 #define HSTATUS_VSXL_SHIFT 32 124 128 #endif ··· 199 195 /* xENVCFG flags */ 200 196 #define ENVCFG_STCE (_AC(1, ULL) << 63) 201 197 #define ENVCFG_PBMTE (_AC(1, ULL) << 62) 198 + #define ENVCFG_PMM (_AC(0x3, ULL) << 32) 199 + #define ENVCFG_PMM_PMLEN_0 (_AC(0x0, ULL) << 32) 200 + #define ENVCFG_PMM_PMLEN_7 (_AC(0x2, ULL) << 32) 201 + #define ENVCFG_PMM_PMLEN_16 (_AC(0x3, ULL) << 32) 202 202 #define ENVCFG_CBZE (_AC(1, UL) << 7) 203 203 #define ENVCFG_CBCFE (_AC(1, UL) << 6) 204 204 #define ENVCFG_CBIE_SHIFT 4 ··· 223 215 #define SMSTATEEN0_HSENVCFG (_ULL(1) << SMSTATEEN0_HSENVCFG_SHIFT) 224 216 #define SMSTATEEN0_SSTATEEN0_SHIFT 63 225 217 #define SMSTATEEN0_SSTATEEN0 (_ULL(1) << SMSTATEEN0_SSTATEEN0_SHIFT) 218 + 219 + /* mseccfg bits */ 220 + #define MSECCFG_PMM ENVCFG_PMM 221 + #define MSECCFG_PMM_PMLEN_0 ENVCFG_PMM_PMLEN_0 222 + #define MSECCFG_PMM_PMLEN_7 ENVCFG_PMM_PMLEN_7 223 + #define MSECCFG_PMM_PMLEN_16 ENVCFG_PMM_PMLEN_16 226 224 227 225 /* symbolic CSR names: */ 228 226 #define CSR_CYCLE 0xc00 ··· 396 382 #define CSR_MIP 0x344 397 383 #define CSR_PMPCFG0 0x3a0 398 384 #define CSR_PMPADDR0 0x3b0 385 + #define CSR_MSECCFG 0x747 386 + #define CSR_MSECCFGH 0x757 399 387 #define CSR_MVENDORID 0xf11 400 388 #define CSR_MARCHID 0xf12 401 389 #define CSR_MIMPID 0xf13

+1

arch/riscv/include/asm/entry-common.h

··· 33 33 { 34 34 return -1; 35 35 } 36 + 36 37 static inline int handle_misaligned_store(struct pt_regs *regs) 37 38 { 38 39 return -1;

+7

arch/riscv/include/asm/hwcap.h

··· 93 93 #define RISCV_ISA_EXT_ZCMOP 84 94 94 #define RISCV_ISA_EXT_ZAWRS 85 95 95 #define RISCV_ISA_EXT_SVVPTC 86 96 + #define RISCV_ISA_EXT_SMMPM 87 97 + #define RISCV_ISA_EXT_SMNPM 88 98 + #define RISCV_ISA_EXT_SSNPM 89 99 + #define RISCV_ISA_EXT_ZABHA 90 100 + #define RISCV_ISA_EXT_ZICCRSE 91 96 101 97 102 #define RISCV_ISA_EXT_XLINUXENVCFG 127 98 103 ··· 106 101 107 102 #ifdef CONFIG_RISCV_M_MODE 108 103 #define RISCV_ISA_EXT_SxAIA RISCV_ISA_EXT_SMAIA 104 + #define RISCV_ISA_EXT_SUPM RISCV_ISA_EXT_SMNPM 109 105 #else 110 106 #define RISCV_ISA_EXT_SxAIA RISCV_ISA_EXT_SSAIA 107 + #define RISCV_ISA_EXT_SUPM RISCV_ISA_EXT_SSNPM 111 108 #endif 112 109 113 110 #endif /* _ASM_RISCV_HWCAP_H */

+1 -1

arch/riscv/include/asm/hwprobe.h

··· 8 8 9 9 #include <uapi/asm/hwprobe.h> 10 10 11 - #define RISCV_HWPROBE_MAX_KEY 9 11 + #define RISCV_HWPROBE_MAX_KEY 10 12 12 13 13 static inline bool riscv_hwprobe_key_is_valid(__s64 key) 14 14 {

+7

arch/riscv/include/asm/mmu.h

··· 26 26 unsigned long exec_fdpic_loadmap; 27 27 unsigned long interp_fdpic_loadmap; 28 28 #endif 29 + unsigned long flags; 30 + #ifdef CONFIG_RISCV_ISA_SUPM 31 + u8 pmlen; 32 + #endif 29 33 } mm_context_t; 34 + 35 + /* Lock the pointer masking mode because this mm is multithreaded */ 36 + #define MM_CONTEXT_LOCK_PMLEN 0 30 37 31 38 #define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) 32 39 #define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK)

+13

arch/riscv/include/asm/mmu_context.h

··· 20 20 static inline void activate_mm(struct mm_struct *prev, 21 21 struct mm_struct *next) 22 22 { 23 + #ifdef CONFIG_RISCV_ISA_SUPM 24 + next->context.pmlen = 0; 25 + #endif 23 26 switch_mm(prev, next, NULL); 24 27 } 25 28 ··· 33 30 #ifdef CONFIG_MMU 34 31 atomic_long_set(&mm->context.id, 0); 35 32 #endif 33 + if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM)) 34 + clear_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags); 36 35 return 0; 37 36 } 38 37 39 38 DECLARE_STATIC_KEY_FALSE(use_asid_allocator); 39 + 40 + #ifdef CONFIG_RISCV_ISA_SUPM 41 + #define mm_untag_mask mm_untag_mask 42 + static inline unsigned long mm_untag_mask(struct mm_struct *mm) 43 + { 44 + return -1UL >> mm->context.pmlen; 45 + } 46 + #endif 40 47 41 48 #include <asm-generic/mmu_context.h> 42 49

+9

arch/riscv/include/asm/processor.h

··· 102 102 unsigned long s[12]; /* s[0]: frame pointer */ 103 103 struct __riscv_d_ext_state fstate; 104 104 unsigned long bad_cause; 105 + unsigned long envcfg; 105 106 u32 riscv_v_flags; 106 107 u32 vstate_ctrl; 107 108 struct __riscv_v_ext_state vstate; ··· 177 176 178 177 #define RISCV_SET_ICACHE_FLUSH_CTX(arg1, arg2) riscv_set_icache_flush_ctx(arg1, arg2) 179 178 extern int riscv_set_icache_flush_ctx(unsigned long ctx, unsigned long per_thread); 179 + 180 + #ifdef CONFIG_RISCV_ISA_SUPM 181 + /* PR_{SET,GET}_TAGGED_ADDR_CTRL prctl */ 182 + long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg); 183 + long get_tagged_addr_ctrl(struct task_struct *task); 184 + #define SET_TAGGED_ADDR_CTRL(arg) set_tagged_addr_ctrl(current, arg) 185 + #define GET_TAGGED_ADDR_CTRL() get_tagged_addr_ctrl(current) 186 + #endif 180 187 181 188 #endif /* __ASSEMBLY__ */ 182 189

+47

arch/riscv/include/asm/spinlock.h

··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + 3 + #ifndef __ASM_RISCV_SPINLOCK_H 4 + #define __ASM_RISCV_SPINLOCK_H 5 + 6 + #ifdef CONFIG_RISCV_COMBO_SPINLOCKS 7 + #define _Q_PENDING_LOOPS (1 << 9) 8 + 9 + #define __no_arch_spinlock_redefine 10 + #include <asm/ticket_spinlock.h> 11 + #include <asm/qspinlock.h> 12 + #include <asm/jump_label.h> 13 + 14 + /* 15 + * TODO: Use an alternative instead of a static key when we are able to parse 16 + * the extensions string earlier in the boot process. 17 + */ 18 + DECLARE_STATIC_KEY_TRUE(qspinlock_key); 19 + 20 + #define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ 21 + static __always_inline type arch_spin_##op(type_lock lock) \ 22 + { \ 23 + if (static_branch_unlikely(&qspinlock_key)) \ 24 + return queued_spin_##op(lock); \ 25 + return ticket_spin_##op(lock); \ 26 + } 27 + 28 + SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) 29 + SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) 30 + SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) 31 + SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) 32 + SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) 33 + SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) 34 + 35 + #elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) 36 + 37 + #include <asm/qspinlock.h> 38 + 39 + #else 40 + 41 + #include <asm/ticket_spinlock.h> 42 + 43 + #endif 44 + 45 + #include <asm/qrwlock.h> 46 + 47 + #endif /* __ASM_RISCV_SPINLOCK_H */

+19

arch/riscv/include/asm/switch_to.h

··· 70 70 #define __switch_to_fpu(__prev, __next) do { } while (0) 71 71 #endif 72 72 73 + static inline void envcfg_update_bits(struct task_struct *task, 74 + unsigned long mask, unsigned long val) 75 + { 76 + unsigned long envcfg; 77 + 78 + envcfg = (task->thread.envcfg & ~mask) | val; 79 + task->thread.envcfg = envcfg; 80 + if (task == current) 81 + csr_write(CSR_ENVCFG, envcfg); 82 + } 83 + 84 + static inline void __switch_to_envcfg(struct task_struct *next) 85 + { 86 + asm volatile (ALTERNATIVE("nop", "csrw " __stringify(CSR_ENVCFG) ", %0", 87 + 0, RISCV_ISA_EXT_XLINUXENVCFG, 1) 88 + :: "r" (next->thread.envcfg) : "memory"); 89 + } 90 + 73 91 extern struct task_struct *__switch_to(struct task_struct *, 74 92 struct task_struct *); 75 93 ··· 121 103 __switch_to_vector(__prev, __next); \ 122 104 if (switch_to_should_flush_icache(__next)) \ 123 105 local_flush_icache_all(); \ 106 + __switch_to_envcfg(__next); \ 124 107 ((last) = __switch_to(__prev, __next)); \ 125 108 } while (0) 126 109

+38 -5

arch/riscv/include/asm/uaccess.h

··· 9 9 #define _ASM_RISCV_UACCESS_H 10 10 11 11 #include <asm/asm-extable.h> 12 + #include <asm/cpufeature.h> 12 13 #include <asm/pgtable.h> /* for TASK_SIZE */ 14 + 15 + #ifdef CONFIG_RISCV_ISA_SUPM 16 + static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigned long addr) 17 + { 18 + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) { 19 + u8 pmlen = mm->context.pmlen; 20 + 21 + /* Virtual addresses are sign-extended; physical addresses are zero-extended. */ 22 + if (IS_ENABLED(CONFIG_MMU)) 23 + return (long)(addr << pmlen) >> pmlen; 24 + else 25 + return (addr << pmlen) >> pmlen; 26 + } 27 + 28 + return addr; 29 + } 30 + 31 + #define untagged_addr(addr) ({ \ 32 + unsigned long __addr = (__force unsigned long)(addr); \ 33 + (__force __typeof__(addr))__untagged_addr_remote(current->mm, __addr); \ 34 + }) 35 + 36 + #define untagged_addr_remote(mm, addr) ({ \ 37 + unsigned long __addr = (__force unsigned long)(addr); \ 38 + mmap_assert_locked(mm); \ 39 + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ 40 + }) 41 + 42 + #define access_ok(addr, size) likely(__access_ok(untagged_addr(addr), size)) 43 + #else 44 + #define untagged_addr(addr) (addr) 45 + #endif 13 46 14 47 /* 15 48 * User space memory access functions ··· 163 130 */ 164 131 #define __get_user(x, ptr) \ 165 132 ({ \ 166 - const __typeof__(*(ptr)) __user *__gu_ptr = (ptr); \ 133 + const __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \ 167 134 long __gu_err = 0; \ 168 135 \ 169 136 __chk_user_ptr(__gu_ptr); \ ··· 279 246 */ 280 247 #define __put_user(x, ptr) \ 281 248 ({ \ 282 - __typeof__(*(ptr)) __user *__gu_ptr = (ptr); \ 249 + __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \ 283 250 __typeof__(*__gu_ptr) __val = (x); \ 284 251 long __pu_err = 0; \ 285 252 \ ··· 326 293 static inline unsigned long 327 294 raw_copy_from_user(void *to, const void __user *from, unsigned long n) 328 295 { 329 - return __asm_copy_from_user(to, from, n); 296 + return __asm_copy_from_user(to, untagged_addr(from), n); 330 297 } 331 298 332 299 static inline unsigned long 333 300 raw_copy_to_user(void __user *to, const void *from, unsigned long n) 334 301 { 335 - return __asm_copy_to_user(to, from, n); 302 + return __asm_copy_to_user(untagged_addr(to), from, n); 336 303 } 337 304 338 305 extern long strncpy_from_user(char *dest, const char __user *src, long count); ··· 347 314 { 348 315 might_fault(); 349 316 return access_ok(to, n) ? 350 - __clear_user(to, n) : n; 317 + __clear_user(untagged_addr(to), n) : n; 351 318 } 352 319 353 320 #define __get_kernel_nofault(dst, src, type, err_label) \

+2

arch/riscv/include/asm/vector.h

··· 21 21 22 22 extern unsigned long riscv_v_vsize; 23 23 int riscv_v_setup_vsize(void); 24 + bool insn_is_vector(u32 insn_buf); 24 25 bool riscv_v_first_use_handler(struct pt_regs *regs); 25 26 void kernel_vector_begin(void); 26 27 void kernel_vector_end(void); ··· 269 268 270 269 static inline int riscv_v_setup_vsize(void) { return -EOPNOTSUPP; } 271 270 static __always_inline bool has_vector(void) { return false; } 271 + static __always_inline bool insn_is_vector(u32 insn_buf) { return false; } 272 272 static inline bool riscv_v_first_use_handler(struct pt_regs *regs) { return false; } 273 273 static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } 274 274 static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }

+6

arch/riscv/include/uapi/asm/hwprobe.h

··· 72 72 #define RISCV_HWPROBE_EXT_ZCF (1ULL << 46) 73 73 #define RISCV_HWPROBE_EXT_ZCMOP (1ULL << 47) 74 74 #define RISCV_HWPROBE_EXT_ZAWRS (1ULL << 48) 75 + #define RISCV_HWPROBE_EXT_SUPM (1ULL << 49) 75 76 #define RISCV_HWPROBE_KEY_CPUPERF_0 5 76 77 #define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) 77 78 #define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) ··· 89 88 #define RISCV_HWPROBE_MISALIGNED_SCALAR_SLOW 2 90 89 #define RISCV_HWPROBE_MISALIGNED_SCALAR_FAST 3 91 90 #define RISCV_HWPROBE_MISALIGNED_SCALAR_UNSUPPORTED 4 91 + #define RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF 10 92 + #define RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN 0 93 + #define RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW 2 94 + #define RISCV_HWPROBE_MISALIGNED_VECTOR_FAST 3 95 + #define RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED 4 92 96 /* Increase RISCV_HWPROBE_MAX_KEY when adding items. */ 93 97 94 98 /* Flags */

+2

arch/riscv/include/uapi/asm/kvm.h

··· 175 175 KVM_RISCV_ISA_EXT_ZCF, 176 176 KVM_RISCV_ISA_EXT_ZCMOP, 177 177 KVM_RISCV_ISA_EXT_ZAWRS, 178 + KVM_RISCV_ISA_EXT_SMNPM, 179 + KVM_RISCV_ISA_EXT_SSNPM, 178 180 KVM_RISCV_ISA_EXT_MAX, 179 181 }; 180 182

+2 -1

arch/riscv/kernel/Makefile

··· 75 75 76 76 obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o 77 77 obj-$(CONFIG_RISCV_MISALIGNED) += unaligned_access_speed.o 78 - obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) += copy-unaligned.o 78 + obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) += copy-unaligned.o 79 + obj-$(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS) += vec-copy-unaligned.o 79 80 80 81 obj-$(CONFIG_FPU) += fpu.o 81 82 obj-$(CONFIG_FPU) += kernel_mode_fpu.o

+5

arch/riscv/kernel/copy-unaligned.h

··· 10 10 void __riscv_copy_words_unaligned(void *dst, const void *src, size_t size); 11 11 void __riscv_copy_bytes_unaligned(void *dst, const void *src, size_t size); 12 12 13 + #ifdef CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS 14 + void __riscv_copy_vec_words_unaligned(void *dst, const void *src, size_t size); 15 + void __riscv_copy_vec_bytes_unaligned(void *dst, const void *src, size_t size); 16 + #endif 17 + 13 18 #endif /* __RISCV_KERNEL_COPY_UNALIGNED_H */

+13 -3

arch/riscv/kernel/cpufeature.c

··· 29 29 30 30 #define NUM_ALPHA_EXTS ('z' - 'a' + 1) 31 31 32 + static bool any_cpu_has_zicboz; 33 + 32 34 unsigned long elf_hwcap __read_mostly; 33 35 34 36 /* Host ISA bitmap */ ··· 101 99 pr_err("Zicboz disabled as cboz-block-size present, but is not a power-of-2\n"); 102 100 return -EINVAL; 103 101 } 102 + any_cpu_has_zicboz = true; 104 103 return 0; 105 104 } 106 105 ··· 318 315 riscv_ext_zicbom_validate), 319 316 __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, 320 317 riscv_ext_zicboz_validate), 318 + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), 321 319 __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), 322 320 __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), 323 321 __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), ··· 327 323 __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), 328 324 __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), 329 325 __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), 326 + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), 330 327 __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), 331 328 __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), 332 329 __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), ··· 380 375 __RISCV_ISA_EXT_BUNDLE(zvksg, riscv_zvksg_bundled_exts), 381 376 __RISCV_ISA_EXT_DATA(zvkt, RISCV_ISA_EXT_ZVKT), 382 377 __RISCV_ISA_EXT_DATA(smaia, RISCV_ISA_EXT_SMAIA), 378 + __RISCV_ISA_EXT_DATA(smmpm, RISCV_ISA_EXT_SMMPM), 379 + __RISCV_ISA_EXT_SUPERSET(smnpm, RISCV_ISA_EXT_SMNPM, riscv_xlinuxenvcfg_exts), 383 380 __RISCV_ISA_EXT_DATA(smstateen, RISCV_ISA_EXT_SMSTATEEN), 384 381 __RISCV_ISA_EXT_DATA(ssaia, RISCV_ISA_EXT_SSAIA), 385 382 __RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF), 383 + __RISCV_ISA_EXT_SUPERSET(ssnpm, RISCV_ISA_EXT_SSNPM, riscv_xlinuxenvcfg_exts), 386 384 __RISCV_ISA_EXT_DATA(sstc, RISCV_ISA_EXT_SSTC), 387 385 __RISCV_ISA_EXT_DATA(svinval, RISCV_ISA_EXT_SVINVAL), 388 386 __RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT), ··· 926 918 return hwcap; 927 919 } 928 920 929 - void riscv_user_isa_enable(void) 921 + void __init riscv_user_isa_enable(void) 930 922 { 931 - if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_ZICBOZ)) 932 - csr_set(CSR_ENVCFG, ENVCFG_CBZE); 923 + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICBOZ)) 924 + current->thread.envcfg |= ENVCFG_CBZE; 925 + else if (any_cpu_has_zicboz) 926 + pr_warn("Zicboz disabled as it is unavailable on some harts\n"); 933 927 } 934 928 935 929 #ifdef CONFIG_RISCV_ALTERNATIVE

+2 -2

arch/riscv/kernel/fpu.S

··· 170 170 __access_func(f31) 171 171 172 172 173 - #ifdef CONFIG_RISCV_MISALIGNED 173 + #ifdef CONFIG_RISCV_SCALAR_MISALIGNED 174 174 175 175 /* 176 176 * Disable compressed instructions set to keep a constant offset between FP ··· 224 224 fp_access_epilogue 225 225 SYM_FUNC_END(get_f64_reg) 226 226 227 - #endif /* CONFIG_RISCV_MISALIGNED */ 227 + #endif /* CONFIG_RISCV_SCALAR_MISALIGNED */

+154

arch/riscv/kernel/process.c

··· 7 7 * Copyright (C) 2017 SiFive 8 8 */ 9 9 10 + #include <linux/bitfield.h> 10 11 #include <linux/cpu.h> 11 12 #include <linux/kernel.h> 12 13 #include <linux/sched.h> ··· 181 180 memset(&current->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); 182 181 clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); 183 182 #endif 183 + #ifdef CONFIG_RISCV_ISA_SUPM 184 + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) 185 + envcfg_update_bits(current, ENVCFG_PMM, ENVCFG_PMM_PMLEN_0); 186 + #endif 184 187 } 185 188 186 189 void arch_release_task_struct(struct task_struct *tsk) ··· 212 207 unsigned long usp = args->stack; 213 208 unsigned long tls = args->tls; 214 209 struct pt_regs *childregs = task_pt_regs(p); 210 + 211 + /* Ensure all threads in this mm have the same pointer masking mode. */ 212 + if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM) && p->mm && (clone_flags & CLONE_VM)) 213 + set_bit(MM_CONTEXT_LOCK_PMLEN, &p->mm->context.flags); 215 214 216 215 memset(&p->thread.s, 0, sizeof(p->thread.s)); 217 216 ··· 251 242 { 252 243 riscv_v_setup_ctx_cache(); 253 244 } 245 + 246 + #ifdef CONFIG_RISCV_ISA_SUPM 247 + enum { 248 + PMLEN_0 = 0, 249 + PMLEN_7 = 7, 250 + PMLEN_16 = 16, 251 + }; 252 + 253 + static bool have_user_pmlen_7; 254 + static bool have_user_pmlen_16; 255 + 256 + /* 257 + * Control the relaxed ABI allowing tagged user addresses into the kernel. 258 + */ 259 + static unsigned int tagged_addr_disabled; 260 + 261 + long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) 262 + { 263 + unsigned long valid_mask = PR_PMLEN_MASK | PR_TAGGED_ADDR_ENABLE; 264 + struct thread_info *ti = task_thread_info(task); 265 + struct mm_struct *mm = task->mm; 266 + unsigned long pmm; 267 + u8 pmlen; 268 + 269 + if (is_compat_thread(ti)) 270 + return -EINVAL; 271 + 272 + if (arg & ~valid_mask) 273 + return -EINVAL; 274 + 275 + /* 276 + * Prefer the smallest PMLEN that satisfies the user's request, 277 + * in case choosing a larger PMLEN has a performance impact. 278 + */ 279 + pmlen = FIELD_GET(PR_PMLEN_MASK, arg); 280 + if (pmlen == PMLEN_0) { 281 + pmm = ENVCFG_PMM_PMLEN_0; 282 + } else if (pmlen <= PMLEN_7 && have_user_pmlen_7) { 283 + pmlen = PMLEN_7; 284 + pmm = ENVCFG_PMM_PMLEN_7; 285 + } else if (pmlen <= PMLEN_16 && have_user_pmlen_16) { 286 + pmlen = PMLEN_16; 287 + pmm = ENVCFG_PMM_PMLEN_16; 288 + } else { 289 + return -EINVAL; 290 + } 291 + 292 + /* 293 + * Do not allow the enabling of the tagged address ABI if globally 294 + * disabled via sysctl abi.tagged_addr_disabled, if pointer masking 295 + * is disabled for userspace. 296 + */ 297 + if (arg & PR_TAGGED_ADDR_ENABLE && (tagged_addr_disabled || !pmlen)) 298 + return -EINVAL; 299 + 300 + if (!(arg & PR_TAGGED_ADDR_ENABLE)) 301 + pmlen = PMLEN_0; 302 + 303 + if (mmap_write_lock_killable(mm)) 304 + return -EINTR; 305 + 306 + if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pmlen != pmlen) { 307 + mmap_write_unlock(mm); 308 + return -EBUSY; 309 + } 310 + 311 + envcfg_update_bits(task, ENVCFG_PMM, pmm); 312 + mm->context.pmlen = pmlen; 313 + 314 + mmap_write_unlock(mm); 315 + 316 + return 0; 317 + } 318 + 319 + long get_tagged_addr_ctrl(struct task_struct *task) 320 + { 321 + struct thread_info *ti = task_thread_info(task); 322 + long ret = 0; 323 + 324 + if (is_compat_thread(ti)) 325 + return -EINVAL; 326 + 327 + /* 328 + * The mm context's pmlen is set only when the tagged address ABI is 329 + * enabled, so the effective PMLEN must be extracted from envcfg.PMM. 330 + */ 331 + switch (task->thread.envcfg & ENVCFG_PMM) { 332 + case ENVCFG_PMM_PMLEN_7: 333 + ret = FIELD_PREP(PR_PMLEN_MASK, PMLEN_7); 334 + break; 335 + case ENVCFG_PMM_PMLEN_16: 336 + ret = FIELD_PREP(PR_PMLEN_MASK, PMLEN_16); 337 + break; 338 + } 339 + 340 + if (task->mm->context.pmlen) 341 + ret |= PR_TAGGED_ADDR_ENABLE; 342 + 343 + return ret; 344 + } 345 + 346 + static bool try_to_set_pmm(unsigned long value) 347 + { 348 + csr_set(CSR_ENVCFG, value); 349 + return (csr_read_clear(CSR_ENVCFG, ENVCFG_PMM) & ENVCFG_PMM) == value; 350 + } 351 + 352 + /* 353 + * Global sysctl to disable the tagged user addresses support. This control 354 + * only prevents the tagged address ABI enabling via prctl() and does not 355 + * disable it for tasks that already opted in to the relaxed ABI. 356 + */ 357 + 358 + static struct ctl_table tagged_addr_sysctl_table[] = { 359 + { 360 + .procname = "tagged_addr_disabled", 361 + .mode = 0644, 362 + .data = &tagged_addr_disabled, 363 + .maxlen = sizeof(int), 364 + .proc_handler = proc_dointvec_minmax, 365 + .extra1 = SYSCTL_ZERO, 366 + .extra2 = SYSCTL_ONE, 367 + }, 368 + }; 369 + 370 + static int __init tagged_addr_init(void) 371 + { 372 + if (!riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) 373 + return 0; 374 + 375 + /* 376 + * envcfg.PMM is a WARL field. Detect which values are supported. 377 + * Assume the supported PMLEN values are the same on all harts. 378 + */ 379 + csr_clear(CSR_ENVCFG, ENVCFG_PMM); 380 + have_user_pmlen_7 = try_to_set_pmm(ENVCFG_PMM_PMLEN_7); 381 + have_user_pmlen_16 = try_to_set_pmm(ENVCFG_PMM_PMLEN_16); 382 + 383 + if (!register_sysctl("abi", tagged_addr_sysctl_table)) 384 + return -EINVAL; 385 + 386 + return 0; 387 + } 388 + core_initcall(tagged_addr_init); 389 + #endif /* CONFIG_RISCV_ISA_SUPM */

+42

arch/riscv/kernel/ptrace.c

··· 28 28 #ifdef CONFIG_RISCV_ISA_V 29 29 REGSET_V, 30 30 #endif 31 + #ifdef CONFIG_RISCV_ISA_SUPM 32 + REGSET_TAGGED_ADDR_CTRL, 33 + #endif 31 34 }; 32 35 33 36 static int riscv_gpr_get(struct task_struct *target, ··· 155 152 } 156 153 #endif 157 154 155 + #ifdef CONFIG_RISCV_ISA_SUPM 156 + static int tagged_addr_ctrl_get(struct task_struct *target, 157 + const struct user_regset *regset, 158 + struct membuf to) 159 + { 160 + long ctrl = get_tagged_addr_ctrl(target); 161 + 162 + if (IS_ERR_VALUE(ctrl)) 163 + return ctrl; 164 + 165 + return membuf_write(&to, &ctrl, sizeof(ctrl)); 166 + } 167 + 168 + static int tagged_addr_ctrl_set(struct task_struct *target, 169 + const struct user_regset *regset, 170 + unsigned int pos, unsigned int count, 171 + const void *kbuf, const void __user *ubuf) 172 + { 173 + int ret; 174 + long ctrl; 175 + 176 + ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &ctrl, 0, -1); 177 + if (ret) 178 + return ret; 179 + 180 + return set_tagged_addr_ctrl(target, ctrl); 181 + } 182 + #endif 183 + 158 184 static const struct user_regset riscv_user_regset[] = { 159 185 [REGSET_X] = { 160 186 .core_note_type = NT_PRSTATUS, ··· 212 180 .size = sizeof(__u32), 213 181 .regset_get = riscv_vr_get, 214 182 .set = riscv_vr_set, 183 + }, 184 + #endif 185 + #ifdef CONFIG_RISCV_ISA_SUPM 186 + [REGSET_TAGGED_ADDR_CTRL] = { 187 + .core_note_type = NT_RISCV_TAGGED_ADDR_CTRL, 188 + .n = 1, 189 + .size = sizeof(long), 190 + .align = sizeof(long), 191 + .regset_get = tagged_addr_ctrl_get, 192 + .set = tagged_addr_ctrl_set, 215 193 }, 216 194 #endif 217 195 };

+37

arch/riscv/kernel/setup.c

··· 244 244 #endif 245 245 } 246 246 247 + #if defined(CONFIG_RISCV_COMBO_SPINLOCKS) 248 + DEFINE_STATIC_KEY_TRUE(qspinlock_key); 249 + EXPORT_SYMBOL(qspinlock_key); 250 + #endif 251 + 252 + static void __init riscv_spinlock_init(void) 253 + { 254 + char *using_ext = NULL; 255 + 256 + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { 257 + pr_info("Ticket spinlock: enabled\n"); 258 + return; 259 + } 260 + 261 + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && 262 + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && 263 + riscv_isa_extension_available(NULL, ZABHA) && 264 + riscv_isa_extension_available(NULL, ZACAS)) { 265 + using_ext = "using Zabha"; 266 + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { 267 + using_ext = "using Ziccrse"; 268 + } 269 + #if defined(CONFIG_RISCV_COMBO_SPINLOCKS) 270 + else { 271 + static_branch_disable(&qspinlock_key); 272 + pr_info("Ticket spinlock: enabled\n"); 273 + return; 274 + } 275 + #endif 276 + 277 + if (!using_ext) 278 + pr_err("Queued spinlock without Zabha or Ziccrse"); 279 + else 280 + pr_info("Queued spinlock %s: enabled\n", using_ext); 281 + } 282 + 247 283 extern void __init init_rt_signal_env(void); 248 284 249 285 void __init setup_arch(char **cmdline_p) ··· 333 297 riscv_set_dma_cache_alignment(); 334 298 335 299 riscv_user_isa_enable(); 300 + riscv_spinlock_init(); 336 301 } 337 302 338 303 bool arch_cpu_is_hotpluggable(int cpu)

-2

arch/riscv/kernel/smpboot.c

··· 233 233 numa_add_cpu(curr_cpuid); 234 234 set_cpu_online(curr_cpuid, true); 235 235 236 - riscv_user_isa_enable(); 237 - 238 236 /* 239 237 * Remote cache and TLB flushes are ignored while the CPU is offline, 240 238 * so flush them both right now just in case.

+2 -2

arch/riscv/kernel/suspend.c

··· 14 14 15 15 void suspend_save_csrs(struct suspend_context *context) 16 16 { 17 - if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_XLINUXENVCFG)) 17 + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_XLINUXENVCFG)) 18 18 context->envcfg = csr_read(CSR_ENVCFG); 19 19 context->tvec = csr_read(CSR_TVEC); 20 20 context->ie = csr_read(CSR_IE); ··· 37 37 void suspend_restore_csrs(struct suspend_context *context) 38 38 { 39 39 csr_write(CSR_SCRATCH, 0); 40 - if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_XLINUXENVCFG)) 40 + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_XLINUXENVCFG)) 41 41 csr_write(CSR_ENVCFG, context->envcfg); 42 42 csr_write(CSR_TVEC, context->tvec); 43 43 csr_write(CSR_IE, context->ie);

+44

arch/riscv/kernel/sys_hwprobe.c

··· 150 150 EXT_KEY(ZFH); 151 151 EXT_KEY(ZFHMIN); 152 152 } 153 + 154 + if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM)) 155 + EXT_KEY(SUPM); 153 156 #undef EXT_KEY 154 157 } 155 158 ··· 204 201 } 205 202 #endif 206 203 204 + #ifdef CONFIG_RISCV_VECTOR_MISALIGNED 205 + static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) 206 + { 207 + int cpu; 208 + u64 perf = -1ULL; 209 + 210 + /* Return if supported or not even if speed wasn't probed */ 211 + for_each_cpu(cpu, cpus) { 212 + int this_perf = per_cpu(vector_misaligned_access, cpu); 213 + 214 + if (perf == -1ULL) 215 + perf = this_perf; 216 + 217 + if (perf != this_perf) { 218 + perf = RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; 219 + break; 220 + } 221 + } 222 + 223 + if (perf == -1ULL) 224 + return RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; 225 + 226 + return perf; 227 + } 228 + #else 229 + static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) 230 + { 231 + if (IS_ENABLED(CONFIG_RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS)) 232 + return RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; 233 + 234 + if (IS_ENABLED(CONFIG_RISCV_SLOW_VECTOR_UNALIGNED_ACCESS)) 235 + return RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; 236 + 237 + return RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; 238 + } 239 + #endif 240 + 207 241 static void hwprobe_one_pair(struct riscv_hwprobe *pair, 208 242 const struct cpumask *cpus) 209 243 { ··· 267 227 case RISCV_HWPROBE_KEY_CPUPERF_0: 268 228 case RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF: 269 229 pair->value = hwprobe_misaligned(cpus); 230 + break; 231 + 232 + case RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF: 233 + pair->value = hwprobe_vec_misaligned(cpus); 270 234 break; 271 235 272 236 case RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE:

+128 -11

arch/riscv/kernel/traps_misaligned.c

··· 16 16 #include <asm/entry-common.h> 17 17 #include <asm/hwprobe.h> 18 18 #include <asm/cpufeature.h> 19 + #include <asm/vector.h> 19 20 20 21 #define INSN_MATCH_LB 0x3 21 22 #define INSN_MASK_LB 0x707f ··· 321 320 u64 data_u64; 322 321 }; 323 322 324 - static bool unaligned_ctl __read_mostly; 325 - 326 323 /* sysctl hooks */ 327 324 int unaligned_enabled __read_mostly = 1; /* Enabled by default */ 328 325 329 - int handle_misaligned_load(struct pt_regs *regs) 326 + #ifdef CONFIG_RISCV_VECTOR_MISALIGNED 327 + static int handle_vector_misaligned_load(struct pt_regs *regs) 328 + { 329 + unsigned long epc = regs->epc; 330 + unsigned long insn; 331 + 332 + if (get_insn(regs, epc, &insn)) 333 + return -1; 334 + 335 + /* Only return 0 when in check_vector_unaligned_access_emulated */ 336 + if (*this_cpu_ptr(&vector_misaligned_access) == RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN) { 337 + *this_cpu_ptr(&vector_misaligned_access) = RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED; 338 + regs->epc = epc + INSN_LEN(insn); 339 + return 0; 340 + } 341 + 342 + /* If vector instruction we don't emulate it yet */ 343 + regs->epc = epc; 344 + return -1; 345 + } 346 + #else 347 + static int handle_vector_misaligned_load(struct pt_regs *regs) 348 + { 349 + return -1; 350 + } 351 + #endif 352 + 353 + static int handle_scalar_misaligned_load(struct pt_regs *regs) 330 354 { 331 355 union reg_data val; 332 356 unsigned long epc = regs->epc; ··· 459 433 return 0; 460 434 } 461 435 462 - int handle_misaligned_store(struct pt_regs *regs) 436 + static int handle_scalar_misaligned_store(struct pt_regs *regs) 463 437 { 464 438 union reg_data val; 465 439 unsigned long epc = regs->epc; ··· 550 524 return 0; 551 525 } 552 526 553 - static bool check_unaligned_access_emulated(int cpu) 527 + int handle_misaligned_load(struct pt_regs *regs) 554 528 { 529 + unsigned long epc = regs->epc; 530 + unsigned long insn; 531 + 532 + if (IS_ENABLED(CONFIG_RISCV_VECTOR_MISALIGNED)) { 533 + if (get_insn(regs, epc, &insn)) 534 + return -1; 535 + 536 + if (insn_is_vector(insn)) 537 + return handle_vector_misaligned_load(regs); 538 + } 539 + 540 + if (IS_ENABLED(CONFIG_RISCV_SCALAR_MISALIGNED)) 541 + return handle_scalar_misaligned_load(regs); 542 + 543 + return -1; 544 + } 545 + 546 + int handle_misaligned_store(struct pt_regs *regs) 547 + { 548 + if (IS_ENABLED(CONFIG_RISCV_SCALAR_MISALIGNED)) 549 + return handle_scalar_misaligned_store(regs); 550 + 551 + return -1; 552 + } 553 + 554 + #ifdef CONFIG_RISCV_VECTOR_MISALIGNED 555 + void check_vector_unaligned_access_emulated(struct work_struct *work __always_unused) 556 + { 557 + long *mas_ptr = this_cpu_ptr(&vector_misaligned_access); 558 + unsigned long tmp_var; 559 + 560 + *mas_ptr = RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; 561 + 562 + kernel_vector_begin(); 563 + /* 564 + * In pre-13.0.0 versions of GCC, vector registers cannot appear in 565 + * the clobber list. This inline asm clobbers v0, but since we do not 566 + * currently build the kernel with V enabled, the v0 clobber arg is not 567 + * needed (as the compiler will not emit vector code itself). If the kernel 568 + * is changed to build with V enabled, the clobber arg will need to be 569 + * added here. 570 + */ 571 + __asm__ __volatile__ ( 572 + ".balign 4\n\t" 573 + ".option push\n\t" 574 + ".option arch, +zve32x\n\t" 575 + " vsetivli zero, 1, e16, m1, ta, ma\n\t" // Vectors of 16b 576 + " vle16.v v0, (%[ptr])\n\t" // Load bytes 577 + ".option pop\n\t" 578 + : : [ptr] "r" ((u8 *)&tmp_var + 1)); 579 + kernel_vector_end(); 580 + } 581 + 582 + bool check_vector_unaligned_access_emulated_all_cpus(void) 583 + { 584 + int cpu; 585 + 586 + if (!has_vector()) { 587 + for_each_online_cpu(cpu) 588 + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED; 589 + return false; 590 + } 591 + 592 + schedule_on_each_cpu(check_vector_unaligned_access_emulated); 593 + 594 + for_each_online_cpu(cpu) 595 + if (per_cpu(vector_misaligned_access, cpu) 596 + == RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN) 597 + return false; 598 + 599 + return true; 600 + } 601 + #else 602 + bool check_vector_unaligned_access_emulated_all_cpus(void) 603 + { 604 + return false; 605 + } 606 + #endif 607 + 608 + #ifdef CONFIG_RISCV_SCALAR_MISALIGNED 609 + 610 + static bool unaligned_ctl __read_mostly; 611 + 612 + void check_unaligned_access_emulated(struct work_struct *work __always_unused) 613 + { 614 + int cpu = smp_processor_id(); 555 615 long *mas_ptr = per_cpu_ptr(&misaligned_access_speed, cpu); 556 616 unsigned long tmp_var, tmp_val; 557 - bool misaligned_emu_detected; 558 617 559 618 *mas_ptr = RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN; 560 619 ··· 647 536 " "REG_L" %[tmp], 1(%[ptr])\n" 648 537 : [tmp] "=r" (tmp_val) : [ptr] "r" (&tmp_var) : "memory"); 649 538 650 - misaligned_emu_detected = (*mas_ptr == RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED); 651 539 /* 652 540 * If unaligned_ctl is already set, this means that we detected that all 653 541 * CPUS uses emulated misaligned access at boot time. If that changed 654 542 * when hotplugging the new cpu, this is something we don't handle. 655 543 */ 656 - if (unlikely(unaligned_ctl && !misaligned_emu_detected)) { 544 + if (unlikely(unaligned_ctl && (*mas_ptr != RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED))) { 657 545 pr_crit("CPU misaligned accesses non homogeneous (expected all emulated)\n"); 658 546 while (true) 659 547 cpu_relax(); 660 548 } 661 - 662 - return misaligned_emu_detected; 663 549 } 664 550 665 551 bool check_unaligned_access_emulated_all_cpus(void) ··· 668 560 * accesses emulated since tasks requesting such control can run on any 669 561 * CPU. 670 562 */ 563 + schedule_on_each_cpu(check_unaligned_access_emulated); 564 + 671 565 for_each_online_cpu(cpu) 672 - if (!check_unaligned_access_emulated(cpu)) 566 + if (per_cpu(misaligned_access_speed, cpu) 567 + != RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED) 673 568 return false; 674 569 675 570 unaligned_ctl = true; ··· 683 572 { 684 573 return unaligned_ctl; 685 574 } 575 + #else 576 + bool check_unaligned_access_emulated_all_cpus(void) 577 + { 578 + return false; 579 + } 580 + #endif

+150 -10

arch/riscv/kernel/unaligned_access_speed.c

··· 6 6 #include <linux/cpu.h> 7 7 #include <linux/cpumask.h> 8 8 #include <linux/jump_label.h> 9 + #include <linux/kthread.h> 9 10 #include <linux/mm.h> 10 11 #include <linux/smp.h> 11 12 #include <linux/types.h> 12 13 #include <asm/cpufeature.h> 13 14 #include <asm/hwprobe.h> 15 + #include <asm/vector.h> 14 16 15 17 #include "copy-unaligned.h" 16 18 ··· 21 19 #define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) 22 20 #define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) 23 21 24 - DEFINE_PER_CPU(long, misaligned_access_speed); 22 + DEFINE_PER_CPU(long, misaligned_access_speed) = RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN; 23 + DEFINE_PER_CPU(long, vector_misaligned_access) = RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED; 25 24 26 25 #ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS 27 26 static cpumask_t fast_misaligned_access; ··· 194 191 if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN) 195 192 goto exit; 196 193 194 + check_unaligned_access_emulated(NULL); 197 195 buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); 198 196 if (!buf) { 199 197 pr_warn("Allocation failure, not measuring misaligned performance\n"); ··· 263 259 kfree(bufs); 264 260 return 0; 265 261 } 262 + #else /* CONFIG_RISCV_PROBE_UNALIGNED_ACCESS */ 263 + static int check_unaligned_access_speed_all_cpus(void) 264 + { 265 + return 0; 266 + } 267 + #endif 268 + 269 + #ifdef CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS 270 + static void check_vector_unaligned_access(struct work_struct *work __always_unused) 271 + { 272 + int cpu = smp_processor_id(); 273 + u64 start_cycles, end_cycles; 274 + u64 word_cycles; 275 + u64 byte_cycles; 276 + int ratio; 277 + unsigned long start_jiffies, now; 278 + struct page *page; 279 + void *dst; 280 + void *src; 281 + long speed = RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; 282 + 283 + if (per_cpu(vector_misaligned_access, cpu) != RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN) 284 + return; 285 + 286 + page = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); 287 + if (!page) { 288 + pr_warn("Allocation failure, not measuring vector misaligned performance\n"); 289 + return; 290 + } 291 + 292 + /* Make an unaligned destination buffer. */ 293 + dst = (void *)((unsigned long)page_address(page) | 0x1); 294 + /* Unalign src as well, but differently (off by 1 + 2 = 3). */ 295 + src = dst + (MISALIGNED_BUFFER_SIZE / 2); 296 + src += 2; 297 + word_cycles = -1ULL; 298 + 299 + /* Do a warmup. */ 300 + kernel_vector_begin(); 301 + __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 302 + 303 + start_jiffies = jiffies; 304 + while ((now = jiffies) == start_jiffies) 305 + cpu_relax(); 306 + 307 + /* 308 + * For a fixed amount of time, repeatedly try the function, and take 309 + * the best time in cycles as the measurement. 310 + */ 311 + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 312 + start_cycles = get_cycles64(); 313 + /* Ensure the CSR read can't reorder WRT to the copy. */ 314 + mb(); 315 + __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 316 + /* Ensure the copy ends before the end time is snapped. */ 317 + mb(); 318 + end_cycles = get_cycles64(); 319 + if ((end_cycles - start_cycles) < word_cycles) 320 + word_cycles = end_cycles - start_cycles; 321 + } 322 + 323 + byte_cycles = -1ULL; 324 + __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 325 + start_jiffies = jiffies; 326 + while ((now = jiffies) == start_jiffies) 327 + cpu_relax(); 328 + 329 + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 330 + start_cycles = get_cycles64(); 331 + /* Ensure the CSR read can't reorder WRT to the copy. */ 332 + mb(); 333 + __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 334 + /* Ensure the copy ends before the end time is snapped. */ 335 + mb(); 336 + end_cycles = get_cycles64(); 337 + if ((end_cycles - start_cycles) < byte_cycles) 338 + byte_cycles = end_cycles - start_cycles; 339 + } 340 + 341 + kernel_vector_end(); 342 + 343 + /* Don't divide by zero. */ 344 + if (!word_cycles || !byte_cycles) { 345 + pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned vector access speed\n", 346 + cpu); 347 + 348 + return; 349 + } 350 + 351 + if (word_cycles < byte_cycles) 352 + speed = RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; 353 + 354 + ratio = div_u64((byte_cycles * 100), word_cycles); 355 + pr_info("cpu%d: Ratio of vector byte access time to vector unaligned word access is %d.%02d, unaligned accesses are %s\n", 356 + cpu, 357 + ratio / 100, 358 + ratio % 100, 359 + (speed == RISCV_HWPROBE_MISALIGNED_VECTOR_FAST) ? "fast" : "slow"); 360 + 361 + per_cpu(vector_misaligned_access, cpu) = speed; 362 + } 363 + 364 + static int riscv_online_cpu_vec(unsigned int cpu) 365 + { 366 + if (!has_vector()) 367 + return 0; 368 + 369 + if (per_cpu(vector_misaligned_access, cpu) != RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED) 370 + return 0; 371 + 372 + check_vector_unaligned_access_emulated(NULL); 373 + check_vector_unaligned_access(NULL); 374 + return 0; 375 + } 376 + 377 + /* Measure unaligned access speed on all CPUs present at boot in parallel. */ 378 + static int vec_check_unaligned_access_speed_all_cpus(void *unused __always_unused) 379 + { 380 + schedule_on_each_cpu(check_vector_unaligned_access); 381 + 382 + /* 383 + * Setup hotplug callbacks for any new CPUs that come online or go 384 + * offline. 385 + */ 386 + cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", 387 + riscv_online_cpu_vec, NULL); 388 + 389 + return 0; 390 + } 391 + #else /* CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS */ 392 + static int vec_check_unaligned_access_speed_all_cpus(void *unused __always_unused) 393 + { 394 + return 0; 395 + } 396 + #endif 266 397 267 398 static int check_unaligned_access_all_cpus(void) 268 399 { 269 - bool all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); 400 + bool all_cpus_emulated, all_cpus_vec_unsupported; 401 + 402 + all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); 403 + all_cpus_vec_unsupported = check_vector_unaligned_access_emulated_all_cpus(); 404 + 405 + if (!all_cpus_vec_unsupported && 406 + IS_ENABLED(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS)) { 407 + kthread_run(vec_check_unaligned_access_speed_all_cpus, 408 + NULL, "vec_check_unaligned_access_speed_all_cpus"); 409 + } 270 410 271 411 if (!all_cpus_emulated) 272 412 return check_unaligned_access_speed_all_cpus(); 273 413 274 414 return 0; 275 415 } 276 - #else /* CONFIG_RISCV_PROBE_UNALIGNED_ACCESS */ 277 - static int check_unaligned_access_all_cpus(void) 278 - { 279 - check_unaligned_access_emulated_all_cpus(); 280 - 281 - return 0; 282 - } 283 - #endif 284 416 285 417 arch_initcall(check_unaligned_access_all_cpus);

+5 -4

arch/riscv/kernel/vdso/Makefile

··· 45 45 46 46 # link rule for the .so file, .lds has to be first 47 47 $(obj)/vdso.so.dbg: $(obj)/vdso.lds $(obj-vdso) FORCE 48 - $(call if_changed,vdsold) 48 + $(call if_changed,vdsold_and_check) 49 49 LDFLAGS_vdso.so.dbg = -shared -soname=linux-vdso.so.1 \ 50 50 --build-id=sha1 --hash-style=both --eh-frame-hdr 51 51 ··· 65 65 # actual build commands 66 66 # The DSO images are built using a special linker script 67 67 # Make sure only to export the intended __vdso_xxx symbol offsets. 68 - quiet_cmd_vdsold = VDSOLD $@ 69 - cmd_vdsold = $(LD) $(ld_flags) -T $(filter-out FORCE,$^) -o $@.tmp && \ 68 + quiet_cmd_vdsold_and_check = VDSOLD $@ 69 + cmd_vdsold_and_check = $(LD) $(ld_flags) -T $(filter-out FORCE,$^) -o $@.tmp && \ 70 70 $(OBJCOPY) $(patsubst %, -G __vdso_%, $(vdso-syms)) $@.tmp $@ && \ 71 - rm $@.tmp 71 + rm $@.tmp && \ 72 + $(cmd_vdso_check)

+58

arch/riscv/kernel/vec-copy-unaligned.S

··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + /* Copyright (C) 2024 Rivos Inc. */ 3 + 4 + #include <linux/args.h> 5 + #include <linux/linkage.h> 6 + #include <asm/asm.h> 7 + 8 + .text 9 + 10 + #define WORD_EEW 32 11 + 12 + #define WORD_SEW CONCATENATE(e, WORD_EEW) 13 + #define VEC_L CONCATENATE(vle, WORD_EEW).v 14 + #define VEC_S CONCATENATE(vle, WORD_EEW).v 15 + 16 + /* void __riscv_copy_vec_words_unaligned(void *, const void *, size_t) */ 17 + /* Performs a memcpy without aligning buffers, using word loads and stores. */ 18 + /* Note: The size is truncated to a multiple of WORD_EEW */ 19 + SYM_FUNC_START(__riscv_copy_vec_words_unaligned) 20 + andi a4, a2, ~(WORD_EEW-1) 21 + beqz a4, 2f 22 + add a3, a1, a4 23 + .option push 24 + .option arch, +zve32x 25 + 1: 26 + vsetivli t0, 8, WORD_SEW, m8, ta, ma 27 + VEC_L v0, (a1) 28 + VEC_S v0, (a0) 29 + addi a0, a0, WORD_EEW 30 + addi a1, a1, WORD_EEW 31 + bltu a1, a3, 1b 32 + 33 + 2: 34 + .option pop 35 + ret 36 + SYM_FUNC_END(__riscv_copy_vec_words_unaligned) 37 + 38 + /* void __riscv_copy_vec_bytes_unaligned(void *, const void *, size_t) */ 39 + /* Performs a memcpy without aligning buffers, using only byte accesses. */ 40 + /* Note: The size is truncated to a multiple of 8 */ 41 + SYM_FUNC_START(__riscv_copy_vec_bytes_unaligned) 42 + andi a4, a2, ~(8-1) 43 + beqz a4, 2f 44 + add a3, a1, a4 45 + .option push 46 + .option arch, +zve32x 47 + 1: 48 + vsetivli t0, 8, e8, m8, ta, ma 49 + vle8.v v0, (a1) 50 + vse8.v v0, (a0) 51 + addi a0, a0, 8 52 + addi a1, a1, 8 53 + bltu a1, a3, 1b 54 + 55 + 2: 56 + .option pop 57 + ret 58 + SYM_FUNC_END(__riscv_copy_vec_bytes_unaligned)

+1 -1

arch/riscv/kernel/vector.c

··· 66 66 #endif 67 67 } 68 68 69 - static bool insn_is_vector(u32 insn_buf) 69 + bool insn_is_vector(u32 insn_buf) 70 70 { 71 71 u32 opcode = insn_buf & __INSN_OPCODE_MASK; 72 72 u32 width, csr;

+4

arch/riscv/kvm/vcpu_onereg.c

··· 34 34 [KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m, 35 35 [KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v, 36 36 /* Multi letter extensions (alphabetically sorted) */ 37 + [KVM_RISCV_ISA_EXT_SMNPM] = RISCV_ISA_EXT_SSNPM, 37 38 KVM_ISA_EXT_ARR(SMSTATEEN), 38 39 KVM_ISA_EXT_ARR(SSAIA), 39 40 KVM_ISA_EXT_ARR(SSCOFPMF), 41 + KVM_ISA_EXT_ARR(SSNPM), 40 42 KVM_ISA_EXT_ARR(SSTC), 41 43 KVM_ISA_EXT_ARR(SVINVAL), 42 44 KVM_ISA_EXT_ARR(SVNAPOT), ··· 129 127 case KVM_RISCV_ISA_EXT_C: 130 128 case KVM_RISCV_ISA_EXT_I: 131 129 case KVM_RISCV_ISA_EXT_M: 130 + case KVM_RISCV_ISA_EXT_SMNPM: 132 131 /* There is not architectural config bit to disable sscofpmf completely */ 133 132 case KVM_RISCV_ISA_EXT_SSCOFPMF: 133 + case KVM_RISCV_ISA_EXT_SSNPM: 134 134 case KVM_RISCV_ISA_EXT_SSTC: 135 135 case KVM_RISCV_ISA_EXT_SVINVAL: 136 136 case KVM_RISCV_ISA_EXT_SVNAPOT:

+9 -2

drivers/cpuidle/cpuidle-riscv-sbi.c

··· 26 26 #include <asm/smp.h> 27 27 #include <asm/suspend.h> 28 28 29 + #include "cpuidle.h" 29 30 #include "dt_idle_states.h" 30 31 #include "dt_idle_genpd.h" 31 32 ··· 330 329 return ret; 331 330 } 332 331 332 + if (cpuidle_disabled()) 333 + return 0; 334 + 333 335 ret = cpuidle_register(drv, NULL); 334 336 if (ret) 335 337 goto deinit; ··· 542 538 /* Setup CPU hotplut notifiers */ 543 539 sbi_idle_init_cpuhp(); 544 540 545 - pr_info("idle driver registered for all CPUs\n"); 541 + if (cpuidle_disabled()) 542 + pr_info("cpuidle is disabled\n"); 543 + else 544 + pr_info("idle driver registered for all CPUs\n"); 546 545 547 546 return 0; 548 547 ··· 589 582 590 583 return 0; 591 584 } 592 - device_initcall(sbi_cpuidle_init); 585 + arch_initcall(sbi_cpuidle_init);

+2

include/asm-generic/qspinlock.h

··· 136 136 } 137 137 #endif 138 138 139 + #ifndef __no_arch_spinlock_redefine 139 140 /* 140 141 * Remapping spinlock architecture specific functions to the corresponding 141 142 * queued spinlock functions. ··· 147 146 #define arch_spin_lock(l) queued_spin_lock(l) 148 147 #define arch_spin_trylock(l) queued_spin_trylock(l) 149 148 #define arch_spin_unlock(l) queued_spin_unlock(l) 149 + #endif 150 150 151 151 #endif /* __ASM_GENERIC_QSPINLOCK_H */

+1 -86

include/asm-generic/spinlock.h

··· 1 1 /* SPDX-License-Identifier: GPL-2.0 */ 2 2 3 - /* 4 - * 'Generic' ticket-lock implementation. 5 - * 6 - * It relies on atomic_fetch_add() having well defined forward progress 7 - * guarantees under contention. If your architecture cannot provide this, stick 8 - * to a test-and-set lock. 9 - * 10 - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a 11 - * sub-word of the value. This is generally true for anything LL/SC although 12 - * you'd be hard pressed to find anything useful in architecture specifications 13 - * about this. If your architecture cannot do this you might be better off with 14 - * a test-and-set. 15 - * 16 - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence 17 - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with 18 - * a full fence after the spin to upgrade the otherwise-RCpc 19 - * atomic_cond_read_acquire(). 20 - * 21 - * The implementation uses smp_cond_load_acquire() to spin, so if the 22 - * architecture has WFE like instructions to sleep instead of poll for word 23 - * modifications be sure to implement that (see ARM64 for example). 24 - * 25 - */ 26 - 27 3 #ifndef __ASM_GENERIC_SPINLOCK_H 28 4 #define __ASM_GENERIC_SPINLOCK_H 29 5 30 - #include <linux/atomic.h> 31 - #include <asm-generic/spinlock_types.h> 32 - 33 - static __always_inline void arch_spin_lock(arch_spinlock_t *lock) 34 - { 35 - u32 val = atomic_fetch_add(1<<16, lock); 36 - u16 ticket = val >> 16; 37 - 38 - if (ticket == (u16)val) 39 - return; 40 - 41 - /* 42 - * atomic_cond_read_acquire() is RCpc, but rather than defining a 43 - * custom cond_read_rcsc() here we just emit a full fence. We only 44 - * need the prior reads before subsequent writes ordering from 45 - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we 46 - * have no outstanding writes due to the atomic_fetch_add() the extra 47 - * orderings are free. 48 - */ 49 - atomic_cond_read_acquire(lock, ticket == (u16)VAL); 50 - smp_mb(); 51 - } 52 - 53 - static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) 54 - { 55 - u32 old = atomic_read(lock); 56 - 57 - if ((old >> 16) != (old & 0xffff)) 58 - return false; 59 - 60 - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ 61 - } 62 - 63 - static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) 64 - { 65 - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); 66 - u32 val = atomic_read(lock); 67 - 68 - smp_store_release(ptr, (u16)val + 1); 69 - } 70 - 71 - static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) 72 - { 73 - u32 val = lock.counter; 74 - 75 - return ((val >> 16) == (val & 0xffff)); 76 - } 77 - 78 - static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) 79 - { 80 - arch_spinlock_t val = READ_ONCE(*lock); 81 - 82 - return !arch_spin_value_unlocked(val); 83 - } 84 - 85 - static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) 86 - { 87 - u32 val = atomic_read(lock); 88 - 89 - return (s16)((val >> 16) - (val & 0xffff)) > 1; 90 - } 91 - 6 + #include <asm-generic/ticket_spinlock.h> 92 7 #include <asm/qrwlock.h> 93 8 94 9 #endif /* __ASM_GENERIC_SPINLOCK_H */

+2 -10

include/asm-generic/spinlock_types.h

··· 3 3 #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H 4 4 #define __ASM_GENERIC_SPINLOCK_TYPES_H 5 5 6 - #include <linux/types.h> 7 - typedef atomic_t arch_spinlock_t; 8 - 9 - /* 10 - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the 11 - * include. 12 - */ 13 - #include <asm/qrwlock_types.h> 14 - 15 - #define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) 6 + #include <asm-generic/qspinlock_types.h> 7 + #include <asm-generic/qrwlock_types.h> 16 8 17 9 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */

+105

include/asm-generic/ticket_spinlock.h

··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + 3 + /* 4 + * 'Generic' ticket-lock implementation. 5 + * 6 + * It relies on atomic_fetch_add() having well defined forward progress 7 + * guarantees under contention. If your architecture cannot provide this, stick 8 + * to a test-and-set lock. 9 + * 10 + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a 11 + * sub-word of the value. This is generally true for anything LL/SC although 12 + * you'd be hard pressed to find anything useful in architecture specifications 13 + * about this. If your architecture cannot do this you might be better off with 14 + * a test-and-set. 15 + * 16 + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence 17 + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with 18 + * a full fence after the spin to upgrade the otherwise-RCpc 19 + * atomic_cond_read_acquire(). 20 + * 21 + * The implementation uses smp_cond_load_acquire() to spin, so if the 22 + * architecture has WFE like instructions to sleep instead of poll for word 23 + * modifications be sure to implement that (see ARM64 for example). 24 + * 25 + */ 26 + 27 + #ifndef __ASM_GENERIC_TICKET_SPINLOCK_H 28 + #define __ASM_GENERIC_TICKET_SPINLOCK_H 29 + 30 + #include <linux/atomic.h> 31 + #include <asm-generic/spinlock_types.h> 32 + 33 + static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) 34 + { 35 + u32 val = atomic_fetch_add(1<<16, &lock->val); 36 + u16 ticket = val >> 16; 37 + 38 + if (ticket == (u16)val) 39 + return; 40 + 41 + /* 42 + * atomic_cond_read_acquire() is RCpc, but rather than defining a 43 + * custom cond_read_rcsc() here we just emit a full fence. We only 44 + * need the prior reads before subsequent writes ordering from 45 + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we 46 + * have no outstanding writes due to the atomic_fetch_add() the extra 47 + * orderings are free. 48 + */ 49 + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); 50 + smp_mb(); 51 + } 52 + 53 + static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) 54 + { 55 + u32 old = atomic_read(&lock->val); 56 + 57 + if ((old >> 16) != (old & 0xffff)) 58 + return false; 59 + 60 + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ 61 + } 62 + 63 + static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) 64 + { 65 + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); 66 + u32 val = atomic_read(&lock->val); 67 + 68 + smp_store_release(ptr, (u16)val + 1); 69 + } 70 + 71 + static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) 72 + { 73 + u32 val = lock.val.counter; 74 + 75 + return ((val >> 16) == (val & 0xffff)); 76 + } 77 + 78 + static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) 79 + { 80 + arch_spinlock_t val = READ_ONCE(*lock); 81 + 82 + return !ticket_spin_value_unlocked(val); 83 + } 84 + 85 + static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) 86 + { 87 + u32 val = atomic_read(&lock->val); 88 + 89 + return (s16)((val >> 16) - (val & 0xffff)) > 1; 90 + } 91 + 92 + #ifndef __no_arch_spinlock_redefine 93 + /* 94 + * Remapping spinlock architecture specific functions to the corresponding 95 + * ticket spinlock functions. 96 + */ 97 + #define arch_spin_is_locked(l) ticket_spin_is_locked(l) 98 + #define arch_spin_is_contended(l) ticket_spin_is_contended(l) 99 + #define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) 100 + #define arch_spin_lock(l) ticket_spin_lock(l) 101 + #define arch_spin_trylock(l) ticket_spin_trylock(l) 102 + #define arch_spin_unlock(l) ticket_spin_unlock(l) 103 + #endif 104 + 105 + #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */

+1

include/uapi/linux/elf.h

··· 451 451 #define NT_MIPS_MSA 0x802 /* MIPS SIMD registers */ 452 452 #define NT_RISCV_CSR 0x900 /* RISC-V Control and Status Registers */ 453 453 #define NT_RISCV_VECTOR 0x901 /* RISC-V vector registers */ 454 + #define NT_RISCV_TAGGED_ADDR_CTRL 0x902 /* RISC-V tagged address control (prctl()) */ 454 455 #define NT_LOONGARCH_CPUCFG 0xa00 /* LoongArch CPU config registers */ 455 456 #define NT_LOONGARCH_CSR 0xa01 /* LoongArch control and status registers */ 456 457 #define NT_LOONGARCH_LSX 0xa02 /* LoongArch Loongson SIMD Extension registers */

+4 -1

include/uapi/linux/prctl.h

··· 230 230 # define PR_PAC_APDBKEY (1UL << 3) 231 231 # define PR_PAC_APGAKEY (1UL << 4) 232 232 233 - /* Tagged user address controls for arm64 */ 233 + /* Tagged user address controls for arm64 and RISC-V */ 234 234 #define PR_SET_TAGGED_ADDR_CTRL 55 235 235 #define PR_GET_TAGGED_ADDR_CTRL 56 236 236 # define PR_TAGGED_ADDR_ENABLE (1UL << 0) ··· 244 244 # define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT) 245 245 /* Unused; kept only for source compatibility */ 246 246 # define PR_MTE_TCF_SHIFT 1 247 + /* RISC-V pointer masking tag length */ 248 + # define PR_PMLEN_SHIFT 24 249 + # define PR_PMLEN_MASK (0x7fUL << PR_PMLEN_SHIFT) 247 250 248 251 /* Control reclaim behavior when allocating memory */ 249 252 #define PR_SET_IO_FLUSHER 57

+8

tools/testing/selftests/kvm/riscv/get-reg-list.c

··· 41 41 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_I: 42 42 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_M: 43 43 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_V: 44 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SMNPM: 44 45 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SMSTATEEN: 45 46 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SSAIA: 46 47 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SSCOFPMF: 48 + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SSNPM: 47 49 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SSTC: 48 50 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SVINVAL: 49 51 case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_SVNAPOT: ··· 416 414 KVM_ISA_EXT_ARR(I), 417 415 KVM_ISA_EXT_ARR(M), 418 416 KVM_ISA_EXT_ARR(V), 417 + KVM_ISA_EXT_ARR(SMNPM), 419 418 KVM_ISA_EXT_ARR(SMSTATEEN), 420 419 KVM_ISA_EXT_ARR(SSAIA), 421 420 KVM_ISA_EXT_ARR(SSCOFPMF), 421 + KVM_ISA_EXT_ARR(SSNPM), 422 422 KVM_ISA_EXT_ARR(SSTC), 423 423 KVM_ISA_EXT_ARR(SVINVAL), 424 424 KVM_ISA_EXT_ARR(SVNAPOT), ··· 950 946 KVM_ISA_EXT_SUBLIST_CONFIG(fp_f, FP_F); 951 947 KVM_ISA_EXT_SUBLIST_CONFIG(fp_d, FP_D); 952 948 KVM_ISA_EXT_SIMPLE_CONFIG(h, H); 949 + KVM_ISA_EXT_SIMPLE_CONFIG(smnpm, SMNPM); 953 950 KVM_ISA_EXT_SUBLIST_CONFIG(smstateen, SMSTATEEN); 954 951 KVM_ISA_EXT_SIMPLE_CONFIG(sscofpmf, SSCOFPMF); 952 + KVM_ISA_EXT_SIMPLE_CONFIG(ssnpm, SSNPM); 955 953 KVM_ISA_EXT_SIMPLE_CONFIG(sstc, SSTC); 956 954 KVM_ISA_EXT_SIMPLE_CONFIG(svinval, SVINVAL); 957 955 KVM_ISA_EXT_SIMPLE_CONFIG(svnapot, SVNAPOT); ··· 1015 1009 &config_fp_f, 1016 1010 &config_fp_d, 1017 1011 &config_h, 1012 + &config_smnpm, 1018 1013 &config_smstateen, 1019 1014 &config_sscofpmf, 1015 + &config_ssnpm, 1020 1016 &config_sstc, 1021 1017 &config_svinval, 1022 1018 &config_svnapot,

+1 -1

tools/testing/selftests/riscv/Makefile

··· 5 5 ARCH ?= $(shell uname -m 2>/dev/null || echo not) 6 6 7 7 ifneq (,$(filter $(ARCH),riscv)) 8 - RISCV_SUBTARGETS ?= hwprobe vector mm sigreturn 8 + RISCV_SUBTARGETS ?= abi hwprobe mm sigreturn vector 9 9 else 10 10 RISCV_SUBTARGETS := 11 11 endif

+1

tools/testing/selftests/riscv/abi/.gitignore

··· 1 + pointer_masking

+10

tools/testing/selftests/riscv/abi/Makefile

··· 1 + # SPDX-License-Identifier: GPL-2.0 2 + 3 + CFLAGS += -I$(top_srcdir)/tools/include 4 + 5 + TEST_GEN_PROGS := pointer_masking 6 + 7 + include ../../lib.mk 8 + 9 + $(OUTPUT)/pointer_masking: pointer_masking.c 10 + $(CC) -static -o$@ $(CFLAGS) $(LDFLAGS) $^

+332

tools/testing/selftests/riscv/abi/pointer_masking.c

··· 1 + // SPDX-License-Identifier: GPL-2.0-only 2 + 3 + #include <errno.h> 4 + #include <fcntl.h> 5 + #include <setjmp.h> 6 + #include <signal.h> 7 + #include <stdbool.h> 8 + #include <sys/prctl.h> 9 + #include <sys/wait.h> 10 + #include <unistd.h> 11 + 12 + #include "../../kselftest.h" 13 + 14 + #ifndef PR_PMLEN_SHIFT 15 + #define PR_PMLEN_SHIFT 24 16 + #endif 17 + #ifndef PR_PMLEN_MASK 18 + #define PR_PMLEN_MASK (0x7fUL << PR_PMLEN_SHIFT) 19 + #endif 20 + 21 + static int dev_zero; 22 + 23 + static int pipefd[2]; 24 + 25 + static sigjmp_buf jmpbuf; 26 + 27 + static void sigsegv_handler(int sig) 28 + { 29 + siglongjmp(jmpbuf, 1); 30 + } 31 + 32 + static int min_pmlen; 33 + static int max_pmlen; 34 + 35 + static inline bool valid_pmlen(int pmlen) 36 + { 37 + return pmlen == 0 || pmlen == 7 || pmlen == 16; 38 + } 39 + 40 + static void test_pmlen(void) 41 + { 42 + ksft_print_msg("Testing available PMLEN values\n"); 43 + 44 + for (int request = 0; request <= 16; request++) { 45 + int pmlen, ret; 46 + 47 + ret = prctl(PR_SET_TAGGED_ADDR_CTRL, request << PR_PMLEN_SHIFT, 0, 0, 0); 48 + if (ret) 49 + goto pr_set_error; 50 + 51 + ret = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0); 52 + ksft_test_result(ret >= 0, "PMLEN=%d PR_GET_TAGGED_ADDR_CTRL\n", request); 53 + if (ret < 0) 54 + goto pr_get_error; 55 + 56 + pmlen = (ret & PR_PMLEN_MASK) >> PR_PMLEN_SHIFT; 57 + ksft_test_result(pmlen >= request, "PMLEN=%d constraint\n", request); 58 + ksft_test_result(valid_pmlen(pmlen), "PMLEN=%d validity\n", request); 59 + 60 + if (min_pmlen == 0) 61 + min_pmlen = pmlen; 62 + if (max_pmlen < pmlen) 63 + max_pmlen = pmlen; 64 + 65 + continue; 66 + 67 + pr_set_error: 68 + ksft_test_result_skip("PMLEN=%d PR_GET_TAGGED_ADDR_CTRL\n", request); 69 + pr_get_error: 70 + ksft_test_result_skip("PMLEN=%d constraint\n", request); 71 + ksft_test_result_skip("PMLEN=%d validity\n", request); 72 + } 73 + 74 + if (max_pmlen == 0) 75 + ksft_exit_fail_msg("Failed to enable pointer masking\n"); 76 + } 77 + 78 + static int set_tagged_addr_ctrl(int pmlen, bool tagged_addr_abi) 79 + { 80 + int arg, ret; 81 + 82 + arg = pmlen << PR_PMLEN_SHIFT | tagged_addr_abi; 83 + ret = prctl(PR_SET_TAGGED_ADDR_CTRL, arg, 0, 0, 0); 84 + if (!ret) { 85 + ret = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0); 86 + if (ret == arg) 87 + return 0; 88 + } 89 + 90 + return ret < 0 ? -errno : -ENODATA; 91 + } 92 + 93 + static void test_dereference_pmlen(int pmlen) 94 + { 95 + static volatile int i; 96 + volatile int *p; 97 + int ret; 98 + 99 + ret = set_tagged_addr_ctrl(pmlen, false); 100 + if (ret) 101 + return ksft_test_result_error("PMLEN=%d setup (%d)\n", pmlen, ret); 102 + 103 + i = pmlen; 104 + 105 + if (pmlen) { 106 + p = (volatile int *)((uintptr_t)&i | 1UL << (__riscv_xlen - pmlen)); 107 + 108 + /* These dereferences should succeed. */ 109 + if (sigsetjmp(jmpbuf, 1)) 110 + return ksft_test_result_fail("PMLEN=%d valid tag\n", pmlen); 111 + if (*p != pmlen) 112 + return ksft_test_result_fail("PMLEN=%d bad value\n", pmlen); 113 + ++*p; 114 + } 115 + 116 + p = (volatile int *)((uintptr_t)&i | 1UL << (__riscv_xlen - pmlen - 1)); 117 + 118 + /* These dereferences should raise SIGSEGV. */ 119 + if (sigsetjmp(jmpbuf, 1)) 120 + return ksft_test_result_pass("PMLEN=%d dereference\n", pmlen); 121 + ++*p; 122 + ksft_test_result_fail("PMLEN=%d invalid tag\n", pmlen); 123 + } 124 + 125 + static void test_dereference(void) 126 + { 127 + ksft_print_msg("Testing userspace pointer dereference\n"); 128 + 129 + signal(SIGSEGV, sigsegv_handler); 130 + 131 + test_dereference_pmlen(0); 132 + test_dereference_pmlen(min_pmlen); 133 + test_dereference_pmlen(max_pmlen); 134 + 135 + signal(SIGSEGV, SIG_DFL); 136 + } 137 + 138 + static void execve_child_sigsegv_handler(int sig) 139 + { 140 + exit(42); 141 + } 142 + 143 + static int execve_child(void) 144 + { 145 + static volatile int i; 146 + volatile int *p = (volatile int *)((uintptr_t)&i | 1UL << (__riscv_xlen - 7)); 147 + 148 + signal(SIGSEGV, execve_child_sigsegv_handler); 149 + 150 + /* This dereference should raise SIGSEGV. */ 151 + return *p; 152 + } 153 + 154 + static void test_fork_exec(void) 155 + { 156 + int ret, status; 157 + 158 + ksft_print_msg("Testing fork/exec behavior\n"); 159 + 160 + ret = set_tagged_addr_ctrl(min_pmlen, false); 161 + if (ret) 162 + return ksft_test_result_error("setup (%d)\n", ret); 163 + 164 + if (fork()) { 165 + wait(&status); 166 + ksft_test_result(WIFEXITED(status) && WEXITSTATUS(status) == 42, 167 + "dereference after fork\n"); 168 + } else { 169 + static volatile int i = 42; 170 + volatile int *p; 171 + 172 + p = (volatile int *)((uintptr_t)&i | 1UL << (__riscv_xlen - min_pmlen)); 173 + 174 + /* This dereference should succeed. */ 175 + exit(*p); 176 + } 177 + 178 + if (fork()) { 179 + wait(&status); 180 + ksft_test_result(WIFEXITED(status) && WEXITSTATUS(status) == 42, 181 + "dereference after fork+exec\n"); 182 + } else { 183 + /* Will call execve_child(). */ 184 + execve("/proc/self/exe", (char *const []) { "", NULL }, NULL); 185 + } 186 + } 187 + 188 + static void test_tagged_addr_abi_sysctl(void) 189 + { 190 + char value; 191 + int fd; 192 + 193 + ksft_print_msg("Testing tagged address ABI sysctl\n"); 194 + 195 + fd = open("/proc/sys/abi/tagged_addr_disabled", O_WRONLY); 196 + if (fd < 0) { 197 + ksft_test_result_skip("failed to open sysctl file\n"); 198 + ksft_test_result_skip("failed to open sysctl file\n"); 199 + return; 200 + } 201 + 202 + value = '1'; 203 + pwrite(fd, &value, 1, 0); 204 + ksft_test_result(set_tagged_addr_ctrl(min_pmlen, true) == -EINVAL, 205 + "sysctl disabled\n"); 206 + 207 + value = '0'; 208 + pwrite(fd, &value, 1, 0); 209 + ksft_test_result(set_tagged_addr_ctrl(min_pmlen, true) == 0, 210 + "sysctl enabled\n"); 211 + 212 + set_tagged_addr_ctrl(0, false); 213 + 214 + close(fd); 215 + } 216 + 217 + static void test_tagged_addr_abi_pmlen(int pmlen) 218 + { 219 + int i, *p, ret; 220 + 221 + i = ~pmlen; 222 + 223 + if (pmlen) { 224 + p = (int *)((uintptr_t)&i | 1UL << (__riscv_xlen - pmlen)); 225 + 226 + ret = set_tagged_addr_ctrl(pmlen, false); 227 + if (ret) 228 + return ksft_test_result_error("PMLEN=%d ABI disabled setup (%d)\n", 229 + pmlen, ret); 230 + 231 + ret = write(pipefd[1], p, sizeof(*p)); 232 + if (ret >= 0 || errno != EFAULT) 233 + return ksft_test_result_fail("PMLEN=%d ABI disabled write\n", pmlen); 234 + 235 + ret = read(dev_zero, p, sizeof(*p)); 236 + if (ret >= 0 || errno != EFAULT) 237 + return ksft_test_result_fail("PMLEN=%d ABI disabled read\n", pmlen); 238 + 239 + if (i != ~pmlen) 240 + return ksft_test_result_fail("PMLEN=%d ABI disabled value\n", pmlen); 241 + 242 + ret = set_tagged_addr_ctrl(pmlen, true); 243 + if (ret) 244 + return ksft_test_result_error("PMLEN=%d ABI enabled setup (%d)\n", 245 + pmlen, ret); 246 + 247 + ret = write(pipefd[1], p, sizeof(*p)); 248 + if (ret != sizeof(*p)) 249 + return ksft_test_result_fail("PMLEN=%d ABI enabled write\n", pmlen); 250 + 251 + ret = read(dev_zero, p, sizeof(*p)); 252 + if (ret != sizeof(*p)) 253 + return ksft_test_result_fail("PMLEN=%d ABI enabled read\n", pmlen); 254 + 255 + if (i) 256 + return ksft_test_result_fail("PMLEN=%d ABI enabled value\n", pmlen); 257 + 258 + i = ~pmlen; 259 + } else { 260 + /* The tagged address ABI cannot be enabled when PMLEN == 0. */ 261 + ret = set_tagged_addr_ctrl(pmlen, true); 262 + if (ret != -EINVAL) 263 + return ksft_test_result_error("PMLEN=%d ABI setup (%d)\n", 264 + pmlen, ret); 265 + } 266 + 267 + p = (int *)((uintptr_t)&i | 1UL << (__riscv_xlen - pmlen - 1)); 268 + 269 + ret = write(pipefd[1], p, sizeof(*p)); 270 + if (ret >= 0 || errno != EFAULT) 271 + return ksft_test_result_fail("PMLEN=%d invalid tag write (%d)\n", pmlen, errno); 272 + 273 + ret = read(dev_zero, p, sizeof(*p)); 274 + if (ret >= 0 || errno != EFAULT) 275 + return ksft_test_result_fail("PMLEN=%d invalid tag read\n", pmlen); 276 + 277 + if (i != ~pmlen) 278 + return ksft_test_result_fail("PMLEN=%d invalid tag value\n", pmlen); 279 + 280 + ksft_test_result_pass("PMLEN=%d tagged address ABI\n", pmlen); 281 + } 282 + 283 + static void test_tagged_addr_abi(void) 284 + { 285 + ksft_print_msg("Testing tagged address ABI\n"); 286 + 287 + test_tagged_addr_abi_pmlen(0); 288 + test_tagged_addr_abi_pmlen(min_pmlen); 289 + test_tagged_addr_abi_pmlen(max_pmlen); 290 + } 291 + 292 + static struct test_info { 293 + unsigned int nr_tests; 294 + void (*test_fn)(void); 295 + } tests[] = { 296 + { .nr_tests = 17 * 3, test_pmlen }, 297 + { .nr_tests = 3, test_dereference }, 298 + { .nr_tests = 2, test_fork_exec }, 299 + { .nr_tests = 2, test_tagged_addr_abi_sysctl }, 300 + { .nr_tests = 3, test_tagged_addr_abi }, 301 + }; 302 + 303 + int main(int argc, char **argv) 304 + { 305 + unsigned int plan = 0; 306 + int ret; 307 + 308 + /* Check if this is the child process after execve(). */ 309 + if (!argv[0][0]) 310 + return execve_child(); 311 + 312 + dev_zero = open("/dev/zero", O_RDWR); 313 + if (dev_zero < 0) 314 + return 1; 315 + 316 + /* Write to a pipe so the kernel must dereference the buffer pointer. */ 317 + ret = pipe(pipefd); 318 + if (ret) 319 + return 1; 320 + 321 + ksft_print_header(); 322 + 323 + for (int i = 0; i < ARRAY_SIZE(tests); i++) 324 + plan += tests[i].nr_tests; 325 + 326 + ksft_set_plan(plan); 327 + 328 + for (int i = 0; i < ARRAY_SIZE(tests); i++) 329 + tests[i].test_fn(); 330 + 331 + ksft_finished(); 332 + }

Configure Feed

Configure Feed