tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge tag 'riscv-for-linus-7.1-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

Pull RISC-V updates from Paul Walmsley:
"There is one significant change outside arch/riscv in this pull
request: the addition of a set of KUnit tests for strlen(), strnlen(),
and strrchr().

Otherwise, the most notable changes are to add some RISC-V-specific
string function implementations, to remove XIP kernel support, to add
hardware error exception handling, and to optimize our runtime
unaligned access speed testing.

A few comments on the motivation for removing XIP support. It's been
broken in the RISC-V kernel for months. The code is not easy to
maintain. Furthermore, for XIP support to truly be useful for RISC-V,
we think that compile-time feature switches would need to be added for
many of the RISC-V ISA features and microarchitectural properties that
are currently implemented with runtime patching. No one has stepped
forward to take responsibility for that work, so many of us think it's
best to remove it until clear use cases and champions emerge.

Summary:

- Add Kunit correctness testing and microbenchmarks for strlen(),
strnlen(), and strrchr()

- Add RISC-V-specific strnlen(), strchr(), strrchr() implementations

- Add hardware error exception handling

- Clean up and optimize our unaligned access probe code

- Enable HAVE_IOREMAP_PROT to be able to use generic_access_phys()

- Remove XIP kernel support

- Warn when addresses outside the vmemmap range are passed to
vmemmap_populate()

- Update the ACPI FADT revision check to warn if it's not at least
ACPI v6.6, which is when key RISC-V-specific tables were added to
the specification

- Increase COMMAND_LINE_SIZE to 2048 to match ARM64, x86, PowerPC,
etc.

- Make kaslr_offset() a static inline function, since there's no need
for it to show up in the symbol table

- Add KASLR offset and SATP to the VMCOREINFO ELF notes to improve
kdump support

- Add Makefile cleanup rule for vdso_cfi copied source files, and add
a .gitignore for the build artifacts in that directory

- Remove some redundant ifdefs that check Kconfig macros

- Add missing SPDX license tag to the CFI selftest

- Simplify UTS_MACHINE assignment in the RISC-V Makefile

- Clarify some unclear comments and remove some superfluous comments

- Fix various English typos across the RISC-V codebase"

* tag 'riscv-for-linus-7.1-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (31 commits)
riscv: Remove support for XIP kernel
riscv: Reuse compare_unaligned_access() in check_vector_unaligned_access()
riscv: Split out compare_unaligned_access()
riscv: Reuse measure_cycles() in check_vector_unaligned_access()
riscv: Split out measure_cycles() for reuse
riscv: Clean up & optimize unaligned scalar access probe
riscv: lib: add strrchr() implementation
riscv: lib: add strchr() implementation
riscv: lib: add strnlen() implementation
lib/string_kunit: extend benchmarks to strnlen() and chr searches
lib/string_kunit: add performance benchmark for strlen()
lib/string_kunit: add correctness test for strrchr()
lib/string_kunit: add correctness test for strnlen()
lib/string_kunit: add correctness test for strlen()
riscv: vdso_cfi: Add .gitignore for build artifacts
riscv: vdso_cfi: Add clean rule for copied sources
riscv: enable HAVE_IOREMAP_PROT
riscv: mm: WARN_ON() for bad addresses in vmemmap_populate()
riscv: acpi: update FADT revision check to 6.6
riscv: add hardware error trap handler support
...

Linus Torvalds feff82eb ff57d592

+735 -539
+1 -1
Documentation/features/vm/ioremap_prot/arch-support.txt
··· 20 20 | openrisc: | TODO | 21 21 | parisc: | TODO | 22 22 | powerpc: | ok | 23 - | riscv: | TODO | 23 + | riscv: | ok | 24 24 | s390: | ok | 25 25 | sh: | ok | 26 26 | sparc: | TODO |
+24 -63
arch/riscv/Kconfig
··· 47 47 select ARCH_HAS_PTE_SPECIAL 48 48 select ARCH_HAS_SET_DIRECT_MAP if MMU 49 49 select ARCH_HAS_SET_MEMORY if MMU 50 - select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL 51 - select ARCH_HAS_STRICT_MODULE_RWX if MMU && !XIP_KERNEL 50 + select ARCH_HAS_STRICT_KERNEL_RWX if MMU 51 + select ARCH_HAS_STRICT_MODULE_RWX if MMU 52 52 select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE 53 53 select ARCH_HAS_SYSCALL_WRAPPER 54 54 select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST ··· 84 84 select ARCH_WANT_FRAME_POINTERS 85 85 select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT 86 86 select ARCH_WANT_HUGE_PMD_SHARE if 64BIT 87 - select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL 87 + select ARCH_WANT_LD_ORPHAN_WARN 88 88 select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP 89 89 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP 90 90 select ARCH_WANTS_NO_INSTR ··· 112 112 select GENERIC_GETTIMEOFDAY if HAVE_GENERIC_VDSO && 64BIT 113 113 select GENERIC_IDLE_POLL_SETUP 114 114 select GENERIC_IOREMAP if MMU 115 + select HAVE_IOREMAP_PROT if MMU 115 116 select GENERIC_IRQ_IPI if SMP 116 117 select GENERIC_IRQ_IPI_MUX if SMP 117 118 select GENERIC_IRQ_MULTI_HANDLER ··· 130 129 select HAVE_ARCH_AUDITSYSCALL 131 130 select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP 132 131 select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT 133 - select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL 134 - select HAVE_ARCH_JUMP_LABEL_RELATIVE if !XIP_KERNEL 132 + select HAVE_ARCH_JUMP_LABEL 133 + select HAVE_ARCH_JUMP_LABEL_RELATIVE 135 134 select HAVE_ARCH_KASAN if MMU && 64BIT 136 135 select HAVE_ARCH_KASAN_VMALLOC if MMU && 64BIT 137 136 select HAVE_ARCH_KFENCE if MMU && 64BIT 138 137 select HAVE_ARCH_KSTACK_ERASE 139 - select HAVE_ARCH_KGDB if !XIP_KERNEL 138 + select HAVE_ARCH_KGDB 140 139 select HAVE_ARCH_KGDB_QXFER_PKT 141 140 select HAVE_ARCH_MMAP_RND_BITS if MMU 142 141 select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT ··· 154 153 select HAVE_CONTEXT_TRACKING_USER 155 154 select HAVE_DEBUG_KMEMLEAK 156 155 select HAVE_DMA_CONTIGUOUS if MMU 157 - select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) 156 + select HAVE_DYNAMIC_FTRACE if MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) 158 157 select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C 159 158 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS 160 159 select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS && !CFI) ··· 162 161 select HAVE_FTRACE_GRAPH_FUNC 163 162 select HAVE_FUNCTION_GRAPH_TRACER if HAVE_DYNAMIC_FTRACE_WITH_ARGS 164 163 select HAVE_FUNCTION_GRAPH_FREGS 165 - select HAVE_FUNCTION_TRACER if !XIP_KERNEL && HAVE_DYNAMIC_FTRACE 164 + select HAVE_FUNCTION_TRACER if HAVE_DYNAMIC_FTRACE 166 165 select HAVE_EBPF_JIT if MMU 167 166 select HAVE_GENERIC_TIF_BITS 168 167 select HAVE_GUP_FAST if MMU ··· 171 170 select HAVE_GCC_PLUGINS 172 171 select HAVE_GENERIC_VDSO if MMU 173 172 select HAVE_IRQ_TIME_ACCOUNTING 174 - select HAVE_KERNEL_BZIP2 if !XIP_KERNEL && !EFI_ZBOOT 175 - select HAVE_KERNEL_GZIP if !XIP_KERNEL && !EFI_ZBOOT 176 - select HAVE_KERNEL_LZ4 if !XIP_KERNEL && !EFI_ZBOOT 177 - select HAVE_KERNEL_LZMA if !XIP_KERNEL && !EFI_ZBOOT 178 - select HAVE_KERNEL_LZO if !XIP_KERNEL && !EFI_ZBOOT 179 - select HAVE_KERNEL_UNCOMPRESSED if !XIP_KERNEL && !EFI_ZBOOT 180 - select HAVE_KERNEL_ZSTD if !XIP_KERNEL && !EFI_ZBOOT 181 - select HAVE_KERNEL_XZ if !XIP_KERNEL && !EFI_ZBOOT 182 - select HAVE_KPROBES if !XIP_KERNEL 183 - select HAVE_KRETPROBES if !XIP_KERNEL 173 + select HAVE_KERNEL_BZIP2 if !EFI_ZBOOT 174 + select HAVE_KERNEL_GZIP if !EFI_ZBOOT 175 + select HAVE_KERNEL_LZ4 if !EFI_ZBOOT 176 + select HAVE_KERNEL_LZMA if !EFI_ZBOOT 177 + select HAVE_KERNEL_LZO if !EFI_ZBOOT 178 + select HAVE_KERNEL_UNCOMPRESSED if !EFI_ZBOOT 179 + select HAVE_KERNEL_ZSTD if !EFI_ZBOOT 180 + select HAVE_KERNEL_XZ if !EFI_ZBOOT 181 + select HAVE_KPROBES 182 + select HAVE_KRETPROBES 184 183 # https://github.com/ClangBuiltLinux/linux/issues/1881 185 184 select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD 186 185 select HAVE_MOVE_PMD ··· 191 190 select HAVE_PERF_REGS 192 191 select HAVE_PERF_USER_STACK_DUMP 193 192 select HAVE_POSIX_CPU_TIMERS_TASK_WORK 194 - select HAVE_PREEMPT_DYNAMIC_KEY if !XIP_KERNEL 193 + select HAVE_PREEMPT_DYNAMIC_KEY 195 194 select HAVE_REGS_AND_STACK_ACCESS_API 196 - select HAVE_RETHOOK if !XIP_KERNEL 195 + select HAVE_RETHOOK 197 196 select HAVE_RSEQ 198 197 select HAVE_RUST if RUSTC_SUPPORTS_RISCV && CC_IS_CLANG 199 198 select HAVE_SAMPLE_FTRACE_DIRECT ··· 214 213 select PCI_ECAM if (ACPI && PCI) 215 214 select PCI_MSI if PCI 216 215 select RELOCATABLE if !MMU && !PHYS_RAM_BASE_FIXED 217 - select RISCV_ALTERNATIVE if !XIP_KERNEL 216 + select RISCV_ALTERNATIVE 218 217 select RISCV_APLIC 219 218 select RISCV_IMSIC 220 219 select RISCV_INTC ··· 538 537 539 538 config RISCV_ALTERNATIVE 540 539 bool 541 - depends on !XIP_KERNEL 542 540 help 543 541 This Kconfig allows the kernel to automatically patch the 544 542 erratum or cpufeature required by the execution platform at run ··· 1126 1126 1127 1127 config RELOCATABLE 1128 1128 bool "Build a relocatable kernel" 1129 - depends on !XIP_KERNEL 1130 1129 select MODULE_SECTIONS if MODULES 1131 1130 select ARCH_VMLINUX_NEEDS_RELOCS 1132 1131 help ··· 1142 1143 config RANDOMIZE_BASE 1143 1144 bool "Randomize the address of the kernel image" 1144 1145 select RELOCATABLE 1145 - depends on MMU && 64BIT && !XIP_KERNEL 1146 + depends on MMU && 64BIT 1146 1147 help 1147 1148 Randomizes the virtual address at which the kernel image is 1148 1149 loaded, as a security feature that deters exploit attempts ··· 1232 1233 1233 1234 config EFI 1234 1235 bool "UEFI runtime support" 1235 - depends on OF && !XIP_KERNEL 1236 + depends on OF 1236 1237 depends on MMU 1237 1238 default y 1238 1239 select ARCH_SUPPORTS_ACPI if 64BIT ··· 1282 1283 This is the physical address of RAM in the system. It has to be 1283 1284 explicitly specified to run early relocations of read-write data 1284 1285 from flash to RAM. 1285 - 1286 - config XIP_KERNEL 1287 - bool "Kernel Execute-In-Place from ROM" 1288 - depends on MMU && SPARSEMEM && NONPORTABLE 1289 - # This prevents XIP from being enabled by all{yes,mod}config, which 1290 - # fail to build since XIP doesn't support large kernels. 1291 - depends on !COMPILE_TEST 1292 - select PHYS_RAM_BASE_FIXED 1293 - help 1294 - Execute-In-Place allows the kernel to run from non-volatile storage 1295 - directly addressable by the CPU, such as NOR flash. This saves RAM 1296 - space since the text section of the kernel is not loaded from flash 1297 - to RAM. Read-write sections, such as the data section and stack, 1298 - are still copied to RAM. The XIP kernel is not compressed since 1299 - it has to run directly from flash, so it will take more space to 1300 - store it. The flash address used to link the kernel object files, 1301 - and for storing it, is configuration dependent. Therefore, if you 1302 - say Y here, you must know the proper physical address where to 1303 - store the kernel image depending on your own flash memory usage. 1304 - 1305 - Also note that the make target becomes "make xipImage" rather than 1306 - "make zImage" or "make Image". The final kernel binary to put in 1307 - ROM memory will be arch/riscv/boot/xipImage. 1308 - 1309 - SPARSEMEM is required because the kernel text and rodata that are 1310 - flash resident are not backed by memmap, then any attempt to get 1311 - a struct page on those regions will trigger a fault. 1312 - 1313 - If unsure, say N. 1314 - 1315 - config XIP_PHYS_ADDR 1316 - hex "XIP Kernel Physical Location" 1317 - depends on XIP_KERNEL 1318 - default "0x21000000" 1319 - help 1320 - This is the physical address in your flash memory the kernel will 1321 - be linked for and stored to. This address is dependent on your 1322 - own flash usage. 1323 1286 1324 1287 config RISCV_ISA_FALLBACK 1325 1288 bool "Permit falling back to parsing riscv,isa for extension support by default"
+4 -4
arch/riscv/Kconfig.socs
··· 2 2 3 3 config ARCH_ANDES 4 4 bool "Andes SoCs" 5 - depends on MMU && !XIP_KERNEL 5 + depends on MMU 6 6 select ERRATA_ANDES 7 7 help 8 8 This enables support for Andes SoC platform hardware. ··· 33 33 34 34 config ARCH_SIFIVE 35 35 bool "SiFive SoCs" 36 - select ERRATA_SIFIVE if !XIP_KERNEL 36 + select ERRATA_SIFIVE 37 37 help 38 38 This enables support for SiFive SoC platform hardware. 39 39 ··· 61 61 62 62 config ARCH_SUNXI 63 63 bool "Allwinner sun20i SoCs" 64 - depends on MMU && !XIP_KERNEL 64 + depends on MMU 65 65 select ERRATA_THEAD 66 66 select SUN4I_TIMER 67 67 help ··· 78 78 79 79 config ARCH_THEAD 80 80 bool "T-HEAD RISC-V SoCs" 81 - depends on MMU && !XIP_KERNEL 81 + depends on MMU 82 82 select ERRATA_THEAD 83 83 select PM_GENERIC_DOMAINS if PM 84 84 help
+2 -5
arch/riscv/Makefile
··· 28 28 export BITS 29 29 ifeq ($(CONFIG_ARCH_RV64I),y) 30 30 BITS := 64 31 - UTS_MACHINE := riscv64 32 31 33 32 KBUILD_CFLAGS += -mabi=lp64 34 33 KBUILD_AFLAGS += -mabi=lp64 ··· 38 39 -Cno-redzone 39 40 else 40 41 BITS := 32 41 - UTS_MACHINE := riscv32 42 42 43 43 KBUILD_CFLAGS += -mabi=ilp32 44 44 KBUILD_AFLAGS += -mabi=ilp32 45 45 KBUILD_LDFLAGS += -melf32lriscv 46 46 endif 47 + 48 + UTS_MACHINE := riscv$(BITS) 47 49 48 50 # LLVM has an issue with target-features and LTO: https://github.com/llvm/llvm-project/issues/59350 49 51 # Ensure it is aware of linker relaxation with LTO, otherwise relocations may ··· 150 150 boot-image-$(CONFIG_SOC_CANAAN_K210) := loader.bin 151 151 endif 152 152 boot-image-$(CONFIG_EFI_ZBOOT) := vmlinuz.efi 153 - boot-image-$(CONFIG_XIP_KERNEL) := xipImage 154 153 KBUILD_IMAGE := $(boot)/$(boot-image-y) 155 154 156 155 libs-y += arch/riscv/lib/ ··· 217 218 echo ' Image.xz - Compressed kernel image (arch/riscv/boot/Image.xz)' 218 219 echo ' vmlinuz.efi - Compressed EFI kernel image (arch/riscv/boot/vmlinuz.efi)' 219 220 echo ' Default when CONFIG_EFI_ZBOOT=y' 220 - echo ' xipImage - Execute-in-place kernel image (arch/riscv/boot/xipImage)' 221 - echo ' Default when CONFIG_XIP_KERNEL=y' 222 221 echo ' install - Install kernel using (your) ~/bin/$(INSTALLKERNEL) or' 223 222 echo ' (distribution) /sbin/$(INSTALLKERNEL) or install to ' 224 223 echo ' $$(INSTALL_PATH)'
-11
arch/riscv/boot/Makefile
··· 20 20 21 21 targets := Image Image.* loader loader.o loader.lds loader.bin xipImage 22 22 23 - ifeq ($(CONFIG_XIP_KERNEL),y) 24 - 25 - quiet_cmd_mkxip = $(quiet_cmd_objcopy) 26 - cmd_mkxip = $(cmd_objcopy) 27 - 28 - $(obj)/xipImage: vmlinux FORCE 29 - $(call if_changed,mkxip) 30 - @$(kecho) ' Physical Address of xipImage: $(CONFIG_XIP_PHYS_ADDR)' 31 - 32 - endif 33 - 34 23 $(obj)/Image: vmlinux FORCE 35 24 $(call if_changed,objcopy) 36 25
+1 -1
arch/riscv/errata/thead/errata.c
··· 153 153 * target-c9xx cores report arch_id and impid as 0 154 154 * 155 155 * While ghostwrite may not affect all c9xx cores that implement 156 - * xtheadvector, there is no futher granularity than c9xx. Assume 156 + * xtheadvector, there is no further granularity than c9xx. Assume 157 157 * vulnerable for this entire class of processors when xtheadvector is 158 158 * enabled. 159 159 */
+1
arch/riscv/include/asm/asm-prototypes.h
··· 40 40 #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) 41 41 42 42 DECLARE_DO_ERROR_INFO(do_trap_unknown); 43 + DECLARE_DO_ERROR_INFO(do_trap_hardware_error); 43 44 DECLARE_DO_ERROR_INFO(do_trap_insn_misaligned); 44 45 DECLARE_DO_ERROR_INFO(do_trap_insn_fault); 45 46 DECLARE_DO_ERROR_INFO(do_trap_insn_illegal);
+2 -2
arch/riscv/include/asm/atomic.h
··· 46 46 #endif 47 47 48 48 /* 49 - * First, the atomic ops that have no ordering constraints and therefor don't 49 + * First, the atomic ops that have no ordering constraints and therefore don't 50 50 * have the AQ or RL bits set. These don't return anything, so there's only 51 51 * one version to worry about. 52 52 */ ··· 81 81 82 82 /* 83 83 * Atomic ops that have ordered, relaxed, acquire, and release variants. 84 - * There's two flavors of these: the arithmatic ops have both fetch and return 84 + * There's two flavors of these: the arithmetic ops have both fetch and return 85 85 * versions, while the logical ops only have fetch versions. 86 86 */ 87 87 #define ATOMIC_FETCH_OP(op, asm_op, I, asm_type, c_type, prefix) \
+2 -2
arch/riscv/include/asm/elf.h
··· 59 59 #endif 60 60 61 61 /* 62 - * Provides information on the availiable set of ISA extensions to userspace, 63 - * via a bitmap that coorespends to each single-letter ISA extension. This is 62 + * Provides information on the available set of ISA extensions to userspace, 63 + * via a bitmap that corresponds to each single-letter ISA extension. This is 64 64 * essentially defunct, but will remain for compatibility with userspace. 65 65 */ 66 66 #define ELF_HWCAP riscv_get_elf_hwcap()
+4 -30
arch/riscv/include/asm/page.h
··· 29 29 #define PAGE_OFFSET_L5 _AC(0xff60000000000000, UL) 30 30 #define PAGE_OFFSET_L4 _AC(0xffffaf8000000000, UL) 31 31 #define PAGE_OFFSET_L3 _AC(0xffffffd600000000, UL) 32 - #ifdef CONFIG_XIP_KERNEL 33 - #define PAGE_OFFSET PAGE_OFFSET_L3 34 - #else 35 32 #define PAGE_OFFSET kernel_map.page_offset 36 - #endif /* CONFIG_XIP_KERNEL */ 37 33 #else 38 34 #define PAGE_OFFSET _AC(0xc0000000, UL) 39 35 #endif /* CONFIG_64BIT */ ··· 100 104 /* Offset between linear mapping virtual address and kernel load address */ 101 105 unsigned long va_pa_offset; 102 106 /* Offset between kernel mapping virtual address and kernel load address */ 103 - #ifdef CONFIG_XIP_KERNEL 104 - unsigned long va_kernel_xip_text_pa_offset; 105 - unsigned long va_kernel_xip_data_pa_offset; 106 - uintptr_t xiprom; 107 - uintptr_t xiprom_sz; 108 - #else 109 107 unsigned long page_offset; 110 108 unsigned long va_kernel_pa_offset; 111 - #endif 112 109 }; 113 110 114 111 extern struct kernel_mapping kernel_map; ··· 120 131 void *linear_mapping_pa_to_va(unsigned long x); 121 132 #endif 122 133 123 - #ifdef CONFIG_XIP_KERNEL 124 - #define kernel_mapping_pa_to_va(y) ({ \ 125 - unsigned long _y = (unsigned long)(y); \ 126 - (_y < phys_ram_base) ? \ 127 - (void *)(_y + kernel_map.va_kernel_xip_text_pa_offset) : \ 128 - (void *)(_y + kernel_map.va_kernel_xip_data_pa_offset); \ 129 - }) 130 - #else 131 134 #define kernel_mapping_pa_to_va(y) ((void *)((unsigned long)(y) + kernel_map.va_kernel_pa_offset)) 132 - #endif 133 135 134 136 #define __pa_to_va_nodebug(x) linear_mapping_pa_to_va(x) 135 137 ··· 130 150 phys_addr_t linear_mapping_va_to_pa(unsigned long x); 131 151 #endif 132 152 133 - #ifdef CONFIG_XIP_KERNEL 134 - #define kernel_mapping_va_to_pa(y) ({ \ 135 - unsigned long _y = (unsigned long)(y); \ 136 - (_y < kernel_map.virt_addr + kernel_map.xiprom_sz) ? \ 137 - (_y - kernel_map.va_kernel_xip_text_pa_offset) : \ 138 - (_y - kernel_map.va_kernel_xip_data_pa_offset); \ 139 - }) 140 - #else 141 153 #define kernel_mapping_va_to_pa(y) ((unsigned long)(y) - kernel_map.va_kernel_pa_offset) 142 - #endif 143 154 144 155 #define __va_to_pa_nodebug(x) ({ \ 145 156 unsigned long _x = x; \ ··· 161 190 162 191 #define sym_to_pfn(x) __phys_to_pfn(__pa_symbol(x)) 163 192 164 - unsigned long kaslr_offset(void); 193 + static inline unsigned long kaslr_offset(void) 194 + { 195 + return kernel_map.virt_offset; 196 + } 165 197 166 198 static __always_inline void *pfn_to_kaddr(unsigned long pfn) 167 199 {
-20
arch/riscv/include/asm/pgtable.h
··· 134 134 135 135 #include <linux/page_table_check.h> 136 136 137 - #ifdef CONFIG_XIP_KERNEL 138 - #define XIP_FIXUP(addr) ({ \ 139 - extern char _sdata[], _start[], _end[]; \ 140 - uintptr_t __rom_start_data = CONFIG_XIP_PHYS_ADDR \ 141 - + (uintptr_t)&_sdata - (uintptr_t)&_start; \ 142 - uintptr_t __rom_end_data = CONFIG_XIP_PHYS_ADDR \ 143 - + (uintptr_t)&_end - (uintptr_t)&_start; \ 144 - uintptr_t __a = (uintptr_t)(addr); \ 145 - (__a >= __rom_start_data && __a < __rom_end_data) ? \ 146 - __a - __rom_start_data + CONFIG_PHYS_RAM_BASE : __a; \ 147 - }) 148 - #else 149 - #define XIP_FIXUP(addr) (addr) 150 - #endif /* CONFIG_XIP_KERNEL */ 151 - 152 137 struct pt_alloc_ops { 153 138 pte_t *(*get_pte_virt)(phys_addr_t pa); 154 139 phys_addr_t (*alloc_pte)(uintptr_t va); ··· 1257 1272 extern char _start[]; 1258 1273 extern void *_dtb_early_va; 1259 1274 extern uintptr_t _dtb_early_pa; 1260 - #if defined(CONFIG_XIP_KERNEL) && defined(CONFIG_MMU) 1261 - #define dtb_early_va (*(void **)XIP_FIXUP(&_dtb_early_va)) 1262 - #define dtb_early_pa (*(uintptr_t *)XIP_FIXUP(&_dtb_early_pa)) 1263 - #else 1264 1275 #define dtb_early_va _dtb_early_va 1265 1276 #define dtb_early_pa _dtb_early_pa 1266 - #endif /* CONFIG_XIP_KERNEL */ 1267 1277 extern u64 satp_mode; 1268 1278 1269 1279 void paging_init(void);
+1 -1
arch/riscv/include/asm/processor.h
··· 86 86 * preempt_v. All preempt_v context should be dropped in such case because 87 87 * V-regs are caller-saved. Only sstatus.VS=ON is persisted across a 88 88 * schedule() call. 89 - * - bit 30: The in-kernel preempt_v context is saved, and requries to be 89 + * - bit 30: The in-kernel preempt_v context is saved, and is required to be 90 90 * restored when returning to the context that owns the preempt_v. 91 91 * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the 92 92 * trap entry code. Any context switches out-of current task need to save
-1
arch/riscv/include/asm/scs.h
··· 10 10 /* Load init_shadow_call_stack to gp. */ 11 11 .macro scs_load_init_stack 12 12 la gp, init_shadow_call_stack 13 - XIP_FIXUP_OFFSET gp 14 13 .endm 15 14 16 15 /* Load the per-CPU IRQ shadow call stack to gp. */
+1 -1
arch/riscv/include/asm/set_memory.h
··· 47 47 48 48 #endif /* __ASSEMBLER__ */ 49 49 50 - #if defined(CONFIG_STRICT_KERNEL_RWX) || defined(CONFIG_XIP_KERNEL) 50 + #if defined(CONFIG_STRICT_KERNEL_RWX) 51 51 #ifdef CONFIG_64BIT 52 52 #define SECTION_ALIGN (1 << 21) 53 53 #else
+1 -1
arch/riscv/include/asm/smp.h
··· 105 105 106 106 #endif /* CONFIG_SMP */ 107 107 108 - #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) 108 + #if defined(CONFIG_HOTPLUG_CPU) 109 109 bool cpu_has_hotplug(unsigned int cpu); 110 110 #else 111 111 static inline bool cpu_has_hotplug(unsigned int cpu)
+9
arch/riscv/include/asm/string.h
··· 28 28 29 29 #define __HAVE_ARCH_STRNCMP 30 30 extern asmlinkage int strncmp(const char *cs, const char *ct, size_t count); 31 + 32 + #define __HAVE_ARCH_STRNLEN 33 + extern asmlinkage __kernel_size_t strnlen(const char *, size_t); 34 + 35 + #define __HAVE_ARCH_STRCHR 36 + extern asmlinkage char *strchr(const char *, int); 37 + 38 + #define __HAVE_ARCH_STRRCHR 39 + extern asmlinkage char *strrchr(const char *, int); 31 40 #endif 32 41 33 42 /* For those files which don't want to check by kasan. */
+1 -1
arch/riscv/include/asm/thread_info.h
··· 120 120 #include <asm-generic/thread_info_tif.h> 121 121 122 122 #define TIF_32BIT 16 /* compat-mode 32bit process */ 123 - #define TIF_RISCV_V_DEFER_RESTORE 17 /* restore Vector before returing to user */ 123 + #define TIF_RISCV_V_DEFER_RESTORE 17 /* restore Vector before returning to user */ 124 124 125 125 #define _TIF_RISCV_V_DEFER_RESTORE BIT(TIF_RISCV_V_DEFER_RESTORE) 126 126
-49
arch/riscv/include/asm/xip_fixup.h
··· 1 - /* SPDX-License-Identifier: GPL-2.0-only */ 2 - /* 3 - * XIP fixup macros, only useful in assembly. 4 - */ 5 - #ifndef _ASM_RISCV_XIP_FIXUP_H 6 - #define _ASM_RISCV_XIP_FIXUP_H 7 - 8 - #include <linux/pgtable.h> 9 - 10 - #ifdef CONFIG_XIP_KERNEL 11 - .macro XIP_FIXUP_OFFSET reg 12 - /* Fix-up address in Flash into address in RAM early during boot before 13 - * MMU is up. Because generated code "thinks" data is in Flash, but it 14 - * is actually in RAM (actually data is also in Flash, but Flash is 15 - * read-only, thus we need to use the data residing in RAM). 16 - * 17 - * The start of data in Flash is _sdata and the start of data in RAM is 18 - * CONFIG_PHYS_RAM_BASE. So this fix-up essentially does this: 19 - * reg += CONFIG_PHYS_RAM_BASE - _start 20 - */ 21 - li t0, CONFIG_PHYS_RAM_BASE 22 - add \reg, \reg, t0 23 - la t0, _sdata 24 - sub \reg, \reg, t0 25 - .endm 26 - .macro XIP_FIXUP_FLASH_OFFSET reg 27 - /* In linker script, at the transition from read-only section to 28 - * writable section, the VMA is increased while LMA remains the same. 29 - * (See in linker script how _sdata, __data_loc and LOAD_OFFSET is 30 - * changed) 31 - * 32 - * Consequently, early during boot before MMU is up, the generated code 33 - * reads the "writable" section at wrong addresses, because VMA is used 34 - * by compiler to generate code, but the data is located in Flash using 35 - * LMA. 36 - */ 37 - la t0, _sdata 38 - sub \reg, \reg, t0 39 - la t0, __data_loc 40 - add \reg, \reg, t0 41 - .endm 42 - #else 43 - .macro XIP_FIXUP_OFFSET reg 44 - .endm 45 - .macro XIP_FIXUP_FLASH_OFFSET reg 46 - .endm 47 - #endif /* CONFIG_XIP_KERNEL */ 48 - 49 - #endif
+1 -1
arch/riscv/include/uapi/asm/setup.h
··· 3 3 #ifndef _UAPI_ASM_RISCV_SETUP_H 4 4 #define _UAPI_ASM_RISCV_SETUP_H 5 5 6 - #define COMMAND_LINE_SIZE 1024 6 + #define COMMAND_LINE_SIZE 2048 7 7 8 8 #endif /* _UAPI_ASM_RISCV_SETUP_H */
+6 -6
arch/riscv/kernel/acpi.c
··· 69 69 70 70 /* 71 71 * FADT is required on riscv; retrieve it to check its presence 72 - * and carry out revision and ACPI HW reduced compliancy tests 72 + * and carry out revision and ACPI HW reduced compliance tests 73 73 */ 74 74 status = acpi_get_table(ACPI_SIG_FADT, 0, &table); 75 75 if (ACPI_FAILURE(status)) { ··· 85 85 * The revision in the table header is the FADT's Major revision. The 86 86 * FADT also has a minor revision, which is stored in the FADT itself. 87 87 * 88 - * TODO: Currently, we check for 6.5 as the minimum version to check 89 - * for HW_REDUCED flag. However, once RISC-V updates are released in 90 - * the ACPI spec, we need to update this check for exact minor revision 88 + * ACPI 6.6 is required for RISC-V as it introduces RISC-V specific 89 + * tables such as RHCT (RISC-V Hart Capabilities Table) and RIMT 90 + * (RISC-V I/O Mapping Table). 91 91 */ 92 - if (table->revision < 6 || (table->revision == 6 && fadt->minor_revision < 5)) 93 - pr_err(FW_BUG "Unsupported FADT revision %d.%d, should be 6.5+\n", 92 + if (table->revision < 6 || (table->revision == 6 && fadt->minor_revision < 6)) 93 + pr_err(FW_BUG "Unsupported FADT revision %d.%d, should be 6.6+\n", 94 94 table->revision, fadt->minor_revision); 95 95 96 96 if (!(fadt->flags & ACPI_FADT_HW_REDUCED)) {
-2
arch/riscv/kernel/cpu-hotplug.c
··· 43 43 return 0; 44 44 } 45 45 46 - #ifdef CONFIG_HOTPLUG_CPU 47 46 /* 48 47 * Called on the thread which is asking for a CPU to be shutdown, if the 49 48 * CPU reported dead to the hotplug core. ··· 74 75 /* It should never reach here */ 75 76 BUG(); 76 77 } 77 - #endif
+1
arch/riscv/kernel/entry.S
··· 498 498 RISCV_PTR do_trap_unknown /* cause=16 */ 499 499 RISCV_PTR do_trap_unknown /* cause=17 */ 500 500 RISCV_PTR do_trap_software_check /* cause=18 is sw check exception */ 501 + RISCV_PTR do_trap_hardware_error /* hardware error (19) */ 501 502 SYM_DATA_END_LABEL(excp_vect_table, SYM_L_LOCAL, excp_vect_table_end) 502 503 503 504 #ifndef CONFIG_MMU
+1 -1
arch/riscv/kernel/ftrace.c
··· 148 148 149 149 /* 150 150 * This is called early on, and isn't wrapped by 151 - * ftrace_arch_code_modify_{prepare,post_process}() and therefor doesn't hold 151 + * ftrace_arch_code_modify_{prepare,post_process}() and therefore doesn't hold 152 152 * text_mutex, which triggers a lockdep failure. SMP isn't running so we could 153 153 * just directly poke the text, but it's simpler to just take the lock 154 154 * ourselves.
+1 -42
arch/riscv/kernel/head.S
··· 14 14 #include <asm/hwcap.h> 15 15 #include <asm/image.h> 16 16 #include <asm/scs.h> 17 - #include <asm/xip_fixup.h> 18 17 #include <asm/usercfi.h> 19 18 #include "efi-header.S" 20 19 ··· 74 75 relocate_enable_mmu: 75 76 /* Relocate return address */ 76 77 la a1, kernel_map 77 - XIP_FIXUP_OFFSET a1 78 78 REG_L a1, KERNEL_MAP_VIRT_ADDR(a1) 79 79 la a2, _start 80 80 sub a1, a1, a2 81 81 add ra, ra, a1 82 82 83 - /* Point stvec to virtual address of intruction after satp write */ 83 + /* Point stvec to virtual address of instruction after satp write */ 84 84 la a2, 1f 85 85 add a2, a2, a1 86 86 csrw CSR_TVEC, a2 ··· 87 89 /* Compute satp for kernel page tables, but don't load it yet */ 88 90 srl a2, a0, PAGE_SHIFT 89 91 la a1, satp_mode 90 - XIP_FIXUP_OFFSET a1 91 92 REG_L a1, 0(a1) 92 93 or a2, a2, a1 93 94 ··· 97 100 * to ensure the new translations are in use. 98 101 */ 99 102 la a0, trampoline_pg_dir 100 - XIP_FIXUP_OFFSET a0 101 103 srl a0, a0, PAGE_SHIFT 102 104 or a0, a0, a1 103 105 sfence.vma ··· 150 154 151 155 /* a0 contains the hartid & a1 contains boot data */ 152 156 li a2, SBI_HART_BOOT_TASK_PTR_OFFSET 153 - XIP_FIXUP_OFFSET a2 154 157 add a2, a2, a1 155 158 REG_L tp, (a2) 156 159 li a3, SBI_HART_BOOT_STACK_PTR_OFFSET 157 - XIP_FIXUP_OFFSET a3 158 160 add a3, a3, a1 159 161 REG_L sp, (a3) 160 162 ··· 161 167 #ifdef CONFIG_MMU 162 168 /* Enable virtual memory and relocate to virtual address */ 163 169 la a0, swapper_pg_dir 164 - XIP_FIXUP_OFFSET a0 165 170 call relocate_enable_mmu 166 171 #endif 167 172 call .Lsetup_trap_vector ··· 262 269 .Lgood_cores: 263 270 264 271 /* The lottery system is only required for spinwait booting method */ 265 - #ifndef CONFIG_XIP_KERNEL 266 272 /* Pick one hart to run the main boot sequence */ 267 273 la a3, hart_lottery 268 274 li a2, 1 269 275 amoadd.w a3, a2, (a3) 270 276 bnez a3, .Lsecondary_start 271 - 272 - #else 273 - /* hart_lottery in flash contains a magic number */ 274 - la a3, hart_lottery 275 - mv a2, a3 276 - XIP_FIXUP_OFFSET a2 277 - XIP_FIXUP_FLASH_OFFSET a3 278 - lw t1, (a3) 279 - amoswap.w t0, t1, (a2) 280 - /* first time here if hart_lottery in RAM is not set */ 281 - beq t0, t1, .Lsecondary_start 282 - 283 - #endif /* CONFIG_XIP */ 284 277 #endif /* CONFIG_RISCV_BOOT_SPINWAIT */ 285 278 286 - #ifdef CONFIG_XIP_KERNEL 287 - la sp, _end + THREAD_SIZE 288 - XIP_FIXUP_OFFSET sp 289 - mv s0, a0 290 - mv s1, a1 291 - call __copy_data 292 - 293 - /* Restore a0 & a1 copy */ 294 - mv a0, s0 295 - mv a1, s1 296 - #endif 297 - 298 - #ifndef CONFIG_XIP_KERNEL 299 279 /* Clear BSS for flat non-ELF images */ 300 280 la a3, __bss_start 301 281 la a4, __bss_stop ··· 278 312 add a3, a3, RISCV_SZPTR 279 313 blt a3, a4, .Lclear_bss 280 314 .Lclear_bss_done: 281 - #endif 282 315 la a2, boot_cpu_hartid 283 - XIP_FIXUP_OFFSET a2 284 316 REG_S a0, (a2) 285 317 286 318 /* Initialize page tables and relocate to virtual addresses */ 287 319 la tp, init_task 288 320 la sp, init_thread_union + THREAD_SIZE 289 - XIP_FIXUP_OFFSET sp 290 321 addi sp, sp, -PT_SIZE_ON_STACK 291 322 scs_load_init_stack 292 323 #ifdef CONFIG_BUILTIN_DTB 293 324 la a0, __dtb_start 294 - XIP_FIXUP_OFFSET a0 295 325 #else 296 326 mv a0, a1 297 327 #endif /* CONFIG_BUILTIN_DTB */ ··· 297 335 call setup_vm 298 336 #ifdef CONFIG_MMU 299 337 la a0, early_pg_dir 300 - XIP_FIXUP_OFFSET a0 301 338 call relocate_enable_mmu 302 339 #endif /* CONFIG_MMU */ 303 340 ··· 335 374 336 375 slli a3, a0, LGREG 337 376 la a1, __cpu_spinwait_stack_pointer 338 - XIP_FIXUP_OFFSET a1 339 377 la a2, __cpu_spinwait_task_pointer 340 - XIP_FIXUP_OFFSET a2 341 378 add a1, a3, a1 342 379 add a2, a3, a2 343 380
-3
arch/riscv/kernel/head.h
··· 11 11 extern atomic_t hart_lottery; 12 12 13 13 asmlinkage void __init setup_vm(uintptr_t dtb_pa); 14 - #ifdef CONFIG_XIP_KERNEL 15 - asmlinkage void __init __copy_data(void); 16 - #endif 17 14 18 15 #ifdef CONFIG_RISCV_BOOT_SPINWAIT 19 16 extern void *__cpu_spinwait_stack_pointer[];
+1 -1
arch/riscv/kernel/mcount-dyn.S
··· 66 66 * 8(sp) stores the function return address (i.e. parent IP) that 67 67 * can be accessed by &(fregs)->ra in tracing function. 68 68 * 69 - * The other regs are saved at the respective localtion and accessed 69 + * The other regs are saved at the respective location and accessed 70 70 * by the respective ftrace_regs member. 71 71 * 72 72 * Here is the layout of stack for your reference.
+1 -1
arch/riscv/kernel/module-sections.c
··· 148 148 return -ENOEXEC; 149 149 } 150 150 151 - /* Calculate the maxinum number of entries */ 151 + /* Calculate the maximum number of entries */ 152 152 for (i = 0; i < ehdr->e_shnum; i++) { 153 153 size_t num_relas = sechdrs[i].sh_size / sizeof(Elf_Rela); 154 154 Elf_Rela *relas = (void *)ehdr + sechdrs[i].sh_offset;
+1 -1
arch/riscv/kernel/probes/kprobes.c
··· 149 149 150 150 /* 151 151 * Interrupts need to be disabled before single-step mode is set, and not 152 - * reenabled until after single-step mode ends. 152 + * re-enabled until after single-step mode ends. 153 153 * Without disabling interrupt on local CPU, there is a chance of 154 154 * interrupt occurrence in the period of exception return and start of 155 155 * out-of-line single-step, that result in wrongly single stepping
+1 -1
arch/riscv/kernel/probes/uprobes.c
··· 111 111 112 112 current->thread.bad_cause = utask->autask.saved_cause; 113 113 /* 114 - * Task has received a fatal signal, so reset back to probbed 114 + * Task has received a fatal signal, so reset back to probed 115 115 * address. 116 116 */ 117 117 instruction_pointer_set(regs, utask->vaddr);
+1 -5
arch/riscv/kernel/setup.c
··· 46 46 * This is used before the kernel initializes the BSS so it can't be in the 47 47 * BSS. 48 48 */ 49 - atomic_t hart_lottery __section(".sdata") 50 - #ifdef CONFIG_XIP_KERNEL 51 - = ATOMIC_INIT(0xC001BEEF) 52 - #endif 53 - ; 49 + atomic_t hart_lottery __section(".sdata"); 54 50 unsigned long boot_cpu_hartid; 55 51 EXPORT_SYMBOL_GPL(boot_cpu_hartid); 56 52
+2 -6
arch/riscv/kernel/smpboot.c
··· 251 251 set_cpu_online(curr_cpuid, true); 252 252 253 253 /* 254 - * Remote cache and TLB flushes are ignored while the CPU is offline, 255 - * so flush them both right now just in case. 254 + * Remote instruction cache and TLB flushes are ignored while the CPU 255 + * is offline, so flush them both right now just in case. 256 256 */ 257 257 local_flush_icache_all(); 258 258 local_flush_tlb_all(); 259 259 #ifndef CONFIG_HOTPLUG_PARALLEL 260 260 complete(&cpu_running); 261 261 #endif 262 - /* 263 - * Disable preemption before enabling interrupts, so we don't try to 264 - * schedule a CPU that hasn't actually started yet. 265 - */ 266 262 local_irq_enable(); 267 263 cpu_startup_entry(CPUHP_AP_ONLINE_IDLE); 268 264 }
+1 -1
arch/riscv/kernel/soc.c
··· 8 8 #include <asm/soc.h> 9 9 10 10 /* 11 - * This is called extremly early, before parse_dtb(), to allow initializing 11 + * This is called extremely early, before parse_dtb(), to allow initializing 12 12 * SoC hardware before memory or any device driver initialization. 13 13 */ 14 14 void __init soc_early_init(void)
+1 -1
arch/riscv/kernel/suspend.c
··· 78 78 suspend_save_csrs(&context); 79 79 80 80 /* 81 - * Function graph tracer state gets incosistent when the kernel 81 + * Function graph tracer state gets inconsistent when the kernel 82 82 * calls functions that never return (aka finishers) hence disable 83 83 * graph tracing during their execution. 84 84 */
-2
arch/riscv/kernel/suspend_entry.S
··· 10 10 #include <asm/asm-offsets.h> 11 11 #include <asm/assembler.h> 12 12 #include <asm/csr.h> 13 - #include <asm/xip_fixup.h> 14 13 15 14 .text 16 15 .altmacro ··· 69 70 70 71 /* Enable MMU */ 71 72 la a0, swapper_pg_dir 72 - XIP_FIXUP_OFFSET a0 73 73 call relocate_enable_mmu 74 74 75 75 /* Restore A0 and A1 */
+2 -4
arch/riscv/kernel/traps.c
··· 142 142 } 143 143 } 144 144 145 - #if defined(CONFIG_XIP_KERNEL) && defined(CONFIG_RISCV_ALTERNATIVE) 146 - #define __trap_section __noinstr_section(".xip.traps") 147 - #else 148 145 #define __trap_section noinstr 149 - #endif 150 146 #define DO_ERROR_INFO(name, signo, code, str) \ 151 147 asmlinkage __visible __trap_section void name(struct pt_regs *regs) \ 152 148 { \ ··· 161 165 162 166 DO_ERROR_INFO(do_trap_unknown, 163 167 SIGILL, ILL_ILLTRP, "unknown exception"); 168 + DO_ERROR_INFO(do_trap_hardware_error, 169 + SIGBUS, BUS_MCEERR_AR, "hardware error"); 164 170 DO_ERROR_INFO(do_trap_insn_misaligned, 165 171 SIGBUS, BUS_ADRALN, "instruction address misaligned"); 166 172 DO_ERROR_INFO(do_trap_insn_fault,
+86 -137
arch/riscv/kernel/unaligned_access_speed.c
··· 16 16 17 17 #include "copy-unaligned.h" 18 18 19 - #define MISALIGNED_ACCESS_JIFFIES_LG2 1 19 + #define MISALIGNED_ACCESS_NS 8000000 20 20 #define MISALIGNED_BUFFER_SIZE 0x4000 21 21 #define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) 22 22 #define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) ··· 29 29 30 30 static cpumask_t fast_misaligned_access; 31 31 32 - #ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS 33 - static int check_unaligned_access(void *param) 32 + static u64 __maybe_unused 33 + measure_cycles(void (*func)(void *dst, const void *src, size_t len), 34 + void *dst, void *src, size_t len) 34 35 { 35 - int cpu = smp_processor_id(); 36 - u64 start_cycles, end_cycles; 37 - u64 word_cycles; 38 - u64 byte_cycles; 39 - int ratio; 40 - unsigned long start_jiffies, now; 41 - struct page *page = param; 42 - void *dst; 43 - void *src; 44 - long speed = RISCV_HWPROBE_MISALIGNED_SCALAR_SLOW; 36 + u64 start_cycles, end_cycles, cycles = -1ULL; 37 + u64 start_ns; 45 38 46 - if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN) 47 - return 0; 48 - 49 - /* Make an unaligned destination buffer. */ 50 - dst = (void *)((unsigned long)page_address(page) | 0x1); 51 - /* Unalign src as well, but differently (off by 1 + 2 = 3). */ 52 - src = dst + (MISALIGNED_BUFFER_SIZE / 2); 53 - src += 2; 54 - word_cycles = -1ULL; 55 39 /* Do a warmup. */ 56 - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 40 + func(dst, src, len); 41 + 57 42 preempt_disable(); 58 - start_jiffies = jiffies; 59 - while ((now = jiffies) == start_jiffies) 60 - cpu_relax(); 61 43 62 44 /* 63 45 * For a fixed amount of time, repeatedly try the function, and take 64 46 * the best time in cycles as the measurement. 65 47 */ 66 - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 48 + start_ns = ktime_get_mono_fast_ns(); 49 + while (ktime_get_mono_fast_ns() < start_ns + MISALIGNED_ACCESS_NS) { 67 50 start_cycles = get_cycles64(); 68 51 /* Ensure the CSR read can't reorder WRT to the copy. */ 69 52 mb(); 70 - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 53 + func(dst, src, len); 71 54 /* Ensure the copy ends before the end time is snapped. */ 72 55 mb(); 73 56 end_cycles = get_cycles64(); 74 - if ((end_cycles - start_cycles) < word_cycles) 75 - word_cycles = end_cycles - start_cycles; 76 - } 77 - 78 - byte_cycles = -1ULL; 79 - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 80 - start_jiffies = jiffies; 81 - while ((now = jiffies) == start_jiffies) 82 - cpu_relax(); 83 - 84 - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 85 - start_cycles = get_cycles64(); 86 - mb(); 87 - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 88 - mb(); 89 - end_cycles = get_cycles64(); 90 - if ((end_cycles - start_cycles) < byte_cycles) 91 - byte_cycles = end_cycles - start_cycles; 57 + if ((end_cycles - start_cycles) < cycles) 58 + cycles = end_cycles - start_cycles; 92 59 } 93 60 94 61 preempt_enable(); 95 62 63 + return cycles; 64 + } 65 + 66 + /* 67 + * Return: 68 + * 1 if unaligned accesses are fast 69 + * 0 if unaligned accesses are slow 70 + * -1 if check cannot be done 71 + */ 72 + static int __maybe_unused 73 + compare_unaligned_access(void (*word_copy)(void *dst, const void *src, size_t len), 74 + void (*byte_copy)(void *dst, const void *src, size_t len), 75 + void *buf, const char *type) 76 + { 77 + int cpu = smp_processor_id(); 78 + u64 word_cycles; 79 + u64 byte_cycles; 80 + void *dst, *src; 81 + bool fast; 82 + int ratio; 83 + 84 + /* Make an unaligned destination buffer. */ 85 + dst = (void *)((unsigned long)buf | 0x1); 86 + /* Unalign src as well, but differently (off by 1 + 2 = 3). */ 87 + src = dst + (MISALIGNED_BUFFER_SIZE / 2); 88 + src += 2; 89 + 90 + word_cycles = measure_cycles(word_copy, dst, src, MISALIGNED_COPY_SIZE); 91 + byte_cycles = measure_cycles(byte_copy, dst, src, MISALIGNED_COPY_SIZE); 92 + 96 93 /* Don't divide by zero. */ 97 94 if (!word_cycles || !byte_cycles) { 98 - pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", 99 - cpu); 95 + pr_warn("cpu%d: rdtime lacks granularity needed to measure %s unaligned access speed\n", 96 + cpu, type); 100 97 101 - return 0; 98 + return -1; 102 99 } 103 100 104 - if (word_cycles < byte_cycles) 105 - speed = RISCV_HWPROBE_MISALIGNED_SCALAR_FAST; 101 + fast = word_cycles < byte_cycles; 106 102 107 103 ratio = div_u64((byte_cycles * 100), word_cycles); 108 - pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", 104 + pr_info("cpu%d: %s unaligned word access speed is %d.%02dx byte access speed (%s)\n", 109 105 cpu, 106 + type, 110 107 ratio / 100, 111 108 ratio % 100, 112 - (speed == RISCV_HWPROBE_MISALIGNED_SCALAR_FAST) ? "fast" : "slow"); 109 + fast ? "fast" : "slow"); 113 110 114 - per_cpu(misaligned_access_speed, cpu) = speed; 111 + return fast; 112 + } 113 + 114 + #ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS 115 + static int check_unaligned_access(struct page *page) 116 + { 117 + void *buf = page_address(page); 118 + int cpu = smp_processor_id(); 119 + int ret; 120 + 121 + if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN) 122 + return 0; 123 + 124 + ret = compare_unaligned_access(__riscv_copy_words_unaligned, 125 + __riscv_copy_bytes_unaligned, 126 + buf, "scalar"); 127 + if (ret < 0) 128 + return 0; 115 129 116 130 /* 117 131 * Set the value of fast_misaligned_access of a CPU. These operations 118 132 * are atomic to avoid race conditions. 119 133 */ 120 - if (speed == RISCV_HWPROBE_MISALIGNED_SCALAR_FAST) 134 + if (ret) { 135 + per_cpu(misaligned_access_speed, cpu) = RISCV_HWPROBE_MISALIGNED_SCALAR_FAST; 121 136 cpumask_set_cpu(cpu, &fast_misaligned_access); 122 - else 137 + } else { 138 + per_cpu(misaligned_access_speed, cpu) = RISCV_HWPROBE_MISALIGNED_SCALAR_SLOW; 123 139 cpumask_clear_cpu(cpu, &fast_misaligned_access); 140 + } 124 141 125 142 return 0; 126 143 } 127 144 128 - static void __init check_unaligned_access_nonboot_cpu(void *param) 145 + static void __init _check_unaligned_access(void *param) 129 146 { 130 147 unsigned int cpu = smp_processor_id(); 131 148 struct page **pages = param; 132 149 133 - if (smp_processor_id() != 0) 134 - check_unaligned_access(pages[cpu]); 150 + check_unaligned_access(pages[cpu]); 135 151 } 136 152 137 153 /* Measure unaligned access speed on all CPUs present at boot in parallel. */ ··· 174 158 } 175 159 } 176 160 177 - /* Check everybody except 0, who stays behind to tend jiffies. */ 178 - on_each_cpu(check_unaligned_access_nonboot_cpu, bufs, 1); 179 - 180 - /* Check core 0. */ 181 - smp_call_on_cpu(0, check_unaligned_access, bufs[0], true); 161 + on_each_cpu(_check_unaligned_access, bufs, 1); 182 162 183 163 out: 184 164 for_each_cpu(cpu, cpu_online_mask) { ··· 293 281 static void check_vector_unaligned_access(struct work_struct *work __always_unused) 294 282 { 295 283 int cpu = smp_processor_id(); 296 - u64 start_cycles, end_cycles; 297 - u64 word_cycles; 298 - u64 byte_cycles; 299 - int ratio; 300 - unsigned long start_jiffies, now; 301 284 struct page *page; 302 - void *dst; 303 - void *src; 304 - long speed = RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; 285 + int ret; 305 286 306 287 if (per_cpu(vector_misaligned_access, cpu) != RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN) 307 288 return; ··· 305 300 return; 306 301 } 307 302 308 - /* Make an unaligned destination buffer. */ 309 - dst = (void *)((unsigned long)page_address(page) | 0x1); 310 - /* Unalign src as well, but differently (off by 1 + 2 = 3). */ 311 - src = dst + (MISALIGNED_BUFFER_SIZE / 2); 312 - src += 2; 313 - word_cycles = -1ULL; 314 - 315 - /* Do a warmup. */ 316 303 kernel_vector_begin(); 317 - __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 318 304 319 - start_jiffies = jiffies; 320 - while ((now = jiffies) == start_jiffies) 321 - cpu_relax(); 322 - 323 - /* 324 - * For a fixed amount of time, repeatedly try the function, and take 325 - * the best time in cycles as the measurement. 326 - */ 327 - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 328 - start_cycles = get_cycles64(); 329 - /* Ensure the CSR read can't reorder WRT to the copy. */ 330 - mb(); 331 - __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); 332 - /* Ensure the copy ends before the end time is snapped. */ 333 - mb(); 334 - end_cycles = get_cycles64(); 335 - if ((end_cycles - start_cycles) < word_cycles) 336 - word_cycles = end_cycles - start_cycles; 337 - } 338 - 339 - byte_cycles = -1ULL; 340 - __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 341 - start_jiffies = jiffies; 342 - while ((now = jiffies) == start_jiffies) 343 - cpu_relax(); 344 - 345 - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { 346 - start_cycles = get_cycles64(); 347 - /* Ensure the CSR read can't reorder WRT to the copy. */ 348 - mb(); 349 - __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); 350 - /* Ensure the copy ends before the end time is snapped. */ 351 - mb(); 352 - end_cycles = get_cycles64(); 353 - if ((end_cycles - start_cycles) < byte_cycles) 354 - byte_cycles = end_cycles - start_cycles; 355 - } 356 - 305 + ret = compare_unaligned_access(__riscv_copy_vec_words_unaligned, 306 + __riscv_copy_vec_bytes_unaligned, 307 + page_address(page), "vector"); 357 308 kernel_vector_end(); 358 309 359 - /* Don't divide by zero. */ 360 - if (!word_cycles || !byte_cycles) { 361 - pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned vector access speed\n", 362 - cpu); 363 - 310 + if (ret < 0) 364 311 goto free; 365 - } 366 312 367 - if (word_cycles < byte_cycles) 368 - speed = RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; 369 - 370 - ratio = div_u64((byte_cycles * 100), word_cycles); 371 - pr_info("cpu%d: Ratio of vector byte access time to vector unaligned word access is %d.%02d, unaligned accesses are %s\n", 372 - cpu, 373 - ratio / 100, 374 - ratio % 100, 375 - (speed == RISCV_HWPROBE_MISALIGNED_VECTOR_FAST) ? "fast" : "slow"); 376 - 377 - per_cpu(vector_misaligned_access, cpu) = speed; 313 + if (ret) 314 + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; 315 + else 316 + per_cpu(vector_misaligned_access, cpu) = RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; 378 317 379 318 free: 380 319 __free_pages(page, MISALIGNED_BUFFER_ORDER); ··· 443 494 return 0; 444 495 } 445 496 446 - arch_initcall(check_unaligned_access_all_cpus); 497 + late_initcall(check_unaligned_access_all_cpus);
+8
arch/riscv/kernel/vdso_cfi/.gitignore
··· 1 + # SPDX-License-Identifier: GPL-2.0-only 2 + # Copied source files from the main vdso directory 3 + *.c 4 + *.S 5 + !vdso-cfi.S 6 + vdso.lds 7 + *.tmp 8 + vdso-syms.S
+3
arch/riscv/kernel/vdso_cfi/Makefile
··· 23 23 # Include the main VDSO Makefile which contains all the build rules and sources 24 24 # The VDSO_CFI_BUILD variable will be passed to it to enable CFI compilation 25 25 include $(src)/Makefile 26 + 27 + # Clean rules - remove the copied source files 28 + clean-files += $(notdir $(vdso_c_sources)) $(notdir $(vdso_S_sources))
+7 -7
arch/riscv/kernel/vmcore_info.c
··· 3 3 #include <linux/vmcore_info.h> 4 4 #include <linux/pagemap.h> 5 5 6 + static inline u64 get_satp_value(void) 7 + { 8 + return csr_read(CSR_SATP); 9 + } 10 + 6 11 void arch_crash_save_vmcoreinfo(void) 7 12 { 8 13 VMCOREINFO_NUMBER(phys_ram_base); ··· 24 19 #endif 25 20 #endif 26 21 vmcoreinfo_append_str("NUMBER(KERNEL_LINK_ADDR)=0x%lx\n", KERNEL_LINK_ADDR); 27 - #ifdef CONFIG_XIP_KERNEL 28 - /* TODO: Communicate with crash-utility developers on the information to 29 - * export. The XIP case is more complicated, because the virtual-physical 30 - * address offset depends on whether the address is in ROM or in RAM. 31 - */ 32 - #else 33 22 vmcoreinfo_append_str("NUMBER(va_kernel_pa_offset)=0x%lx\n", 34 23 kernel_map.va_kernel_pa_offset); 35 - #endif 24 + vmcoreinfo_append_str("KERNELOFFSET=%lx\n", kaslr_offset()); 25 + vmcoreinfo_append_str("NUMBER(satp)=0x%llx\n", get_satp_value()); 36 26 }
-5
arch/riscv/kernel/vmlinux.lds.S
··· 7 7 #define RO_EXCEPTION_TABLE_ALIGN 4 8 8 #define RUNTIME_DISCARD_EXIT 9 9 10 - #ifdef CONFIG_XIP_KERNEL 11 - #include "vmlinux-xip.lds.S" 12 - #else 13 - 14 10 #include <asm/pgtable.h> 15 11 #define LOAD_OFFSET KERNEL_LINK_ADDR 16 12 ··· 172 176 173 177 DISCARDS 174 178 } 175 - #endif /* CONFIG_XIP_KERNEL */
+1 -1
arch/riscv/kvm/tlb.c
··· 182 182 183 183 /* 184 184 * Flush VS-stage TLB entries for implementation where VS-stage 185 - * TLB does not cahce guest physical address and VMID. 185 + * TLB does not cache guest physical address and VMID. 186 186 */ 187 187 if (static_branch_unlikely(&kvm_riscv_vsstage_tlb_no_gpa)) 188 188 kvm_riscv_local_hfence_vvma_all(vmid);
+1 -1
arch/riscv/kvm/vcpu_pmu.c
··· 694 694 pmc->counter_val += perf_event_read_value(pmc->perf_event, 695 695 &enabled, &running); 696 696 /* 697 - * The counter and overflow indicies in the snapshot region are w.r.to 697 + * The counter and overflow indices in the snapshot region are w.r.to 698 698 * cbase. Modify the set bit in the counter mask instead of the pmc_index 699 699 * which indicates the absolute counter index. 700 700 */
+3
arch/riscv/lib/Makefile
··· 7 7 lib-y += strcmp.o 8 8 lib-y += strlen.o 9 9 lib-y += strncmp.o 10 + lib-y += strnlen.o 11 + lib-y += strchr.o 12 + lib-y += strrchr.o 10 13 endif 11 14 lib-y += csum.o 12 15 ifeq ($(CONFIG_MMU), y)
+1 -1
arch/riscv/lib/csum.c
··· 269 269 * on machines with fast misaligned accesses. 270 270 * 271 271 * There is some duplicate code between the "with_alignment" and 272 - * "no_alignment" implmentations, but the overlap is too awkward to be 272 + * "no_alignment" implementations, but the overlap is too awkward to be 273 273 * able to fit in one function without introducing multiple static 274 274 * branches. The largest chunk of overlap was delegated into the 275 275 * do_csum_common function.
+2 -2
arch/riscv/lib/memmove.S
··· 40 40 * Both Copy Modes: t1 - Temporary for load-store 41 41 * Both Copy Modes: t2 - Temporary for load-store 42 42 * Both Copy Modes: a5 - dest to src alignment offset 43 - * Both Copy Modes: a6 - Shift ammount 44 - * Both Copy Modes: a7 - Inverse Shift ammount 43 + * Both Copy Modes: a6 - Shift amount 44 + * Both Copy Modes: a7 - Inverse Shift amount 45 45 * Both Copy Modes: a2 - Alternate breakpoint for unrolled loops 46 46 */ 47 47
+35
arch/riscv/lib/strchr.S
··· 1 + /* SPDX-License-Identifier: GPL-2.0-only */ 2 + 3 + /* 4 + * Copyright (C) 2025 Feng Jiang <jiangfeng@kylinos.cn> 5 + */ 6 + 7 + #include <linux/linkage.h> 8 + #include <asm/asm.h> 9 + 10 + /* char *strchr(const char *s, int c) */ 11 + SYM_FUNC_START(strchr) 12 + /* 13 + * Parameters 14 + * a0 - The string to be searched 15 + * a1 - The character to search for 16 + * 17 + * Returns 18 + * a0 - Address of first occurrence of 'c' or 0 19 + * 20 + * Clobbers 21 + * t0 22 + */ 23 + andi a1, a1, 0xff 24 + 1: 25 + lbu t0, 0(a0) 26 + beq t0, a1, 2f 27 + addi a0, a0, 1 28 + bnez t0, 1b 29 + li a0, 0 30 + 2: 31 + ret 32 + SYM_FUNC_END(strchr) 33 + 34 + SYM_FUNC_ALIAS_WEAK(__pi_strchr, strchr) 35 + EXPORT_SYMBOL(strchr)
+164
arch/riscv/lib/strnlen.S
··· 1 + /* SPDX-License-Identifier: GPL-2.0-only */ 2 + 3 + /* 4 + * Base on arch/riscv/lib/strlen.S 5 + * 6 + * Copyright (C) Feng Jiang <jiangfeng@kylinos.cn> 7 + */ 8 + 9 + #include <linux/linkage.h> 10 + #include <asm/asm.h> 11 + #include <asm/alternative-macros.h> 12 + #include <asm/hwcap.h> 13 + 14 + /* size_t strnlen(const char *s, size_t count) */ 15 + SYM_FUNC_START(strnlen) 16 + 17 + __ALTERNATIVE_CFG("nop", "j strnlen_zbb", 0, RISCV_ISA_EXT_ZBB, 18 + IS_ENABLED(CONFIG_RISCV_ISA_ZBB) && IS_ENABLED(CONFIG_TOOLCHAIN_HAS_ZBB)) 19 + 20 + 21 + /* 22 + * Returns 23 + * a0 - String length 24 + * 25 + * Parameters 26 + * a0 - String to measure 27 + * a1 - Max length of string 28 + * 29 + * Clobbers 30 + * t0, t1, t2 31 + */ 32 + addi t1, a0, -1 33 + add t2, a0, a1 34 + 1: 35 + addi t1, t1, 1 36 + beq t1, t2, 2f 37 + lbu t0, 0(t1) 38 + bnez t0, 1b 39 + 2: 40 + sub a0, t1, a0 41 + ret 42 + 43 + 44 + /* 45 + * Variant of strnlen using the ZBB extension if available 46 + */ 47 + #if defined(CONFIG_RISCV_ISA_ZBB) && defined(CONFIG_TOOLCHAIN_HAS_ZBB) 48 + strnlen_zbb: 49 + 50 + #ifdef CONFIG_CPU_BIG_ENDIAN 51 + # define CZ clz 52 + # define SHIFT sll 53 + #else 54 + # define CZ ctz 55 + # define SHIFT srl 56 + #endif 57 + 58 + .option push 59 + .option arch,+zbb 60 + 61 + /* 62 + * Returns 63 + * a0 - String length 64 + * 65 + * Parameters 66 + * a0 - String to measure 67 + * a1 - Max length of string 68 + * 69 + * Clobbers 70 + * t0, t1, t2, t3, t4 71 + */ 72 + 73 + /* If maxlen is 0, return 0. */ 74 + beqz a1, 3f 75 + 76 + /* Number of irrelevant bytes in the first word. */ 77 + andi t2, a0, SZREG-1 78 + 79 + /* Align pointer. */ 80 + andi t0, a0, -SZREG 81 + 82 + li t3, SZREG 83 + sub t3, t3, t2 84 + slli t2, t2, 3 85 + 86 + /* Aligned boundary. */ 87 + add t4, a0, a1 88 + andi t4, t4, -SZREG 89 + 90 + /* Get the first word. */ 91 + REG_L t1, 0(t0) 92 + 93 + /* 94 + * Shift away the partial data we loaded to remove the irrelevant bytes 95 + * preceding the string with the effect of adding NUL bytes at the 96 + * end of the string's first word. 97 + */ 98 + SHIFT t1, t1, t2 99 + 100 + /* Convert non-NUL into 0xff and NUL into 0x00. */ 101 + orc.b t1, t1 102 + 103 + /* Convert non-NUL into 0x00 and NUL into 0xff. */ 104 + not t1, t1 105 + 106 + /* 107 + * Search for the first set bit (corresponding to a NUL byte in the 108 + * original chunk). 109 + */ 110 + CZ t1, t1 111 + 112 + /* 113 + * The first chunk is special: compare against the number 114 + * of valid bytes in this chunk. 115 + */ 116 + srli a0, t1, 3 117 + 118 + /* Limit the result by maxlen. */ 119 + minu a0, a0, a1 120 + 121 + bgtu t3, a0, 2f 122 + 123 + /* Prepare for the word comparison loop. */ 124 + addi t2, t0, SZREG 125 + li t3, -1 126 + 127 + /* 128 + * Our critical loop is 4 instructions and processes data in 129 + * 4 byte or 8 byte chunks. 130 + */ 131 + .p2align 3 132 + 1: 133 + REG_L t1, SZREG(t0) 134 + addi t0, t0, SZREG 135 + orc.b t1, t1 136 + bgeu t0, t4, 4f 137 + beq t1, t3, 1b 138 + 4: 139 + not t1, t1 140 + CZ t1, t1 141 + srli t1, t1, 3 142 + 143 + /* Get number of processed bytes. */ 144 + sub t2, t0, t2 145 + 146 + /* Add number of characters in the first word. */ 147 + add a0, a0, t2 148 + 149 + /* Add number of characters in the last word. */ 150 + add a0, a0, t1 151 + 152 + /* Ensure the final result does not exceed maxlen. */ 153 + minu a0, a0, a1 154 + 2: 155 + ret 156 + 3: 157 + mv a0, a1 158 + ret 159 + 160 + .option pop 161 + #endif 162 + SYM_FUNC_END(strnlen) 163 + SYM_FUNC_ALIAS(__pi_strnlen, strnlen) 164 + EXPORT_SYMBOL(strnlen)
+37
arch/riscv/lib/strrchr.S
··· 1 + /* SPDX-License-Identifier: GPL-2.0-only */ 2 + 3 + /* 4 + * Copyright (C) 2025 Feng Jiang <jiangfeng@kylinos.cn> 5 + */ 6 + 7 + #include <linux/linkage.h> 8 + #include <asm/asm.h> 9 + 10 + /* char *strrchr(const char *s, int c) */ 11 + SYM_FUNC_START(strrchr) 12 + /* 13 + * Parameters 14 + * a0 - The string to be searched 15 + * a1 - The character to seaerch for 16 + * 17 + * Returns 18 + * a0 - Address of last occurrence of 'c' or 0 19 + * 20 + * Clobbers 21 + * t0, t1 22 + */ 23 + andi a1, a1, 0xff 24 + mv t1, a0 25 + li a0, 0 26 + 1: 27 + lbu t0, 0(t1) 28 + bne t0, a1, 2f 29 + mv a0, t1 30 + 2: 31 + addi t1, t1, 1 32 + bnez t0, 1b 33 + ret 34 + SYM_FUNC_END(strrchr) 35 + 36 + SYM_FUNC_ALIAS_WEAK(__pi_strrchr, strrchr) 37 + EXPORT_SYMBOL(strrchr)
+1 -1
arch/riscv/mm/cacheflush.c
··· 29 29 * Make sure all previous writes to the D$ are ordered before making 30 30 * the IPI. The RISC-V spec states that a hart must execute a data fence 31 31 * before triggering a remote fence.i in order to make the modification 32 - * visable for remote harts. 32 + * visible for remote harts. 33 33 * 34 34 * IPIs on RISC-V are triggered by MMIO writes to either CLINT or 35 35 * S-IMSIC, so the fence ensures previous data writes "happen before"
+10 -108
arch/riscv/mm/init.c
··· 41 41 42 42 struct kernel_mapping kernel_map __ro_after_init; 43 43 EXPORT_SYMBOL(kernel_map); 44 - #ifdef CONFIG_XIP_KERNEL 45 - #define kernel_map (*(struct kernel_mapping *)XIP_FIXUP(&kernel_map)) 46 - #endif 47 44 48 45 #ifdef CONFIG_64BIT 49 - u64 satp_mode __ro_after_init = !IS_ENABLED(CONFIG_XIP_KERNEL) ? SATP_MODE_57 : SATP_MODE_39; 46 + u64 satp_mode __ro_after_init = SATP_MODE_57; 50 47 #else 51 48 u64 satp_mode __ro_after_init = SATP_MODE_32; 52 49 #endif 53 50 EXPORT_SYMBOL(satp_mode); 54 51 55 52 #ifdef CONFIG_64BIT 56 - bool pgtable_l4_enabled __ro_after_init = !IS_ENABLED(CONFIG_XIP_KERNEL); 57 - bool pgtable_l5_enabled __ro_after_init = !IS_ENABLED(CONFIG_XIP_KERNEL); 53 + bool pgtable_l4_enabled __ro_after_init = true; 54 + bool pgtable_l5_enabled __ro_after_init = true; 58 55 EXPORT_SYMBOL(pgtable_l4_enabled); 59 56 EXPORT_SYMBOL(pgtable_l5_enabled); 60 57 #endif ··· 187 190 188 191 /* Limit the memory size via mem. */ 189 192 static phys_addr_t memory_limit; 190 - #ifdef CONFIG_XIP_KERNEL 191 - #define memory_limit (*(phys_addr_t *)XIP_FIXUP(&memory_limit)) 192 - #endif /* CONFIG_XIP_KERNEL */ 193 193 194 194 static int __init early_mem(char *p) 195 195 { ··· 210 216 phys_addr_t max_mapped_addr; 211 217 phys_addr_t phys_ram_end, vmlinux_start; 212 218 213 - if (IS_ENABLED(CONFIG_XIP_KERNEL)) 214 - vmlinux_start = __pa_symbol(&_sdata); 215 - else 216 - vmlinux_start = __pa_symbol(&_start); 219 + vmlinux_start = __pa_symbol(&_start); 217 220 218 221 memblock_enforce_memory_limit(memory_limit); 219 222 ··· 230 239 * Make sure we align the start of the memory on a PMD boundary so that 231 240 * at worst, we map the linear mapping with PMD mappings. 232 241 */ 233 - if (!IS_ENABLED(CONFIG_XIP_KERNEL)) { 234 - phys_ram_base = memblock_start_of_DRAM() & PMD_MASK; 242 + phys_ram_base = memblock_start_of_DRAM() & PMD_MASK; 235 243 #ifdef CONFIG_SPARSEMEM_VMEMMAP 236 - vmemmap_start_pfn = round_down(phys_ram_base, VMEMMAP_ADDR_ALIGN) >> PAGE_SHIFT; 244 + vmemmap_start_pfn = round_down(phys_ram_base, VMEMMAP_ADDR_ALIGN) >> PAGE_SHIFT; 237 245 #endif 238 - } 239 246 240 247 /* 241 248 * In 64-bit, any use of __va/__pa before this point is wrong as we ··· 346 357 347 358 pgd_t early_pg_dir[PTRS_PER_PGD] __initdata __aligned(PAGE_SIZE); 348 359 349 - #ifdef CONFIG_XIP_KERNEL 350 - #define pt_ops (*(struct pt_alloc_ops *)XIP_FIXUP(&pt_ops)) 351 - #define trampoline_pg_dir ((pgd_t *)XIP_FIXUP(trampoline_pg_dir)) 352 - #define fixmap_pte ((pte_t *)XIP_FIXUP(fixmap_pte)) 353 - #define early_pg_dir ((pgd_t *)XIP_FIXUP(early_pg_dir)) 354 - #endif /* CONFIG_XIP_KERNEL */ 355 - 356 360 static const pgprot_t protection_map[16] = { 357 361 [VM_NONE] = PAGE_NONE, 358 362 [VM_READ] = PAGE_READ, ··· 442 460 static pmd_t fixmap_pmd[PTRS_PER_PMD] __page_aligned_bss; 443 461 static pmd_t early_pmd[PTRS_PER_PMD] __initdata __aligned(PAGE_SIZE); 444 462 445 - #ifdef CONFIG_XIP_KERNEL 446 - #define trampoline_pmd ((pmd_t *)XIP_FIXUP(trampoline_pmd)) 447 - #define fixmap_pmd ((pmd_t *)XIP_FIXUP(fixmap_pmd)) 448 - #define early_pmd ((pmd_t *)XIP_FIXUP(early_pmd)) 449 - #endif /* CONFIG_XIP_KERNEL */ 450 - 451 463 static p4d_t trampoline_p4d[PTRS_PER_P4D] __page_aligned_bss; 452 464 static p4d_t fixmap_p4d[PTRS_PER_P4D] __page_aligned_bss; 453 465 static p4d_t early_p4d[PTRS_PER_P4D] __initdata __aligned(PAGE_SIZE); 454 466 455 - #ifdef CONFIG_XIP_KERNEL 456 - #define trampoline_p4d ((p4d_t *)XIP_FIXUP(trampoline_p4d)) 457 - #define fixmap_p4d ((p4d_t *)XIP_FIXUP(fixmap_p4d)) 458 - #define early_p4d ((p4d_t *)XIP_FIXUP(early_p4d)) 459 - #endif /* CONFIG_XIP_KERNEL */ 460 - 461 467 static pud_t trampoline_pud[PTRS_PER_PUD] __page_aligned_bss; 462 468 static pud_t fixmap_pud[PTRS_PER_PUD] __page_aligned_bss; 463 469 static pud_t early_pud[PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); 464 - 465 - #ifdef CONFIG_XIP_KERNEL 466 - #define trampoline_pud ((pud_t *)XIP_FIXUP(trampoline_pud)) 467 - #define fixmap_pud ((pud_t *)XIP_FIXUP(fixmap_pud)) 468 - #define early_pud ((pud_t *)XIP_FIXUP(early_pud)) 469 - #endif /* CONFIG_XIP_KERNEL */ 470 470 471 471 static pmd_t *__init get_pmd_virt_early(phys_addr_t pa) 472 472 { ··· 720 756 return PAGE_SIZE; 721 757 } 722 758 723 - #ifdef CONFIG_XIP_KERNEL 724 - #define phys_ram_base (*(phys_addr_t *)XIP_FIXUP(&phys_ram_base)) 725 - extern char _xiprom[], _exiprom[], __data_loc; 726 - 727 - /* called from head.S with MMU off */ 728 - asmlinkage void __init __copy_data(void) 729 - { 730 - void *from = (void *)(&__data_loc); 731 - void *to = (void *)CONFIG_PHYS_RAM_BASE; 732 - size_t sz = (size_t)((uintptr_t)(&_end) - (uintptr_t)(&_sdata)); 733 - 734 - memcpy(to, from, sz); 735 - } 736 - #endif 737 - 738 759 #ifdef CONFIG_STRICT_KERNEL_RWX 739 760 static __meminit pgprot_t pgprot_from_va(uintptr_t va) 740 761 { ··· 755 806 } 756 807 #endif /* CONFIG_STRICT_KERNEL_RWX */ 757 808 758 - #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) 809 + #if defined(CONFIG_64BIT) 759 810 u64 __pi_set_satp_mode_from_cmdline(uintptr_t dtb_pa); 760 811 u64 __pi_set_satp_mode_from_fdt(uintptr_t dtb_pa); 761 812 ··· 880 931 #error "setup_vm() is called from head.S before relocate so it should not use absolute addressing." 881 932 #endif 882 933 883 - #ifdef CONFIG_XIP_KERNEL 884 - static void __init create_kernel_page_table(pgd_t *pgdir, 885 - __always_unused bool early) 886 - { 887 - uintptr_t va, start_va, end_va; 888 - 889 - /* Map the flash resident part */ 890 - end_va = kernel_map.virt_addr + kernel_map.xiprom_sz; 891 - for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) 892 - create_pgd_mapping(pgdir, va, 893 - kernel_map.xiprom + (va - kernel_map.virt_addr), 894 - PMD_SIZE, PAGE_KERNEL_EXEC); 895 - 896 - /* Map the data in RAM */ 897 - start_va = kernel_map.virt_addr + (uintptr_t)&_sdata - (uintptr_t)&_start; 898 - end_va = kernel_map.virt_addr + kernel_map.size; 899 - for (va = start_va; va < end_va; va += PMD_SIZE) 900 - create_pgd_mapping(pgdir, va, 901 - kernel_map.phys_addr + (va - start_va), 902 - PMD_SIZE, PAGE_KERNEL); 903 - } 904 - #else 905 934 static void __init create_kernel_page_table(pgd_t *pgdir, bool early) 906 935 { 907 936 uintptr_t va, end_va; ··· 892 965 early ? 893 966 PAGE_KERNEL_EXEC : pgprot_from_va(va)); 894 967 } 895 - #endif 896 968 897 969 /* 898 970 * Setup a 4MB mapping that encompasses the device tree: for 64-bit kernel, ··· 1001 1075 return 0; 1002 1076 } 1003 1077 early_param("nokaslr", print_nokaslr); 1004 - 1005 - unsigned long kaslr_offset(void) 1006 - { 1007 - return kernel_map.virt_offset; 1008 - } 1009 1078 #endif 1010 1079 1011 1080 asmlinkage void __init setup_vm(uintptr_t dtb_pa) ··· 1028 1107 1029 1108 kernel_map.virt_addr = KERNEL_LINK_ADDR + kernel_map.virt_offset; 1030 1109 1031 - #ifdef CONFIG_XIP_KERNEL 1032 - kernel_map.xiprom = (uintptr_t)CONFIG_XIP_PHYS_ADDR; 1033 - kernel_map.xiprom_sz = (uintptr_t)(&_exiprom) - (uintptr_t)(&_xiprom); 1034 - 1035 - phys_ram_base = CONFIG_PHYS_RAM_BASE; 1036 - #ifdef CONFIG_SPARSEMEM_VMEMMAP 1037 - vmemmap_start_pfn = round_down(phys_ram_base, VMEMMAP_ADDR_ALIGN) >> PAGE_SHIFT; 1038 - #endif 1039 - kernel_map.phys_addr = (uintptr_t)CONFIG_PHYS_RAM_BASE; 1040 - kernel_map.size = (uintptr_t)(&_end) - (uintptr_t)(&_start); 1041 - 1042 - kernel_map.va_kernel_xip_text_pa_offset = kernel_map.virt_addr - kernel_map.xiprom; 1043 - kernel_map.va_kernel_xip_data_pa_offset = kernel_map.virt_addr - kernel_map.phys_addr 1044 - + (uintptr_t)&_sdata - (uintptr_t)&_start; 1045 - #else 1046 1110 kernel_map.phys_addr = (uintptr_t)(&_start); 1047 1111 kernel_map.size = (uintptr_t)(&_end) - kernel_map.phys_addr; 1048 1112 kernel_map.va_kernel_pa_offset = kernel_map.virt_addr - kernel_map.phys_addr; 1049 - #endif 1050 1113 1051 - #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) 1114 + #if defined(CONFIG_64BIT) 1052 1115 set_satp_mode(dtb_pa); 1053 1116 set_mmap_rnd_bits_max(); 1054 1117 #endif ··· 1105 1200 if (pgtable_l4_enabled) 1106 1201 create_pud_mapping(trampoline_pud, kernel_map.virt_addr, 1107 1202 (uintptr_t)trampoline_pmd, PUD_SIZE, PAGE_TABLE); 1108 - #ifdef CONFIG_XIP_KERNEL 1109 - create_pmd_mapping(trampoline_pmd, kernel_map.virt_addr, 1110 - kernel_map.xiprom, PMD_SIZE, PAGE_KERNEL_EXEC); 1111 - #else 1112 1203 create_pmd_mapping(trampoline_pmd, kernel_map.virt_addr, 1113 1204 kernel_map.phys_addr, PMD_SIZE, PAGE_KERNEL_EXEC); 1114 - #endif 1115 1205 #else 1116 1206 /* Setup trampoline PGD */ 1117 1207 create_pgd_mapping(trampoline_pg_dir, kernel_map.virt_addr, ··· 1350 1450 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, 1351 1451 struct vmem_altmap *altmap) 1352 1452 { 1453 + WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END)); 1454 + 1353 1455 /* 1354 1456 * Note that SPARSEMEM_VMEMMAP is only selected for rv64 and that we 1355 1457 * can't use hugepage mappings for 2-level page table because in case of
+2 -2
arch/riscv/mm/physaddr.c
··· 9 9 phys_addr_t __virt_to_phys(unsigned long x) 10 10 { 11 11 /* 12 - * Boundary checking aginst the kernel linear mapping space. 12 + * Boundary checking against the kernel linear mapping space. 13 13 */ 14 14 WARN(!is_linear_mapping(x) && !is_kernel_mapping(x), 15 15 "virt_to_phys used for non-linear address: %p (%pS)\n", ··· 25 25 unsigned long kernel_end = kernel_start + kernel_map.size; 26 26 27 27 /* 28 - * Boundary checking aginst the kernel image mapping. 28 + * Boundary checking against the kernel image mapping. 29 29 * __pa_symbol should only be used on kernel symbol addresses. 30 30 */ 31 31 VIRTUAL_BUG_ON(x < kernel_start || x > kernel_end);
+10 -1
arch/riscv/purgatory/Makefile
··· 2 2 3 3 purgatory-y := purgatory.o sha256.o entry.o string.o ctype.o memcpy.o memset.o 4 4 ifeq ($(CONFIG_KASAN_GENERIC)$(CONFIG_KASAN_SW_TAGS),) 5 - purgatory-y += strcmp.o strlen.o strncmp.o 5 + purgatory-y += strcmp.o strlen.o strncmp.o strnlen.o strchr.o strrchr.o 6 6 endif 7 7 8 8 targets += $(purgatory-y) ··· 31 31 32 32 $(obj)/sha256.o: $(srctree)/lib/crypto/sha256.c FORCE 33 33 $(call if_changed_rule,cc_o_c) 34 + 35 + $(obj)/strnlen.o: $(srctree)/arch/riscv/lib/strnlen.S FORCE 36 + $(call if_changed_rule,as_o_S) 37 + 38 + $(obj)/strchr.o: $(srctree)/arch/riscv/lib/strchr.S FORCE 39 + $(call if_changed_rule,as_o_S) 40 + 41 + $(obj)/strrchr.o: $(srctree)/arch/riscv/lib/strrchr.S FORCE 42 + $(call if_changed_rule,as_o_S) 34 43 35 44 CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY 36 45 CFLAGS_string.o := -D__DISABLE_EXPORTS
+2 -2
arch/riscv/purgatory/kexec-purgatory.S
··· 6 6 kexec_purgatory: 7 7 .globl kexec_purgatory 8 8 .incbin "arch/riscv/purgatory/purgatory.ro" 9 - .Lkexec_purgatroy_end: 9 + .Lkexec_purgatory_end: 10 10 11 11 .align 8 12 12 kexec_purgatory_size: 13 13 .globl kexec_purgatory_size 14 - .quad .Lkexec_purgatroy_end - kexec_purgatory 14 + .quad .Lkexec_purgatory_end - kexec_purgatory
+11
lib/Kconfig.debug
··· 2526 2526 depends on KUNIT 2527 2527 default KUNIT_ALL_TESTS 2528 2528 2529 + config STRING_KUNIT_BENCH 2530 + bool "Benchmark string functions at runtime" 2531 + depends on STRING_KUNIT_TEST 2532 + help 2533 + Enable performance measurement for string functions. 2534 + 2535 + This measures the execution efficiency of string functions 2536 + during the KUnit test run. 2537 + 2538 + If unsure, say N. 2539 + 2529 2540 config FFS_KUNIT_TEST 2530 2541 tristate "KUnit test ffs-family functions at runtime" if !KUNIT_ALL_TESTS 2531 2542 depends on KUNIT
+274
lib/tests/string_kunit.c
··· 6 6 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt 7 7 8 8 #include <kunit/test.h> 9 + #include <linux/ktime.h> 10 + #include <linux/math64.h> 11 + #include <linux/minmax.h> 12 + #include <linux/mm.h> 9 13 #include <linux/module.h> 14 + #include <linux/prandom.h> 10 15 #include <linux/printk.h> 11 16 #include <linux/slab.h> 12 17 #include <linux/string.h> 18 + #include <linux/time64.h> 19 + #include <linux/units.h> 20 + #include <linux/vmalloc.h> 13 21 14 22 #define STRCMP_LARGE_BUF_LEN 2048 15 23 #define STRCMP_CHANGE_POINT 1337 16 24 #define STRCMP_TEST_EXPECT_EQUAL(test, fn, ...) KUNIT_EXPECT_EQ(test, fn(__VA_ARGS__), 0) 17 25 #define STRCMP_TEST_EXPECT_LOWER(test, fn, ...) KUNIT_EXPECT_LT(test, fn(__VA_ARGS__), 0) 18 26 #define STRCMP_TEST_EXPECT_GREATER(test, fn, ...) KUNIT_EXPECT_GT(test, fn(__VA_ARGS__), 0) 27 + 28 + #define STRING_TEST_MAX_LEN 128 29 + #define STRING_TEST_MAX_OFFSET 16 30 + 31 + #define STRING_BENCH_SEED 888 32 + #define STRING_BENCH_WORKLOAD (1 * MEGA) 19 33 20 34 static void string_test_memset16(struct kunit *test) 21 35 { ··· 118 104 } 119 105 } 120 106 107 + static void string_test_strlen(struct kunit *test) 108 + { 109 + size_t buf_size; 110 + char *buf, *s; 111 + 112 + buf_size = PAGE_ALIGN(STRING_TEST_MAX_LEN + STRING_TEST_MAX_OFFSET + 1); 113 + buf = vmalloc(buf_size); 114 + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); 115 + 116 + memset(buf, 'A', buf_size); 117 + 118 + for (size_t offset = 0; offset < STRING_TEST_MAX_OFFSET; offset++) { 119 + for (size_t len = 0; len <= STRING_TEST_MAX_LEN; len++) { 120 + s = buf + buf_size - 1 - offset - len; 121 + s[len] = '\0'; 122 + KUNIT_EXPECT_EQ_MSG(test, strlen(s), len, 123 + "offset:%zu len:%zu", offset, len); 124 + s[len] = 'A'; 125 + } 126 + } 127 + 128 + vfree(buf); 129 + } 130 + 131 + static void string_test_strnlen(struct kunit *test) 132 + { 133 + size_t buf_size; 134 + char *buf, *s; 135 + 136 + buf_size = PAGE_ALIGN(STRING_TEST_MAX_LEN + STRING_TEST_MAX_OFFSET + 1); 137 + buf = vmalloc(buf_size); 138 + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); 139 + 140 + memset(buf, 'A', buf_size); 141 + 142 + for (size_t offset = 0; offset < STRING_TEST_MAX_OFFSET; offset++) { 143 + for (size_t len = 0; len <= STRING_TEST_MAX_LEN; len++) { 144 + s = buf + buf_size - 1 - offset - len; 145 + s[len] = '\0'; 146 + 147 + if (len > 0) 148 + KUNIT_EXPECT_EQ(test, strnlen(s, len - 1), len - 1); 149 + if (len > 1) 150 + KUNIT_EXPECT_EQ(test, strnlen(s, len - 2), len - 2); 151 + 152 + KUNIT_EXPECT_EQ(test, strnlen(s, len), len); 153 + 154 + KUNIT_EXPECT_EQ(test, strnlen(s, len + 1), len); 155 + KUNIT_EXPECT_EQ(test, strnlen(s, len + 2), len); 156 + KUNIT_EXPECT_EQ(test, strnlen(s, len + 10), len); 157 + 158 + s[len] = 'A'; 159 + } 160 + } 161 + 162 + vfree(buf); 163 + } 164 + 121 165 static void string_test_strchr(struct kunit *test) 122 166 { 123 167 const char *test_string = "abcdefghijkl"; ··· 197 125 198 126 result = strchr(test_string, 'z'); 199 127 KUNIT_ASSERT_NULL(test, result); 128 + } 129 + 130 + static void string_test_strrchr(struct kunit *test) 131 + { 132 + size_t buf_size; 133 + char *buf, *s; 134 + 135 + buf_size = PAGE_ALIGN(STRING_TEST_MAX_LEN + STRING_TEST_MAX_OFFSET + 1); 136 + buf = vmalloc(buf_size); 137 + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); 138 + 139 + memset(buf, 'A', buf_size); 140 + 141 + for (size_t offset = 0; offset < STRING_TEST_MAX_OFFSET; offset++) { 142 + for (size_t len = 0; len <= STRING_TEST_MAX_LEN; len++) { 143 + s = buf + buf_size - 1 - offset - len; 144 + s[len] = '\0'; 145 + 146 + KUNIT_EXPECT_PTR_EQ(test, strrchr(s, 'Z'), NULL); 147 + 148 + if (len > 0) 149 + KUNIT_EXPECT_PTR_EQ(test, strrchr(s, 'A'), s + len - 1); 150 + else 151 + KUNIT_EXPECT_PTR_EQ(test, strrchr(s, 'A'), NULL); 152 + 153 + s[len] = 'A'; 154 + } 155 + } 156 + 157 + vfree(buf); 200 158 } 201 159 202 160 static void string_test_strnchr(struct kunit *test) ··· 716 614 KUNIT_EXPECT_TRUE(test, strends("", "")); 717 615 } 718 616 617 + #if IS_ENABLED(CONFIG_STRING_KUNIT_BENCH) 618 + /* Target string lengths for benchmarking */ 619 + static const size_t bench_lens[] = { 620 + 0, 1, 7, 8, 16, 31, 64, 127, 512, 1024, 3173, 4096, 621 + }; 622 + 623 + /** 624 + * alloc_max_bench_buffer() - Allocate buffer for the max test case. 625 + * @test: KUnit context for managed allocation. 626 + * @lens: Array of lengths used in the benchmark cases. 627 + * @count: Number of elements in the @lens array. 628 + * @buf_len: [out] Pointer to store the actually allocated buffer 629 + * size (including NUL character). 630 + * 631 + * Return: Pointer to the allocated memory, or NULL on failure. 632 + */ 633 + static void *alloc_max_bench_buffer(struct kunit *test, const size_t *lens, 634 + size_t count, size_t *buf_len) 635 + { 636 + size_t max_len = 0; 637 + void *buf; 638 + 639 + for (size_t i = 0; i < count; i++) 640 + max_len = max(lens[i], max_len); 641 + 642 + /* Add space for NUL character */ 643 + max_len += 1; 644 + 645 + buf = kunit_kzalloc(test, max_len, GFP_KERNEL); 646 + if (!buf) 647 + return NULL; 648 + 649 + if (buf_len) 650 + *buf_len = max_len; 651 + 652 + return buf; 653 + } 654 + 655 + /** 656 + * fill_random_string() - Populate a buffer with a random NUL-terminated string. 657 + * @buf: Buffer to fill. 658 + * @len: Length of the buffer in bytes. 659 + * 660 + * Fills the buffer with random non-NUL bytes and ensures the string is 661 + * properly NUL-terminated. 662 + */ 663 + static void fill_random_string(char *buf, size_t len) 664 + { 665 + struct rnd_state state; 666 + 667 + if (!buf || !len) 668 + return; 669 + 670 + /* Use a fixed seed to ensure deterministic benchmark results */ 671 + prandom_seed_state(&state, STRING_BENCH_SEED); 672 + prandom_bytes_state(&state, buf, len); 673 + 674 + /* Replace NUL characters to avoid early string termination */ 675 + for (size_t i = 0; i < len; i++) { 676 + if (buf[i] == '\0') 677 + buf[i] = 0x01; 678 + } 679 + 680 + buf[len - 1] = '\0'; 681 + } 682 + 683 + /** 684 + * STRING_BENCH() - Benchmark string functions. 685 + * @iters: Number of iterations to run. 686 + * @func: Function to benchmark. 687 + * @...: Variable arguments passed to @func. 688 + * 689 + * Disables preemption and measures the total time in nanoseconds to execute 690 + * @func(@__VA_ARGS__) for @iters times, including a small warm-up phase. 691 + * 692 + * Context: Disables preemption during measurement. 693 + * Return: Total execution time in nanoseconds (u64). 694 + */ 695 + #define STRING_BENCH(iters, func, ...) \ 696 + ({ \ 697 + /* Volatile function pointer prevents dead code elimination */ \ 698 + typeof(func) (* volatile __func) = (func); \ 699 + size_t __bn_iters = (iters); \ 700 + size_t __bn_warm_iters; \ 701 + u64 __bn_t; \ 702 + \ 703 + /* Use 10% of the given iterations (maximum 50) to warm up */ \ 704 + __bn_warm_iters = max(__bn_iters / 10, 50U); \ 705 + \ 706 + for (size_t __bn_i = 0; __bn_i < __bn_warm_iters; __bn_i++) \ 707 + (void)__func(__VA_ARGS__); \ 708 + \ 709 + preempt_disable(); \ 710 + __bn_t = ktime_get_ns(); \ 711 + for (size_t __bn_i = 0; __bn_i < __bn_iters; __bn_i++) \ 712 + (void)__func(__VA_ARGS__); \ 713 + __bn_t = ktime_get_ns() - __bn_t; \ 714 + preempt_enable(); \ 715 + __bn_t; \ 716 + }) 717 + 718 + /** 719 + * STRING_BENCH_BUF() - Benchmark harness for single-buffer functions. 720 + * @test: KUnit context. 721 + * @buf_name: Local char * variable name to be defined. 722 + * @buf_size: Local size_t variable name to be defined. 723 + * @func: Function to benchmark. 724 + * @...: Extra arguments for @func. 725 + * 726 + * Prepares a randomized, NUL-terminated buffer and iterates through lengths 727 + * in bench_lens, defining @buf_name and @buf_size in each loop. 728 + */ 729 + #define STRING_BENCH_BUF(test, buf_name, buf_size, func, ...) \ 730 + do { \ 731 + size_t _bn_i, _bn_iters, _bn_size = 0; \ 732 + u64 _bn_t, _bn_mbps = 0, _bn_lat = 0; \ 733 + char *_bn_buf; \ 734 + \ 735 + _bn_buf = alloc_max_bench_buffer(test, bench_lens, \ 736 + ARRAY_SIZE(bench_lens), &_bn_size); \ 737 + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, _bn_buf); \ 738 + \ 739 + fill_random_string(_bn_buf, _bn_size); \ 740 + \ 741 + for (_bn_i = 0; _bn_i < ARRAY_SIZE(bench_lens); _bn_i++) { \ 742 + size_t buf_size = bench_lens[_bn_i]; \ 743 + char *buf_name = _bn_buf + _bn_size - buf_size - 1; \ 744 + _bn_iters = STRING_BENCH_WORKLOAD / max(buf_size, 1U); \ 745 + \ 746 + _bn_t = STRING_BENCH(_bn_iters, func, ##__VA_ARGS__); \ 747 + if (_bn_t > 0) { \ 748 + _bn_mbps = (u64)(buf_size) * _bn_iters * \ 749 + (NSEC_PER_SEC / MEGA); \ 750 + _bn_mbps = div64_u64(_bn_mbps, _bn_t); \ 751 + _bn_lat = div64_u64(_bn_t, _bn_iters); \ 752 + } \ 753 + kunit_info(test, "len=%zu: %llu MB/s (%llu ns/call)\n", \ 754 + buf_size, _bn_mbps, _bn_lat); \ 755 + } \ 756 + } while (0) 757 + #else 758 + #define STRING_BENCH_BUF(test, buf_name, buf_size, func, ...) \ 759 + kunit_skip(test, "not enabled") 760 + #endif /* IS_ENABLED(CONFIG_STRING_KUNIT_BENCH) */ 761 + 762 + static void string_bench_strlen(struct kunit *test) 763 + { 764 + STRING_BENCH_BUF(test, buf, len, strlen, buf); 765 + } 766 + 767 + static void string_bench_strnlen(struct kunit *test) 768 + { 769 + STRING_BENCH_BUF(test, buf, len, strnlen, buf, len); 770 + } 771 + 772 + static void string_bench_strchr(struct kunit *test) 773 + { 774 + STRING_BENCH_BUF(test, buf, len, strchr, buf, '\0'); 775 + } 776 + 777 + static void string_bench_strrchr(struct kunit *test) 778 + { 779 + STRING_BENCH_BUF(test, buf, len, strrchr, buf, '\0'); 780 + } 781 + 719 782 static struct kunit_case string_test_cases[] = { 720 783 KUNIT_CASE(string_test_memset16), 721 784 KUNIT_CASE(string_test_memset32), 722 785 KUNIT_CASE(string_test_memset64), 786 + KUNIT_CASE(string_test_strlen), 787 + KUNIT_CASE(string_test_strnlen), 723 788 KUNIT_CASE(string_test_strchr), 724 789 KUNIT_CASE(string_test_strnchr), 790 + KUNIT_CASE(string_test_strrchr), 725 791 KUNIT_CASE(string_test_strspn), 726 792 KUNIT_CASE(string_test_strcmp), 727 793 KUNIT_CASE(string_test_strcmp_long_strings), ··· 906 636 KUNIT_CASE(string_test_strtomem), 907 637 KUNIT_CASE(string_test_memtostr), 908 638 KUNIT_CASE(string_test_strends), 639 + KUNIT_CASE(string_bench_strlen), 640 + KUNIT_CASE(string_bench_strnlen), 641 + KUNIT_CASE(string_bench_strchr), 642 + KUNIT_CASE(string_bench_strrchr), 909 643 {} 910 644 }; 911 645
+2
tools/testing/selftests/riscv/cfi/Makefile
··· 1 + # SPDX-License-Identifier: GPL-2.0 2 + 1 3 CFLAGS += $(KHDR_INCLUDES) 2 4 CFLAGS += -I$(top_srcdir)/tools/include 3 5