tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

membarrier: riscv: Provide core serializing command

RISC-V uses xRET instructions on return from interrupt and to go back
to user-space; the xRET instruction is not core serializing.

Use FENCE.I for providing core serialization as follows:

- by calling sync_core_before_usermode() on return from interrupt (cf.
ipi_sync_core()),

- via switch_mm() and sync_core_before_usermode() (respectively, for
uthread->uthread and kthread->uthread transitions) before returning
to user-space.

On RISC-V, the serialization in switch_mm() is activated by resetting
the icache_stale_mask of the mm at prepare_sync_core_cmd().

Suggested-by: Palmer Dabbelt <palmer@dabbelt.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://lore.kernel.org/r/20240131144936.29190-5-parri.andrea@gmail.com
Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>

authored by

Andrea Parri and committed by
Palmer Dabbelt cd9b2901 4ff4c745

+77 -1
+17 -1
Documentation/features/sched/membarrier-sync-core/arch-support.txt
··· 10 10 # Rely on implicit context synchronization as a result of exception return 11 11 # when returning from IPI handler, and when returning to user-space. 12 12 # 13 + # * riscv 14 + # 15 + # riscv uses xRET as return from interrupt and to return to user-space. 16 + # 17 + # Given that xRET is not core serializing, we rely on FENCE.I for providing 18 + # core serialization: 19 + # 20 + # - by calling sync_core_before_usermode() on return from interrupt (cf. 21 + # ipi_sync_core()), 22 + # 23 + # - via switch_mm() and sync_core_before_usermode() (respectively, for 24 + # uthread->uthread and kthread->uthread transitions) before returning 25 + # to user-space. 26 + # 27 + # The serialization in switch_mm() is activated by prepare_sync_core_cmd(). 28 + # 13 29 # * x86 14 30 # 15 31 # x86-32 uses IRET as return from interrupt, which takes care of the IPI. ··· 59 43 | openrisc: | TODO | 60 44 | parisc: | TODO | 61 45 | powerpc: | ok | 62 - | riscv: | TODO | 46 + | riscv: | ok | 63 47 | s390: | ok | 64 48 | sh: | TODO | 65 49 | sparc: | TODO |
+1
MAINTAINERS
··· 14041 14041 S: Supported 14042 14042 F: Documentation/scheduler/membarrier.rst 14043 14043 F: arch/*/include/asm/membarrier.h 14044 + F: arch/*/include/asm/sync_core.h 14044 14045 F: include/uapi/linux/membarrier.h 14045 14046 F: kernel/sched/membarrier.c 14046 14047
+3
arch/riscv/Kconfig
··· 28 28 select ARCH_HAS_GIGANTIC_PAGE 29 29 select ARCH_HAS_KCOV 30 30 select ARCH_HAS_MEMBARRIER_CALLBACKS 31 + select ARCH_HAS_MEMBARRIER_SYNC_CORE 31 32 select ARCH_HAS_MMIOWB 32 33 select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE 33 34 select ARCH_HAS_PMEM_API 35 + select ARCH_HAS_PREPARE_SYNC_CORE_CMD 34 36 select ARCH_HAS_PTE_SPECIAL 35 37 select ARCH_HAS_SET_DIRECT_MAP if MMU 36 38 select ARCH_HAS_SET_MEMORY if MMU 37 39 select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL 38 40 select ARCH_HAS_STRICT_MODULE_RWX if MMU && !XIP_KERNEL 41 + select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE 39 42 select ARCH_HAS_SYSCALL_WRAPPER 40 43 select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST 41 44 select ARCH_HAS_UBSAN_SANITIZE_ALL
+19
arch/riscv/include/asm/membarrier.h
··· 22 22 /* 23 23 * The membarrier system call requires a full memory barrier 24 24 * after storing to rq->curr, before going back to user-space. 25 + * 26 + * This barrier is also needed for the SYNC_CORE command when 27 + * switching between processes; in particular, on a transition 28 + * from a thread belonging to another mm to a thread belonging 29 + * to the mm for which a membarrier SYNC_CORE is done on CPU0: 30 + * 31 + * - [CPU0] sets all bits in the mm icache_stale_mask (in 32 + * prepare_sync_core_cmd()); 33 + * 34 + * - [CPU1] stores to rq->curr (by the scheduler); 35 + * 36 + * - [CPU0] loads rq->curr within membarrier and observes 37 + * cpu_rq(1)->curr->mm != mm, so the IPI is skipped on 38 + * CPU1; this means membarrier relies on switch_mm() to 39 + * issue the sync-core; 40 + * 41 + * - [CPU1] switch_mm() loads icache_stale_mask; if the bit 42 + * is zero, switch_mm() may incorrectly skip the sync-core. 43 + * 25 44 * Matches a full barrier in the proximity of the membarrier 26 45 * system call entry. 27 46 */
+29
arch/riscv/include/asm/sync_core.h
··· 1 + /* SPDX-License-Identifier: GPL-2.0 */ 2 + #ifndef _ASM_RISCV_SYNC_CORE_H 3 + #define _ASM_RISCV_SYNC_CORE_H 4 + 5 + /* 6 + * RISC-V implements return to user-space through an xRET instruction, 7 + * which is not core serializing. 8 + */ 9 + static inline void sync_core_before_usermode(void) 10 + { 11 + asm volatile ("fence.i" ::: "memory"); 12 + } 13 + 14 + #ifdef CONFIG_SMP 15 + /* 16 + * Ensure the next switch_mm() on every CPU issues a core serializing 17 + * instruction for the given @mm. 18 + */ 19 + static inline void prepare_sync_core_cmd(struct mm_struct *mm) 20 + { 21 + cpumask_setall(&mm->context.icache_stale_mask); 22 + } 23 + #else 24 + static inline void prepare_sync_core_cmd(struct mm_struct *mm) 25 + { 26 + } 27 + #endif /* CONFIG_SMP */ 28 + 29 + #endif /* _ASM_RISCV_SYNC_CORE_H */
+4
kernel/sched/core.c
··· 6721 6721 * 6722 6722 * The barrier matches a full barrier in the proximity of 6723 6723 * the membarrier system call entry. 6724 + * 6725 + * On RISC-V, this barrier pairing is also needed for the 6726 + * SYNC_CORE command when switching between processes, cf. 6727 + * the inline comments in membarrier_arch_switch_mm(). 6724 6728 */ 6725 6729 ++*switch_count; 6726 6730
+4
kernel/sched/membarrier.c
··· 342 342 /* 343 343 * Matches memory barriers after rq->curr modification in 344 344 * scheduler. 345 + * 346 + * On RISC-V, this barrier pairing is also needed for the 347 + * SYNC_CORE command when switching between processes, cf. 348 + * the inline comments in membarrier_arch_switch_mm(). 345 349 */ 346 350 smp_mb(); /* system call entry is not a mb. */ 347 351