tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

arm64/irqflags: __always_inline the arch_local_irq_*() helpers

The arch_local_irq_*() wrappers in <asm/irqflags.h> dispatch between two
underlying primitives: the __daif_* path on most systems, and the
__pmr_* path on builds that use GIC PMR-based masking (Pseudo-NMI). The
leaf primitives are already __always_inline, but the wrappers themselves
are plain "static inline".

That is unsafe for noinstr callers: nothing prevents the compiler from
emitting an out-of-line copy of e.g. arch_local_irq_disable(), and an
out-of-line copy can be instrumented (ftrace, kcov, sanitizers), which
breaks the noinstr contract on the entry/idle paths that rely on these
helpers.

x86 hit and fixed exactly this class of bug in commit 7a745be1cc90
("x86/entry: __always_inline irqflags for noinstr").

Force-inline all of the arch_local_irq_*() wrappers so they cannot be
emitted out-of-line:

- arch_local_irq_enable()
- arch_local_irq_disable()
- arch_local_save_flags()
- arch_irqs_disabled_flags()
- arch_irqs_disabled()
- arch_local_irq_save()
- arch_local_irq_restore()

The primary motivation is noinstr safety. There is a useful side effect
for fleet-wide profiling: when the wrapper is emitted out-of-line,
samples taken inside it during the post-WFI IRQ unmask in
default_idle_call() are attributed to arch_local_irq_enable rather than
default_idle_call(), and the FP-unwinder loses default_idle_call() from
the chain.

Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Leonardo Bras <leo.bras@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

authored by

Breno Leitao and committed by
Catalin Marinas caecde11 4023b742

+7 -7
+7 -7
arch/arm64/include/asm/irqflags.h
··· 40 40 barrier(); 41 41 } 42 42 43 - static inline void arch_local_irq_enable(void) 43 + static __always_inline void arch_local_irq_enable(void) 44 44 { 45 45 if (system_uses_irq_prio_masking()) { 46 46 __pmr_local_irq_enable(); ··· 68 68 barrier(); 69 69 } 70 70 71 - static inline void arch_local_irq_disable(void) 71 + static __always_inline void arch_local_irq_disable(void) 72 72 { 73 73 if (system_uses_irq_prio_masking()) { 74 74 __pmr_local_irq_disable(); ··· 90 90 /* 91 91 * Save the current interrupt enable state. 92 92 */ 93 - static inline unsigned long arch_local_save_flags(void) 93 + static __always_inline unsigned long arch_local_save_flags(void) 94 94 { 95 95 if (system_uses_irq_prio_masking()) { 96 96 return __pmr_local_save_flags(); ··· 109 109 return flags != GIC_PRIO_IRQON; 110 110 } 111 111 112 - static inline bool arch_irqs_disabled_flags(unsigned long flags) 112 + static __always_inline bool arch_irqs_disabled_flags(unsigned long flags) 113 113 { 114 114 if (system_uses_irq_prio_masking()) { 115 115 return __pmr_irqs_disabled_flags(flags); ··· 128 128 return __pmr_irqs_disabled_flags(__pmr_local_save_flags()); 129 129 } 130 130 131 - static inline bool arch_irqs_disabled(void) 131 + static __always_inline bool arch_irqs_disabled(void) 132 132 { 133 133 if (system_uses_irq_prio_masking()) { 134 134 return __pmr_irqs_disabled(); ··· 160 160 return flags; 161 161 } 162 162 163 - static inline unsigned long arch_local_irq_save(void) 163 + static __always_inline unsigned long arch_local_irq_save(void) 164 164 { 165 165 if (system_uses_irq_prio_masking()) { 166 166 return __pmr_local_irq_save(); ··· 187 187 /* 188 188 * restore saved IRQ state 189 189 */ 190 - static inline void arch_local_irq_restore(unsigned long flags) 190 + static __always_inline void arch_local_irq_restore(unsigned long flags) 191 191 { 192 192 if (system_uses_irq_prio_masking()) { 193 193 __pmr_local_irq_restore(flags);