Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

kbuild: rust: provide an option to inline C helpers into Rust

A new experimental Kconfig option, `RUST_INLINE_HELPERS` is added to
allow C helpers (which were created to allow Rust to call into
inline/macro C functions without having to re-implement the logic in
Rust) to be inlined into Rust crates without performing global LTO.

If the option is enabled, the following is performed:
* For helpers, instead of compiling them to an object file to be linked
into vmlinux, they're compiled to LLVM IR bitcode. Two versions are
generated: one for built-in code (`helpers.bc`) and one for modules
(`helpers_module.bc`, with -DMODULE defined). This ensures that C
macros/inlines that behave differently for modules (e.g. static calls)
function correctly when inlined.
* When a Rust crate or object is compiled, instead of generating an
object file, LLVM bitcode is generated.
* llvm-link is invoked with --internalize to combine the helper bitcode
with the crate bitcode. This step is similar to LTO, but this is much
faster since it only needs to inline the helpers.
* clang is invoked to turn the combined bitcode into a final object file.
* Since clang may produce LLVM bitcode when LTO is enabled, and objtool
requires ELF input, $(cmd_ld_single) is invoked to ensure the object
is converted to ELF before objtool runs.

The --internalize flag tells llvm-link to treat all symbols in
helpers.bc using `internal` linkage [1]. This matches the behavior of
`clang` on `static inline` functions, and avoids exporting the symbol
from the object file.

To ensure that RUST_INLINE_HELPERS is not incompatible with BTF, we pass
the -g0 flag when building helpers. See commit 5daa0c35a1f0 ("rust:
Disallow BTF generation with Rust + LTO") for details.

We have an intended triple mismatch of `aarch64-unknown-none` vs
`aarch64-unknown-linux-gnu`, so we pass --suppress-warnings to llvm-link
to suppress it.

I considered adding some sort of check that KBUILD_MODNAME is not
present in helpers_module.bc, but this is actually not so easy to carry
out because .bc files store strings in a weird binary format, so you
cannot just grep it for a string to check whether it ended up using
KBUILD_MODNAME anywhere.

[ Andreas writes:

For the rnull driver, enabling helper inlining with this patch
gives an average speedup of 2% over the set of 120 workloads that
we publish on [2].

Link: https://rust-for-linux.com/null-block-driver [2]

This series also uncovered a pre-existing UB instance thanks to an
`objtool` warning which I noticed while testing the series (details
in the mailing list).

- Miguel ]

Link: https://github.com/llvm/llvm-project/pull/170397 [1]
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Co-developed-by: Matthew Maurer <mmaurer@google.com>
Signed-off-by: Matthew Maurer <mmaurer@google.com>
Signed-off-by: Gary Guo <gary@garyguo.net>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Andreas Hindborg <a.hindborg@kernel.org>
Reviewed-by: Andreas Hindborg <a.hindborg@kernel.org>
Link: https://patch.msgid.link/20260203-inline-helpers-v2-3-beb8547a03c9@google.com
[ Some changes, apart from the rebase:

- Added "(EXPERIMENTAL)" to Kconfig as the commit mentions.

- Added `depends on ARM64 || X86_64` and `!UML` for now, since this is
experimental, other architectures may require other changes (e.g.
the issues I mentioned in the mailing list for ARM and UML) and they
are not really tested so far. So let arch maintainers pick this up
if they think it is worth it.

- Gated the `cmd_ld_single` step also into the new mode, which also
means that any possible future `objcopy` step is done after the
translation, as expected.

- Added `.gitignore` for `.bc` with exception for existing script.

- Added `part-of-*` for helpers bitcode files as discussed, and
dropped `$(if $(filter %_module.bc,$@),-DMODULE)` since `-DMODULE`
is already there (would be duplicated otherwise).

- Moved `LLVM_LINK` to keep binutils list alphabetized.

- Fixed typo in title.

- Dropped second `cmd_ld_single` commit message paragraph.

- Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>

authored by

Gary Guo and committed by
Miguel Ojeda
3a2486cc db702816

+60 -7
+4
.gitignore
··· 13 13 .* 14 14 *.a 15 15 *.asn1.[ch] 16 + *.bc 16 17 *.bin 17 18 *.bz2 18 19 *.c.[012]*.* ··· 185 184 186 185 # Rust analyzer configuration 187 186 /rust-project.json 187 + 188 + # bc language scripts (not LLVM bitcode) 189 + !kernel/time/timeconst.bc
+2 -1
Makefile
··· 515 515 CC = $(LLVM_PREFIX)clang$(LLVM_SUFFIX) 516 516 LD = $(LLVM_PREFIX)ld.lld$(LLVM_SUFFIX) 517 517 AR = $(LLVM_PREFIX)llvm-ar$(LLVM_SUFFIX) 518 + LLVM_LINK = $(LLVM_PREFIX)llvm-link$(LLVM_SUFFIX) 518 519 NM = $(LLVM_PREFIX)llvm-nm$(LLVM_SUFFIX) 519 520 OBJCOPY = $(LLVM_PREFIX)llvm-objcopy$(LLVM_SUFFIX) 520 521 OBJDUMP = $(LLVM_PREFIX)llvm-objdump$(LLVM_SUFFIX) ··· 629 628 export CLIPPY_CONF_DIR := $(srctree) 630 629 631 630 export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE LD CC HOSTPKG_CONFIG 632 - export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN 631 + export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN LLVM_LINK 633 632 export HOSTRUSTC KBUILD_HOSTRUSTFLAGS 634 633 export CPP AR NM STRIP OBJCOPY OBJDUMP READELF PAHOLE RESOLVE_BTFIDS LEX YACC AWK INSTALLKERNEL 635 634 export PERL PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX
+17
lib/Kconfig.debug
··· 3577 3577 3578 3578 If unsure, say N. 3579 3579 3580 + config RUST_INLINE_HELPERS 3581 + bool "Inline C helpers into Rust code (EXPERIMENTAL)" 3582 + depends on RUST && RUSTC_CLANG_LLVM_COMPATIBLE 3583 + depends on EXPERT 3584 + depends on ARM64 || X86_64 3585 + depends on !UML 3586 + help 3587 + Inlines C helpers into Rust code using Link Time Optimization. 3588 + 3589 + If this option is enabled, C helper functions declared in 3590 + rust/helpers/ are inlined into Rust code, which is helpful for 3591 + performance of Rust code. This requires a matching LLVM version for 3592 + Clang and rustc. 3593 + 3594 + If you are sure that you're using Clang and rustc with matching LLVM 3595 + versions, say Y. Otherwise say N. 3596 + 3580 3597 endmenu # "Rust" 3581 3598 3582 3599 endmenu # Kernel hacking
+27 -4
rust/Makefile
··· 6 6 obj-$(CONFIG_RUST) += core.o compiler_builtins.o ffi.o 7 7 always-$(CONFIG_RUST) += exports_core_generated.h 8 8 9 + ifdef CONFIG_RUST_INLINE_HELPERS 10 + always-$(CONFIG_RUST) += helpers/helpers.bc helpers/helpers_module.bc 11 + else 12 + obj-$(CONFIG_RUST) += helpers/helpers.o 13 + always-$(CONFIG_RUST) += exports_helpers_generated.h 14 + endif 9 15 # Missing prototypes are expected in the helpers since these are exported 10 16 # for Rust only, thus there is no header nor prototypes. 11 - obj-$(CONFIG_RUST) += helpers/helpers.o 12 17 CFLAGS_REMOVE_helpers/helpers.o = -Wmissing-prototypes -Wmissing-declarations 13 18 14 19 always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs 15 20 obj-$(CONFIG_RUST) += bindings.o pin_init.o kernel.o 16 - always-$(CONFIG_RUST) += exports_helpers_generated.h \ 17 - exports_bindings_generated.h exports_kernel_generated.h 21 + always-$(CONFIG_RUST) += exports_bindings_generated.h exports_kernel_generated.h 18 22 19 23 always-$(CONFIG_RUST) += uapi/uapi_generated.rs 20 24 obj-$(CONFIG_RUST) += uapi.o ··· 499 495 $(obj)/bindings/bindings_helpers_generated.rs: $(src)/helpers/helpers.c FORCE 500 496 $(call if_changed_dep,bindgen) 501 497 498 + quiet_cmd_rust_helper = HELPER $@ 499 + cmd_rust_helper = \ 500 + $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) \ 501 + -c -g0 $< -emit-llvm -o $@ 502 + 503 + $(obj)/helpers/helpers.bc: private part-of-builtin := y 504 + $(obj)/helpers/helpers_module.bc: private part-of-module := y 505 + $(obj)/helpers/helpers.bc $(obj)/helpers/helpers_module.bc: $(src)/helpers/helpers.c FORCE 506 + +$(call if_changed_dep,rust_helper) 507 + 502 508 rust_exports = $(NM) -p --defined-only $(1) | awk '$$2~/(T|R|D|B)/ && $$3!~/__(pfx|cfi|odr_asan)/ { printf $(2),$$3 }' 503 509 504 510 quiet_cmd_exports = EXPORTS $@ ··· 591 577 OBJTREE=$(abspath $(objtree)) \ 592 578 $(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \ 593 579 $(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) \ 594 - --emit=dep-info=$(depfile) --emit=obj=$@ \ 580 + --emit=dep-info=$(depfile) --emit=$(if $(link_helper),llvm-bc=$(patsubst %.o,%.bc,$@),obj=$@) \ 595 581 --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \ 596 582 --crate-type rlib -L$(objtree)/$(obj) \ 597 583 --crate-name $(patsubst %.o,%,$(notdir $@)) $< \ 598 584 --sysroot=/dev/null \ 599 585 -Zunstable-options \ 586 + $(if $(link_helper),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \ 587 + $(obj)/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \ 588 + $(CC) $(CLANG_FLAGS) $(KBUILD_CFLAGS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \ 589 + $(cmd_ld_single)) \ 600 590 $(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@) \ 601 591 $(cmd_objtool) 602 592 ··· 728 710 ifdef CONFIG_BUG 729 711 $(obj)/kernel.o: $(obj)/kernel/generated_arch_warn_asm.rs $(obj)/kernel/generated_arch_reachable_asm.rs 730 712 endif 713 + endif 714 + 715 + ifdef CONFIG_RUST_INLINE_HELPERS 716 + $(obj)/kernel.o: private link_helper = 1 717 + $(obj)/kernel.o: $(obj)/helpers/helpers.bc 731 718 endif 732 719 733 720 endif # CONFIG_RUST
+4 -1
rust/exports.c
··· 16 16 #define EXPORT_SYMBOL_RUST_GPL(sym) extern int sym; EXPORT_SYMBOL_GPL(sym) 17 17 18 18 #include "exports_core_generated.h" 19 - #include "exports_helpers_generated.h" 20 19 #include "exports_bindings_generated.h" 21 20 #include "exports_kernel_generated.h" 21 + 22 + #ifndef CONFIG_RUST_INLINE_HELPERS 23 + #include "exports_helpers_generated.h" 24 + #endif 22 25 23 26 // For modules using `rust/build_error.rs`. 24 27 #ifdef CONFIG_RUST_BUILD_ASSERT_ALLOW
+6 -1
scripts/Makefile.build
··· 346 346 # would not match each other. 347 347 348 348 quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@ 349 - cmd_rustc_o_rs = $(rust_common_cmd) --emit=obj=$@ $< $(cmd_objtool) 349 + cmd_rustc_o_rs = $(rust_common_cmd) --emit=$(if $(CONFIG_RUST_INLINE_HELPERS),llvm-bc=$(patsubst %.o,%.bc,$@),obj=$@) $< \ 350 + $(if $(CONFIG_RUST_INLINE_HELPERS),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \ 351 + $(objtree)/rust/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \ 352 + $(CC) $(CLANG_FLAGS) $(KBUILD_CFLAGS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@ \ 353 + $(cmd_ld_single)) \ 354 + $(cmd_objtool) 350 355 351 356 define rule_rustc_o_rs 352 357 $(call cmd_and_fixdep,rustc_o_rs)