Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'rust-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux
Pull Rust updates from Miguel Ojeda:
"Toolchain and infrastructure:
- Extract the 'pin-init' API from the 'kernel' crate and make it into
a standalone crate.
In order to do this, the contents are rearranged so that they can
easily be kept in sync with the version maintained out-of-tree that
other projects have started to use too (or plan to, like QEMU).
This will reduce the maintenance burden for Benno, who will now
have his own sub-tree, and will simplify future expected changes
like the move to use 'syn' to simplify the implementation.
- Add '#[test]'-like support based on KUnit.
We already had doctests support based on KUnit, which takes the
examples in our Rust documentation and runs them under KUnit.
Now, we are adding the beginning of the support for "normal" tests,
similar to those the '#[test]' tests in userspace Rust. For
instance:
#[kunit_tests(my_suite)]
mod tests {
#[test]
fn my_test() {
assert_eq!(1 + 1, 2);
}
}
Unlike with doctests, the 'assert*!'s do not map to the KUnit
assertion APIs yet.
- Check Rust signatures at compile time for functions called from C
by name.
In particular, introduce a new '#[export]' macro that can be placed
in the Rust function definition. It will ensure that the function
declaration on the C side matches the signature on the Rust
function:
#[export]
pub unsafe extern "C" fn my_function(a: u8, b: i32) -> usize {
// ...
}
The macro essentially forces the compiler to compare the types of
the actual Rust function and the 'bindgen'-processed C signature.
These cases are rare so far. In the future, we may consider
introducing another tool, 'cbindgen', to generate C headers
automatically. Even then, having these functions explicitly marked
may be a good idea anyway.
- Enable the 'raw_ref_op' Rust feature: it is already stable, and
allows us to use the new '&raw' syntax, avoiding a couple macros.
After everyone has migrated, we will disallow the macros.
- Pass the correct target to 'bindgen' on Usermode Linux.
- Fix 'rusttest' build in macOS.
'kernel' crate:
- New 'hrtimer' module: add support for setting up intrusive timers
without allocating when starting the timer. Add support for
'Pin<Box<_>>', 'Arc<_>', 'Pin<&_>' and 'Pin<&mut _>' as pointer
types for use with timer callbacks. Add support for setting clock
source and timer mode.
- New 'dma' module: add a simple DMA coherent allocator abstraction
and a test sample driver.
- 'list' module: make the linked list 'Cursor' point between
elements, rather than at an element, which is more convenient to us
and allows for cursors to empty lists; and document it with
examples of how to perform common operations with the provided
methods.
- 'str' module: implement a few traits for 'BStr' as well as the
'strip_prefix()' method.
- 'sync' module: add 'Arc::as_ptr'.
- 'alloc' module: add 'Box::into_pin'.
- 'error' module: extend the 'Result' documentation, including a few
examples on different ways of handling errors, a warning about
using methods that may panic, and links to external documentation.
'macros' crate:
- 'module' macro: add the 'authors' key to support multiple authors.
The original key will be kept until everyone has migrated.
Documentation:
- Add error handling sections.
MAINTAINERS:
- Add Danilo Krummrich as reviewer of the Rust "subsystem".
- Add 'RUST [PIN-INIT]' entry with Benno Lossin as maintainer. It has
its own sub-tree.
- Add sub-tree for 'RUST [ALLOC]'.
- Add 'DMA MAPPING HELPERS DEVICE DRIVER API [RUST]' entry with
Abdiel Janulgue as primary maintainer. It will go through the
sub-tree of the 'RUST [ALLOC]' entry.
- Add 'HIGH-RESOLUTION TIMERS [RUST]' entry with Andreas Hindborg as
maintainer. It has its own sub-tree.
And a few other cleanups and improvements"
* tag 'rust-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux: (71 commits)
rust: dma: add `Send` implementation for `CoherentAllocation`
rust: macros: fix `make rusttest` build on macOS
rust: block: refactor to use `&raw mut`
rust: enable `raw_ref_op` feature
rust: uaccess: name the correct function
rust: rbtree: fix comments referring to Box instead of KBox
rust: hrtimer: add maintainer entry
rust: hrtimer: add clocksource selection through `ClockId`
rust: hrtimer: add `HrTimerMode`
rust: hrtimer: implement `HrTimerPointer` for `Pin<Box<T>>`
rust: alloc: add `Box::into_pin`
rust: hrtimer: implement `UnsafeHrTimerPointer` for `Pin<&mut T>`
rust: hrtimer: implement `UnsafeHrTimerPointer` for `Pin<&T>`
rust: hrtimer: add `hrtimer::ScopedHrTimerPointer`
rust: hrtimer: add `UnsafeHrTimerPointer`
rust: hrtimer: allow timer restart from timer handler
rust: str: implement `strip_prefix` for `BStr`
rust: str: implement `AsRef<BStr>` for `[u8]` and `BStr`
rust: str: implement `Index` for `BStr`
rust: str: implement `PartialEq` for `BStr`
...
+6008
-1854
+8
Documentation/rust/coding-guidelines.rst
+8
Documentation/rust/coding-guidelines.rst
···
373
373
For more information about diagnostics in Rust, please see:
374
374
375
375
https://doc.rust-lang.org/stable/reference/attributes/diagnostics.html
376
+
377
+
Error handling
378
+
--------------
379
+
380
+
For some background and guidelines about Rust for Linux specific error handling,
381
+
please see:
382
+
383
+
https://rust.docs.kernel.org/kernel/error/type.Result.html#error-codes-in-c-and-rust
+7
Documentation/rust/testing.rst
+7
Documentation/rust/testing.rst
···
123
123
Thus, we presently simply print an error to the kernel log if an assertion
124
124
actually failed. Additionally, doctests are not run for nonpublic functions.
125
125
126
+
Since these tests are examples, i.e. they are part of the documentation, they
127
+
should generally be written like "real code". Thus, for example, instead of
128
+
using ``unwrap()`` or ``expect()``, use the ``?`` operator. For more background,
129
+
please see:
130
+
131
+
https://rust.docs.kernel.org/kernel/error/type.Result.html#error-codes-in-c-and-rust
132
+
126
133
The ``#[test]`` tests
127
134
---------------------
128
135
+44
MAINTAINERS
+44
MAINTAINERS
···
6981
6981
F: include/linux/swiotlb.h
6982
6982
F: kernel/dma/
6983
6983
6984
+
DMA MAPPING HELPERS DEVICE DRIVER API [RUST]
6985
+
M: Abdiel Janulgue <abdiel.janulgue@gmail.com>
6986
+
M: Danilo Krummrich <dakr@kernel.org>
6987
+
R: Daniel Almeida <daniel.almeida@collabora.com>
6988
+
R: Robin Murphy <robin.murphy@arm.com>
6989
+
R: Andreas Hindborg <a.hindborg@kernel.org>
6990
+
L: rust-for-linux@vger.kernel.org
6991
+
S: Supported
6992
+
W: https://rust-for-linux.com
6993
+
T: git https://github.com/Rust-for-Linux/linux.git alloc-next
6994
+
F: rust/kernel/dma.rs
6995
+
F: samples/rust/rust_dma.rs
6996
+
6984
6997
DMA-BUF HEAPS FRAMEWORK
6985
6998
M: Sumit Semwal <sumit.semwal@linaro.org>
6986
6999
R: Benjamin Gaignard <benjamin.gaignard@collabora.com>
···
10552
10539
F: kernel/time/timer_migration.*
10553
10540
F: tools/testing/selftests/timers/
10554
10541
10542
+
HIGH-RESOLUTION TIMERS [RUST]
10543
+
M: Andreas Hindborg <a.hindborg@kernel.org>
10544
+
R: Boqun Feng <boqun.feng@gmail.com>
10545
+
R: Frederic Weisbecker <frederic@kernel.org>
10546
+
R: Lyude Paul <lyude@redhat.com>
10547
+
R: Thomas Gleixner <tglx@linutronix.de>
10548
+
R: Anna-Maria Behnsen <anna-maria@linutronix.de>
10549
+
L: rust-for-linux@vger.kernel.org
10550
+
S: Supported
10551
+
W: https://rust-for-linux.com
10552
+
B: https://github.com/Rust-for-Linux/linux/issues
10553
+
T: git https://github.com/Rust-for-Linux/linux.git hrtimer-next
10554
+
F: rust/kernel/time/hrtimer.rs
10555
+
F: rust/kernel/time/hrtimer/
10556
+
10555
10557
HIGH-SPEED SCC DRIVER FOR AX.25
10556
10558
L: linux-hams@vger.kernel.org
10557
10559
S: Orphan
···
12930
12902
F: include/kunit/
12931
12903
F: lib/kunit/
12932
12904
F: rust/kernel/kunit.rs
12905
+
F: rust/macros/kunit.rs
12933
12906
F: scripts/rustdoc_test_*
12934
12907
F: tools/testing/kunit/
12935
12908
···
21022
20993
R: Andreas Hindborg <a.hindborg@kernel.org>
21023
20994
R: Alice Ryhl <aliceryhl@google.com>
21024
20995
R: Trevor Gross <tmgross@umich.edu>
20996
+
R: Danilo Krummrich <dakr@kernel.org>
21025
20997
L: rust-for-linux@vger.kernel.org
21026
20998
S: Supported
21027
20999
W: https://rust-for-linux.com
···
21043
21013
M: Danilo Krummrich <dakr@kernel.org>
21044
21014
L: rust-for-linux@vger.kernel.org
21045
21015
S: Maintained
21016
+
T: git https://github.com/Rust-for-Linux/linux.git alloc-next
21046
21017
F: rust/kernel/alloc.rs
21047
21018
F: rust/kernel/alloc/
21019
+
21020
+
RUST [PIN-INIT]
21021
+
M: Benno Lossin <benno.lossin@proton.me>
21022
+
L: rust-for-linux@vger.kernel.org
21023
+
S: Maintained
21024
+
W: https://rust-for-linux.com/pin-init
21025
+
B: https://github.com/Rust-for-Linux/pin-init/issues
21026
+
C: zulip://rust-for-linux.zulipchat.com
21027
+
P: rust/pin-init/CONTRIBUTING.md
21028
+
T: git https://github.com/Rust-for-Linux/linux.git pin-init-next
21029
+
F: rust/kernel/init.rs
21030
+
F: rust/pin-init/
21031
+
K: \bpin-init\b|pin_init\b|PinInit
21048
21032
21049
21033
RXRPC SOCKETS (AF_RXRPC)
21050
21034
M: David Howells <dhowells@redhat.com>
+1
-1
drivers/block/rnull.rs
+1
-1
drivers/block/rnull.rs
-5
drivers/gpu/drm/drm_panic.c
-5
drivers/gpu/drm/drm_panic.c
···
486
486
stream.workspace = NULL;
487
487
}
488
488
489
-
extern size_t drm_panic_qr_max_data_size(u8 version, size_t url_len);
490
-
491
-
extern u8 drm_panic_qr_generate(const char *url, u8 *data, size_t data_len, size_t data_size,
492
-
u8 *tmp, size_t tmp_size);
493
-
494
489
static int drm_panic_get_qr_code_url(u8 **qr_image)
495
490
{
496
491
struct kmsg_dump_iter iter;
+9
-4
drivers/gpu/drm/drm_panic_qr.rs
+9
-4
drivers/gpu/drm/drm_panic_qr.rs
···
27
27
//! * <https://github.com/erwanvivien/fast_qr>
28
28
//! * <https://github.com/bjguillot/qr>
29
29
30
-
use kernel::str::CStr;
30
+
use kernel::{prelude::*, str::CStr};
31
31
32
32
#[derive(Debug, Clone, Copy, PartialEq, Eq, Ord, PartialOrd)]
33
33
struct Version(usize);
···
891
891
/// * `tmp` must be valid for reading and writing for `tmp_size` bytes.
892
892
///
893
893
/// They must remain valid for the duration of the function call.
894
-
#[no_mangle]
894
+
#[export]
895
895
pub unsafe extern "C" fn drm_panic_qr_generate(
896
896
url: *const kernel::ffi::c_char,
897
897
data: *mut u8,
···
942
942
/// * If `url_len` > 0, remove the 2 segments header/length and also count the
943
943
/// conversion to numeric segments.
944
944
/// * If `url_len` = 0, only removes 3 bytes for 1 binary segment.
945
-
#[no_mangle]
946
-
pub extern "C" fn drm_panic_qr_max_data_size(version: u8, url_len: usize) -> usize {
945
+
///
946
+
/// # Safety
947
+
///
948
+
/// Always safe to call.
949
+
// Required to be unsafe due to the `#[export]` annotation.
950
+
#[export]
951
+
pub unsafe extern "C" fn drm_panic_qr_max_data_size(version: u8, url_len: usize) -> usize {
947
952
#[expect(clippy::manual_range_contains)]
948
953
if version < 1 || version > 40 {
949
954
return 0;
+1
-1
drivers/net/phy/ax88796b_rust.rs
+1
-1
drivers/net/phy/ax88796b_rust.rs
+1
-1
drivers/net/phy/qt2025.rs
+1
-1
drivers/net/phy/qt2025.rs
···
26
26
phy::DeviceId::new_with_driver::<PhyQT2025>(),
27
27
],
28
28
name: "qt2025_phy",
29
-
author: "FUJITA Tomonori <fujita.tomonori@gmail.com>",
29
+
authors: ["FUJITA Tomonori <fujita.tomonori@gmail.com>"],
30
30
description: "AMCC QT2025 PHY driver",
31
31
license: "GPL",
32
32
firmware: ["qt2025-2.0.3.3.fw"],
+7
include/drm/drm_panic.h
+7
include/drm/drm_panic.h
···
163
163
164
164
#endif
165
165
166
+
#if defined(CONFIG_DRM_PANIC_SCREEN_QR_CODE)
167
+
size_t drm_panic_qr_max_data_size(u8 version, size_t url_len);
168
+
169
+
u8 drm_panic_qr_generate(const char *url, u8 *data, size_t data_len, size_t data_size,
170
+
u8 *tmp, size_t tmp_size);
171
+
#endif
172
+
166
173
#endif /* __DRM_PANIC_H__ */
+3
include/linux/sprintf.h
+3
include/linux/sprintf.h
-3
lib/vsprintf.c
-3
lib/vsprintf.c
···
2291
2291
}
2292
2292
early_param("no_hash_pointers", no_hash_pointers_enable);
2293
2293
2294
-
/* Used for Rust formatting ('%pA'). */
2295
-
char *rust_fmt_argument(char *buf, char *end, void *ptr);
2296
-
2297
2294
/*
2298
2295
* Show a '%p' thing. A kernel extension is that the '%p' is followed
2299
2296
* by an extra set of alphanumeric characters that are extended format
+58
-18
rust/Makefile
+58
-18
rust/Makefile
···
12
12
CFLAGS_REMOVE_helpers/helpers.o = -Wmissing-prototypes -Wmissing-declarations
13
13
14
14
always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs
15
-
obj-$(CONFIG_RUST) += bindings.o kernel.o
15
+
obj-$(CONFIG_RUST) += bindings.o pin_init.o kernel.o
16
16
always-$(CONFIG_RUST) += exports_helpers_generated.h \
17
17
exports_bindings_generated.h exports_kernel_generated.h
18
18
···
41
41
libmacros_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name macros --crate-type proc-macro - </dev/null)
42
42
libmacros_extension := $(patsubst libmacros.%,%,$(libmacros_name))
43
43
44
-
always-$(CONFIG_RUST) += $(libmacros_name)
44
+
libpin_init_internal_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name pin_init_internal --crate-type proc-macro - </dev/null)
45
+
libpin_init_internal_extension := $(patsubst libpin_init_internal.%,%,$(libpin_init_internal_name))
46
+
47
+
always-$(CONFIG_RUST) += $(libmacros_name) $(libpin_init_internal_name)
45
48
46
49
# `$(rust_flags)` is passed in case the user added `--sysroot`.
47
50
rustc_sysroot := $(shell MAKEFLAGS= $(RUSTC) $(rust_flags) --print sysroot)
···
83
80
# command-like flags to solve the issue. Meanwhile, we use the non-custom case
84
81
# and then retouch the generated files.
85
82
rustdoc: rustdoc-core rustdoc-macros rustdoc-compiler_builtins \
86
-
rustdoc-kernel
83
+
rustdoc-kernel rustdoc-pin_init
87
84
$(Q)cp $(srctree)/Documentation/images/logo.svg $(rustdoc_output)/static.files/
88
85
$(Q)cp $(srctree)/Documentation/images/COPYING-logo $(rustdoc_output)/static.files/
89
86
$(Q)find $(rustdoc_output) -name '*.html' -type f -print0 | xargs -0 sed -Ei \
···
113
110
rustdoc-ffi: $(src)/ffi.rs rustdoc-core FORCE
114
111
+$(call if_changed,rustdoc)
115
112
116
-
rustdoc-kernel: private rustc_target_flags = --extern ffi \
113
+
rustdoc-pin_init_internal: private rustdoc_host = yes
114
+
rustdoc-pin_init_internal: private rustc_target_flags = --cfg kernel \
115
+
--extern proc_macro --crate-type proc-macro
116
+
rustdoc-pin_init_internal: $(src)/pin-init/internal/src/lib.rs FORCE
117
+
+$(call if_changed,rustdoc)
118
+
119
+
rustdoc-pin_init: private rustdoc_host = yes
120
+
rustdoc-pin_init: private rustc_target_flags = --extern pin_init_internal \
121
+
--extern macros --extern alloc --cfg kernel --cfg feature=\"alloc\"
122
+
rustdoc-pin_init: $(src)/pin-init/src/lib.rs rustdoc-pin_init_internal \
123
+
rustdoc-macros FORCE
124
+
+$(call if_changed,rustdoc)
125
+
126
+
rustdoc-kernel: private rustc_target_flags = --extern ffi --extern pin_init \
117
127
--extern build_error --extern macros \
118
128
--extern bindings --extern uapi
119
129
rustdoc-kernel: $(src)/kernel/lib.rs rustdoc-core rustdoc-ffi rustdoc-macros \
120
-
rustdoc-compiler_builtins $(obj)/$(libmacros_name) \
130
+
rustdoc-pin_init rustdoc-compiler_builtins $(obj)/$(libmacros_name) \
121
131
$(obj)/bindings.o FORCE
122
132
+$(call if_changed,rustdoc)
123
133
···
155
139
rusttestlib-macros: $(src)/macros/lib.rs FORCE
156
140
+$(call if_changed,rustc_test_library)
157
141
142
+
rusttestlib-pin_init_internal: private rustc_target_flags = --cfg kernel \
143
+
--extern proc_macro
144
+
rusttestlib-pin_init_internal: private rustc_test_library_proc = yes
145
+
rusttestlib-pin_init_internal: $(src)/pin-init/internal/src/lib.rs FORCE
146
+
+$(call if_changed,rustc_test_library)
147
+
148
+
rusttestlib-pin_init: private rustc_target_flags = --extern pin_init_internal \
149
+
--extern macros --cfg kernel
150
+
rusttestlib-pin_init: $(src)/pin-init/src/lib.rs rusttestlib-macros \
151
+
rusttestlib-pin_init_internal $(obj)/$(libpin_init_internal_name) FORCE
152
+
+$(call if_changed,rustc_test_library)
153
+
158
154
rusttestlib-kernel: private rustc_target_flags = --extern ffi \
159
-
--extern build_error --extern macros \
155
+
--extern build_error --extern macros --extern pin_init \
160
156
--extern bindings --extern uapi
161
-
rusttestlib-kernel: $(src)/kernel/lib.rs \
162
-
rusttestlib-bindings rusttestlib-uapi rusttestlib-build_error \
163
-
$(obj)/$(libmacros_name) $(obj)/bindings.o FORCE
157
+
rusttestlib-kernel: $(src)/kernel/lib.rs rusttestlib-bindings rusttestlib-uapi \
158
+
rusttestlib-build_error rusttestlib-pin_init $(obj)/$(libmacros_name) \
159
+
$(obj)/bindings.o FORCE
164
160
+$(call if_changed,rustc_test_library)
165
161
166
162
rusttestlib-bindings: private rustc_target_flags = --extern ffi
···
200
172
mkdir -p $(objtree)/$(obj)/test/doctests/kernel; \
201
173
OBJTREE=$(abspath $(objtree)) \
202
174
$(RUSTDOC) --test $(rust_flags) \
203
-
-L$(objtree)/$(obj) --extern ffi --extern kernel \
204
-
--extern build_error --extern macros \
175
+
-L$(objtree)/$(obj) --extern ffi --extern pin_init \
176
+
--extern kernel --extern build_error --extern macros \
205
177
--extern bindings --extern uapi \
206
178
--no-run --crate-name kernel -Zunstable-options \
207
179
--sysroot=/dev/null \
···
231
203
rusttest: rusttest-macros rusttest-kernel
232
204
233
205
rusttest-macros: private rustc_target_flags = --extern proc_macro \
234
-
--extern macros --extern kernel
206
+
--extern macros --extern kernel --extern pin_init
235
207
rusttest-macros: private rustdoc_test_target_flags = --crate-type proc-macro
236
208
rusttest-macros: $(src)/macros/lib.rs \
237
-
rusttestlib-macros rusttestlib-kernel FORCE
209
+
rusttestlib-macros rusttestlib-kernel rusttestlib-pin_init FORCE
238
210
+$(call if_changed,rustc_test)
239
211
+$(call if_changed,rustdoc_test)
240
212
241
-
rusttest-kernel: private rustc_target_flags = --extern ffi \
213
+
rusttest-kernel: private rustc_target_flags = --extern ffi --extern pin_init \
242
214
--extern build_error --extern macros --extern bindings --extern uapi
243
215
rusttest-kernel: $(src)/kernel/lib.rs rusttestlib-ffi rusttestlib-kernel \
244
216
rusttestlib-build_error rusttestlib-macros rusttestlib-bindings \
245
-
rusttestlib-uapi FORCE
217
+
rusttestlib-uapi rusttestlib-pin_init FORCE
246
218
+$(call if_changed,rustc_test)
247
219
248
220
ifdef CONFIG_CC_IS_CLANG
···
274
246
# Derived from `scripts/Makefile.clang`.
275
247
BINDGEN_TARGET_x86 := x86_64-linux-gnu
276
248
BINDGEN_TARGET_arm64 := aarch64-linux-gnu
249
+
BINDGEN_TARGET_um := $(BINDGEN_TARGET_$(SUBARCH))
277
250
BINDGEN_TARGET := $(BINDGEN_TARGET_$(SRCARCH))
278
251
279
252
# All warnings are inhibited since GCC builds are very experimental,
···
390
361
391
362
quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
392
363
cmd_rustc_procmacro = \
393
-
$(RUSTC_OR_CLIPPY) $(rust_common_flags) \
364
+
$(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \
394
365
-Clinker-flavor=gcc -Clinker=$(HOSTCC) \
395
366
-Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \
396
367
--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
···
399
370
400
371
# Procedural macros can only be used with the `rustc` that compiled it.
401
372
$(obj)/$(libmacros_name): $(src)/macros/lib.rs FORCE
373
+
+$(call if_changed_dep,rustc_procmacro)
374
+
375
+
$(obj)/$(libpin_init_internal_name): private rustc_target_flags = --cfg kernel
376
+
$(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs FORCE
402
377
+$(call if_changed_dep,rustc_procmacro)
403
378
404
379
quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L $@
···
484
451
$(obj)/compiler_builtins.o: $(src)/compiler_builtins.rs $(obj)/core.o FORCE
485
452
+$(call if_changed_rule,rustc_library)
486
453
454
+
$(obj)/pin_init.o: private skip_gendwarfksyms = 1
455
+
$(obj)/pin_init.o: private rustc_target_flags = --extern pin_init_internal \
456
+
--extern macros --cfg kernel
457
+
$(obj)/pin_init.o: $(src)/pin-init/src/lib.rs $(obj)/compiler_builtins.o \
458
+
$(obj)/$(libpin_init_internal_name) $(obj)/$(libmacros_name) FORCE
459
+
+$(call if_changed_rule,rustc_library)
460
+
487
461
$(obj)/build_error.o: private skip_gendwarfksyms = 1
488
462
$(obj)/build_error.o: $(src)/build_error.rs $(obj)/compiler_builtins.o FORCE
489
463
+$(call if_changed_rule,rustc_library)
···
513
473
$(obj)/uapi/uapi_generated.rs FORCE
514
474
+$(call if_changed_rule,rustc_library)
515
475
516
-
$(obj)/kernel.o: private rustc_target_flags = --extern ffi \
476
+
$(obj)/kernel.o: private rustc_target_flags = --extern ffi --extern pin_init \
517
477
--extern build_error --extern macros --extern bindings --extern uapi
518
-
$(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/build_error.o \
478
+
$(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/build_error.o $(obj)/pin_init.o \
519
479
$(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o FORCE
520
480
+$(call if_changed_rule,rustc_library)
521
481
+6
rust/bindings/bindings_helper.h
+6
rust/bindings/bindings_helper.h
···
13
13
#include <linux/cpumask.h>
14
14
#include <linux/cred.h>
15
15
#include <linux/device/faux.h>
16
+
#include <linux/dma-mapping.h>
16
17
#include <linux/errname.h>
17
18
#include <linux/ethtool.h>
18
19
#include <linux/file.h>
···
38
37
#include <linux/wait.h>
39
38
#include <linux/workqueue.h>
40
39
#include <trace/events/rust_sample.h>
40
+
41
+
#if defined(CONFIG_DRM_PANIC_SCREEN_QR_CODE)
42
+
// Used by `#[export]` in `drivers/gpu/drm/drm_panic_qr.rs`.
43
+
#include <drm/drm_panic.h>
44
+
#endif
41
45
42
46
/* `bindgen` gets confused at certain things. */
43
47
const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
+1
rust/kernel/alloc/allocator.rs
+1
rust/kernel/alloc/allocator.rs
···
80
80
/// This method has the same guarantees as `Allocator::realloc`. Additionally
81
81
/// - it accepts any pointer to a valid memory allocation allocated by this function.
82
82
/// - memory allocated by this function remains valid until it is passed to this function.
83
+
#[inline]
83
84
unsafe fn call(
84
85
&self,
85
86
ptr: Option<NonNull<u8>>,
+12
-1
rust/kernel/alloc/kbox.rs
+12
-1
rust/kernel/alloc/kbox.rs
···
15
15
use core::ptr::NonNull;
16
16
use core::result::Result;
17
17
18
-
use crate::init::{InPlaceInit, InPlaceWrite, Init, PinInit};
18
+
use crate::init::InPlaceInit;
19
19
use crate::types::ForeignOwnable;
20
+
use pin_init::{InPlaceWrite, Init, PinInit, ZeroableOption};
20
21
21
22
/// The kernel's [`Box`] type -- a heap allocation for a single value of type `T`.
22
23
///
···
99
98
/// # Ok::<(), Error>(())
100
99
/// ```
101
100
pub type KVBox<T> = Box<T, super::allocator::KVmalloc>;
101
+
102
+
// SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee:
103
+
// https://doc.rust-lang.org/stable/std/option/index.html#representation).
104
+
unsafe impl<T, A: Allocator> ZeroableOption for Box<T, A> {}
102
105
103
106
// SAFETY: `Box` is `Send` if `T` is `Send` because the `Box` owns a `T`.
104
107
unsafe impl<T, A> Send for Box<T, A>
···
248
243
A: 'static,
249
244
{
250
245
Ok(Self::new(x, flags)?.into())
246
+
}
247
+
248
+
/// Convert a [`Box<T,A>`] to a [`Pin<Box<T,A>>`]. If `T` does not implement
249
+
/// [`Unpin`], then `x` will be pinned in memory and can't be moved.
250
+
pub fn into_pin(this: Self) -> Pin<Self> {
251
+
this.into()
251
252
}
252
253
253
254
/// Forgets the contents (does not run the destructor), but keeps the allocation.
+2
-2
rust/kernel/block/mq/request.rs
+2
-2
rust/kernel/block/mq/request.rs
···
12
12
};
13
13
use core::{
14
14
marker::PhantomData,
15
-
ptr::{addr_of_mut, NonNull},
15
+
ptr::NonNull,
16
16
sync::atomic::{AtomicU64, Ordering},
17
17
};
18
18
···
187
187
pub(crate) unsafe fn refcount_ptr(this: *mut Self) -> *mut AtomicU64 {
188
188
// SAFETY: Because of the safety requirements of this function, the
189
189
// field projection is safe.
190
-
unsafe { addr_of_mut!((*this).refcount) }
190
+
unsafe { &raw mut (*this).refcount }
191
191
}
192
192
}
193
193
+2
-3
rust/kernel/block/mq/tag_set.rs
+2
-3
rust/kernel/block/mq/tag_set.rs
···
10
10
bindings,
11
11
block::mq::{operations::OperationsVTable, request::RequestDataWrapper, Operations},
12
12
error,
13
-
prelude::PinInit,
14
-
try_pin_init,
13
+
prelude::try_pin_init,
15
14
types::Opaque,
16
15
};
17
16
use core::{convert::TryInto, marker::PhantomData};
18
-
use macros::{pin_data, pinned_drop};
17
+
use pin_init::{pin_data, pinned_drop, PinInit};
19
18
20
19
/// A wrapper for the C `struct blk_mq_tag_set`.
21
20
///
+391
rust/kernel/dma.rs
+391
rust/kernel/dma.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
//! Direct memory access (DMA).
4
+
//!
5
+
//! C header: [`include/linux/dma-mapping.h`](srctree/include/linux/dma-mapping.h)
6
+
7
+
use crate::{
8
+
bindings, build_assert,
9
+
device::Device,
10
+
error::code::*,
11
+
error::Result,
12
+
transmute::{AsBytes, FromBytes},
13
+
types::ARef,
14
+
};
15
+
16
+
/// Possible attributes associated with a DMA mapping.
17
+
///
18
+
/// They can be combined with the operators `|`, `&`, and `!`.
19
+
///
20
+
/// Values can be used from the [`attrs`] module.
21
+
///
22
+
/// # Examples
23
+
///
24
+
/// ```
25
+
/// use kernel::device::Device;
26
+
/// use kernel::dma::{attrs::*, CoherentAllocation};
27
+
///
28
+
/// # fn test(dev: &Device) -> Result {
29
+
/// let attribs = DMA_ATTR_FORCE_CONTIGUOUS | DMA_ATTR_NO_WARN;
30
+
/// let c: CoherentAllocation<u64> =
31
+
/// CoherentAllocation::alloc_attrs(dev, 4, GFP_KERNEL, attribs)?;
32
+
/// # Ok::<(), Error>(()) }
33
+
/// ```
34
+
#[derive(Clone, Copy, PartialEq)]
35
+
#[repr(transparent)]
36
+
pub struct Attrs(u32);
37
+
38
+
impl Attrs {
39
+
/// Get the raw representation of this attribute.
40
+
pub(crate) fn as_raw(self) -> crate::ffi::c_ulong {
41
+
self.0 as _
42
+
}
43
+
44
+
/// Check whether `flags` is contained in `self`.
45
+
pub fn contains(self, flags: Attrs) -> bool {
46
+
(self & flags) == flags
47
+
}
48
+
}
49
+
50
+
impl core::ops::BitOr for Attrs {
51
+
type Output = Self;
52
+
fn bitor(self, rhs: Self) -> Self::Output {
53
+
Self(self.0 | rhs.0)
54
+
}
55
+
}
56
+
57
+
impl core::ops::BitAnd for Attrs {
58
+
type Output = Self;
59
+
fn bitand(self, rhs: Self) -> Self::Output {
60
+
Self(self.0 & rhs.0)
61
+
}
62
+
}
63
+
64
+
impl core::ops::Not for Attrs {
65
+
type Output = Self;
66
+
fn not(self) -> Self::Output {
67
+
Self(!self.0)
68
+
}
69
+
}
70
+
71
+
/// DMA mapping attributes.
72
+
pub mod attrs {
73
+
use super::Attrs;
74
+
75
+
/// Specifies that reads and writes to the mapping may be weakly ordered, that is that reads
76
+
/// and writes may pass each other.
77
+
pub const DMA_ATTR_WEAK_ORDERING: Attrs = Attrs(bindings::DMA_ATTR_WEAK_ORDERING);
78
+
79
+
/// Specifies that writes to the mapping may be buffered to improve performance.
80
+
pub const DMA_ATTR_WRITE_COMBINE: Attrs = Attrs(bindings::DMA_ATTR_WRITE_COMBINE);
81
+
82
+
/// Lets the platform to avoid creating a kernel virtual mapping for the allocated buffer.
83
+
pub const DMA_ATTR_NO_KERNEL_MAPPING: Attrs = Attrs(bindings::DMA_ATTR_NO_KERNEL_MAPPING);
84
+
85
+
/// Allows platform code to skip synchronization of the CPU cache for the given buffer assuming
86
+
/// that it has been already transferred to 'device' domain.
87
+
pub const DMA_ATTR_SKIP_CPU_SYNC: Attrs = Attrs(bindings::DMA_ATTR_SKIP_CPU_SYNC);
88
+
89
+
/// Forces contiguous allocation of the buffer in physical memory.
90
+
pub const DMA_ATTR_FORCE_CONTIGUOUS: Attrs = Attrs(bindings::DMA_ATTR_FORCE_CONTIGUOUS);
91
+
92
+
/// This is a hint to the DMA-mapping subsystem that it's probably not worth the time to try
93
+
/// to allocate memory to in a way that gives better TLB efficiency.
94
+
pub const DMA_ATTR_ALLOC_SINGLE_PAGES: Attrs = Attrs(bindings::DMA_ATTR_ALLOC_SINGLE_PAGES);
95
+
96
+
/// This tells the DMA-mapping subsystem to suppress allocation failure reports (similarly to
97
+
/// __GFP_NOWARN).
98
+
pub const DMA_ATTR_NO_WARN: Attrs = Attrs(bindings::DMA_ATTR_NO_WARN);
99
+
100
+
/// Used to indicate that the buffer is fully accessible at an elevated privilege level (and
101
+
/// ideally inaccessible or at least read-only at lesser-privileged levels).
102
+
pub const DMA_ATTR_PRIVILEGED: Attrs = Attrs(bindings::DMA_ATTR_PRIVILEGED);
103
+
}
104
+
105
+
/// An abstraction of the `dma_alloc_coherent` API.
106
+
///
107
+
/// This is an abstraction around the `dma_alloc_coherent` API which is used to allocate and map
108
+
/// large consistent DMA regions.
109
+
///
110
+
/// A [`CoherentAllocation`] instance contains a pointer to the allocated region (in the
111
+
/// processor's virtual address space) and the device address which can be given to the device
112
+
/// as the DMA address base of the region. The region is released once [`CoherentAllocation`]
113
+
/// is dropped.
114
+
///
115
+
/// # Invariants
116
+
///
117
+
/// For the lifetime of an instance of [`CoherentAllocation`], the `cpu_addr` is a valid pointer
118
+
/// to an allocated region of consistent memory and `dma_handle` is the DMA address base of
119
+
/// the region.
120
+
// TODO
121
+
//
122
+
// DMA allocations potentially carry device resources (e.g.IOMMU mappings), hence for soundness
123
+
// reasons DMA allocation would need to be embedded in a `Devres` container, in order to ensure
124
+
// that device resources can never survive device unbind.
125
+
//
126
+
// However, it is neither desirable nor necessary to protect the allocated memory of the DMA
127
+
// allocation from surviving device unbind; it would require RCU read side critical sections to
128
+
// access the memory, which may require subsequent unnecessary copies.
129
+
//
130
+
// Hence, find a way to revoke the device resources of a `CoherentAllocation`, but not the
131
+
// entire `CoherentAllocation` including the allocated memory itself.
132
+
pub struct CoherentAllocation<T: AsBytes + FromBytes> {
133
+
dev: ARef<Device>,
134
+
dma_handle: bindings::dma_addr_t,
135
+
count: usize,
136
+
cpu_addr: *mut T,
137
+
dma_attrs: Attrs,
138
+
}
139
+
140
+
impl<T: AsBytes + FromBytes> CoherentAllocation<T> {
141
+
/// Allocates a region of `size_of::<T> * count` of consistent memory.
142
+
///
143
+
/// # Examples
144
+
///
145
+
/// ```
146
+
/// use kernel::device::Device;
147
+
/// use kernel::dma::{attrs::*, CoherentAllocation};
148
+
///
149
+
/// # fn test(dev: &Device) -> Result {
150
+
/// let c: CoherentAllocation<u64> =
151
+
/// CoherentAllocation::alloc_attrs(dev, 4, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
152
+
/// # Ok::<(), Error>(()) }
153
+
/// ```
154
+
pub fn alloc_attrs(
155
+
dev: &Device,
156
+
count: usize,
157
+
gfp_flags: kernel::alloc::Flags,
158
+
dma_attrs: Attrs,
159
+
) -> Result<CoherentAllocation<T>> {
160
+
build_assert!(
161
+
core::mem::size_of::<T>() > 0,
162
+
"It doesn't make sense for the allocated type to be a ZST"
163
+
);
164
+
165
+
let size = count
166
+
.checked_mul(core::mem::size_of::<T>())
167
+
.ok_or(EOVERFLOW)?;
168
+
let mut dma_handle = 0;
169
+
// SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
170
+
let ret = unsafe {
171
+
bindings::dma_alloc_attrs(
172
+
dev.as_raw(),
173
+
size,
174
+
&mut dma_handle,
175
+
gfp_flags.as_raw(),
176
+
dma_attrs.as_raw(),
177
+
)
178
+
};
179
+
if ret.is_null() {
180
+
return Err(ENOMEM);
181
+
}
182
+
// INVARIANT: We just successfully allocated a coherent region which is accessible for
183
+
// `count` elements, hence the cpu address is valid. We also hold a refcounted reference
184
+
// to the device.
185
+
Ok(Self {
186
+
dev: dev.into(),
187
+
dma_handle,
188
+
count,
189
+
cpu_addr: ret as *mut T,
190
+
dma_attrs,
191
+
})
192
+
}
193
+
194
+
/// Performs the same functionality as [`CoherentAllocation::alloc_attrs`], except the
195
+
/// `dma_attrs` is 0 by default.
196
+
pub fn alloc_coherent(
197
+
dev: &Device,
198
+
count: usize,
199
+
gfp_flags: kernel::alloc::Flags,
200
+
) -> Result<CoherentAllocation<T>> {
201
+
CoherentAllocation::alloc_attrs(dev, count, gfp_flags, Attrs(0))
202
+
}
203
+
204
+
/// Returns the base address to the allocated region in the CPU's virtual address space.
205
+
pub fn start_ptr(&self) -> *const T {
206
+
self.cpu_addr
207
+
}
208
+
209
+
/// Returns the base address to the allocated region in the CPU's virtual address space as
210
+
/// a mutable pointer.
211
+
pub fn start_ptr_mut(&mut self) -> *mut T {
212
+
self.cpu_addr
213
+
}
214
+
215
+
/// Returns a DMA handle which may given to the device as the DMA address base of
216
+
/// the region.
217
+
pub fn dma_handle(&self) -> bindings::dma_addr_t {
218
+
self.dma_handle
219
+
}
220
+
221
+
/// Returns a pointer to an element from the region with bounds checking. `offset` is in
222
+
/// units of `T`, not the number of bytes.
223
+
///
224
+
/// Public but hidden since it should only be used from [`dma_read`] and [`dma_write`] macros.
225
+
#[doc(hidden)]
226
+
pub fn item_from_index(&self, offset: usize) -> Result<*mut T> {
227
+
if offset >= self.count {
228
+
return Err(EINVAL);
229
+
}
230
+
// SAFETY:
231
+
// - The pointer is valid due to type invariant on `CoherentAllocation`
232
+
// and we've just checked that the range and index is within bounds.
233
+
// - `offset` can't overflow since it is smaller than `self.count` and we've checked
234
+
// that `self.count` won't overflow early in the constructor.
235
+
Ok(unsafe { self.cpu_addr.add(offset) })
236
+
}
237
+
238
+
/// Reads the value of `field` and ensures that its type is [`FromBytes`].
239
+
///
240
+
/// # Safety
241
+
///
242
+
/// This must be called from the [`dma_read`] macro which ensures that the `field` pointer is
243
+
/// validated beforehand.
244
+
///
245
+
/// Public but hidden since it should only be used from [`dma_read`] macro.
246
+
#[doc(hidden)]
247
+
pub unsafe fn field_read<F: FromBytes>(&self, field: *const F) -> F {
248
+
// SAFETY:
249
+
// - By the safety requirements field is valid.
250
+
// - Using read_volatile() here is not sound as per the usual rules, the usage here is
251
+
// a special exception with the following notes in place. When dealing with a potential
252
+
// race from a hardware or code outside kernel (e.g. user-space program), we need that
253
+
// read on a valid memory is not UB. Currently read_volatile() is used for this, and the
254
+
// rationale behind is that it should generate the same code as READ_ONCE() which the
255
+
// kernel already relies on to avoid UB on data races. Note that the usage of
256
+
// read_volatile() is limited to this particular case, it cannot be used to prevent
257
+
// the UB caused by racing between two kernel functions nor do they provide atomicity.
258
+
unsafe { field.read_volatile() }
259
+
}
260
+
261
+
/// Writes a value to `field` and ensures that its type is [`AsBytes`].
262
+
///
263
+
/// # Safety
264
+
///
265
+
/// This must be called from the [`dma_write`] macro which ensures that the `field` pointer is
266
+
/// validated beforehand.
267
+
///
268
+
/// Public but hidden since it should only be used from [`dma_write`] macro.
269
+
#[doc(hidden)]
270
+
pub unsafe fn field_write<F: AsBytes>(&self, field: *mut F, val: F) {
271
+
// SAFETY:
272
+
// - By the safety requirements field is valid.
273
+
// - Using write_volatile() here is not sound as per the usual rules, the usage here is
274
+
// a special exception with the following notes in place. When dealing with a potential
275
+
// race from a hardware or code outside kernel (e.g. user-space program), we need that
276
+
// write on a valid memory is not UB. Currently write_volatile() is used for this, and the
277
+
// rationale behind is that it should generate the same code as WRITE_ONCE() which the
278
+
// kernel already relies on to avoid UB on data races. Note that the usage of
279
+
// write_volatile() is limited to this particular case, it cannot be used to prevent
280
+
// the UB caused by racing between two kernel functions nor do they provide atomicity.
281
+
unsafe { field.write_volatile(val) }
282
+
}
283
+
}
284
+
285
+
/// Note that the device configured to do DMA must be halted before this object is dropped.
286
+
impl<T: AsBytes + FromBytes> Drop for CoherentAllocation<T> {
287
+
fn drop(&mut self) {
288
+
let size = self.count * core::mem::size_of::<T>();
289
+
// SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
290
+
// The cpu address, and the dma handle are valid due to the type invariants on
291
+
// `CoherentAllocation`.
292
+
unsafe {
293
+
bindings::dma_free_attrs(
294
+
self.dev.as_raw(),
295
+
size,
296
+
self.cpu_addr as _,
297
+
self.dma_handle,
298
+
self.dma_attrs.as_raw(),
299
+
)
300
+
}
301
+
}
302
+
}
303
+
304
+
// SAFETY: It is safe to send a `CoherentAllocation` to another thread if `T`
305
+
// can be sent to another thread.
306
+
unsafe impl<T: AsBytes + FromBytes + Send> Send for CoherentAllocation<T> {}
307
+
308
+
/// Reads a field of an item from an allocated region of structs.
309
+
///
310
+
/// # Examples
311
+
///
312
+
/// ```
313
+
/// use kernel::device::Device;
314
+
/// use kernel::dma::{attrs::*, CoherentAllocation};
315
+
///
316
+
/// struct MyStruct { field: u32, }
317
+
///
318
+
/// // SAFETY: All bit patterns are acceptable values for `MyStruct`.
319
+
/// unsafe impl kernel::transmute::FromBytes for MyStruct{};
320
+
/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
321
+
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
322
+
///
323
+
/// # fn test(alloc: &kernel::dma::CoherentAllocation<MyStruct>) -> Result {
324
+
/// let whole = kernel::dma_read!(alloc[2]);
325
+
/// let field = kernel::dma_read!(alloc[1].field);
326
+
/// # Ok::<(), Error>(()) }
327
+
/// ```
328
+
#[macro_export]
329
+
macro_rules! dma_read {
330
+
($dma:expr, $idx: expr, $($field:tt)*) => {{
331
+
let item = $crate::dma::CoherentAllocation::item_from_index(&$dma, $idx)?;
332
+
// SAFETY: `item_from_index` ensures that `item` is always a valid pointer and can be
333
+
// dereferenced. The compiler also further validates the expression on whether `field`
334
+
// is a member of `item` when expanded by the macro.
335
+
unsafe {
336
+
let ptr_field = ::core::ptr::addr_of!((*item) $($field)*);
337
+
$crate::dma::CoherentAllocation::field_read(&$dma, ptr_field)
338
+
}
339
+
}};
340
+
($dma:ident [ $idx:expr ] $($field:tt)* ) => {
341
+
$crate::dma_read!($dma, $idx, $($field)*);
342
+
};
343
+
($($dma:ident).* [ $idx:expr ] $($field:tt)* ) => {
344
+
$crate::dma_read!($($dma).*, $idx, $($field)*);
345
+
};
346
+
}
347
+
348
+
/// Writes to a field of an item from an allocated region of structs.
349
+
///
350
+
/// # Examples
351
+
///
352
+
/// ```
353
+
/// use kernel::device::Device;
354
+
/// use kernel::dma::{attrs::*, CoherentAllocation};
355
+
///
356
+
/// struct MyStruct { member: u32, }
357
+
///
358
+
/// // SAFETY: All bit patterns are acceptable values for `MyStruct`.
359
+
/// unsafe impl kernel::transmute::FromBytes for MyStruct{};
360
+
/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
361
+
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
362
+
///
363
+
/// # fn test(alloc: &kernel::dma::CoherentAllocation<MyStruct>) -> Result {
364
+
/// kernel::dma_write!(alloc[2].member = 0xf);
365
+
/// kernel::dma_write!(alloc[1] = MyStruct { member: 0xf });
366
+
/// # Ok::<(), Error>(()) }
367
+
/// ```
368
+
#[macro_export]
369
+
macro_rules! dma_write {
370
+
($dma:ident [ $idx:expr ] $($field:tt)*) => {{
371
+
$crate::dma_write!($dma, $idx, $($field)*);
372
+
}};
373
+
($($dma:ident).* [ $idx:expr ] $($field:tt)* ) => {{
374
+
$crate::dma_write!($($dma).*, $idx, $($field)*);
375
+
}};
376
+
($dma:expr, $idx: expr, = $val:expr) => {
377
+
let item = $crate::dma::CoherentAllocation::item_from_index(&$dma, $idx)?;
378
+
// SAFETY: `item_from_index` ensures that `item` is always a valid item.
379
+
unsafe { $crate::dma::CoherentAllocation::field_write(&$dma, item, $val) }
380
+
};
381
+
($dma:expr, $idx: expr, $(.$field:ident)* = $val:expr) => {
382
+
let item = $crate::dma::CoherentAllocation::item_from_index(&$dma, $idx)?;
383
+
// SAFETY: `item_from_index` ensures that `item` is always a valid pointer and can be
384
+
// dereferenced. The compiler also further validates the expression on whether `field`
385
+
// is a member of `item` when expanded by the macro.
386
+
unsafe {
387
+
let ptr_field = ::core::ptr::addr_of_mut!((*item) $(.$field)*);
388
+
$crate::dma::CoherentAllocation::field_write(&$dma, ptr_field, $val)
389
+
}
390
+
};
391
+
}
+3
-3
rust/kernel/driver.rs
+3
-3
rust/kernel/driver.rs
···
6
6
//! register using the [`Registration`] class.
7
7
8
8
use crate::error::{Error, Result};
9
-
use crate::{device, init::PinInit, of, str::CStr, try_pin_init, types::Opaque, ThisModule};
9
+
use crate::{device, of, str::CStr, try_pin_init, types::Opaque, ThisModule};
10
10
use core::pin::Pin;
11
-
use macros::{pin_data, pinned_drop};
11
+
use pin_init::{pin_data, pinned_drop, PinInit};
12
12
13
13
/// The [`RegistrationOps`] trait serves as generic interface for subsystems (e.g., PCI, Platform,
14
14
/// Amba, etc.) to provide the corresponding subsystem specific implementation to register /
···
114
114
impl $crate::InPlaceModule for DriverModule {
115
115
fn init(
116
116
module: &'static $crate::ThisModule
117
-
) -> impl $crate::init::PinInit<Self, $crate::error::Error> {
117
+
) -> impl ::pin_init::PinInit<Self, $crate::error::Error> {
118
118
$crate::try_pin_init!(Self {
119
119
_driver <- $crate::driver::Registration::new(
120
120
<Self as $crate::ModuleMetadata>::NAME,
+123
-1
rust/kernel/error.rs
+123
-1
rust/kernel/error.rs
···
64
64
declare_err!(EPIPE, "Broken pipe.");
65
65
declare_err!(EDOM, "Math argument out of domain of func.");
66
66
declare_err!(ERANGE, "Math result not representable.");
67
+
declare_err!(EOVERFLOW, "Value too large for defined data type.");
67
68
declare_err!(ERESTARTSYS, "Restart the system call.");
68
69
declare_err!(ERESTARTNOINTR, "System call was interrupted by a signal and will be restarted.");
69
70
declare_err!(ERESTARTNOHAND, "Restart if no handler.");
···
249
248
/// [`Error`] as its error type.
250
249
///
251
250
/// Note that even if a function does not return anything when it succeeds,
252
-
/// it should still be modeled as returning a `Result` rather than
251
+
/// it should still be modeled as returning a [`Result`] rather than
253
252
/// just an [`Error`].
253
+
///
254
+
/// Calling a function that returns [`Result`] forces the caller to handle
255
+
/// the returned [`Result`].
256
+
///
257
+
/// This can be done "manually" by using [`match`]. Using [`match`] to decode
258
+
/// the [`Result`] is similar to C where all the return value decoding and the
259
+
/// error handling is done explicitly by writing handling code for each
260
+
/// error to cover. Using [`match`] the error and success handling can be
261
+
/// implemented in all detail as required. For example (inspired by
262
+
/// [`samples/rust/rust_minimal.rs`]):
263
+
///
264
+
/// ```
265
+
/// # #[allow(clippy::single_match)]
266
+
/// fn example() -> Result {
267
+
/// let mut numbers = KVec::new();
268
+
///
269
+
/// match numbers.push(72, GFP_KERNEL) {
270
+
/// Err(e) => {
271
+
/// pr_err!("Error pushing 72: {e:?}");
272
+
/// return Err(e.into());
273
+
/// }
274
+
/// // Do nothing, continue.
275
+
/// Ok(()) => (),
276
+
/// }
277
+
///
278
+
/// match numbers.push(108, GFP_KERNEL) {
279
+
/// Err(e) => {
280
+
/// pr_err!("Error pushing 108: {e:?}");
281
+
/// return Err(e.into());
282
+
/// }
283
+
/// // Do nothing, continue.
284
+
/// Ok(()) => (),
285
+
/// }
286
+
///
287
+
/// match numbers.push(200, GFP_KERNEL) {
288
+
/// Err(e) => {
289
+
/// pr_err!("Error pushing 200: {e:?}");
290
+
/// return Err(e.into());
291
+
/// }
292
+
/// // Do nothing, continue.
293
+
/// Ok(()) => (),
294
+
/// }
295
+
///
296
+
/// Ok(())
297
+
/// }
298
+
/// # example()?;
299
+
/// # Ok::<(), Error>(())
300
+
/// ```
301
+
///
302
+
/// An alternative to be more concise is the [`if let`] syntax:
303
+
///
304
+
/// ```
305
+
/// fn example() -> Result {
306
+
/// let mut numbers = KVec::new();
307
+
///
308
+
/// if let Err(e) = numbers.push(72, GFP_KERNEL) {
309
+
/// pr_err!("Error pushing 72: {e:?}");
310
+
/// return Err(e.into());
311
+
/// }
312
+
///
313
+
/// if let Err(e) = numbers.push(108, GFP_KERNEL) {
314
+
/// pr_err!("Error pushing 108: {e:?}");
315
+
/// return Err(e.into());
316
+
/// }
317
+
///
318
+
/// if let Err(e) = numbers.push(200, GFP_KERNEL) {
319
+
/// pr_err!("Error pushing 200: {e:?}");
320
+
/// return Err(e.into());
321
+
/// }
322
+
///
323
+
/// Ok(())
324
+
/// }
325
+
/// # example()?;
326
+
/// # Ok::<(), Error>(())
327
+
/// ```
328
+
///
329
+
/// Instead of these verbose [`match`]/[`if let`], the [`?`] operator can
330
+
/// be used to handle the [`Result`]. Using the [`?`] operator is often
331
+
/// the best choice to handle [`Result`] in a non-verbose way as done in
332
+
/// [`samples/rust/rust_minimal.rs`]:
333
+
///
334
+
/// ```
335
+
/// fn example() -> Result {
336
+
/// let mut numbers = KVec::new();
337
+
///
338
+
/// numbers.push(72, GFP_KERNEL)?;
339
+
/// numbers.push(108, GFP_KERNEL)?;
340
+
/// numbers.push(200, GFP_KERNEL)?;
341
+
///
342
+
/// Ok(())
343
+
/// }
344
+
/// # example()?;
345
+
/// # Ok::<(), Error>(())
346
+
/// ```
347
+
///
348
+
/// Another possibility is to call [`unwrap()`](Result::unwrap) or
349
+
/// [`expect()`](Result::expect). However, use of these functions is
350
+
/// *heavily discouraged* in the kernel because they trigger a Rust
351
+
/// [`panic!`] if an error happens, which may destabilize the system or
352
+
/// entirely break it as a result -- just like the C [`BUG()`] macro.
353
+
/// Please see the documentation for the C macro [`BUG()`] for guidance
354
+
/// on when to use these functions.
355
+
///
356
+
/// Alternatively, depending on the use case, using [`unwrap_or()`],
357
+
/// [`unwrap_or_else()`], [`unwrap_or_default()`] or [`unwrap_unchecked()`]
358
+
/// might be an option, as well.
359
+
///
360
+
/// For even more details, please see the [Rust documentation].
361
+
///
362
+
/// [`match`]: https://doc.rust-lang.org/reference/expressions/match-expr.html
363
+
/// [`samples/rust/rust_minimal.rs`]: srctree/samples/rust/rust_minimal.rs
364
+
/// [`if let`]: https://doc.rust-lang.org/reference/expressions/if-expr.html#if-let-expressions
365
+
/// [`?`]: https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator
366
+
/// [`unwrap()`]: Result::unwrap
367
+
/// [`expect()`]: Result::expect
368
+
/// [`BUG()`]: https://docs.kernel.org/process/deprecated.html#bug-and-bug-on
369
+
/// [`unwrap_or()`]: Result::unwrap_or
370
+
/// [`unwrap_or_else()`]: Result::unwrap_or_else
371
+
/// [`unwrap_or_default()`]: Result::unwrap_or_default
372
+
/// [`unwrap_unchecked()`]: Result::unwrap_unchecked
373
+
/// [Rust documentation]: https://doc.rust-lang.org/book/ch09-02-recoverable-errors-with-result.html
254
374
pub type Result<T = (), E = Error> = core::result::Result<T, E>;
255
375
256
376
/// Converts an integer as returned by a C kernel function to an error if it's negative, and
+2
-2
rust/kernel/fs/file.rs
+2
-2
rust/kernel/fs/file.rs
···
267
267
/// # Safety
268
268
///
269
269
/// * The caller must ensure that `ptr` points at a valid file and that the file's refcount is
270
-
/// positive for the duration of 'a.
270
+
/// positive for the duration of `'a`.
271
271
/// * The caller must ensure that if there is an active call to `fdget_pos` that did not take
272
272
/// the `f_pos_lock` mutex, then that call is on the current thread.
273
273
#[inline]
···
341
341
/// # Safety
342
342
///
343
343
/// * The caller must ensure that `ptr` points at a valid file and that the file's refcount is
344
-
/// positive for the duration of 'a.
344
+
/// positive for the duration of `'a`.
345
345
/// * The caller must ensure that if there are active `fdget_pos` calls on this file, then they
346
346
/// took the `f_pos_lock` mutex.
347
347
#[inline]
+155
-1306
rust/kernel/init.rs
+155
-1306
rust/kernel/init.rs
···
1
-
// SPDX-License-Identifier: Apache-2.0 OR MIT
1
+
// SPDX-License-Identifier: GPL-2.0
2
2
3
-
//! API to safely and fallibly initialize pinned `struct`s using in-place constructors.
4
-
//!
5
-
//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
6
-
//! overflow.
3
+
//! Extensions to the [`pin-init`] crate.
7
4
//!
8
5
//! Most `struct`s from the [`sync`] module need to be pinned, because they contain self-referential
9
6
//! `struct`s from C. [Pinning][pinning] is Rust's way of ensuring data does not move.
10
7
//!
11
-
//! # Overview
8
+
//! The [`pin-init`] crate is the way such structs are initialized on the Rust side. Please refer
9
+
//! to its documentation to better understand how to use it. Additionally, there are many examples
10
+
//! throughout the kernel, such as the types from the [`sync`] module. And the ones presented
11
+
//! below.
12
12
//!
13
-
//! To initialize a `struct` with an in-place constructor you will need two things:
14
-
//! - an in-place constructor,
15
-
//! - a memory location that can hold your `struct` (this can be the [stack], an [`Arc<T>`],
16
-
//! [`UniqueArc<T>`], [`KBox<T>`] or any other smart pointer that implements [`InPlaceInit`]).
13
+
//! [`sync`]: crate::sync
14
+
//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
15
+
//! [`pin-init`]: https://rust.docs.kernel.org/pin_init/
17
16
//!
18
-
//! To get an in-place constructor there are generally three options:
19
-
//! - directly creating an in-place constructor using the [`pin_init!`] macro,
20
-
//! - a custom function/macro returning an in-place constructor provided by someone else,
21
-
//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
17
+
//! # [`Opaque<T>`]
22
18
//!
23
-
//! Aside from pinned initialization, this API also supports in-place construction without pinning,
24
-
//! the macros/types/functions are generally named like the pinned variants without the `pin`
25
-
//! prefix.
19
+
//! For the special case where initializing a field is a single FFI-function call that cannot fail,
20
+
//! there exist the helper function [`Opaque::ffi_init`]. This function initialize a single
21
+
//! [`Opaque<T>`] field by just delegating to the supplied closure. You can use these in
22
+
//! combination with [`pin_init!`].
23
+
//!
24
+
//! [`Opaque<T>`]: crate::types::Opaque
25
+
//! [`Opaque::ffi_init`]: crate::types::Opaque::ffi_init
26
+
//! [`pin_init!`]: pin_init::pin_init
26
27
//!
27
28
//! # Examples
28
29
//!
29
-
//! ## Using the [`pin_init!`] macro
30
+
//! ## General Examples
30
31
//!
31
-
//! If you want to use [`PinInit`], then you will have to annotate your `struct` with
32
-
//! `#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
33
-
//! [structurally pinned fields]. After doing this, you can then create an in-place constructor via
34
-
//! [`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
35
-
//! that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
32
+
//! ```rust,ignore
33
+
//! # #![allow(clippy::disallowed_names)]
34
+
//! use kernel::types::Opaque;
35
+
//! use pin_init::pin_init_from_closure;
36
36
//!
37
-
//! ```rust
38
-
//! # #![expect(clippy::disallowed_names)]
39
-
//! use kernel::sync::{new_mutex, Mutex};
40
-
//! # use core::pin::Pin;
37
+
//! // assume we have some `raw_foo` type in C:
38
+
//! #[repr(C)]
39
+
//! struct RawFoo([u8; 16]);
40
+
//! extern {
41
+
//! fn init_foo(_: *mut RawFoo);
42
+
//! }
43
+
//!
41
44
//! #[pin_data]
42
45
//! struct Foo {
43
46
//! #[pin]
44
-
//! a: Mutex<usize>,
45
-
//! b: u32,
47
+
//! raw: Opaque<RawFoo>,
48
+
//! }
49
+
//!
50
+
//! impl Foo {
51
+
//! fn setup(self: Pin<&mut Self>) {
52
+
//! pr_info!("Setting up foo\n");
53
+
//! }
46
54
//! }
47
55
//!
48
56
//! let foo = pin_init!(Foo {
49
-
//! a <- new_mutex!(42, "Foo::a"),
50
-
//! b: 24,
57
+
//! raw <- unsafe {
58
+
//! Opaque::ffi_init(|s| {
59
+
//! // note that this cannot fail.
60
+
//! init_foo(s);
61
+
//! })
62
+
//! },
63
+
//! }).pin_chain(|foo| {
64
+
//! foo.setup();
65
+
//! Ok(())
51
66
//! });
52
67
//! ```
53
68
//!
54
-
//! `foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
55
-
//! (or just the stack) to actually initialize a `Foo`:
56
-
//!
57
-
//! ```rust
58
-
//! # #![expect(clippy::disallowed_names)]
59
-
//! # use kernel::sync::{new_mutex, Mutex};
60
-
//! # use core::pin::Pin;
61
-
//! # #[pin_data]
62
-
//! # struct Foo {
63
-
//! # #[pin]
64
-
//! # a: Mutex<usize>,
65
-
//! # b: u32,
66
-
//! # }
67
-
//! # let foo = pin_init!(Foo {
68
-
//! # a <- new_mutex!(42, "Foo::a"),
69
-
//! # b: 24,
70
-
//! # });
71
-
//! let foo: Result<Pin<KBox<Foo>>> = KBox::pin_init(foo, GFP_KERNEL);
72
-
//! ```
73
-
//!
74
-
//! For more information see the [`pin_init!`] macro.
75
-
//!
76
-
//! ## Using a custom function/macro that returns an initializer
77
-
//!
78
-
//! Many types from the kernel supply a function/macro that returns an initializer, because the
79
-
//! above method only works for types where you can access the fields.
80
-
//!
81
-
//! ```rust
82
-
//! # use kernel::sync::{new_mutex, Arc, Mutex};
83
-
//! let mtx: Result<Arc<Mutex<usize>>> =
84
-
//! Arc::pin_init(new_mutex!(42, "example::mtx"), GFP_KERNEL);
85
-
//! ```
86
-
//!
87
-
//! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
88
-
//!
89
-
//! ```rust
90
-
//! # use kernel::{sync::Mutex, new_mutex, init::PinInit, try_pin_init};
91
-
//! #[pin_data]
92
-
//! struct DriverData {
93
-
//! #[pin]
94
-
//! status: Mutex<i32>,
95
-
//! buffer: KBox<[u8; 1_000_000]>,
96
-
//! }
97
-
//!
98
-
//! impl DriverData {
99
-
//! fn new() -> impl PinInit<Self, Error> {
100
-
//! try_pin_init!(Self {
101
-
//! status <- new_mutex!(0, "DriverData::status"),
102
-
//! buffer: KBox::init(kernel::init::zeroed(), GFP_KERNEL)?,
103
-
//! })
104
-
//! }
105
-
//! }
106
-
//! ```
107
-
//!
108
-
//! ## Manual creation of an initializer
109
-
//!
110
-
//! Often when working with primitives the previous approaches are not sufficient. That is where
111
-
//! [`pin_init_from_closure()`] comes in. This `unsafe` function allows you to create a
112
-
//! [`impl PinInit<T, E>`] directly from a closure. Of course you have to ensure that the closure
113
-
//! actually does the initialization in the correct way. Here are the things to look out for
114
-
//! (we are calling the parameter to the closure `slot`):
115
-
//! - when the closure returns `Ok(())`, then it has completed the initialization successfully, so
116
-
//! `slot` now contains a valid bit pattern for the type `T`,
117
-
//! - when the closure returns `Err(e)`, then the caller may deallocate the memory at `slot`, so
118
-
//! you need to take care to clean up anything if your initialization fails mid-way,
119
-
//! - you may assume that `slot` will stay pinned even after the closure returns until `drop` of
120
-
//! `slot` gets called.
121
-
//!
122
-
//! ```rust
123
-
//! # #![expect(unreachable_pub, clippy::disallowed_names)]
124
-
//! use kernel::{init, types::Opaque};
69
+
//! ```rust,ignore
70
+
//! # #![allow(unreachable_pub, clippy::disallowed_names)]
71
+
//! use kernel::{prelude::*, types::Opaque};
125
72
//! use core::{ptr::addr_of_mut, marker::PhantomPinned, pin::Pin};
126
73
//! # mod bindings {
127
-
//! # #![expect(non_camel_case_types)]
128
-
//! # #![expect(clippy::missing_safety_doc)]
74
+
//! # #![allow(non_camel_case_types)]
129
75
//! # pub struct foo;
130
76
//! # pub unsafe fn init_foo(_ptr: *mut foo) {}
131
77
//! # pub unsafe fn destroy_foo(_ptr: *mut foo) {}
···
79
133
//! # }
80
134
//! # // `Error::from_errno` is `pub(crate)` in the `kernel` crate, thus provide a workaround.
81
135
//! # trait FromErrno {
82
-
//! # fn from_errno(errno: kernel::ffi::c_int) -> Error {
136
+
//! # fn from_errno(errno: core::ffi::c_int) -> Error {
83
137
//! # // Dummy error that can be constructed outside the `kernel` crate.
84
138
//! # Error::from(core::fmt::Error)
85
139
//! # }
···
103
157
//! // enabled `foo`,
104
158
//! // - when it returns `Err(e)`, then it has cleaned up before
105
159
//! unsafe {
106
-
//! init::pin_init_from_closure(move |slot: *mut Self| {
160
+
//! pin_init::pin_init_from_closure(move |slot: *mut Self| {
107
161
//! // `slot` contains uninit memory, avoid creating a reference.
108
162
//! let foo = addr_of_mut!((*slot).foo);
109
163
//!
···
133
187
//! }
134
188
//! }
135
189
//! ```
136
-
//!
137
-
//! For the special case where initializing a field is a single FFI-function call that cannot fail,
138
-
//! there exist the helper function [`Opaque::ffi_init`]. This function initialize a single
139
-
//! [`Opaque`] field by just delegating to the supplied closure. You can use these in combination
140
-
//! with [`pin_init!`].
141
-
//!
142
-
//! For more information on how to use [`pin_init_from_closure()`], take a look at the uses inside
143
-
//! the `kernel` crate. The [`sync`] module is a good starting point.
144
-
//!
145
-
//! [`sync`]: kernel::sync
146
-
//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
147
-
//! [structurally pinned fields]:
148
-
//! https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
149
-
//! [stack]: crate::stack_pin_init
150
-
//! [`Arc<T>`]: crate::sync::Arc
151
-
//! [`impl PinInit<Foo>`]: PinInit
152
-
//! [`impl PinInit<T, E>`]: PinInit
153
-
//! [`impl Init<T, E>`]: Init
154
-
//! [`Opaque`]: kernel::types::Opaque
155
-
//! [`Opaque::ffi_init`]: kernel::types::Opaque::ffi_init
156
-
//! [`pin_data`]: ::macros::pin_data
157
-
//! [`pin_init!`]: crate::pin_init!
158
190
159
191
use crate::{
160
-
alloc::{AllocError, Flags, KBox},
192
+
alloc::{AllocError, Flags},
161
193
error::{self, Error},
162
-
sync::Arc,
163
-
sync::UniqueArc,
164
-
types::{Opaque, ScopeGuard},
165
194
};
166
-
use core::{
167
-
cell::UnsafeCell,
168
-
convert::Infallible,
169
-
marker::PhantomData,
170
-
mem::MaybeUninit,
171
-
num::*,
172
-
pin::Pin,
173
-
ptr::{self, NonNull},
174
-
};
175
-
176
-
#[doc(hidden)]
177
-
pub mod __internal;
178
-
#[doc(hidden)]
179
-
pub mod macros;
180
-
181
-
/// Initialize and pin a type directly on the stack.
182
-
///
183
-
/// # Examples
184
-
///
185
-
/// ```rust
186
-
/// # #![expect(clippy::disallowed_names)]
187
-
/// # use kernel::{init, macros::pin_data, pin_init, stack_pin_init, init::*, sync::Mutex, new_mutex};
188
-
/// # use core::pin::Pin;
189
-
/// #[pin_data]
190
-
/// struct Foo {
191
-
/// #[pin]
192
-
/// a: Mutex<usize>,
193
-
/// b: Bar,
194
-
/// }
195
-
///
196
-
/// #[pin_data]
197
-
/// struct Bar {
198
-
/// x: u32,
199
-
/// }
200
-
///
201
-
/// stack_pin_init!(let foo = pin_init!(Foo {
202
-
/// a <- new_mutex!(42),
203
-
/// b: Bar {
204
-
/// x: 64,
205
-
/// },
206
-
/// }));
207
-
/// let foo: Pin<&mut Foo> = foo;
208
-
/// pr_info!("a: {}\n", &*foo.a.lock());
209
-
/// ```
210
-
///
211
-
/// # Syntax
212
-
///
213
-
/// A normal `let` binding with optional type annotation. The expression is expected to implement
214
-
/// [`PinInit`]/[`Init`] with the error type [`Infallible`]. If you want to use a different error
215
-
/// type, then use [`stack_try_pin_init!`].
216
-
///
217
-
/// [`stack_try_pin_init!`]: crate::stack_try_pin_init!
218
-
#[macro_export]
219
-
macro_rules! stack_pin_init {
220
-
(let $var:ident $(: $t:ty)? = $val:expr) => {
221
-
let val = $val;
222
-
let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
223
-
let mut $var = match $crate::init::__internal::StackInit::init($var, val) {
224
-
Ok(res) => res,
225
-
Err(x) => {
226
-
let x: ::core::convert::Infallible = x;
227
-
match x {}
228
-
}
229
-
};
230
-
};
231
-
}
232
-
233
-
/// Initialize and pin a type directly on the stack.
234
-
///
235
-
/// # Examples
236
-
///
237
-
/// ```rust,ignore
238
-
/// # #![expect(clippy::disallowed_names)]
239
-
/// # use kernel::{
240
-
/// # init,
241
-
/// # pin_init,
242
-
/// # stack_try_pin_init,
243
-
/// # init::*,
244
-
/// # sync::Mutex,
245
-
/// # new_mutex,
246
-
/// # alloc::AllocError,
247
-
/// # };
248
-
/// # use macros::pin_data;
249
-
/// # use core::pin::Pin;
250
-
/// #[pin_data]
251
-
/// struct Foo {
252
-
/// #[pin]
253
-
/// a: Mutex<usize>,
254
-
/// b: KBox<Bar>,
255
-
/// }
256
-
///
257
-
/// struct Bar {
258
-
/// x: u32,
259
-
/// }
260
-
///
261
-
/// stack_try_pin_init!(let foo: Result<Pin<&mut Foo>, AllocError> = pin_init!(Foo {
262
-
/// a <- new_mutex!(42),
263
-
/// b: KBox::new(Bar {
264
-
/// x: 64,
265
-
/// }, GFP_KERNEL)?,
266
-
/// }));
267
-
/// let foo = foo.unwrap();
268
-
/// pr_info!("a: {}\n", &*foo.a.lock());
269
-
/// ```
270
-
///
271
-
/// ```rust,ignore
272
-
/// # #![expect(clippy::disallowed_names)]
273
-
/// # use kernel::{
274
-
/// # init,
275
-
/// # pin_init,
276
-
/// # stack_try_pin_init,
277
-
/// # init::*,
278
-
/// # sync::Mutex,
279
-
/// # new_mutex,
280
-
/// # alloc::AllocError,
281
-
/// # };
282
-
/// # use macros::pin_data;
283
-
/// # use core::pin::Pin;
284
-
/// #[pin_data]
285
-
/// struct Foo {
286
-
/// #[pin]
287
-
/// a: Mutex<usize>,
288
-
/// b: KBox<Bar>,
289
-
/// }
290
-
///
291
-
/// struct Bar {
292
-
/// x: u32,
293
-
/// }
294
-
///
295
-
/// stack_try_pin_init!(let foo: Pin<&mut Foo> =? pin_init!(Foo {
296
-
/// a <- new_mutex!(42),
297
-
/// b: KBox::new(Bar {
298
-
/// x: 64,
299
-
/// }, GFP_KERNEL)?,
300
-
/// }));
301
-
/// pr_info!("a: {}\n", &*foo.a.lock());
302
-
/// # Ok::<_, AllocError>(())
303
-
/// ```
304
-
///
305
-
/// # Syntax
306
-
///
307
-
/// A normal `let` binding with optional type annotation. The expression is expected to implement
308
-
/// [`PinInit`]/[`Init`]. This macro assigns a result to the given variable, adding a `?` after the
309
-
/// `=` will propagate this error.
310
-
#[macro_export]
311
-
macro_rules! stack_try_pin_init {
312
-
(let $var:ident $(: $t:ty)? = $val:expr) => {
313
-
let val = $val;
314
-
let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
315
-
let mut $var = $crate::init::__internal::StackInit::init($var, val);
316
-
};
317
-
(let $var:ident $(: $t:ty)? =? $val:expr) => {
318
-
let val = $val;
319
-
let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
320
-
let mut $var = $crate::init::__internal::StackInit::init($var, val)?;
321
-
};
322
-
}
323
-
324
-
/// Construct an in-place, pinned initializer for `struct`s.
325
-
///
326
-
/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
327
-
/// [`try_pin_init!`].
328
-
///
329
-
/// The syntax is almost identical to that of a normal `struct` initializer:
330
-
///
331
-
/// ```rust
332
-
/// # use kernel::{init, pin_init, macros::pin_data, init::*};
333
-
/// # use core::pin::Pin;
334
-
/// #[pin_data]
335
-
/// struct Foo {
336
-
/// a: usize,
337
-
/// b: Bar,
338
-
/// }
339
-
///
340
-
/// #[pin_data]
341
-
/// struct Bar {
342
-
/// x: u32,
343
-
/// }
344
-
///
345
-
/// # fn demo() -> impl PinInit<Foo> {
346
-
/// let a = 42;
347
-
///
348
-
/// let initializer = pin_init!(Foo {
349
-
/// a,
350
-
/// b: Bar {
351
-
/// x: 64,
352
-
/// },
353
-
/// });
354
-
/// # initializer }
355
-
/// # KBox::pin_init(demo(), GFP_KERNEL).unwrap();
356
-
/// ```
357
-
///
358
-
/// Arbitrary Rust expressions can be used to set the value of a variable.
359
-
///
360
-
/// The fields are initialized in the order that they appear in the initializer. So it is possible
361
-
/// to read already initialized fields using raw pointers.
362
-
///
363
-
/// IMPORTANT: You are not allowed to create references to fields of the struct inside of the
364
-
/// initializer.
365
-
///
366
-
/// # Init-functions
367
-
///
368
-
/// When working with this API it is often desired to let others construct your types without
369
-
/// giving access to all fields. This is where you would normally write a plain function `new`
370
-
/// that would return a new instance of your type. With this API that is also possible.
371
-
/// However, there are a few extra things to keep in mind.
372
-
///
373
-
/// To create an initializer function, simply declare it like this:
374
-
///
375
-
/// ```rust
376
-
/// # use kernel::{init, pin_init, init::*};
377
-
/// # use core::pin::Pin;
378
-
/// # #[pin_data]
379
-
/// # struct Foo {
380
-
/// # a: usize,
381
-
/// # b: Bar,
382
-
/// # }
383
-
/// # #[pin_data]
384
-
/// # struct Bar {
385
-
/// # x: u32,
386
-
/// # }
387
-
/// impl Foo {
388
-
/// fn new() -> impl PinInit<Self> {
389
-
/// pin_init!(Self {
390
-
/// a: 42,
391
-
/// b: Bar {
392
-
/// x: 64,
393
-
/// },
394
-
/// })
395
-
/// }
396
-
/// }
397
-
/// ```
398
-
///
399
-
/// Users of `Foo` can now create it like this:
400
-
///
401
-
/// ```rust
402
-
/// # #![expect(clippy::disallowed_names)]
403
-
/// # use kernel::{init, pin_init, macros::pin_data, init::*};
404
-
/// # use core::pin::Pin;
405
-
/// # #[pin_data]
406
-
/// # struct Foo {
407
-
/// # a: usize,
408
-
/// # b: Bar,
409
-
/// # }
410
-
/// # #[pin_data]
411
-
/// # struct Bar {
412
-
/// # x: u32,
413
-
/// # }
414
-
/// # impl Foo {
415
-
/// # fn new() -> impl PinInit<Self> {
416
-
/// # pin_init!(Self {
417
-
/// # a: 42,
418
-
/// # b: Bar {
419
-
/// # x: 64,
420
-
/// # },
421
-
/// # })
422
-
/// # }
423
-
/// # }
424
-
/// let foo = KBox::pin_init(Foo::new(), GFP_KERNEL);
425
-
/// ```
426
-
///
427
-
/// They can also easily embed it into their own `struct`s:
428
-
///
429
-
/// ```rust
430
-
/// # use kernel::{init, pin_init, macros::pin_data, init::*};
431
-
/// # use core::pin::Pin;
432
-
/// # #[pin_data]
433
-
/// # struct Foo {
434
-
/// # a: usize,
435
-
/// # b: Bar,
436
-
/// # }
437
-
/// # #[pin_data]
438
-
/// # struct Bar {
439
-
/// # x: u32,
440
-
/// # }
441
-
/// # impl Foo {
442
-
/// # fn new() -> impl PinInit<Self> {
443
-
/// # pin_init!(Self {
444
-
/// # a: 42,
445
-
/// # b: Bar {
446
-
/// # x: 64,
447
-
/// # },
448
-
/// # })
449
-
/// # }
450
-
/// # }
451
-
/// #[pin_data]
452
-
/// struct FooContainer {
453
-
/// #[pin]
454
-
/// foo1: Foo,
455
-
/// #[pin]
456
-
/// foo2: Foo,
457
-
/// other: u32,
458
-
/// }
459
-
///
460
-
/// impl FooContainer {
461
-
/// fn new(other: u32) -> impl PinInit<Self> {
462
-
/// pin_init!(Self {
463
-
/// foo1 <- Foo::new(),
464
-
/// foo2 <- Foo::new(),
465
-
/// other,
466
-
/// })
467
-
/// }
468
-
/// }
469
-
/// ```
470
-
///
471
-
/// Here we see that when using `pin_init!` with `PinInit`, one needs to write `<-` instead of `:`.
472
-
/// This signifies that the given field is initialized in-place. As with `struct` initializers, just
473
-
/// writing the field (in this case `other`) without `:` or `<-` means `other: other,`.
474
-
///
475
-
/// # Syntax
476
-
///
477
-
/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
478
-
/// the following modifications is expected:
479
-
/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
480
-
/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
481
-
/// pointer named `this` inside of the initializer.
482
-
/// - Using struct update syntax one can place `..Zeroable::zeroed()` at the very end of the
483
-
/// struct, this initializes every field with 0 and then runs all initializers specified in the
484
-
/// body. This can only be done if [`Zeroable`] is implemented for the struct.
485
-
///
486
-
/// For instance:
487
-
///
488
-
/// ```rust
489
-
/// # use kernel::{macros::{Zeroable, pin_data}, pin_init};
490
-
/// # use core::{ptr::addr_of_mut, marker::PhantomPinned};
491
-
/// #[pin_data]
492
-
/// #[derive(Zeroable)]
493
-
/// struct Buf {
494
-
/// // `ptr` points into `buf`.
495
-
/// ptr: *mut u8,
496
-
/// buf: [u8; 64],
497
-
/// #[pin]
498
-
/// pin: PhantomPinned,
499
-
/// }
500
-
/// pin_init!(&this in Buf {
501
-
/// buf: [0; 64],
502
-
/// // SAFETY: TODO.
503
-
/// ptr: unsafe { addr_of_mut!((*this.as_ptr()).buf).cast() },
504
-
/// pin: PhantomPinned,
505
-
/// });
506
-
/// pin_init!(Buf {
507
-
/// buf: [1; 64],
508
-
/// ..Zeroable::zeroed()
509
-
/// });
510
-
/// ```
511
-
///
512
-
/// [`try_pin_init!`]: kernel::try_pin_init
513
-
/// [`NonNull<Self>`]: core::ptr::NonNull
514
-
// For a detailed example of how this macro works, see the module documentation of the hidden
515
-
// module `__internal` inside of `init/__internal.rs`.
516
-
#[macro_export]
517
-
macro_rules! pin_init {
518
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
519
-
$($fields:tt)*
520
-
}) => {
521
-
$crate::__init_internal!(
522
-
@this($($this)?),
523
-
@typ($t $(::<$($generics),*>)?),
524
-
@fields($($fields)*),
525
-
@error(::core::convert::Infallible),
526
-
@data(PinData, use_data),
527
-
@has_data(HasPinData, __pin_data),
528
-
@construct_closure(pin_init_from_closure),
529
-
@munch_fields($($fields)*),
530
-
)
531
-
};
532
-
}
533
-
534
-
/// Construct an in-place, fallible pinned initializer for `struct`s.
535
-
///
536
-
/// If the initialization can complete without error (or [`Infallible`]), then use [`pin_init!`].
537
-
///
538
-
/// You can use the `?` operator or use `return Err(err)` inside the initializer to stop
539
-
/// initialization and return the error.
540
-
///
541
-
/// IMPORTANT: if you have `unsafe` code inside of the initializer you have to ensure that when
542
-
/// initialization fails, the memory can be safely deallocated without any further modifications.
543
-
///
544
-
/// This macro defaults the error to [`Error`].
545
-
///
546
-
/// The syntax is identical to [`pin_init!`] with the following exception: you can append `? $type`
547
-
/// after the `struct` initializer to specify the error type you want to use.
548
-
///
549
-
/// # Examples
550
-
///
551
-
/// ```rust
552
-
/// use kernel::{init::{self, PinInit}, error::Error};
553
-
/// #[pin_data]
554
-
/// struct BigBuf {
555
-
/// big: KBox<[u8; 1024 * 1024 * 1024]>,
556
-
/// small: [u8; 1024 * 1024],
557
-
/// ptr: *mut u8,
558
-
/// }
559
-
///
560
-
/// impl BigBuf {
561
-
/// fn new() -> impl PinInit<Self, Error> {
562
-
/// try_pin_init!(Self {
563
-
/// big: KBox::init(init::zeroed(), GFP_KERNEL)?,
564
-
/// small: [0; 1024 * 1024],
565
-
/// ptr: core::ptr::null_mut(),
566
-
/// }? Error)
567
-
/// }
568
-
/// }
569
-
/// ```
570
-
// For a detailed example of how this macro works, see the module documentation of the hidden
571
-
// module `__internal` inside of `init/__internal.rs`.
572
-
#[macro_export]
573
-
macro_rules! try_pin_init {
574
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
575
-
$($fields:tt)*
576
-
}) => {
577
-
$crate::__init_internal!(
578
-
@this($($this)?),
579
-
@typ($t $(::<$($generics),*>)? ),
580
-
@fields($($fields)*),
581
-
@error($crate::error::Error),
582
-
@data(PinData, use_data),
583
-
@has_data(HasPinData, __pin_data),
584
-
@construct_closure(pin_init_from_closure),
585
-
@munch_fields($($fields)*),
586
-
)
587
-
};
588
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
589
-
$($fields:tt)*
590
-
}? $err:ty) => {
591
-
$crate::__init_internal!(
592
-
@this($($this)?),
593
-
@typ($t $(::<$($generics),*>)? ),
594
-
@fields($($fields)*),
595
-
@error($err),
596
-
@data(PinData, use_data),
597
-
@has_data(HasPinData, __pin_data),
598
-
@construct_closure(pin_init_from_closure),
599
-
@munch_fields($($fields)*),
600
-
)
601
-
};
602
-
}
603
-
604
-
/// Construct an in-place initializer for `struct`s.
605
-
///
606
-
/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
607
-
/// [`try_init!`].
608
-
///
609
-
/// The syntax is identical to [`pin_init!`] and its safety caveats also apply:
610
-
/// - `unsafe` code must guarantee either full initialization or return an error and allow
611
-
/// deallocation of the memory.
612
-
/// - the fields are initialized in the order given in the initializer.
613
-
/// - no references to fields are allowed to be created inside of the initializer.
614
-
///
615
-
/// This initializer is for initializing data in-place that might later be moved. If you want to
616
-
/// pin-initialize, use [`pin_init!`].
617
-
///
618
-
/// [`try_init!`]: crate::try_init!
619
-
// For a detailed example of how this macro works, see the module documentation of the hidden
620
-
// module `__internal` inside of `init/__internal.rs`.
621
-
#[macro_export]
622
-
macro_rules! init {
623
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
624
-
$($fields:tt)*
625
-
}) => {
626
-
$crate::__init_internal!(
627
-
@this($($this)?),
628
-
@typ($t $(::<$($generics),*>)?),
629
-
@fields($($fields)*),
630
-
@error(::core::convert::Infallible),
631
-
@data(InitData, /*no use_data*/),
632
-
@has_data(HasInitData, __init_data),
633
-
@construct_closure(init_from_closure),
634
-
@munch_fields($($fields)*),
635
-
)
636
-
}
637
-
}
638
-
639
-
/// Construct an in-place fallible initializer for `struct`s.
640
-
///
641
-
/// This macro defaults the error to [`Error`]. If you need [`Infallible`], then use
642
-
/// [`init!`].
643
-
///
644
-
/// The syntax is identical to [`try_pin_init!`]. If you want to specify a custom error,
645
-
/// append `? $type` after the `struct` initializer.
646
-
/// The safety caveats from [`try_pin_init!`] also apply:
647
-
/// - `unsafe` code must guarantee either full initialization or return an error and allow
648
-
/// deallocation of the memory.
649
-
/// - the fields are initialized in the order given in the initializer.
650
-
/// - no references to fields are allowed to be created inside of the initializer.
651
-
///
652
-
/// # Examples
653
-
///
654
-
/// ```rust
655
-
/// use kernel::{alloc::KBox, init::{PinInit, zeroed}, error::Error};
656
-
/// struct BigBuf {
657
-
/// big: KBox<[u8; 1024 * 1024 * 1024]>,
658
-
/// small: [u8; 1024 * 1024],
659
-
/// }
660
-
///
661
-
/// impl BigBuf {
662
-
/// fn new() -> impl Init<Self, Error> {
663
-
/// try_init!(Self {
664
-
/// big: KBox::init(zeroed(), GFP_KERNEL)?,
665
-
/// small: [0; 1024 * 1024],
666
-
/// }? Error)
667
-
/// }
668
-
/// }
669
-
/// ```
670
-
// For a detailed example of how this macro works, see the module documentation of the hidden
671
-
// module `__internal` inside of `init/__internal.rs`.
672
-
#[macro_export]
673
-
macro_rules! try_init {
674
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
675
-
$($fields:tt)*
676
-
}) => {
677
-
$crate::__init_internal!(
678
-
@this($($this)?),
679
-
@typ($t $(::<$($generics),*>)?),
680
-
@fields($($fields)*),
681
-
@error($crate::error::Error),
682
-
@data(InitData, /*no use_data*/),
683
-
@has_data(HasInitData, __init_data),
684
-
@construct_closure(init_from_closure),
685
-
@munch_fields($($fields)*),
686
-
)
687
-
};
688
-
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
689
-
$($fields:tt)*
690
-
}? $err:ty) => {
691
-
$crate::__init_internal!(
692
-
@this($($this)?),
693
-
@typ($t $(::<$($generics),*>)?),
694
-
@fields($($fields)*),
695
-
@error($err),
696
-
@data(InitData, /*no use_data*/),
697
-
@has_data(HasInitData, __init_data),
698
-
@construct_closure(init_from_closure),
699
-
@munch_fields($($fields)*),
700
-
)
701
-
};
702
-
}
703
-
704
-
/// Asserts that a field on a struct using `#[pin_data]` is marked with `#[pin]` ie. that it is
705
-
/// structurally pinned.
706
-
///
707
-
/// # Example
708
-
///
709
-
/// This will succeed:
710
-
/// ```
711
-
/// use kernel::assert_pinned;
712
-
/// #[pin_data]
713
-
/// struct MyStruct {
714
-
/// #[pin]
715
-
/// some_field: u64,
716
-
/// }
717
-
///
718
-
/// assert_pinned!(MyStruct, some_field, u64);
719
-
/// ```
720
-
///
721
-
/// This will fail:
722
-
// TODO: replace with `compile_fail` when supported.
723
-
/// ```ignore
724
-
/// use kernel::assert_pinned;
725
-
/// #[pin_data]
726
-
/// struct MyStruct {
727
-
/// some_field: u64,
728
-
/// }
729
-
///
730
-
/// assert_pinned!(MyStruct, some_field, u64);
731
-
/// ```
732
-
///
733
-
/// Some uses of the macro may trigger the `can't use generic parameters from outer item` error. To
734
-
/// work around this, you may pass the `inline` parameter to the macro. The `inline` parameter can
735
-
/// only be used when the macro is invoked from a function body.
736
-
/// ```
737
-
/// use kernel::assert_pinned;
738
-
/// #[pin_data]
739
-
/// struct Foo<T> {
740
-
/// #[pin]
741
-
/// elem: T,
742
-
/// }
743
-
///
744
-
/// impl<T> Foo<T> {
745
-
/// fn project(self: Pin<&mut Self>) -> Pin<&mut T> {
746
-
/// assert_pinned!(Foo<T>, elem, T, inline);
747
-
///
748
-
/// // SAFETY: The field is structurally pinned.
749
-
/// unsafe { self.map_unchecked_mut(|me| &mut me.elem) }
750
-
/// }
751
-
/// }
752
-
/// ```
753
-
#[macro_export]
754
-
macro_rules! assert_pinned {
755
-
($ty:ty, $field:ident, $field_ty:ty, inline) => {
756
-
let _ = move |ptr: *mut $field_ty| {
757
-
// SAFETY: This code is unreachable.
758
-
let data = unsafe { <$ty as $crate::init::__internal::HasPinData>::__pin_data() };
759
-
let init = $crate::init::__internal::AlwaysFail::<$field_ty>::new();
760
-
// SAFETY: This code is unreachable.
761
-
unsafe { data.$field(ptr, init) }.ok();
762
-
};
763
-
};
764
-
765
-
($ty:ty, $field:ident, $field_ty:ty) => {
766
-
const _: () = {
767
-
$crate::assert_pinned!($ty, $field, $field_ty, inline);
768
-
};
769
-
};
770
-
}
771
-
772
-
/// A pin-initializer for the type `T`.
773
-
///
774
-
/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
775
-
/// be [`KBox<T>`], [`Arc<T>`], [`UniqueArc<T>`] or even the stack (see [`stack_pin_init!`]). Use
776
-
/// the [`InPlaceInit::pin_init`] function of a smart pointer like [`Arc<T>`] on this.
777
-
///
778
-
/// Also see the [module description](self).
779
-
///
780
-
/// # Safety
781
-
///
782
-
/// When implementing this trait you will need to take great care. Also there are probably very few
783
-
/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
784
-
///
785
-
/// The [`PinInit::__pinned_init`] function:
786
-
/// - returns `Ok(())` if it initialized every field of `slot`,
787
-
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
788
-
/// - `slot` can be deallocated without UB occurring,
789
-
/// - `slot` does not need to be dropped,
790
-
/// - `slot` is not partially initialized.
791
-
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
792
-
///
793
-
/// [`Arc<T>`]: crate::sync::Arc
794
-
/// [`Arc::pin_init`]: crate::sync::Arc::pin_init
795
-
#[must_use = "An initializer must be used in order to create its value."]
796
-
pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
797
-
/// Initializes `slot`.
798
-
///
799
-
/// # Safety
800
-
///
801
-
/// - `slot` is a valid pointer to uninitialized memory.
802
-
/// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
803
-
/// deallocate.
804
-
/// - `slot` will not move until it is dropped, i.e. it will be pinned.
805
-
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E>;
806
-
807
-
/// First initializes the value using `self` then calls the function `f` with the initialized
808
-
/// value.
809
-
///
810
-
/// If `f` returns an error the value is dropped and the initializer will forward the error.
811
-
///
812
-
/// # Examples
813
-
///
814
-
/// ```rust
815
-
/// # #![expect(clippy::disallowed_names)]
816
-
/// use kernel::{types::Opaque, init::pin_init_from_closure};
817
-
/// #[repr(C)]
818
-
/// struct RawFoo([u8; 16]);
819
-
/// extern "C" {
820
-
/// fn init_foo(_: *mut RawFoo);
821
-
/// }
822
-
///
823
-
/// #[pin_data]
824
-
/// struct Foo {
825
-
/// #[pin]
826
-
/// raw: Opaque<RawFoo>,
827
-
/// }
828
-
///
829
-
/// impl Foo {
830
-
/// fn setup(self: Pin<&mut Self>) {
831
-
/// pr_info!("Setting up foo\n");
832
-
/// }
833
-
/// }
834
-
///
835
-
/// let foo = pin_init!(Foo {
836
-
/// // SAFETY: TODO.
837
-
/// raw <- unsafe {
838
-
/// Opaque::ffi_init(|s| {
839
-
/// init_foo(s);
840
-
/// })
841
-
/// },
842
-
/// }).pin_chain(|foo| {
843
-
/// foo.setup();
844
-
/// Ok(())
845
-
/// });
846
-
/// ```
847
-
fn pin_chain<F>(self, f: F) -> ChainPinInit<Self, F, T, E>
848
-
where
849
-
F: FnOnce(Pin<&mut T>) -> Result<(), E>,
850
-
{
851
-
ChainPinInit(self, f, PhantomData)
852
-
}
853
-
}
854
-
855
-
/// An initializer returned by [`PinInit::pin_chain`].
856
-
pub struct ChainPinInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, KBox<T>)>);
857
-
858
-
// SAFETY: The `__pinned_init` function is implemented such that it
859
-
// - returns `Ok(())` on successful initialization,
860
-
// - returns `Err(err)` on error and in this case `slot` will be dropped.
861
-
// - considers `slot` pinned.
862
-
unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainPinInit<I, F, T, E>
863
-
where
864
-
I: PinInit<T, E>,
865
-
F: FnOnce(Pin<&mut T>) -> Result<(), E>,
866
-
{
867
-
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
868
-
// SAFETY: All requirements fulfilled since this function is `__pinned_init`.
869
-
unsafe { self.0.__pinned_init(slot)? };
870
-
// SAFETY: The above call initialized `slot` and we still have unique access.
871
-
let val = unsafe { &mut *slot };
872
-
// SAFETY: `slot` is considered pinned.
873
-
let val = unsafe { Pin::new_unchecked(val) };
874
-
// SAFETY: `slot` was initialized above.
875
-
(self.1)(val).inspect_err(|_| unsafe { core::ptr::drop_in_place(slot) })
876
-
}
877
-
}
878
-
879
-
/// An initializer for `T`.
880
-
///
881
-
/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
882
-
/// be [`KBox<T>`], [`Arc<T>`], [`UniqueArc<T>`] or even the stack (see [`stack_pin_init!`]). Use
883
-
/// the [`InPlaceInit::init`] function of a smart pointer like [`Arc<T>`] on this. Because
884
-
/// [`PinInit<T, E>`] is a super trait, you can use every function that takes it as well.
885
-
///
886
-
/// Also see the [module description](self).
887
-
///
888
-
/// # Safety
889
-
///
890
-
/// When implementing this trait you will need to take great care. Also there are probably very few
891
-
/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
892
-
///
893
-
/// The [`Init::__init`] function:
894
-
/// - returns `Ok(())` if it initialized every field of `slot`,
895
-
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
896
-
/// - `slot` can be deallocated without UB occurring,
897
-
/// - `slot` does not need to be dropped,
898
-
/// - `slot` is not partially initialized.
899
-
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
900
-
///
901
-
/// The `__pinned_init` function from the supertrait [`PinInit`] needs to execute the exact same
902
-
/// code as `__init`.
903
-
///
904
-
/// Contrary to its supertype [`PinInit<T, E>`] the caller is allowed to
905
-
/// move the pointee after initialization.
906
-
///
907
-
/// [`Arc<T>`]: crate::sync::Arc
908
-
#[must_use = "An initializer must be used in order to create its value."]
909
-
pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
910
-
/// Initializes `slot`.
911
-
///
912
-
/// # Safety
913
-
///
914
-
/// - `slot` is a valid pointer to uninitialized memory.
915
-
/// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
916
-
/// deallocate.
917
-
unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
918
-
919
-
/// First initializes the value using `self` then calls the function `f` with the initialized
920
-
/// value.
921
-
///
922
-
/// If `f` returns an error the value is dropped and the initializer will forward the error.
923
-
///
924
-
/// # Examples
925
-
///
926
-
/// ```rust
927
-
/// # #![expect(clippy::disallowed_names)]
928
-
/// use kernel::{types::Opaque, init::{self, init_from_closure}};
929
-
/// struct Foo {
930
-
/// buf: [u8; 1_000_000],
931
-
/// }
932
-
///
933
-
/// impl Foo {
934
-
/// fn setup(&mut self) {
935
-
/// pr_info!("Setting up foo\n");
936
-
/// }
937
-
/// }
938
-
///
939
-
/// let foo = init!(Foo {
940
-
/// buf <- init::zeroed()
941
-
/// }).chain(|foo| {
942
-
/// foo.setup();
943
-
/// Ok(())
944
-
/// });
945
-
/// ```
946
-
fn chain<F>(self, f: F) -> ChainInit<Self, F, T, E>
947
-
where
948
-
F: FnOnce(&mut T) -> Result<(), E>,
949
-
{
950
-
ChainInit(self, f, PhantomData)
951
-
}
952
-
}
953
-
954
-
/// An initializer returned by [`Init::chain`].
955
-
pub struct ChainInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, KBox<T>)>);
956
-
957
-
// SAFETY: The `__init` function is implemented such that it
958
-
// - returns `Ok(())` on successful initialization,
959
-
// - returns `Err(err)` on error and in this case `slot` will be dropped.
960
-
unsafe impl<T: ?Sized, E, I, F> Init<T, E> for ChainInit<I, F, T, E>
961
-
where
962
-
I: Init<T, E>,
963
-
F: FnOnce(&mut T) -> Result<(), E>,
964
-
{
965
-
unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
966
-
// SAFETY: All requirements fulfilled since this function is `__init`.
967
-
unsafe { self.0.__pinned_init(slot)? };
968
-
// SAFETY: The above call initialized `slot` and we still have unique access.
969
-
(self.1)(unsafe { &mut *slot }).inspect_err(|_|
970
-
// SAFETY: `slot` was initialized above.
971
-
unsafe { core::ptr::drop_in_place(slot) })
972
-
}
973
-
}
974
-
975
-
// SAFETY: `__pinned_init` behaves exactly the same as `__init`.
976
-
unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainInit<I, F, T, E>
977
-
where
978
-
I: Init<T, E>,
979
-
F: FnOnce(&mut T) -> Result<(), E>,
980
-
{
981
-
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
982
-
// SAFETY: `__init` has less strict requirements compared to `__pinned_init`.
983
-
unsafe { self.__init(slot) }
984
-
}
985
-
}
986
-
987
-
/// Creates a new [`PinInit<T, E>`] from the given closure.
988
-
///
989
-
/// # Safety
990
-
///
991
-
/// The closure:
992
-
/// - returns `Ok(())` if it initialized every field of `slot`,
993
-
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
994
-
/// - `slot` can be deallocated without UB occurring,
995
-
/// - `slot` does not need to be dropped,
996
-
/// - `slot` is not partially initialized.
997
-
/// - may assume that the `slot` does not move if `T: !Unpin`,
998
-
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
999
-
#[inline]
1000
-
pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
1001
-
f: impl FnOnce(*mut T) -> Result<(), E>,
1002
-
) -> impl PinInit<T, E> {
1003
-
__internal::InitClosure(f, PhantomData)
1004
-
}
1005
-
1006
-
/// Creates a new [`Init<T, E>`] from the given closure.
1007
-
///
1008
-
/// # Safety
1009
-
///
1010
-
/// The closure:
1011
-
/// - returns `Ok(())` if it initialized every field of `slot`,
1012
-
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1013
-
/// - `slot` can be deallocated without UB occurring,
1014
-
/// - `slot` does not need to be dropped,
1015
-
/// - `slot` is not partially initialized.
1016
-
/// - the `slot` may move after initialization.
1017
-
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1018
-
#[inline]
1019
-
pub const unsafe fn init_from_closure<T: ?Sized, E>(
1020
-
f: impl FnOnce(*mut T) -> Result<(), E>,
1021
-
) -> impl Init<T, E> {
1022
-
__internal::InitClosure(f, PhantomData)
1023
-
}
1024
-
1025
-
/// An initializer that leaves the memory uninitialized.
1026
-
///
1027
-
/// The initializer is a no-op. The `slot` memory is not changed.
1028
-
#[inline]
1029
-
pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
1030
-
// SAFETY: The memory is allowed to be uninitialized.
1031
-
unsafe { init_from_closure(|_| Ok(())) }
1032
-
}
1033
-
1034
-
/// Initializes an array by initializing each element via the provided initializer.
1035
-
///
1036
-
/// # Examples
1037
-
///
1038
-
/// ```rust
1039
-
/// use kernel::{alloc::KBox, error::Error, init::init_array_from_fn};
1040
-
/// let array: KBox<[usize; 1_000]> =
1041
-
/// KBox::init::<Error>(init_array_from_fn(|i| i), GFP_KERNEL)?;
1042
-
/// assert_eq!(array.len(), 1_000);
1043
-
/// # Ok::<(), Error>(())
1044
-
/// ```
1045
-
pub fn init_array_from_fn<I, const N: usize, T, E>(
1046
-
mut make_init: impl FnMut(usize) -> I,
1047
-
) -> impl Init<[T; N], E>
1048
-
where
1049
-
I: Init<T, E>,
1050
-
{
1051
-
let init = move |slot: *mut [T; N]| {
1052
-
let slot = slot.cast::<T>();
1053
-
// Counts the number of initialized elements and when dropped drops that many elements from
1054
-
// `slot`.
1055
-
let mut init_count = ScopeGuard::new_with_data(0, |i| {
1056
-
// We now free every element that has been initialized before.
1057
-
// SAFETY: The loop initialized exactly the values from 0..i and since we
1058
-
// return `Err` below, the caller will consider the memory at `slot` as
1059
-
// uninitialized.
1060
-
unsafe { ptr::drop_in_place(ptr::slice_from_raw_parts_mut(slot, i)) };
1061
-
});
1062
-
for i in 0..N {
1063
-
let init = make_init(i);
1064
-
// SAFETY: Since 0 <= `i` < N, it is still in bounds of `[T; N]`.
1065
-
let ptr = unsafe { slot.add(i) };
1066
-
// SAFETY: The pointer is derived from `slot` and thus satisfies the `__init`
1067
-
// requirements.
1068
-
unsafe { init.__init(ptr) }?;
1069
-
*init_count += 1;
1070
-
}
1071
-
init_count.dismiss();
1072
-
Ok(())
1073
-
};
1074
-
// SAFETY: The initializer above initializes every element of the array. On failure it drops
1075
-
// any initialized elements and returns `Err`.
1076
-
unsafe { init_from_closure(init) }
1077
-
}
1078
-
1079
-
/// Initializes an array by initializing each element via the provided initializer.
1080
-
///
1081
-
/// # Examples
1082
-
///
1083
-
/// ```rust
1084
-
/// use kernel::{sync::{Arc, Mutex}, init::pin_init_array_from_fn, new_mutex};
1085
-
/// let array: Arc<[Mutex<usize>; 1_000]> =
1086
-
/// Arc::pin_init(pin_init_array_from_fn(|i| new_mutex!(i)), GFP_KERNEL)?;
1087
-
/// assert_eq!(array.len(), 1_000);
1088
-
/// # Ok::<(), Error>(())
1089
-
/// ```
1090
-
pub fn pin_init_array_from_fn<I, const N: usize, T, E>(
1091
-
mut make_init: impl FnMut(usize) -> I,
1092
-
) -> impl PinInit<[T; N], E>
1093
-
where
1094
-
I: PinInit<T, E>,
1095
-
{
1096
-
let init = move |slot: *mut [T; N]| {
1097
-
let slot = slot.cast::<T>();
1098
-
// Counts the number of initialized elements and when dropped drops that many elements from
1099
-
// `slot`.
1100
-
let mut init_count = ScopeGuard::new_with_data(0, |i| {
1101
-
// We now free every element that has been initialized before.
1102
-
// SAFETY: The loop initialized exactly the values from 0..i and since we
1103
-
// return `Err` below, the caller will consider the memory at `slot` as
1104
-
// uninitialized.
1105
-
unsafe { ptr::drop_in_place(ptr::slice_from_raw_parts_mut(slot, i)) };
1106
-
});
1107
-
for i in 0..N {
1108
-
let init = make_init(i);
1109
-
// SAFETY: Since 0 <= `i` < N, it is still in bounds of `[T; N]`.
1110
-
let ptr = unsafe { slot.add(i) };
1111
-
// SAFETY: The pointer is derived from `slot` and thus satisfies the `__init`
1112
-
// requirements.
1113
-
unsafe { init.__pinned_init(ptr) }?;
1114
-
*init_count += 1;
1115
-
}
1116
-
init_count.dismiss();
1117
-
Ok(())
1118
-
};
1119
-
// SAFETY: The initializer above initializes every element of the array. On failure it drops
1120
-
// any initialized elements and returns `Err`.
1121
-
unsafe { pin_init_from_closure(init) }
1122
-
}
1123
-
1124
-
// SAFETY: Every type can be initialized by-value.
1125
-
unsafe impl<T, E> Init<T, E> for T {
1126
-
unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1127
-
// SAFETY: TODO.
1128
-
unsafe { slot.write(self) };
1129
-
Ok(())
1130
-
}
1131
-
}
1132
-
1133
-
// SAFETY: Every type can be initialized by-value. `__pinned_init` calls `__init`.
1134
-
unsafe impl<T, E> PinInit<T, E> for T {
1135
-
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
1136
-
// SAFETY: TODO.
1137
-
unsafe { self.__init(slot) }
1138
-
}
1139
-
}
195
+
use pin_init::{init_from_closure, pin_init_from_closure, Init, PinInit};
1140
196
1141
197
/// Smart pointer that can initialize memory in-place.
1142
198
pub trait InPlaceInit<T>: Sized {
···
189
1241
}
190
1242
}
191
1243
192
-
impl<T> InPlaceInit<T> for Arc<T> {
193
-
type PinnedSelf = Self;
194
-
195
-
#[inline]
196
-
fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::PinnedSelf, E>
197
-
where
198
-
E: From<AllocError>,
199
-
{
200
-
UniqueArc::try_pin_init(init, flags).map(|u| u.into())
201
-
}
202
-
203
-
#[inline]
204
-
fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
205
-
where
206
-
E: From<AllocError>,
207
-
{
208
-
UniqueArc::try_init(init, flags).map(|u| u.into())
209
-
}
210
-
}
211
-
212
-
impl<T> InPlaceInit<T> for UniqueArc<T> {
213
-
type PinnedSelf = Pin<Self>;
214
-
215
-
#[inline]
216
-
fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::PinnedSelf, E>
217
-
where
218
-
E: From<AllocError>,
219
-
{
220
-
UniqueArc::new_uninit(flags)?.write_pin_init(init)
221
-
}
222
-
223
-
#[inline]
224
-
fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
225
-
where
226
-
E: From<AllocError>,
227
-
{
228
-
UniqueArc::new_uninit(flags)?.write_init(init)
229
-
}
230
-
}
231
-
232
-
/// Smart pointer containing uninitialized memory and that can write a value.
233
-
pub trait InPlaceWrite<T> {
234
-
/// The type `Self` turns into when the contents are initialized.
235
-
type Initialized;
236
-
237
-
/// Use the given initializer to write a value into `self`.
238
-
///
239
-
/// Does not drop the current value and considers it as uninitialized memory.
240
-
fn write_init<E>(self, init: impl Init<T, E>) -> Result<Self::Initialized, E>;
241
-
242
-
/// Use the given pin-initializer to write a value into `self`.
243
-
///
244
-
/// Does not drop the current value and considers it as uninitialized memory.
245
-
fn write_pin_init<E>(self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E>;
246
-
}
247
-
248
-
impl<T> InPlaceWrite<T> for UniqueArc<MaybeUninit<T>> {
249
-
type Initialized = UniqueArc<T>;
250
-
251
-
fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
252
-
let slot = self.as_mut_ptr();
253
-
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
254
-
// slot is valid.
255
-
unsafe { init.__init(slot)? };
256
-
// SAFETY: All fields have been initialized.
257
-
Ok(unsafe { self.assume_init() })
258
-
}
259
-
260
-
fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
261
-
let slot = self.as_mut_ptr();
262
-
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
263
-
// slot is valid and will not be moved, because we pin it later.
264
-
unsafe { init.__pinned_init(slot)? };
265
-
// SAFETY: All fields have been initialized.
266
-
Ok(unsafe { self.assume_init() }.into())
267
-
}
268
-
}
269
-
270
-
/// Trait facilitating pinned destruction.
1244
+
/// Construct an in-place fallible initializer for `struct`s.
271
1245
///
272
-
/// Use [`pinned_drop`] to implement this trait safely:
1246
+
/// This macro defaults the error to [`Error`]. If you need [`Infallible`], then use
1247
+
/// [`init!`].
1248
+
///
1249
+
/// The syntax is identical to [`try_pin_init!`]. If you want to specify a custom error,
1250
+
/// append `? $type` after the `struct` initializer.
1251
+
/// The safety caveats from [`try_pin_init!`] also apply:
1252
+
/// - `unsafe` code must guarantee either full initialization or return an error and allow
1253
+
/// deallocation of the memory.
1254
+
/// - the fields are initialized in the order given in the initializer.
1255
+
/// - no references to fields are allowed to be created inside of the initializer.
1256
+
///
1257
+
/// # Examples
273
1258
///
274
1259
/// ```rust
275
-
/// # use kernel::sync::Mutex;
276
-
/// use kernel::macros::pinned_drop;
277
-
/// use core::pin::Pin;
278
-
/// #[pin_data(PinnedDrop)]
279
-
/// struct Foo {
280
-
/// #[pin]
281
-
/// mtx: Mutex<usize>,
1260
+
/// use kernel::error::Error;
1261
+
/// use pin_init::zeroed;
1262
+
/// struct BigBuf {
1263
+
/// big: KBox<[u8; 1024 * 1024 * 1024]>,
1264
+
/// small: [u8; 1024 * 1024],
282
1265
/// }
283
1266
///
284
-
/// #[pinned_drop]
285
-
/// impl PinnedDrop for Foo {
286
-
/// fn drop(self: Pin<&mut Self>) {
287
-
/// pr_info!("Foo is being dropped!\n");
1267
+
/// impl BigBuf {
1268
+
/// fn new() -> impl Init<Self, Error> {
1269
+
/// try_init!(Self {
1270
+
/// big: KBox::init(zeroed(), GFP_KERNEL)?,
1271
+
/// small: [0; 1024 * 1024],
1272
+
/// }? Error)
288
1273
/// }
289
1274
/// }
290
1275
/// ```
291
1276
///
292
-
/// # Safety
293
-
///
294
-
/// This trait must be implemented via the [`pinned_drop`] proc-macro attribute on the impl.
295
-
///
296
-
/// [`pinned_drop`]: kernel::macros::pinned_drop
297
-
pub unsafe trait PinnedDrop: __internal::HasPinData {
298
-
/// Executes the pinned destructor of this type.
299
-
///
300
-
/// While this function is marked safe, it is actually unsafe to call it manually. For this
301
-
/// reason it takes an additional parameter. This type can only be constructed by `unsafe` code
302
-
/// and thus prevents this function from being called where it should not.
303
-
///
304
-
/// This extra parameter will be generated by the `#[pinned_drop]` proc-macro attribute
305
-
/// automatically.
306
-
fn drop(self: Pin<&mut Self>, only_call_from_drop: __internal::OnlyCallFromDrop);
307
-
}
308
-
309
-
/// Marker trait for types that can be initialized by writing just zeroes.
310
-
///
311
-
/// # Safety
312
-
///
313
-
/// The bit pattern consisting of only zeroes is a valid bit pattern for this type. In other words,
314
-
/// this is not UB:
315
-
///
316
-
/// ```rust,ignore
317
-
/// let val: Self = unsafe { core::mem::zeroed() };
318
-
/// ```
319
-
pub unsafe trait Zeroable {}
320
-
321
-
/// Create a new zeroed T.
322
-
///
323
-
/// The returned initializer will write `0x00` to every byte of the given `slot`.
324
-
#[inline]
325
-
pub fn zeroed<T: Zeroable>() -> impl Init<T> {
326
-
// SAFETY: Because `T: Zeroable`, all bytes zero is a valid bit pattern for `T`
327
-
// and because we write all zeroes, the memory is initialized.
328
-
unsafe {
329
-
init_from_closure(|slot: *mut T| {
330
-
slot.write_bytes(0, 1);
331
-
Ok(())
332
-
})
333
-
}
334
-
}
335
-
336
-
macro_rules! impl_zeroable {
337
-
($($({$($generics:tt)*})? $t:ty, )*) => {
338
-
// SAFETY: Safety comments written in the macro invocation.
339
-
$(unsafe impl$($($generics)*)? Zeroable for $t {})*
1277
+
/// [`Infallible`]: core::convert::Infallible
1278
+
/// [`init!`]: pin_init::init
1279
+
/// [`try_pin_init!`]: crate::try_pin_init!
1280
+
/// [`Error`]: crate::error::Error
1281
+
#[macro_export]
1282
+
macro_rules! try_init {
1283
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
1284
+
$($fields:tt)*
1285
+
}) => {
1286
+
::pin_init::try_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
1287
+
$($fields)*
1288
+
}? $crate::error::Error)
1289
+
};
1290
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
1291
+
$($fields:tt)*
1292
+
}? $err:ty) => {
1293
+
::pin_init::try_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
1294
+
$($fields)*
1295
+
}? $err)
340
1296
};
341
1297
}
342
1298
343
-
impl_zeroable! {
344
-
// SAFETY: All primitives that are allowed to be zero.
345
-
bool,
346
-
char,
347
-
u8, u16, u32, u64, u128, usize,
348
-
i8, i16, i32, i64, i128, isize,
349
-
f32, f64,
350
-
351
-
// Note: do not add uninhabited types (such as `!` or `core::convert::Infallible`) to this list;
352
-
// creating an instance of an uninhabited type is immediate undefined behavior. For more on
353
-
// uninhabited/empty types, consult The Rustonomicon:
354
-
// <https://doc.rust-lang.org/stable/nomicon/exotic-sizes.html#empty-types>. The Rust Reference
355
-
// also has information on undefined behavior:
356
-
// <https://doc.rust-lang.org/stable/reference/behavior-considered-undefined.html>.
357
-
//
358
-
// SAFETY: These are inhabited ZSTs; there is nothing to zero and a valid value exists.
359
-
{<T: ?Sized>} PhantomData<T>, core::marker::PhantomPinned, (),
360
-
361
-
// SAFETY: Type is allowed to take any value, including all zeros.
362
-
{<T>} MaybeUninit<T>,
363
-
// SAFETY: Type is allowed to take any value, including all zeros.
364
-
{<T>} Opaque<T>,
365
-
366
-
// SAFETY: `T: Zeroable` and `UnsafeCell` is `repr(transparent)`.
367
-
{<T: ?Sized + Zeroable>} UnsafeCell<T>,
368
-
369
-
// SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee:
370
-
// https://doc.rust-lang.org/stable/std/option/index.html#representation).
371
-
Option<NonZeroU8>, Option<NonZeroU16>, Option<NonZeroU32>, Option<NonZeroU64>,
372
-
Option<NonZeroU128>, Option<NonZeroUsize>,
373
-
Option<NonZeroI8>, Option<NonZeroI16>, Option<NonZeroI32>, Option<NonZeroI64>,
374
-
Option<NonZeroI128>, Option<NonZeroIsize>,
375
-
{<T>} Option<NonNull<T>>,
376
-
{<T>} Option<KBox<T>>,
377
-
378
-
// SAFETY: `null` pointer is valid.
379
-
//
380
-
// We cannot use `T: ?Sized`, since the VTABLE pointer part of fat pointers is not allowed to be
381
-
// null.
382
-
//
383
-
// When `Pointee` gets stabilized, we could use
384
-
// `T: ?Sized where <T as Pointee>::Metadata: Zeroable`
385
-
{<T>} *mut T, {<T>} *const T,
386
-
387
-
// SAFETY: `null` pointer is valid and the metadata part of these fat pointers is allowed to be
388
-
// zero.
389
-
{<T>} *mut [T], {<T>} *const [T], *mut str, *const str,
390
-
391
-
// SAFETY: `T` is `Zeroable`.
392
-
{<const N: usize, T: Zeroable>} [T; N], {<T: Zeroable>} Wrapping<T>,
1299
+
/// Construct an in-place, fallible pinned initializer for `struct`s.
1300
+
///
1301
+
/// If the initialization can complete without error (or [`Infallible`]), then use [`pin_init!`].
1302
+
///
1303
+
/// You can use the `?` operator or use `return Err(err)` inside the initializer to stop
1304
+
/// initialization and return the error.
1305
+
///
1306
+
/// IMPORTANT: if you have `unsafe` code inside of the initializer you have to ensure that when
1307
+
/// initialization fails, the memory can be safely deallocated without any further modifications.
1308
+
///
1309
+
/// This macro defaults the error to [`Error`].
1310
+
///
1311
+
/// The syntax is identical to [`pin_init!`] with the following exception: you can append `? $type`
1312
+
/// after the `struct` initializer to specify the error type you want to use.
1313
+
///
1314
+
/// # Examples
1315
+
///
1316
+
/// ```rust
1317
+
/// # #![feature(new_uninit)]
1318
+
/// use kernel::error::Error;
1319
+
/// use pin_init::zeroed;
1320
+
/// #[pin_data]
1321
+
/// struct BigBuf {
1322
+
/// big: KBox<[u8; 1024 * 1024 * 1024]>,
1323
+
/// small: [u8; 1024 * 1024],
1324
+
/// ptr: *mut u8,
1325
+
/// }
1326
+
///
1327
+
/// impl BigBuf {
1328
+
/// fn new() -> impl PinInit<Self, Error> {
1329
+
/// try_pin_init!(Self {
1330
+
/// big: KBox::init(zeroed(), GFP_KERNEL)?,
1331
+
/// small: [0; 1024 * 1024],
1332
+
/// ptr: core::ptr::null_mut(),
1333
+
/// }? Error)
1334
+
/// }
1335
+
/// }
1336
+
/// ```
1337
+
///
1338
+
/// [`Infallible`]: core::convert::Infallible
1339
+
/// [`pin_init!`]: pin_init::pin_init
1340
+
/// [`Error`]: crate::error::Error
1341
+
#[macro_export]
1342
+
macro_rules! try_pin_init {
1343
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
1344
+
$($fields:tt)*
1345
+
}) => {
1346
+
::pin_init::try_pin_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
1347
+
$($fields)*
1348
+
}? $crate::error::Error)
1349
+
};
1350
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
1351
+
$($fields:tt)*
1352
+
}? $err:ty) => {
1353
+
::pin_init::try_pin_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
1354
+
$($fields)*
1355
+
}? $err)
1356
+
};
393
1357
}
394
-
395
-
macro_rules! impl_tuple_zeroable {
396
-
($(,)?) => {};
397
-
($first:ident, $($t:ident),* $(,)?) => {
398
-
// SAFETY: All elements are zeroable and padding can be zero.
399
-
unsafe impl<$first: Zeroable, $($t: Zeroable),*> Zeroable for ($first, $($t),*) {}
400
-
impl_tuple_zeroable!($($t),* ,);
401
-
}
402
-
}
403
-
404
-
impl_tuple_zeroable!(A, B, C, D, E, F, G, H, I, J);
+37
-9
rust/kernel/init/__internal.rs
rust/pin-init/src/__internal.rs
+37
-9
rust/kernel/init/__internal.rs
rust/pin-init/src/__internal.rs
···
1
1
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
2
3
-
//! This module contains API-internal items for pin-init.
3
+
//! This module contains library internal items.
4
4
//!
5
-
//! These items must not be used outside of
6
-
//! - `kernel/init.rs`
7
-
//! - `macros/pin_data.rs`
8
-
//! - `macros/pinned_drop.rs`
5
+
//! These items must not be used outside of this crate and the pin-init-internal crate located at
6
+
//! `../internal`.
9
7
10
8
use super::*;
11
9
12
10
/// See the [nomicon] for what subtyping is. See also [this table].
13
11
///
12
+
/// The reason for not using `PhantomData<*mut T>` is that that type never implements [`Send`] and
13
+
/// [`Sync`]. Hence `fn(*mut T) -> *mut T` is used, as that type always implements them.
14
+
///
14
15
/// [nomicon]: https://doc.rust-lang.org/nomicon/subtyping.html
15
16
/// [this table]: https://doc.rust-lang.org/nomicon/phantom-data.html#table-of-phantomdata-patterns
16
-
pub(super) type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
17
+
pub(crate) type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
17
18
18
19
/// Module-internal type implementing `PinInit` and `Init`.
19
20
///
···
106
105
}
107
106
}
108
107
109
-
pub struct AllData<T: ?Sized>(PhantomData<fn(KBox<T>) -> KBox<T>>);
108
+
pub struct AllData<T: ?Sized>(Invariant<T>);
110
109
111
110
impl<T: ?Sized> Clone for AllData<T> {
112
111
fn clone(&self) -> Self {
···
136
135
///
137
136
/// If `self.is_init` is true, then `self.value` is initialized.
138
137
///
139
-
/// [`stack_pin_init`]: kernel::stack_pin_init
138
+
/// [`stack_pin_init`]: crate::stack_pin_init
140
139
pub struct StackInit<T> {
141
140
value: MaybeUninit<T>,
142
141
is_init: bool,
···
157
156
/// Creates a new [`StackInit<T>`] that is uninitialized. Use [`stack_pin_init`] instead of this
158
157
/// primitive.
159
158
///
160
-
/// [`stack_pin_init`]: kernel::stack_pin_init
159
+
/// [`stack_pin_init`]: crate::stack_pin_init
161
160
#[inline]
162
161
pub fn uninit() -> Self {
163
162
Self {
···
185
184
// SAFETY: The slot is now pinned, since we will never give access to `&mut T`.
186
185
Ok(unsafe { Pin::new_unchecked(this.value.assume_init_mut()) })
187
186
}
187
+
}
188
+
189
+
#[test]
190
+
fn stack_init_reuse() {
191
+
use ::std::{borrow::ToOwned, println, string::String};
192
+
use core::pin::pin;
193
+
194
+
#[derive(Debug)]
195
+
struct Foo {
196
+
a: usize,
197
+
b: String,
198
+
}
199
+
let mut slot: Pin<&mut StackInit<Foo>> = pin!(StackInit::uninit());
200
+
let value: Result<Pin<&mut Foo>, core::convert::Infallible> =
201
+
slot.as_mut().init(crate::init!(Foo {
202
+
a: 42,
203
+
b: "Hello".to_owned(),
204
+
}));
205
+
let value = value.unwrap();
206
+
println!("{value:?}");
207
+
let value: Result<Pin<&mut Foo>, core::convert::Infallible> =
208
+
slot.as_mut().init(crate::init!(Foo {
209
+
a: 24,
210
+
b: "world!".to_owned(),
211
+
}));
212
+
let value = value.unwrap();
213
+
println!("{value:?}");
188
214
}
189
215
190
216
/// When a value of this type is dropped, it drops a `T`.
+67
-62
rust/kernel/init/macros.rs
rust/pin-init/src/macros.rs
+67
-62
rust/kernel/init/macros.rs
rust/pin-init/src/macros.rs
···
1
1
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
2
3
3
//! This module provides the macros that actually implement the proc-macros `pin_data` and
4
-
//! `pinned_drop`. It also contains `__init_internal` the implementation of the `{try_}{pin_}init!`
5
-
//! macros.
4
+
//! `pinned_drop`. It also contains `__init_internal`, the implementation of the
5
+
//! `{try_}{pin_}init!` macros.
6
6
//!
7
7
//! These macros should never be called directly, since they expect their input to be
8
8
//! in a certain format which is internal. If used incorrectly, these macros can lead to UB even in
···
11
11
//! This architecture has been chosen because the kernel does not yet have access to `syn` which
12
12
//! would make matters a lot easier for implementing these as proc-macros.
13
13
//!
14
+
//! Since this library and the kernel implementation should diverge as little as possible, the same
15
+
//! approach has been taken here.
16
+
//!
14
17
//! # Macro expansion example
15
18
//!
16
19
//! This section is intended for readers trying to understand the macros in this module and the
17
-
//! `pin_init!` macros from `init.rs`.
20
+
//! `[try_][pin_]init!` macros from `lib.rs`.
18
21
//!
19
22
//! We will look at the following example:
20
23
//!
21
24
//! ```rust,ignore
22
-
//! # use kernel::init::*;
23
-
//! # use core::pin::Pin;
24
25
//! #[pin_data]
25
26
//! #[repr(C)]
26
27
//! struct Bar<T> {
···
46
45
//! #[pinned_drop]
47
46
//! impl PinnedDrop for Foo {
48
47
//! fn drop(self: Pin<&mut Self>) {
49
-
//! pr_info!("{self:p} is getting dropped.\n");
48
+
//! println!("{self:p} is getting dropped.");
50
49
//! }
51
50
//! }
52
51
//!
···
76
75
//! Here is the definition of `Bar` from our example:
77
76
//!
78
77
//! ```rust,ignore
79
-
//! # use kernel::init::*;
80
78
//! #[pin_data]
81
79
//! #[repr(C)]
82
80
//! struct Bar<T> {
···
121
121
//! self,
122
122
//! slot: *mut T,
123
123
//! // Since `t` is `#[pin]`, this is `PinInit`.
124
-
//! init: impl ::kernel::init::PinInit<T, E>,
124
+
//! init: impl ::pin_init::PinInit<T, E>,
125
125
//! ) -> ::core::result::Result<(), E> {
126
-
//! unsafe { ::kernel::init::PinInit::__pinned_init(init, slot) }
126
+
//! unsafe { ::pin_init::PinInit::__pinned_init(init, slot) }
127
127
//! }
128
128
//! pub unsafe fn x<E>(
129
129
//! self,
130
130
//! slot: *mut usize,
131
131
//! // Since `x` is not `#[pin]`, this is `Init`.
132
-
//! init: impl ::kernel::init::Init<usize, E>,
132
+
//! init: impl ::pin_init::Init<usize, E>,
133
133
//! ) -> ::core::result::Result<(), E> {
134
-
//! unsafe { ::kernel::init::Init::__init(init, slot) }
134
+
//! unsafe { ::pin_init::Init::__init(init, slot) }
135
135
//! }
136
136
//! }
137
137
//! // Implement the internal `HasPinData` trait that associates `Bar` with the pin-data struct
138
138
//! // that we constructed above.
139
-
//! unsafe impl<T> ::kernel::init::__internal::HasPinData for Bar<T> {
139
+
//! unsafe impl<T> ::pin_init::__internal::HasPinData for Bar<T> {
140
140
//! type PinData = __ThePinData<T>;
141
141
//! unsafe fn __pin_data() -> Self::PinData {
142
142
//! __ThePinData {
···
147
147
//! // Implement the internal `PinData` trait that marks the pin-data struct as a pin-data
148
148
//! // struct. This is important to ensure that no user can implement a rogue `__pin_data`
149
149
//! // function without using `unsafe`.
150
-
//! unsafe impl<T> ::kernel::init::__internal::PinData for __ThePinData<T> {
150
+
//! unsafe impl<T> ::pin_init::__internal::PinData for __ThePinData<T> {
151
151
//! type Datee = Bar<T>;
152
152
//! }
153
153
//! // Now we only want to implement `Unpin` for `Bar` when every structurally pinned field is
···
191
191
//! #[expect(non_camel_case_types)]
192
192
//! trait UselessPinnedDropImpl_you_need_to_specify_PinnedDrop {}
193
193
//! impl<
194
-
//! T: ::kernel::init::PinnedDrop,
194
+
//! T: ::pin_init::PinnedDrop,
195
195
//! > UselessPinnedDropImpl_you_need_to_specify_PinnedDrop for T {}
196
196
//! impl<T> UselessPinnedDropImpl_you_need_to_specify_PinnedDrop for Bar<T> {}
197
197
//! };
···
227
227
//! // - we `use` the `HasPinData` trait in the block, it is only available in that
228
228
//! // scope.
229
229
//! let data = unsafe {
230
-
//! use ::kernel::init::__internal::HasPinData;
230
+
//! use ::pin_init::__internal::HasPinData;
231
231
//! Self::__pin_data()
232
232
//! };
233
233
//! // Ensure that `data` really is of type `PinData` and help with type inference:
234
-
//! let init = ::kernel::init::__internal::PinData::make_closure::<
234
+
//! let init = ::pin_init::__internal::PinData::make_closure::<
235
235
//! _,
236
236
//! __InitOk,
237
237
//! ::core::convert::Infallible,
···
251
251
//! // is an error later. This `DropGuard` will drop the field when it gets
252
252
//! // dropped and has not yet been forgotten.
253
253
//! let __t_guard = unsafe {
254
-
//! ::pinned_init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).t))
254
+
//! ::pin_init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).t))
255
255
//! };
256
256
//! // Expansion of `x: 0,`:
257
257
//! // Since this can be an arbitrary expression we cannot place it inside
···
262
262
//! }
263
263
//! // We again create a `DropGuard`.
264
264
//! let __x_guard = unsafe {
265
-
//! ::kernel::init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).x))
265
+
//! ::pin_init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).x))
266
266
//! };
267
267
//! // Since initialization has successfully completed, we can now forget
268
268
//! // the guards. This is not `mem::forget`, since we only have
···
303
303
//! };
304
304
//! // Construct the initializer.
305
305
//! let init = unsafe {
306
-
//! ::kernel::init::pin_init_from_closure::<
306
+
//! ::pin_init::pin_init_from_closure::<
307
307
//! _,
308
308
//! ::core::convert::Infallible,
309
309
//! >(init)
···
350
350
//! unsafe fn b<E>(
351
351
//! self,
352
352
//! slot: *mut Bar<u32>,
353
-
//! init: impl ::kernel::init::PinInit<Bar<u32>, E>,
353
+
//! init: impl ::pin_init::PinInit<Bar<u32>, E>,
354
354
//! ) -> ::core::result::Result<(), E> {
355
-
//! unsafe { ::kernel::init::PinInit::__pinned_init(init, slot) }
355
+
//! unsafe { ::pin_init::PinInit::__pinned_init(init, slot) }
356
356
//! }
357
357
//! unsafe fn a<E>(
358
358
//! self,
359
359
//! slot: *mut usize,
360
-
//! init: impl ::kernel::init::Init<usize, E>,
360
+
//! init: impl ::pin_init::Init<usize, E>,
361
361
//! ) -> ::core::result::Result<(), E> {
362
-
//! unsafe { ::kernel::init::Init::__init(init, slot) }
362
+
//! unsafe { ::pin_init::Init::__init(init, slot) }
363
363
//! }
364
364
//! }
365
-
//! unsafe impl ::kernel::init::__internal::HasPinData for Foo {
365
+
//! unsafe impl ::pin_init::__internal::HasPinData for Foo {
366
366
//! type PinData = __ThePinData;
367
367
//! unsafe fn __pin_data() -> Self::PinData {
368
368
//! __ThePinData {
···
370
370
//! }
371
371
//! }
372
372
//! }
373
-
//! unsafe impl ::kernel::init::__internal::PinData for __ThePinData {
373
+
//! unsafe impl ::pin_init::__internal::PinData for __ThePinData {
374
374
//! type Datee = Foo;
375
375
//! }
376
376
//! #[allow(dead_code)]
···
394
394
//! let pinned = unsafe { ::core::pin::Pin::new_unchecked(self) };
395
395
//! // Create the unsafe token that proves that we are inside of a destructor, this
396
396
//! // type is only allowed to be created in a destructor.
397
-
//! let token = unsafe { ::kernel::init::__internal::OnlyCallFromDrop::new() };
398
-
//! ::kernel::init::PinnedDrop::drop(pinned, token);
397
+
//! let token = unsafe { ::pin_init::__internal::OnlyCallFromDrop::new() };
398
+
//! ::pin_init::PinnedDrop::drop(pinned, token);
399
399
//! }
400
400
//! }
401
401
//! };
···
412
412
//! #[pinned_drop]
413
413
//! impl PinnedDrop for Foo {
414
414
//! fn drop(self: Pin<&mut Self>) {
415
-
//! pr_info!("{self:p} is getting dropped.\n");
415
+
//! println!("{self:p} is getting dropped.");
416
416
//! }
417
417
//! }
418
418
//! ```
···
421
421
//!
422
422
//! ```rust,ignore
423
423
//! // `unsafe`, full path and the token parameter are added, everything else stays the same.
424
-
//! unsafe impl ::kernel::init::PinnedDrop for Foo {
425
-
//! fn drop(self: Pin<&mut Self>, _: ::kernel::init::__internal::OnlyCallFromDrop) {
426
-
//! pr_info!("{self:p} is getting dropped.\n");
424
+
//! unsafe impl ::pin_init::PinnedDrop for Foo {
425
+
//! fn drop(self: Pin<&mut Self>, _: ::pin_init::__internal::OnlyCallFromDrop) {
426
+
//! println!("{self:p} is getting dropped.");
427
427
//! }
428
428
//! }
429
429
//! ```
···
448
448
//! let initializer = {
449
449
//! struct __InitOk;
450
450
//! let data = unsafe {
451
-
//! use ::kernel::init::__internal::HasPinData;
451
+
//! use ::pin_init::__internal::HasPinData;
452
452
//! Foo::__pin_data()
453
453
//! };
454
-
//! let init = ::kernel::init::__internal::PinData::make_closure::<
454
+
//! let init = ::pin_init::__internal::PinData::make_closure::<
455
455
//! _,
456
456
//! __InitOk,
457
457
//! ::core::convert::Infallible,
···
462
462
//! unsafe { ::core::ptr::write(::core::addr_of_mut!((*slot).a), a) };
463
463
//! }
464
464
//! let __a_guard = unsafe {
465
-
//! ::kernel::init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).a))
465
+
//! ::pin_init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).a))
466
466
//! };
467
467
//! let init = Bar::new(36);
468
468
//! unsafe { data.b(::core::addr_of_mut!((*slot).b), b)? };
469
469
//! let __b_guard = unsafe {
470
-
//! ::kernel::init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).b))
470
+
//! ::pin_init::__internal::DropGuard::new(::core::addr_of_mut!((*slot).b))
471
471
//! };
472
472
//! ::core::mem::forget(__b_guard);
473
473
//! ::core::mem::forget(__a_guard);
···
492
492
//! init(slot).map(|__InitOk| ())
493
493
//! };
494
494
//! let init = unsafe {
495
-
//! ::kernel::init::pin_init_from_closure::<_, ::core::convert::Infallible>(init)
495
+
//! ::pin_init::pin_init_from_closure::<_, ::core::convert::Infallible>(init)
496
496
//! };
497
497
//! init
498
498
//! };
499
499
//! ```
500
+
501
+
#[cfg(kernel)]
502
+
pub use ::macros::paste;
503
+
#[cfg(not(kernel))]
504
+
pub use ::paste::paste;
500
505
501
506
/// Creates a `unsafe impl<...> PinnedDrop for $type` block.
502
507
///
···
522
517
unsafe $($impl_sig)* {
523
518
// Inherit all attributes and the type/ident tokens for the signature.
524
519
$(#[$($attr)*])*
525
-
fn drop($($sig)*, _: $crate::init::__internal::OnlyCallFromDrop) {
520
+
fn drop($($sig)*, _: $crate::__internal::OnlyCallFromDrop) {
526
521
$($inner)*
527
522
}
528
523
}
···
868
863
// SAFETY: We have added the correct projection functions above to `__ThePinData` and
869
864
// we also use the least restrictive generics possible.
870
865
unsafe impl<$($impl_generics)*>
871
-
$crate::init::__internal::HasPinData for $name<$($ty_generics)*>
866
+
$crate::__internal::HasPinData for $name<$($ty_generics)*>
872
867
where $($whr)*
873
868
{
874
869
type PinData = __ThePinData<$($ty_generics)*>;
···
880
875
881
876
// SAFETY: TODO.
882
877
unsafe impl<$($impl_generics)*>
883
-
$crate::init::__internal::PinData for __ThePinData<$($ty_generics)*>
878
+
$crate::__internal::PinData for __ThePinData<$($ty_generics)*>
884
879
where $($whr)*
885
880
{
886
881
type Datee = $name<$($ty_generics)*>;
···
939
934
// `PinnedDrop` as the parameter to `#[pin_data]`.
940
935
#[expect(non_camel_case_types)]
941
936
trait UselessPinnedDropImpl_you_need_to_specify_PinnedDrop {}
942
-
impl<T: $crate::init::PinnedDrop>
937
+
impl<T: $crate::PinnedDrop>
943
938
UselessPinnedDropImpl_you_need_to_specify_PinnedDrop for T {}
944
939
impl<$($impl_generics)*>
945
940
UselessPinnedDropImpl_you_need_to_specify_PinnedDrop for $name<$($ty_generics)*>
···
962
957
let pinned = unsafe { ::core::pin::Pin::new_unchecked(self) };
963
958
// SAFETY: Since this is a drop function, we can create this token to call the
964
959
// pinned destructor of this type.
965
-
let token = unsafe { $crate::init::__internal::OnlyCallFromDrop::new() };
966
-
$crate::init::PinnedDrop::drop(pinned, token);
960
+
let token = unsafe { $crate::__internal::OnlyCallFromDrop::new() };
961
+
$crate::PinnedDrop::drop(pinned, token);
967
962
}
968
963
}
969
964
};
···
1003
998
$pvis unsafe fn $p_field<E>(
1004
999
self,
1005
1000
slot: *mut $p_type,
1006
-
init: impl $crate::init::PinInit<$p_type, E>,
1001
+
init: impl $crate::PinInit<$p_type, E>,
1007
1002
) -> ::core::result::Result<(), E> {
1008
1003
// SAFETY: TODO.
1009
-
unsafe { $crate::init::PinInit::__pinned_init(init, slot) }
1004
+
unsafe { $crate::PinInit::__pinned_init(init, slot) }
1010
1005
}
1011
1006
)*
1012
1007
$(
···
1014
1009
$fvis unsafe fn $field<E>(
1015
1010
self,
1016
1011
slot: *mut $type,
1017
-
init: impl $crate::init::Init<$type, E>,
1012
+
init: impl $crate::Init<$type, E>,
1018
1013
) -> ::core::result::Result<(), E> {
1019
1014
// SAFETY: TODO.
1020
-
unsafe { $crate::init::Init::__init(init, slot) }
1015
+
unsafe { $crate::Init::__init(init, slot) }
1021
1016
}
1022
1017
)*
1023
1018
}
···
1134
1129
//
1135
1130
// SAFETY: TODO.
1136
1131
let data = unsafe {
1137
-
use $crate::init::__internal::$has_data;
1132
+
use $crate::__internal::$has_data;
1138
1133
// Here we abuse `paste!` to retokenize `$t`. Declarative macros have some internal
1139
1134
// information that is associated to already parsed fragments, so a path fragment
1140
1135
// cannot be used in this position. Doing the retokenization results in valid rust
1141
1136
// code.
1142
-
::kernel::macros::paste!($t::$get_data())
1137
+
$crate::macros::paste!($t::$get_data())
1143
1138
};
1144
1139
// Ensure that `data` really is of type `$data` and help with type inference:
1145
-
let init = $crate::init::__internal::$data::make_closure::<_, __InitOk, $err>(
1140
+
let init = $crate::__internal::$data::make_closure::<_, __InitOk, $err>(
1146
1141
data,
1147
1142
move |slot| {
1148
1143
{
···
1152
1147
// error when fields are missing (since they will be zeroed). We also have to
1153
1148
// check that the type actually implements `Zeroable`.
1154
1149
$({
1155
-
fn assert_zeroable<T: $crate::init::Zeroable>(_: *mut T) {}
1150
+
fn assert_zeroable<T: $crate::Zeroable>(_: *mut T) {}
1156
1151
// Ensure that the struct is indeed `Zeroable`.
1157
1152
assert_zeroable(slot);
1158
1153
// SAFETY: The type implements `Zeroable` by the check above.
···
1189
1184
init(slot).map(|__InitOk| ())
1190
1185
};
1191
1186
// SAFETY: TODO.
1192
-
let init = unsafe { $crate::init::$construct_closure::<_, $err>(init) };
1187
+
let init = unsafe { $crate::$construct_closure::<_, $err>(init) };
1193
1188
init
1194
1189
}};
1195
1190
(init_slot($($use_data:ident)?):
···
1220
1215
//
1221
1216
// We rely on macro hygiene to make it impossible for users to access this local variable.
1222
1217
// We use `paste!` to create new hygiene for `$field`.
1223
-
::kernel::macros::paste! {
1218
+
$crate::macros::paste! {
1224
1219
// SAFETY: We forget the guard later when initialization has succeeded.
1225
1220
let [< __ $field _guard >] = unsafe {
1226
-
$crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1221
+
$crate::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1227
1222
};
1228
1223
1229
1224
$crate::__init_internal!(init_slot($use_data):
···
1246
1241
//
1247
1242
// SAFETY: `slot` is valid, because we are inside of an initializer closure, we
1248
1243
// return when an error/panic occurs.
1249
-
unsafe { $crate::init::Init::__init(init, ::core::ptr::addr_of_mut!((*$slot).$field))? };
1244
+
unsafe { $crate::Init::__init(init, ::core::ptr::addr_of_mut!((*$slot).$field))? };
1250
1245
// Create the drop guard:
1251
1246
//
1252
1247
// We rely on macro hygiene to make it impossible for users to access this local variable.
1253
1248
// We use `paste!` to create new hygiene for `$field`.
1254
-
::kernel::macros::paste! {
1249
+
$crate::macros::paste! {
1255
1250
// SAFETY: We forget the guard later when initialization has succeeded.
1256
1251
let [< __ $field _guard >] = unsafe {
1257
-
$crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1252
+
$crate::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1258
1253
};
1259
1254
1260
1255
$crate::__init_internal!(init_slot():
···
1283
1278
//
1284
1279
// We rely on macro hygiene to make it impossible for users to access this local variable.
1285
1280
// We use `paste!` to create new hygiene for `$field`.
1286
-
::kernel::macros::paste! {
1281
+
$crate::macros::paste! {
1287
1282
// SAFETY: We forget the guard later when initialization has succeeded.
1288
1283
let [< __ $field _guard >] = unsafe {
1289
-
$crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1284
+
$crate::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
1290
1285
};
1291
1286
1292
1287
$crate::__init_internal!(init_slot($($use_data)?):
···
1320
1315
// information that is associated to already parsed fragments, so a path fragment
1321
1316
// cannot be used in this position. Doing the retokenization results in valid rust
1322
1317
// code.
1323
-
::kernel::macros::paste!(
1318
+
$crate::macros::paste!(
1324
1319
::core::ptr::write($slot, $t {
1325
1320
$($acc)*
1326
1321
..zeroed
···
1344
1339
// information that is associated to already parsed fragments, so a path fragment
1345
1340
// cannot be used in this position. Doing the retokenization results in valid rust
1346
1341
// code.
1347
-
::kernel::macros::paste!(
1342
+
$crate::macros::paste!(
1348
1343
::core::ptr::write($slot, $t {
1349
1344
$($acc)*
1350
1345
});
···
1399
1394
) => {
1400
1395
// SAFETY: Every field type implements `Zeroable` and padding bytes may be zero.
1401
1396
#[automatically_derived]
1402
-
unsafe impl<$($impl_generics)*> $crate::init::Zeroable for $name<$($ty_generics)*>
1397
+
unsafe impl<$($impl_generics)*> $crate::Zeroable for $name<$($ty_generics)*>
1403
1398
where
1404
1399
$($($whr)*)?
1405
1400
{}
1406
1401
const _: () = {
1407
-
fn assert_zeroable<T: ?::core::marker::Sized + $crate::init::Zeroable>() {}
1402
+
fn assert_zeroable<T: ?::core::marker::Sized + $crate::Zeroable>() {}
1408
1403
fn ensure_zeroable<$($impl_generics)*>()
1409
1404
where $($($whr)*)?
1410
1405
{
+171
rust/kernel/kunit.rs
+171
rust/kernel/kunit.rs
···
40
40
}
41
41
}
42
42
43
+
use macros::kunit_tests;
44
+
43
45
/// Asserts that a boolean expression is `true` at runtime.
44
46
///
45
47
/// Public but hidden since it should only be used from generated tests.
···
162
160
// KUnit supports only a few types (e.g. integers).
163
161
$crate::kunit_assert!($name, $file, $diff, $left == $right);
164
162
}};
163
+
}
164
+
165
+
/// Represents an individual test case.
166
+
///
167
+
/// The [`kunit_unsafe_test_suite!`] macro expects a NULL-terminated list of valid test cases.
168
+
/// Use [`kunit_case_null`] to generate such a delimiter.
169
+
#[doc(hidden)]
170
+
pub const fn kunit_case(
171
+
name: &'static kernel::str::CStr,
172
+
run_case: unsafe extern "C" fn(*mut kernel::bindings::kunit),
173
+
) -> kernel::bindings::kunit_case {
174
+
kernel::bindings::kunit_case {
175
+
run_case: Some(run_case),
176
+
name: name.as_char_ptr(),
177
+
attr: kernel::bindings::kunit_attributes {
178
+
speed: kernel::bindings::kunit_speed_KUNIT_SPEED_NORMAL,
179
+
},
180
+
generate_params: None,
181
+
status: kernel::bindings::kunit_status_KUNIT_SUCCESS,
182
+
module_name: core::ptr::null_mut(),
183
+
log: core::ptr::null_mut(),
184
+
}
185
+
}
186
+
187
+
/// Represents the NULL test case delimiter.
188
+
///
189
+
/// The [`kunit_unsafe_test_suite!`] macro expects a NULL-terminated list of test cases. This
190
+
/// function returns such a delimiter.
191
+
#[doc(hidden)]
192
+
pub const fn kunit_case_null() -> kernel::bindings::kunit_case {
193
+
kernel::bindings::kunit_case {
194
+
run_case: None,
195
+
name: core::ptr::null_mut(),
196
+
generate_params: None,
197
+
attr: kernel::bindings::kunit_attributes {
198
+
speed: kernel::bindings::kunit_speed_KUNIT_SPEED_NORMAL,
199
+
},
200
+
status: kernel::bindings::kunit_status_KUNIT_SUCCESS,
201
+
module_name: core::ptr::null_mut(),
202
+
log: core::ptr::null_mut(),
203
+
}
204
+
}
205
+
206
+
/// Registers a KUnit test suite.
207
+
///
208
+
/// # Safety
209
+
///
210
+
/// `test_cases` must be a NULL terminated array of valid test cases,
211
+
/// whose lifetime is at least that of the test suite (i.e., static).
212
+
///
213
+
/// # Examples
214
+
///
215
+
/// ```ignore
216
+
/// extern "C" fn test_fn(_test: *mut kernel::bindings::kunit) {
217
+
/// let actual = 1 + 1;
218
+
/// let expected = 2;
219
+
/// assert_eq!(actual, expected);
220
+
/// }
221
+
///
222
+
/// static mut KUNIT_TEST_CASES: [kernel::bindings::kunit_case; 2] = [
223
+
/// kernel::kunit::kunit_case(kernel::c_str!("name"), test_fn),
224
+
/// kernel::kunit::kunit_case_null(),
225
+
/// ];
226
+
/// kernel::kunit_unsafe_test_suite!(suite_name, KUNIT_TEST_CASES);
227
+
/// ```
228
+
#[doc(hidden)]
229
+
#[macro_export]
230
+
macro_rules! kunit_unsafe_test_suite {
231
+
($name:ident, $test_cases:ident) => {
232
+
const _: () = {
233
+
const KUNIT_TEST_SUITE_NAME: [::kernel::ffi::c_char; 256] = {
234
+
let name_u8 = ::core::stringify!($name).as_bytes();
235
+
let mut ret = [0; 256];
236
+
237
+
if name_u8.len() > 255 {
238
+
panic!(concat!(
239
+
"The test suite name `",
240
+
::core::stringify!($name),
241
+
"` exceeds the maximum length of 255 bytes."
242
+
));
243
+
}
244
+
245
+
let mut i = 0;
246
+
while i < name_u8.len() {
247
+
ret[i] = name_u8[i] as ::kernel::ffi::c_char;
248
+
i += 1;
249
+
}
250
+
251
+
ret
252
+
};
253
+
254
+
static mut KUNIT_TEST_SUITE: ::kernel::bindings::kunit_suite =
255
+
::kernel::bindings::kunit_suite {
256
+
name: KUNIT_TEST_SUITE_NAME,
257
+
#[allow(unused_unsafe)]
258
+
// SAFETY: `$test_cases` is passed in by the user, and
259
+
// (as documented) must be valid for the lifetime of
260
+
// the suite (i.e., static).
261
+
test_cases: unsafe {
262
+
::core::ptr::addr_of_mut!($test_cases)
263
+
.cast::<::kernel::bindings::kunit_case>()
264
+
},
265
+
suite_init: None,
266
+
suite_exit: None,
267
+
init: None,
268
+
exit: None,
269
+
attr: ::kernel::bindings::kunit_attributes {
270
+
speed: ::kernel::bindings::kunit_speed_KUNIT_SPEED_NORMAL,
271
+
},
272
+
status_comment: [0; 256usize],
273
+
debugfs: ::core::ptr::null_mut(),
274
+
log: ::core::ptr::null_mut(),
275
+
suite_init_err: 0,
276
+
is_init: false,
277
+
};
278
+
279
+
#[used]
280
+
#[allow(unused_unsafe)]
281
+
#[cfg_attr(not(target_os = "macos"), link_section = ".kunit_test_suites")]
282
+
static mut KUNIT_TEST_SUITE_ENTRY: *const ::kernel::bindings::kunit_suite =
283
+
// SAFETY: `KUNIT_TEST_SUITE` is static.
284
+
unsafe { ::core::ptr::addr_of_mut!(KUNIT_TEST_SUITE) };
285
+
};
286
+
};
287
+
}
288
+
289
+
/// Returns whether we are currently running a KUnit test.
290
+
///
291
+
/// In some cases, you need to call test-only code from outside the test case, for example, to
292
+
/// create a function mock. This function allows to change behavior depending on whether we are
293
+
/// currently running a KUnit test or not.
294
+
///
295
+
/// # Examples
296
+
///
297
+
/// This example shows how a function can be mocked to return a well-known value while testing:
298
+
///
299
+
/// ```
300
+
/// # use kernel::kunit::in_kunit_test;
301
+
/// fn fn_mock_example(n: i32) -> i32 {
302
+
/// if in_kunit_test() {
303
+
/// return 100;
304
+
/// }
305
+
///
306
+
/// n + 1
307
+
/// }
308
+
///
309
+
/// let mock_res = fn_mock_example(5);
310
+
/// assert_eq!(mock_res, 100);
311
+
/// ```
312
+
pub fn in_kunit_test() -> bool {
313
+
// SAFETY: `kunit_get_current_test()` is always safe to call (it has fallbacks for
314
+
// when KUnit is not enabled).
315
+
!unsafe { bindings::kunit_get_current_test() }.is_null()
316
+
}
317
+
318
+
#[kunit_tests(rust_kernel_kunit)]
319
+
mod tests {
320
+
use super::*;
321
+
322
+
#[test]
323
+
fn rust_test_kunit_example_test() {
324
+
#![expect(clippy::eq_op)]
325
+
assert_eq!(1 + 1, 2);
326
+
}
327
+
328
+
#[test]
329
+
fn rust_test_kunit_in_kunit_test() {
330
+
assert!(in_kunit_test());
331
+
}
165
332
}
+6
-3
rust/kernel/lib.rs
+6
-3
rust/kernel/lib.rs
···
19
19
#![cfg_attr(not(CONFIG_RUSTC_HAS_COERCE_POINTEE), feature(unsize))]
20
20
#![feature(inline_const)]
21
21
#![feature(lint_reasons)]
22
+
// Stable in Rust 1.82
23
+
#![feature(raw_ref_op)]
22
24
// Stable in Rust 1.83
23
25
#![feature(const_maybe_uninit_as_mut_ptr)]
24
26
#![feature(const_mut_refs)]
···
46
44
pub mod device;
47
45
pub mod device_id;
48
46
pub mod devres;
47
+
pub mod dma;
49
48
pub mod driver;
50
49
pub mod error;
51
50
pub mod faux;
···
115
112
/// Creates an initialiser for the module.
116
113
///
117
114
/// It is called when the module is loaded.
118
-
fn init(module: &'static ThisModule) -> impl init::PinInit<Self, error::Error>;
115
+
fn init(module: &'static ThisModule) -> impl pin_init::PinInit<Self, error::Error>;
119
116
}
120
117
121
118
impl<T: Module> InPlaceModule for T {
122
-
fn init(module: &'static ThisModule) -> impl init::PinInit<Self, error::Error> {
119
+
fn init(module: &'static ThisModule) -> impl pin_init::PinInit<Self, error::Error> {
123
120
let initer = move |slot: *mut Self| {
124
121
let m = <Self as Module>::init(module)?;
125
122
···
129
126
};
130
127
131
128
// SAFETY: On success, `initer` always fully initialises an instance of `Self`.
132
-
unsafe { init::pin_init_from_closure(initer) }
129
+
unsafe { pin_init::pin_init_from_closure(initer) }
133
130
}
134
131
}
135
132
+382
-95
rust/kernel/list.rs
+382
-95
rust/kernel/list.rs
···
4
4
5
5
//! A linked list implementation.
6
6
7
-
use crate::init::PinInit;
8
7
use crate::sync::ArcBorrow;
9
8
use crate::types::Opaque;
10
9
use core::iter::{DoubleEndedIterator, FusedIterator};
11
10
use core::marker::PhantomData;
12
11
use core::ptr;
12
+
use pin_init::PinInit;
13
13
14
14
mod impl_list_item_mod;
15
15
pub use self::impl_list_item_mod::{
···
245
245
self.first.is_null()
246
246
}
247
247
248
-
/// Add the provided item to the back of the list.
249
-
pub fn push_back(&mut self, item: ListArc<T, ID>) {
248
+
/// Inserts `item` before `next` in the cycle.
249
+
///
250
+
/// Returns a pointer to the newly inserted element. Never changes `self.first` unless the list
251
+
/// is empty.
252
+
///
253
+
/// # Safety
254
+
///
255
+
/// * `next` must be an element in this list or null.
256
+
/// * if `next` is null, then the list must be empty.
257
+
unsafe fn insert_inner(
258
+
&mut self,
259
+
item: ListArc<T, ID>,
260
+
next: *mut ListLinksFields,
261
+
) -> *mut ListLinksFields {
250
262
let raw_item = ListArc::into_raw(item);
251
263
// SAFETY:
252
264
// * We just got `raw_item` from a `ListArc`, so it's in an `Arc`.
···
271
259
// SAFETY: We have not yet called `post_remove`, so `list_links` is still valid.
272
260
let item = unsafe { ListLinks::fields(list_links) };
273
261
274
-
if self.first.is_null() {
275
-
self.first = item;
262
+
// Check if the list is empty.
263
+
if next.is_null() {
276
264
// SAFETY: The caller just gave us ownership of these fields.
277
265
// INVARIANT: A linked list with one item should be cyclic.
278
266
unsafe {
279
267
(*item).next = item;
280
268
(*item).prev = item;
281
269
}
270
+
self.first = item;
282
271
} else {
283
-
let next = self.first;
284
272
// SAFETY: By the type invariant, this pointer is valid or null. We just checked that
285
273
// it's not null, so it must be valid.
286
274
let prev = unsafe { (*next).prev };
···
294
282
(*next).prev = item;
295
283
}
296
284
}
285
+
286
+
item
287
+
}
288
+
289
+
/// Add the provided item to the back of the list.
290
+
pub fn push_back(&mut self, item: ListArc<T, ID>) {
291
+
// SAFETY:
292
+
// * `self.first` is null or in the list.
293
+
// * `self.first` is only null if the list is empty.
294
+
unsafe { self.insert_inner(item, self.first) };
297
295
}
298
296
299
297
/// Add the provided item to the front of the list.
300
298
pub fn push_front(&mut self, item: ListArc<T, ID>) {
301
-
let raw_item = ListArc::into_raw(item);
302
299
// SAFETY:
303
-
// * We just got `raw_item` from a `ListArc`, so it's in an `Arc`.
304
-
// * If this requirement is violated, then the previous caller of `prepare_to_insert`
305
-
// violated the safety requirement that they can't give up ownership of the `ListArc`
306
-
// until they call `post_remove`.
307
-
// * We own the `ListArc`.
308
-
// * Removing items] from this list is always done using `remove_internal_inner`, which
309
-
// calls `post_remove` before giving up ownership.
310
-
let list_links = unsafe { T::prepare_to_insert(raw_item) };
311
-
// SAFETY: We have not yet called `post_remove`, so `list_links` is still valid.
312
-
let item = unsafe { ListLinks::fields(list_links) };
300
+
// * `self.first` is null or in the list.
301
+
// * `self.first` is only null if the list is empty.
302
+
let new_elem = unsafe { self.insert_inner(item, self.first) };
313
303
314
-
if self.first.is_null() {
315
-
// SAFETY: The caller just gave us ownership of these fields.
316
-
// INVARIANT: A linked list with one item should be cyclic.
317
-
unsafe {
318
-
(*item).next = item;
319
-
(*item).prev = item;
320
-
}
321
-
} else {
322
-
let next = self.first;
323
-
// SAFETY: We just checked that `next` is non-null.
324
-
let prev = unsafe { (*next).prev };
325
-
// SAFETY: Pointers in a linked list are never dangling, and the caller just gave us
326
-
// ownership of the fields on `item`.
327
-
// INVARIANT: This correctly inserts `item` between `prev` and `next`.
328
-
unsafe {
329
-
(*item).next = next;
330
-
(*item).prev = prev;
331
-
(*prev).next = item;
332
-
(*next).prev = item;
333
-
}
334
-
}
335
-
self.first = item;
304
+
// INVARIANT: `new_elem` is in the list because we just inserted it.
305
+
self.first = new_elem;
336
306
}
337
307
338
308
/// Removes the last item from this list.
···
483
489
other.first = ptr::null_mut();
484
490
}
485
491
486
-
/// Returns a cursor to the first element of the list.
487
-
///
488
-
/// If the list is empty, this returns `None`.
489
-
pub fn cursor_front(&mut self) -> Option<Cursor<'_, T, ID>> {
490
-
if self.first.is_null() {
491
-
None
492
-
} else {
493
-
Some(Cursor {
494
-
current: self.first,
495
-
list: self,
496
-
})
492
+
/// Returns a cursor that points before the first element of the list.
493
+
pub fn cursor_front(&mut self) -> Cursor<'_, T, ID> {
494
+
// INVARIANT: `self.first` is in this list.
495
+
Cursor {
496
+
next: self.first,
497
+
list: self,
498
+
}
499
+
}
500
+
501
+
/// Returns a cursor that points after the last element in the list.
502
+
pub fn cursor_back(&mut self) -> Cursor<'_, T, ID> {
503
+
// INVARIANT: `next` is allowed to be null.
504
+
Cursor {
505
+
next: core::ptr::null_mut(),
506
+
list: self,
497
507
}
498
508
}
499
509
···
577
579
578
580
/// A cursor into a [`List`].
579
581
///
582
+
/// A cursor always rests between two elements in the list. This means that a cursor has a previous
583
+
/// and next element, but no current element. It also means that it's possible to have a cursor
584
+
/// into an empty list.
585
+
///
586
+
/// # Examples
587
+
///
588
+
/// ```
589
+
/// use kernel::prelude::*;
590
+
/// use kernel::list::{List, ListArc, ListLinks};
591
+
///
592
+
/// #[pin_data]
593
+
/// struct ListItem {
594
+
/// value: u32,
595
+
/// #[pin]
596
+
/// links: ListLinks,
597
+
/// }
598
+
///
599
+
/// impl ListItem {
600
+
/// fn new(value: u32) -> Result<ListArc<Self>> {
601
+
/// ListArc::pin_init(try_pin_init!(Self {
602
+
/// value,
603
+
/// links <- ListLinks::new(),
604
+
/// }), GFP_KERNEL)
605
+
/// }
606
+
/// }
607
+
///
608
+
/// kernel::list::impl_has_list_links! {
609
+
/// impl HasListLinks<0> for ListItem { self.links }
610
+
/// }
611
+
/// kernel::list::impl_list_arc_safe! {
612
+
/// impl ListArcSafe<0> for ListItem { untracked; }
613
+
/// }
614
+
/// kernel::list::impl_list_item! {
615
+
/// impl ListItem<0> for ListItem { using ListLinks; }
616
+
/// }
617
+
///
618
+
/// // Use a cursor to remove the first element with the given value.
619
+
/// fn remove_first(list: &mut List<ListItem>, value: u32) -> Option<ListArc<ListItem>> {
620
+
/// let mut cursor = list.cursor_front();
621
+
/// while let Some(next) = cursor.peek_next() {
622
+
/// if next.value == value {
623
+
/// return Some(next.remove());
624
+
/// }
625
+
/// cursor.move_next();
626
+
/// }
627
+
/// None
628
+
/// }
629
+
///
630
+
/// // Use a cursor to remove the last element with the given value.
631
+
/// fn remove_last(list: &mut List<ListItem>, value: u32) -> Option<ListArc<ListItem>> {
632
+
/// let mut cursor = list.cursor_back();
633
+
/// while let Some(prev) = cursor.peek_prev() {
634
+
/// if prev.value == value {
635
+
/// return Some(prev.remove());
636
+
/// }
637
+
/// cursor.move_prev();
638
+
/// }
639
+
/// None
640
+
/// }
641
+
///
642
+
/// // Use a cursor to remove all elements with the given value. The removed elements are moved to
643
+
/// // a new list.
644
+
/// fn remove_all(list: &mut List<ListItem>, value: u32) -> List<ListItem> {
645
+
/// let mut out = List::new();
646
+
/// let mut cursor = list.cursor_front();
647
+
/// while let Some(next) = cursor.peek_next() {
648
+
/// if next.value == value {
649
+
/// out.push_back(next.remove());
650
+
/// } else {
651
+
/// cursor.move_next();
652
+
/// }
653
+
/// }
654
+
/// out
655
+
/// }
656
+
///
657
+
/// // Use a cursor to insert a value at a specific index. Returns an error if the index is out of
658
+
/// // bounds.
659
+
/// fn insert_at(list: &mut List<ListItem>, new: ListArc<ListItem>, idx: usize) -> Result {
660
+
/// let mut cursor = list.cursor_front();
661
+
/// for _ in 0..idx {
662
+
/// if !cursor.move_next() {
663
+
/// return Err(EINVAL);
664
+
/// }
665
+
/// }
666
+
/// cursor.insert_next(new);
667
+
/// Ok(())
668
+
/// }
669
+
///
670
+
/// // Merge two sorted lists into a single sorted list.
671
+
/// fn merge_sorted(list: &mut List<ListItem>, merge: List<ListItem>) {
672
+
/// let mut cursor = list.cursor_front();
673
+
/// for to_insert in merge {
674
+
/// while let Some(next) = cursor.peek_next() {
675
+
/// if to_insert.value < next.value {
676
+
/// break;
677
+
/// }
678
+
/// cursor.move_next();
679
+
/// }
680
+
/// cursor.insert_prev(to_insert);
681
+
/// }
682
+
/// }
683
+
///
684
+
/// let mut list = List::new();
685
+
/// list.push_back(ListItem::new(14)?);
686
+
/// list.push_back(ListItem::new(12)?);
687
+
/// list.push_back(ListItem::new(10)?);
688
+
/// list.push_back(ListItem::new(12)?);
689
+
/// list.push_back(ListItem::new(15)?);
690
+
/// list.push_back(ListItem::new(14)?);
691
+
/// assert_eq!(remove_all(&mut list, 12).iter().count(), 2);
692
+
/// // [14, 10, 15, 14]
693
+
/// assert!(remove_first(&mut list, 14).is_some());
694
+
/// // [10, 15, 14]
695
+
/// insert_at(&mut list, ListItem::new(12)?, 2)?;
696
+
/// // [10, 15, 12, 14]
697
+
/// assert!(remove_last(&mut list, 15).is_some());
698
+
/// // [10, 12, 14]
699
+
///
700
+
/// let mut list2 = List::new();
701
+
/// list2.push_back(ListItem::new(11)?);
702
+
/// list2.push_back(ListItem::new(13)?);
703
+
/// merge_sorted(&mut list, list2);
704
+
///
705
+
/// let mut items = list.into_iter();
706
+
/// assert_eq!(items.next().unwrap().value, 10);
707
+
/// assert_eq!(items.next().unwrap().value, 11);
708
+
/// assert_eq!(items.next().unwrap().value, 12);
709
+
/// assert_eq!(items.next().unwrap().value, 13);
710
+
/// assert_eq!(items.next().unwrap().value, 14);
711
+
/// assert!(items.next().is_none());
712
+
/// # Result::<(), Error>::Ok(())
713
+
/// ```
714
+
///
580
715
/// # Invariants
581
716
///
582
-
/// The `current` pointer points a value in `list`.
717
+
/// The `next` pointer is null or points a value in `list`.
583
718
pub struct Cursor<'a, T: ?Sized + ListItem<ID>, const ID: u64 = 0> {
584
-
current: *mut ListLinksFields,
585
719
list: &'a mut List<T, ID>,
720
+
/// Points at the element after this cursor, or null if the cursor is after the last element.
721
+
next: *mut ListLinksFields,
586
722
}
587
723
588
724
impl<'a, T: ?Sized + ListItem<ID>, const ID: u64> Cursor<'a, T, ID> {
589
-
/// Access the current element of this cursor.
590
-
pub fn current(&self) -> ArcBorrow<'_, T> {
591
-
// SAFETY: The `current` pointer points a value in the list.
592
-
let me = unsafe { T::view_value(ListLinks::from_fields(self.current)) };
725
+
/// Returns a pointer to the element before the cursor.
726
+
///
727
+
/// Returns null if there is no element before the cursor.
728
+
fn prev_ptr(&self) -> *mut ListLinksFields {
729
+
let mut next = self.next;
730
+
let first = self.list.first;
731
+
if next == first {
732
+
// We are before the first element.
733
+
return core::ptr::null_mut();
734
+
}
735
+
736
+
if next.is_null() {
737
+
// We are after the last element, so we need a pointer to the last element, which is
738
+
// the same as `(*first).prev`.
739
+
next = first;
740
+
}
741
+
742
+
// SAFETY: `next` can't be null, because then `first` must also be null, but in that case
743
+
// we would have exited at the `next == first` check. Thus, `next` is an element in the
744
+
// list, so we can access its `prev` pointer.
745
+
unsafe { (*next).prev }
746
+
}
747
+
748
+
/// Access the element after this cursor.
749
+
pub fn peek_next(&mut self) -> Option<CursorPeek<'_, 'a, T, true, ID>> {
750
+
if self.next.is_null() {
751
+
return None;
752
+
}
753
+
754
+
// INVARIANT:
755
+
// * We just checked that `self.next` is non-null, so it must be in `self.list`.
756
+
// * `ptr` is equal to `self.next`.
757
+
Some(CursorPeek {
758
+
ptr: self.next,
759
+
cursor: self,
760
+
})
761
+
}
762
+
763
+
/// Access the element before this cursor.
764
+
pub fn peek_prev(&mut self) -> Option<CursorPeek<'_, 'a, T, false, ID>> {
765
+
let prev = self.prev_ptr();
766
+
767
+
if prev.is_null() {
768
+
return None;
769
+
}
770
+
771
+
// INVARIANT:
772
+
// * We just checked that `prev` is non-null, so it must be in `self.list`.
773
+
// * `self.prev_ptr()` never returns `self.next`.
774
+
Some(CursorPeek {
775
+
ptr: prev,
776
+
cursor: self,
777
+
})
778
+
}
779
+
780
+
/// Move the cursor one element forward.
781
+
///
782
+
/// If the cursor is after the last element, then this call does nothing. This call returns
783
+
/// `true` if the cursor's position was changed.
784
+
pub fn move_next(&mut self) -> bool {
785
+
if self.next.is_null() {
786
+
return false;
787
+
}
788
+
789
+
// SAFETY: `self.next` is an element in the list and we borrow the list mutably, so we can
790
+
// access the `next` field.
791
+
let mut next = unsafe { (*self.next).next };
792
+
793
+
if next == self.list.first {
794
+
next = core::ptr::null_mut();
795
+
}
796
+
797
+
// INVARIANT: `next` is either null or the next element after an element in the list.
798
+
self.next = next;
799
+
true
800
+
}
801
+
802
+
/// Move the cursor one element backwards.
803
+
///
804
+
/// If the cursor is before the first element, then this call does nothing. This call returns
805
+
/// `true` if the cursor's position was changed.
806
+
pub fn move_prev(&mut self) -> bool {
807
+
if self.next == self.list.first {
808
+
return false;
809
+
}
810
+
811
+
// INVARIANT: `prev_ptr()` always returns a pointer that is null or in the list.
812
+
self.next = self.prev_ptr();
813
+
true
814
+
}
815
+
816
+
/// Inserts an element where the cursor is pointing and get a pointer to the new element.
817
+
fn insert_inner(&mut self, item: ListArc<T, ID>) -> *mut ListLinksFields {
818
+
let ptr = if self.next.is_null() {
819
+
self.list.first
820
+
} else {
821
+
self.next
822
+
};
823
+
// SAFETY:
824
+
// * `ptr` is an element in the list or null.
825
+
// * if `ptr` is null, then `self.list.first` is null so the list is empty.
826
+
let item = unsafe { self.list.insert_inner(item, ptr) };
827
+
if self.next == self.list.first {
828
+
// INVARIANT: We just inserted `item`, so it's a member of list.
829
+
self.list.first = item;
830
+
}
831
+
item
832
+
}
833
+
834
+
/// Insert an element at this cursor's location.
835
+
pub fn insert(mut self, item: ListArc<T, ID>) {
836
+
// This is identical to `insert_prev`, but consumes the cursor. This is helpful because it
837
+
// reduces confusion when the last operation on the cursor is an insertion; in that case,
838
+
// you just want to insert the element at the cursor, and it is confusing that the call
839
+
// involves the word prev or next.
840
+
self.insert_inner(item);
841
+
}
842
+
843
+
/// Inserts an element after this cursor.
844
+
///
845
+
/// After insertion, the new element will be after the cursor.
846
+
pub fn insert_next(&mut self, item: ListArc<T, ID>) {
847
+
self.next = self.insert_inner(item);
848
+
}
849
+
850
+
/// Inserts an element before this cursor.
851
+
///
852
+
/// After insertion, the new element will be before the cursor.
853
+
pub fn insert_prev(&mut self, item: ListArc<T, ID>) {
854
+
self.insert_inner(item);
855
+
}
856
+
857
+
/// Remove the next element from the list.
858
+
pub fn remove_next(&mut self) -> Option<ListArc<T, ID>> {
859
+
self.peek_next().map(|v| v.remove())
860
+
}
861
+
862
+
/// Remove the previous element from the list.
863
+
pub fn remove_prev(&mut self) -> Option<ListArc<T, ID>> {
864
+
self.peek_prev().map(|v| v.remove())
865
+
}
866
+
}
867
+
868
+
/// References the element in the list next to the cursor.
869
+
///
870
+
/// # Invariants
871
+
///
872
+
/// * `ptr` is an element in `self.cursor.list`.
873
+
/// * `ISNEXT == (self.ptr == self.cursor.next)`.
874
+
pub struct CursorPeek<'a, 'b, T: ?Sized + ListItem<ID>, const ISNEXT: bool, const ID: u64> {
875
+
cursor: &'a mut Cursor<'b, T, ID>,
876
+
ptr: *mut ListLinksFields,
877
+
}
878
+
879
+
impl<'a, 'b, T: ?Sized + ListItem<ID>, const ISNEXT: bool, const ID: u64>
880
+
CursorPeek<'a, 'b, T, ISNEXT, ID>
881
+
{
882
+
/// Remove the element from the list.
883
+
pub fn remove(self) -> ListArc<T, ID> {
884
+
if ISNEXT {
885
+
self.cursor.move_next();
886
+
}
887
+
888
+
// INVARIANT: `self.ptr` is not equal to `self.cursor.next` due to the above `move_next`
889
+
// call.
890
+
// SAFETY: By the type invariants of `Self`, `next` is not null, so `next` is an element of
891
+
// `self.cursor.list` by the type invariants of `Cursor`.
892
+
unsafe { self.cursor.list.remove_internal(self.ptr) }
893
+
}
894
+
895
+
/// Access this value as an [`ArcBorrow`].
896
+
pub fn arc(&self) -> ArcBorrow<'_, T> {
897
+
// SAFETY: `self.ptr` points at an element in `self.cursor.list`.
898
+
let me = unsafe { T::view_value(ListLinks::from_fields(self.ptr)) };
593
899
// SAFETY:
594
900
// * All values in a list are stored in an `Arc`.
595
901
// * The value cannot be removed from the list for the duration of the lifetime annotated
596
902
// on the returned `ArcBorrow`, because removing it from the list would require mutable
597
-
// access to the cursor or the list. However, the `ArcBorrow` holds an immutable borrow
598
-
// on the cursor, which in turn holds a mutable borrow on the list, so any such
599
-
// mutable access requires first releasing the immutable borrow on the cursor.
903
+
// access to the `CursorPeek`, the `Cursor` or the `List`. However, the `ArcBorrow` holds
904
+
// an immutable borrow on the `CursorPeek`, which in turn holds a mutable borrow on the
905
+
// `Cursor`, which in turn holds a mutable borrow on the `List`, so any such mutable
906
+
// access requires first releasing the immutable borrow on the `CursorPeek`.
600
907
// * Values in a list never have a `UniqueArc` reference, because the list has a `ListArc`
601
908
// reference, and `UniqueArc` references must be unique.
602
909
unsafe { ArcBorrow::from_raw(me) }
603
910
}
911
+
}
604
912
605
-
/// Move the cursor to the next element.
606
-
pub fn next(self) -> Option<Cursor<'a, T, ID>> {
607
-
// SAFETY: The `current` field is always in a list.
608
-
let next = unsafe { (*self.current).next };
913
+
impl<'a, 'b, T: ?Sized + ListItem<ID>, const ISNEXT: bool, const ID: u64> core::ops::Deref
914
+
for CursorPeek<'a, 'b, T, ISNEXT, ID>
915
+
{
916
+
// If you change the `ptr` field to have type `ArcBorrow<'a, T>`, it might seem like you could
917
+
// get rid of the `CursorPeek::arc` method and change the deref target to `ArcBorrow<'a, T>`.
918
+
// However, that doesn't work because 'a is too long. You could obtain an `ArcBorrow<'a, T>`
919
+
// and then call `CursorPeek::remove` without giving up the `ArcBorrow<'a, T>`, which would be
920
+
// unsound.
921
+
type Target = T;
609
922
610
-
if next == self.list.first {
611
-
None
612
-
} else {
613
-
// INVARIANT: Since `self.current` is in the `list`, its `next` pointer is also in the
614
-
// `list`.
615
-
Some(Cursor {
616
-
current: next,
617
-
list: self.list,
618
-
})
619
-
}
620
-
}
923
+
fn deref(&self) -> &T {
924
+
// SAFETY: `self.ptr` points at an element in `self.cursor.list`.
925
+
let me = unsafe { T::view_value(ListLinks::from_fields(self.ptr)) };
621
926
622
-
/// Move the cursor to the previous element.
623
-
pub fn prev(self) -> Option<Cursor<'a, T, ID>> {
624
-
// SAFETY: The `current` field is always in a list.
625
-
let prev = unsafe { (*self.current).prev };
626
-
627
-
if self.current == self.list.first {
628
-
None
629
-
} else {
630
-
// INVARIANT: Since `self.current` is in the `list`, its `prev` pointer is also in the
631
-
// `list`.
632
-
Some(Cursor {
633
-
current: prev,
634
-
list: self.list,
635
-
})
636
-
}
637
-
}
638
-
639
-
/// Remove the current element from the list.
640
-
pub fn remove(self) -> ListArc<T, ID> {
641
-
// SAFETY: The `current` pointer always points at a member of the list.
642
-
unsafe { self.list.remove_internal(self.current) }
927
+
// SAFETY: The value cannot be removed from the list for the duration of the lifetime
928
+
// annotated on the returned `&T`, because removing it from the list would require mutable
929
+
// access to the `CursorPeek`, the `Cursor` or the `List`. However, the `&T` holds an
930
+
// immutable borrow on the `CursorPeek`, which in turn holds a mutable borrow on the
931
+
// `Cursor`, which in turn holds a mutable borrow on the `List`, so any such mutable access
932
+
// requires first releasing the immutable borrow on the `CursorPeek`.
933
+
unsafe { &*me }
643
934
}
644
935
}
645
936
+2
-2
rust/kernel/net/phy.rs
+2
-2
rust/kernel/net/phy.rs
···
790
790
/// DeviceId::new_with_driver::<PhySample>()
791
791
/// ],
792
792
/// name: "rust_sample_phy",
793
-
/// author: "Rust for Linux Contributors",
793
+
/// authors: ["Rust for Linux Contributors"],
794
794
/// description: "Rust sample PHYs driver",
795
795
/// license: "GPL",
796
796
/// }
···
819
819
/// module! {
820
820
/// type: Module,
821
821
/// name: "rust_sample_phy",
822
-
/// author: "Rust for Linux Contributors",
822
+
/// authors: ["Rust for Linux Contributors"],
823
823
/// description: "Rust sample PHYs driver",
824
824
/// license: "GPL",
825
825
/// }
+1
-1
rust/kernel/pci.rs
+1
-1
rust/kernel/pci.rs
+1
-1
rust/kernel/platform.rs
+1
-1
rust/kernel/platform.rs
+5
-3
rust/kernel/prelude.rs
+5
-3
rust/kernel/prelude.rs
···
17
17
pub use crate::alloc::{flags::*, Box, KBox, KVBox, KVVec, KVec, VBox, VVec, Vec};
18
18
19
19
#[doc(no_inline)]
20
-
pub use macros::{module, pin_data, pinned_drop, vtable, Zeroable};
20
+
pub use macros::{export, module, vtable};
21
+
22
+
pub use pin_init::{init, pin_data, pin_init, pinned_drop, InPlaceWrite, Init, PinInit, Zeroable};
21
23
22
24
pub use super::{build_assert, build_error};
23
25
···
30
28
pub use super::{dev_alert, dev_crit, dev_dbg, dev_emerg, dev_err, dev_info, dev_notice, dev_warn};
31
29
pub use super::{pr_alert, pr_crit, pr_debug, pr_emerg, pr_err, pr_info, pr_notice, pr_warn};
32
30
33
-
pub use super::{init, pin_init, try_init, try_pin_init};
31
+
pub use super::{try_init, try_pin_init};
34
32
35
33
pub use super::static_assert;
36
34
···
38
36
39
37
pub use super::{str::CStr, ThisModule};
40
38
41
-
pub use super::init::{InPlaceInit, InPlaceWrite, Init, PinInit};
39
+
pub use super::init::InPlaceInit;
42
40
43
41
pub use super::current;
+5
-5
rust/kernel/print.rs
+5
-5
rust/kernel/print.rs
···
6
6
//!
7
7
//! Reference: <https://docs.kernel.org/core-api/printk-basics.html>
8
8
9
-
use core::{
9
+
use crate::{
10
10
ffi::{c_char, c_void},
11
-
fmt,
11
+
prelude::*,
12
+
str::RawFormatter,
12
13
};
13
-
14
-
use crate::str::RawFormatter;
14
+
use core::fmt;
15
15
16
16
// Called from `vsprintf` with format specifier `%pA`.
17
17
#[expect(clippy::missing_safety_doc)]
18
-
#[no_mangle]
18
+
#[export]
19
19
unsafe extern "C" fn rust_fmt_argument(
20
20
buf: *mut c_char,
21
21
end: *mut c_char,
+6
-6
rust/kernel/rbtree.rs
+6
-6
rust/kernel/rbtree.rs
···
886
886
/// # Safety
887
887
///
888
888
/// - `node` must be a valid pointer to a node in an [`RBTree`].
889
-
/// - The caller has immutable access to `node` for the duration of 'b.
889
+
/// - The caller has immutable access to `node` for the duration of `'b`.
890
890
unsafe fn to_key_value<'b>(node: NonNull<bindings::rb_node>) -> (&'b K, &'b V) {
891
891
// SAFETY: the caller guarantees that `node` is a valid pointer in an `RBTree`.
892
892
let (k, v) = unsafe { Self::to_key_value_raw(node) };
···
897
897
/// # Safety
898
898
///
899
899
/// - `node` must be a valid pointer to a node in an [`RBTree`].
900
-
/// - The caller has mutable access to `node` for the duration of 'b.
900
+
/// - The caller has mutable access to `node` for the duration of `'b`.
901
901
unsafe fn to_key_value_mut<'b>(node: NonNull<bindings::rb_node>) -> (&'b K, &'b mut V) {
902
902
// SAFETY: the caller guarantees that `node` is a valid pointer in an `RBTree`.
903
903
let (k, v) = unsafe { Self::to_key_value_raw(node) };
···
908
908
/// # Safety
909
909
///
910
910
/// - `node` must be a valid pointer to a node in an [`RBTree`].
911
-
/// - The caller has immutable access to the key for the duration of 'b.
911
+
/// - The caller has immutable access to the key for the duration of `'b`.
912
912
unsafe fn to_key_value_raw<'b>(node: NonNull<bindings::rb_node>) -> (&'b K, *mut V) {
913
913
// SAFETY: By the type invariant of `Self`, all non-null `rb_node` pointers stored in `self`
914
914
// point to the links field of `Node<K, V>` objects.
···
1168
1168
fn insert(self, node: RBTreeNode<K, V>) -> &'a mut V {
1169
1169
let node = KBox::into_raw(node.node);
1170
1170
1171
-
// SAFETY: `node` is valid at least until we call `Box::from_raw`, which only happens when
1171
+
// SAFETY: `node` is valid at least until we call `KBox::from_raw`, which only happens when
1172
1172
// the node is removed or replaced.
1173
1173
let node_links = unsafe { addr_of_mut!((*node).links) };
1174
1174
1175
1175
// INVARIANT: We are linking in a new node, which is valid. It remains valid because we
1176
-
// "forgot" it with `Box::into_raw`.
1176
+
// "forgot" it with `KBox::into_raw`.
1177
1177
// SAFETY: The type invariants of `RawVacantEntry` are exactly the safety requirements of `rb_link_node`.
1178
1178
unsafe { bindings::rb_link_node(node_links, self.parent, self.child_field_of_parent) };
1179
1179
···
1259
1259
fn replace(self, node: RBTreeNode<K, V>) -> RBTreeNode<K, V> {
1260
1260
let node = KBox::into_raw(node.node);
1261
1261
1262
-
// SAFETY: `node` is valid at least until we call `Box::from_raw`, which only happens when
1262
+
// SAFETY: `node` is valid at least until we call `KBox::from_raw`, which only happens when
1263
1263
// the node is removed or replaced.
1264
1264
let new_node_links = unsafe { addr_of_mut!((*node).links) };
1265
1265
+1
-1
rust/kernel/seq_file.rs
+1
-1
rust/kernel/seq_file.rs
···
18
18
///
19
19
/// # Safety
20
20
///
21
-
/// The caller must ensure that for the duration of 'a the following is satisfied:
21
+
/// The caller must ensure that for the duration of `'a` the following is satisfied:
22
22
/// * The pointer points at a valid `struct seq_file`.
23
23
/// * The `struct seq_file` is not accessed from any other thread.
24
24
pub unsafe fn from_raw<'a>(ptr: *mut bindings::seq_file) -> &'a SeqFile {
+46
rust/kernel/str.rs
+46
rust/kernel/str.rs
···
31
31
// SAFETY: `BStr` is transparent to `[u8]`.
32
32
unsafe { &*(bytes as *const [u8] as *const BStr) }
33
33
}
34
+
35
+
/// Strip a prefix from `self`. Delegates to [`slice::strip_prefix`].
36
+
///
37
+
/// # Examples
38
+
///
39
+
/// ```
40
+
/// # use kernel::b_str;
41
+
/// assert_eq!(Some(b_str!("bar")), b_str!("foobar").strip_prefix(b_str!("foo")));
42
+
/// assert_eq!(None, b_str!("foobar").strip_prefix(b_str!("bar")));
43
+
/// assert_eq!(Some(b_str!("foobar")), b_str!("foobar").strip_prefix(b_str!("")));
44
+
/// assert_eq!(Some(b_str!("")), b_str!("foobar").strip_prefix(b_str!("foobar")));
45
+
/// ```
46
+
pub fn strip_prefix(&self, pattern: impl AsRef<Self>) -> Option<&BStr> {
47
+
self.deref()
48
+
.strip_prefix(pattern.as_ref().deref())
49
+
.map(Self::from_bytes)
50
+
}
34
51
}
35
52
36
53
impl fmt::Display for BStr {
···
122
105
#[inline]
123
106
fn deref(&self) -> &Self::Target {
124
107
&self.0
108
+
}
109
+
}
110
+
111
+
impl PartialEq for BStr {
112
+
fn eq(&self, other: &Self) -> bool {
113
+
self.deref().eq(other.deref())
114
+
}
115
+
}
116
+
117
+
impl<Idx> Index<Idx> for BStr
118
+
where
119
+
[u8]: Index<Idx, Output = [u8]>,
120
+
{
121
+
type Output = Self;
122
+
123
+
fn index(&self, index: Idx) -> &Self::Output {
124
+
BStr::from_bytes(&self.0[index])
125
+
}
126
+
}
127
+
128
+
impl AsRef<BStr> for [u8] {
129
+
fn as_ref(&self) -> &BStr {
130
+
BStr::from_bytes(self)
131
+
}
132
+
}
133
+
134
+
impl AsRef<BStr> for BStr {
135
+
fn as_ref(&self) -> &BStr {
136
+
self
125
137
}
126
138
}
127
139
+3
-2
rust/kernel/sync.rs
+3
-2
rust/kernel/sync.rs
···
5
5
//! This module contains the kernel APIs related to synchronisation that have been ported or
6
6
//! wrapped for usage by Rust code in the kernel.
7
7
8
-
use crate::pin_init;
9
8
use crate::prelude::*;
10
9
use crate::types::Opaque;
10
+
use pin_init;
11
11
12
12
mod arc;
13
13
mod condvar;
···
41
41
///
42
42
/// # Example
43
43
/// ```
44
-
/// # use kernel::{c_str, stack_pin_init};
44
+
/// # use kernel::c_str;
45
45
/// # use kernel::alloc::KBox;
46
46
/// # use kernel::types::ForeignOwnable;
47
47
/// # use kernel::sync::{LockClassKey, SpinLock};
48
+
/// # use pin_init::stack_pin_init;
48
49
///
49
50
/// let key = KBox::pin_init(LockClassKey::new_dynamic(), GFP_KERNEL)?;
50
51
/// let key_ptr = key.into_foreign();
+76
-5
rust/kernel/sync/arc.rs
+76
-5
rust/kernel/sync/arc.rs
···
19
19
use crate::{
20
20
alloc::{AllocError, Flags, KBox},
21
21
bindings,
22
-
init::{self, InPlaceInit, Init, PinInit},
22
+
init::InPlaceInit,
23
23
try_init,
24
24
types::{ForeignOwnable, Opaque},
25
25
};
···
32
32
pin::Pin,
33
33
ptr::NonNull,
34
34
};
35
-
use macros::pin_data;
35
+
use pin_init::{self, pin_data, InPlaceWrite, Init, PinInit};
36
36
37
37
mod std_vendor;
38
38
···
202
202
// the reference count reaches zero and `T` is dropped.
203
203
unsafe impl<T: ?Sized + Sync + Send> Sync for Arc<T> {}
204
204
205
+
impl<T> InPlaceInit<T> for Arc<T> {
206
+
type PinnedSelf = Self;
207
+
208
+
#[inline]
209
+
fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::PinnedSelf, E>
210
+
where
211
+
E: From<AllocError>,
212
+
{
213
+
UniqueArc::try_pin_init(init, flags).map(|u| u.into())
214
+
}
215
+
216
+
#[inline]
217
+
fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
218
+
where
219
+
E: From<AllocError>,
220
+
{
221
+
UniqueArc::try_init(init, flags).map(|u| u.into())
222
+
}
223
+
}
224
+
205
225
impl<T> Arc<T> {
206
226
/// Constructs a new reference counted instance of `T`.
207
227
pub fn new(contents: T, flags: Flags) -> Result<Self, AllocError> {
···
263
243
let ptr = self.ptr.as_ptr();
264
244
core::mem::forget(self);
265
245
// SAFETY: The pointer is valid.
246
+
unsafe { core::ptr::addr_of!((*ptr).data) }
247
+
}
248
+
249
+
/// Return a raw pointer to the data in this arc.
250
+
pub fn as_ptr(this: &Self) -> *const T {
251
+
let ptr = this.ptr.as_ptr();
252
+
253
+
// SAFETY: As `ptr` points to a valid allocation of type `ArcInner`,
254
+
// field projection to `data`is within bounds of the allocation.
266
255
unsafe { core::ptr::addr_of!((*ptr).data) }
267
256
}
268
257
···
568
539
}
569
540
570
541
/// Creates an [`ArcBorrow`] to an [`Arc`] that has previously been deconstructed with
571
-
/// [`Arc::into_raw`].
542
+
/// [`Arc::into_raw`] or [`Arc::as_ptr`].
572
543
///
573
544
/// # Safety
574
545
///
575
-
/// * The provided pointer must originate from a call to [`Arc::into_raw`].
546
+
/// * The provided pointer must originate from a call to [`Arc::into_raw`] or [`Arc::as_ptr`].
576
547
/// * For the duration of the lifetime annotated on this `ArcBorrow`, the reference count must
577
548
/// not hit zero.
578
549
/// * For the duration of the lifetime annotated on this `ArcBorrow`, there must not be a
···
688
659
inner: Arc<T>,
689
660
}
690
661
662
+
impl<T> InPlaceInit<T> for UniqueArc<T> {
663
+
type PinnedSelf = Pin<Self>;
664
+
665
+
#[inline]
666
+
fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::PinnedSelf, E>
667
+
where
668
+
E: From<AllocError>,
669
+
{
670
+
UniqueArc::new_uninit(flags)?.write_pin_init(init)
671
+
}
672
+
673
+
#[inline]
674
+
fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
675
+
where
676
+
E: From<AllocError>,
677
+
{
678
+
UniqueArc::new_uninit(flags)?.write_init(init)
679
+
}
680
+
}
681
+
682
+
impl<T> InPlaceWrite<T> for UniqueArc<MaybeUninit<T>> {
683
+
type Initialized = UniqueArc<T>;
684
+
685
+
fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
686
+
let slot = self.as_mut_ptr();
687
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
688
+
// slot is valid.
689
+
unsafe { init.__init(slot)? };
690
+
// SAFETY: All fields have been initialized.
691
+
Ok(unsafe { self.assume_init() })
692
+
}
693
+
694
+
fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
695
+
let slot = self.as_mut_ptr();
696
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
697
+
// slot is valid and will not be moved, because we pin it later.
698
+
unsafe { init.__pinned_init(slot)? };
699
+
// SAFETY: All fields have been initialized.
700
+
Ok(unsafe { self.assume_init() }.into())
701
+
}
702
+
}
703
+
691
704
impl<T> UniqueArc<T> {
692
705
/// Tries to allocate a new [`UniqueArc`] instance.
693
706
pub fn new(value: T, flags: Flags) -> Result<Self, AllocError> {
···
746
675
try_init!(ArcInner {
747
676
// SAFETY: There are no safety requirements for this FFI call.
748
677
refcount: Opaque::new(unsafe { bindings::REFCOUNT_INIT(1) }),
749
-
data <- init::uninit::<T, AllocError>(),
678
+
data <- pin_init::uninit::<T, AllocError>(),
750
679
}? AllocError),
751
680
flags,
752
681
)?;
+2
-4
rust/kernel/sync/condvar.rs
+2
-4
rust/kernel/sync/condvar.rs
···
8
8
use super::{lock::Backend, lock::Guard, LockClassKey};
9
9
use crate::{
10
10
ffi::{c_int, c_long},
11
-
init::PinInit,
12
-
pin_init,
13
11
str::CStr,
14
12
task::{
15
13
MAX_SCHEDULE_TIMEOUT, TASK_FREEZABLE, TASK_INTERRUPTIBLE, TASK_NORMAL, TASK_UNINTERRUPTIBLE,
···
16
18
types::Opaque,
17
19
};
18
20
use core::{marker::PhantomPinned, pin::Pin, ptr};
19
-
use macros::pin_data;
21
+
use pin_init::{pin_data, pin_init, PinInit};
20
22
21
23
/// Creates a [`CondVar`] initialiser with the given name and a newly-created lock class.
22
24
#[macro_export]
···
36
38
/// spuriously.
37
39
///
38
40
/// Instances of [`CondVar`] need a lock class and to be pinned. The recommended way to create such
39
-
/// instances is with the [`pin_init`](crate::pin_init) and [`new_condvar`] macros.
41
+
/// instances is with the [`pin_init`](crate::pin_init!) and [`new_condvar`] macros.
40
42
///
41
43
/// # Examples
42
44
///
+3
-4
rust/kernel/sync/lock.rs
+3
-4
rust/kernel/sync/lock.rs
···
7
7
8
8
use super::LockClassKey;
9
9
use crate::{
10
-
init::PinInit,
11
-
pin_init,
12
10
str::CStr,
13
11
types::{NotThreadSafe, Opaque, ScopeGuard},
14
12
};
15
13
use core::{cell::UnsafeCell, marker::PhantomPinned, pin::Pin};
16
-
use macros::pin_data;
14
+
use pin_init::{pin_data, pin_init, PinInit};
17
15
18
16
pub mod mutex;
19
17
pub mod spinlock;
···
206
208
/// lock is held.
207
209
///
208
210
/// ```
209
-
/// # use kernel::{new_spinlock, stack_pin_init, sync::lock::{Backend, Guard, Lock}};
211
+
/// # use kernel::{new_spinlock, sync::lock::{Backend, Guard, Lock}};
212
+
/// # use pin_init::stack_pin_init;
210
213
///
211
214
/// fn assert_held<T, B: Backend>(guard: &Guard<'_, T, B>, lock: &Lock<T, B>) {
212
215
/// // Address-equal means the same lock.
+1
-1
rust/kernel/sync/lock/mutex.rs
+1
-1
rust/kernel/sync/lock/mutex.rs
···
26
26
/// Since it may block, [`Mutex`] needs to be used with care in atomic contexts.
27
27
///
28
28
/// Instances of [`Mutex`] need a lock class and to be pinned. The recommended way to create such
29
-
/// instances is with the [`pin_init`](crate::pin_init) and [`new_mutex`] macros.
29
+
/// instances is with the [`pin_init`](pin_init::pin_init) and [`new_mutex`] macros.
30
30
///
31
31
/// # Examples
32
32
///
+1
-1
rust/kernel/sync/lock/spinlock.rs
+1
-1
rust/kernel/sync/lock/spinlock.rs
···
24
24
/// unlocked, at which point another CPU will be allowed to make progress.
25
25
///
26
26
/// Instances of [`SpinLock`] need a lock class and to be pinned. The recommended way to create such
27
-
/// instances is with the [`pin_init`](crate::pin_init) and [`new_spinlock`] macros.
27
+
/// instances is with the [`pin_init`](pin_init::pin_init) and [`new_spinlock`] macros.
28
28
///
29
29
/// # Examples
30
30
///
+2
-2
rust/kernel/sync/poll.rs
+2
-2
rust/kernel/sync/poll.rs
···
43
43
///
44
44
/// # Safety
45
45
///
46
-
/// The caller must ensure that for the duration of 'a, the pointer will point at a valid poll
46
+
/// The caller must ensure that for the duration of `'a`, the pointer will point at a valid poll
47
47
/// table (as defined in the type invariants).
48
48
///
49
49
/// The caller must also ensure that the `poll_table` is only accessed via the returned
50
-
/// reference for the duration of 'a.
50
+
/// reference for the duration of `'a`.
51
51
pub unsafe fn from_ptr<'a>(ptr: *mut bindings::poll_table) -> &'a mut PollTable {
52
52
// SAFETY: The safety requirements guarantee the validity of the dereference, while the
53
53
// `PollTable` type being transparent makes the cast ok.
+1
-1
rust/kernel/task.rs
+1
-1
rust/kernel/task.rs
+68
rust/kernel/time.rs
+68
rust/kernel/time.rs
···
8
8
//! C header: [`include/linux/jiffies.h`](srctree/include/linux/jiffies.h).
9
9
//! C header: [`include/linux/ktime.h`](srctree/include/linux/ktime.h).
10
10
11
+
pub mod hrtimer;
12
+
11
13
/// The number of nanoseconds per millisecond.
12
14
pub const NSEC_PER_MSEC: i64 = bindings::NSEC_PER_MSEC as i64;
13
15
···
81
79
Self {
82
80
inner: self.inner - other.inner,
83
81
}
82
+
}
83
+
}
84
+
85
+
/// An identifier for a clock. Used when specifying clock sources.
86
+
///
87
+
///
88
+
/// Selection of the clock depends on the use case. In some cases the usage of a
89
+
/// particular clock is mandatory, e.g. in network protocols, filesystems.In other
90
+
/// cases the user of the clock has to decide which clock is best suited for the
91
+
/// purpose. In most scenarios clock [`ClockId::Monotonic`] is the best choice as it
92
+
/// provides a accurate monotonic notion of time (leap second smearing ignored).
93
+
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
94
+
#[repr(u32)]
95
+
pub enum ClockId {
96
+
/// A settable system-wide clock that measures real (i.e., wall-clock) time.
97
+
///
98
+
/// Setting this clock requires appropriate privileges. This clock is
99
+
/// affected by discontinuous jumps in the system time (e.g., if the system
100
+
/// administrator manually changes the clock), and by frequency adjustments
101
+
/// performed by NTP and similar applications via adjtime(3), adjtimex(2),
102
+
/// clock_adjtime(2), and ntp_adjtime(3). This clock normally counts the
103
+
/// number of seconds since 1970-01-01 00:00:00 Coordinated Universal Time
104
+
/// (UTC) except that it ignores leap seconds; near a leap second it may be
105
+
/// adjusted by leap second smearing to stay roughly in sync with UTC. Leap
106
+
/// second smearing applies frequency adjustments to the clock to speed up
107
+
/// or slow down the clock to account for the leap second without
108
+
/// discontinuities in the clock. If leap second smearing is not applied,
109
+
/// the clock will experience discontinuity around leap second adjustment.
110
+
RealTime = bindings::CLOCK_REALTIME,
111
+
/// A monotonically increasing clock.
112
+
///
113
+
/// A nonsettable system-wide clock that represents monotonic time since—as
114
+
/// described by POSIX—"some unspecified point in the past". On Linux, that
115
+
/// point corresponds to the number of seconds that the system has been
116
+
/// running since it was booted.
117
+
///
118
+
/// The CLOCK_MONOTONIC clock is not affected by discontinuous jumps in the
119
+
/// CLOCK_REAL (e.g., if the system administrator manually changes the
120
+
/// clock), but is affected by frequency adjustments. This clock does not
121
+
/// count time that the system is suspended.
122
+
Monotonic = bindings::CLOCK_MONOTONIC,
123
+
/// A monotonic that ticks while system is suspended.
124
+
///
125
+
/// A nonsettable system-wide clock that is identical to CLOCK_MONOTONIC,
126
+
/// except that it also includes any time that the system is suspended. This
127
+
/// allows applications to get a suspend-aware monotonic clock without
128
+
/// having to deal with the complications of CLOCK_REALTIME, which may have
129
+
/// discontinuities if the time is changed using settimeofday(2) or similar.
130
+
BootTime = bindings::CLOCK_BOOTTIME,
131
+
/// International Atomic Time.
132
+
///
133
+
/// A system-wide clock derived from wall-clock time but counting leap seconds.
134
+
///
135
+
/// This clock is coupled to CLOCK_REALTIME and will be set when CLOCK_REALTIME is
136
+
/// set, or when the offset to CLOCK_REALTIME is changed via adjtimex(2). This
137
+
/// usually happens during boot and **should** not happen during normal operations.
138
+
/// However, if NTP or another application adjusts CLOCK_REALTIME by leap second
139
+
/// smearing, this clock will not be precise during leap second smearing.
140
+
///
141
+
/// The acronym TAI refers to International Atomic Time.
142
+
TAI = bindings::CLOCK_TAI,
143
+
}
144
+
145
+
impl ClockId {
146
+
fn into_c(self) -> bindings::clockid_t {
147
+
self as bindings::clockid_t
84
148
}
85
149
}
+520
rust/kernel/time/hrtimer.rs
+520
rust/kernel/time/hrtimer.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
//! Intrusive high resolution timers.
4
+
//!
5
+
//! Allows running timer callbacks without doing allocations at the time of
6
+
//! starting the timer. For now, only one timer per type is allowed.
7
+
//!
8
+
//! # Vocabulary
9
+
//!
10
+
//! States:
11
+
//!
12
+
//! - Stopped: initialized but not started, or cancelled, or not restarted.
13
+
//! - Started: initialized and started or restarted.
14
+
//! - Running: executing the callback.
15
+
//!
16
+
//! Operations:
17
+
//!
18
+
//! * Start
19
+
//! * Cancel
20
+
//! * Restart
21
+
//!
22
+
//! Events:
23
+
//!
24
+
//! * Expire
25
+
//!
26
+
//! ## State Diagram
27
+
//!
28
+
//! ```text
29
+
//! Return NoRestart
30
+
//! +---------------------------------------------------------------------+
31
+
//! | |
32
+
//! | |
33
+
//! | |
34
+
//! | Return Restart |
35
+
//! | +------------------------+ |
36
+
//! | | | |
37
+
//! | | | |
38
+
//! v v | |
39
+
//! +-----------------+ Start +------------------+ +--------+-----+--+
40
+
//! | +---------------->| | | |
41
+
//! Init | | | | Expire | |
42
+
//! --------->| Stopped | | Started +---------->| Running |
43
+
//! | | Cancel | | | |
44
+
//! | |<----------------+ | | |
45
+
//! +-----------------+ +---------------+--+ +-----------------+
46
+
//! ^ |
47
+
//! | |
48
+
//! +---------+
49
+
//! Restart
50
+
//! ```
51
+
//!
52
+
//!
53
+
//! A timer is initialized in the **stopped** state. A stopped timer can be
54
+
//! **started** by the `start` operation, with an **expiry** time. After the
55
+
//! `start` operation, the timer is in the **started** state. When the timer
56
+
//! **expires**, the timer enters the **running** state and the handler is
57
+
//! executed. After the handler has returned, the timer may enter the
58
+
//! **started* or **stopped** state, depending on the return value of the
59
+
//! handler. A timer in the **started** or **running** state may be **canceled**
60
+
//! by the `cancel` operation. A timer that is cancelled enters the **stopped**
61
+
//! state.
62
+
//!
63
+
//! A `cancel` or `restart` operation on a timer in the **running** state takes
64
+
//! effect after the handler has returned and the timer has transitioned
65
+
//! out of the **running** state.
66
+
//!
67
+
//! A `restart` operation on a timer in the **stopped** state is equivalent to a
68
+
//! `start` operation.
69
+
70
+
use super::ClockId;
71
+
use crate::{prelude::*, time::Ktime, types::Opaque};
72
+
use core::marker::PhantomData;
73
+
use pin_init::PinInit;
74
+
75
+
/// A timer backed by a C `struct hrtimer`.
76
+
///
77
+
/// # Invariants
78
+
///
79
+
/// * `self.timer` is initialized by `bindings::hrtimer_setup`.
80
+
#[pin_data]
81
+
#[repr(C)]
82
+
pub struct HrTimer<T> {
83
+
#[pin]
84
+
timer: Opaque<bindings::hrtimer>,
85
+
mode: HrTimerMode,
86
+
_t: PhantomData<T>,
87
+
}
88
+
89
+
// SAFETY: Ownership of an `HrTimer` can be moved to other threads and
90
+
// used/dropped from there.
91
+
unsafe impl<T> Send for HrTimer<T> {}
92
+
93
+
// SAFETY: Timer operations are locked on the C side, so it is safe to operate
94
+
// on a timer from multiple threads.
95
+
unsafe impl<T> Sync for HrTimer<T> {}
96
+
97
+
impl<T> HrTimer<T> {
98
+
/// Return an initializer for a new timer instance.
99
+
pub fn new(mode: HrTimerMode, clock: ClockId) -> impl PinInit<Self>
100
+
where
101
+
T: HrTimerCallback,
102
+
{
103
+
pin_init!(Self {
104
+
// INVARIANT: We initialize `timer` with `hrtimer_setup` below.
105
+
timer <- Opaque::ffi_init(move |place: *mut bindings::hrtimer| {
106
+
// SAFETY: By design of `pin_init!`, `place` is a pointer to a
107
+
// live allocation. hrtimer_setup will initialize `place` and
108
+
// does not require `place` to be initialized prior to the call.
109
+
unsafe {
110
+
bindings::hrtimer_setup(
111
+
place,
112
+
Some(T::Pointer::run),
113
+
clock.into_c(),
114
+
mode.into_c(),
115
+
);
116
+
}
117
+
}),
118
+
mode: mode,
119
+
_t: PhantomData,
120
+
})
121
+
}
122
+
123
+
/// Get a pointer to the contained `bindings::hrtimer`.
124
+
///
125
+
/// This function is useful to get access to the value without creating
126
+
/// intermediate references.
127
+
///
128
+
/// # Safety
129
+
///
130
+
/// `this` must point to a live allocation of at least the size of `Self`.
131
+
unsafe fn raw_get(this: *const Self) -> *mut bindings::hrtimer {
132
+
// SAFETY: The field projection to `timer` does not go out of bounds,
133
+
// because the caller of this function promises that `this` points to an
134
+
// allocation of at least the size of `Self`.
135
+
unsafe { Opaque::raw_get(core::ptr::addr_of!((*this).timer)) }
136
+
}
137
+
138
+
/// Cancel an initialized and potentially running timer.
139
+
///
140
+
/// If the timer handler is running, this function will block until the
141
+
/// handler returns.
142
+
///
143
+
/// Note that the timer might be started by a concurrent start operation. If
144
+
/// so, the timer might not be in the **stopped** state when this function
145
+
/// returns.
146
+
///
147
+
/// Users of the `HrTimer` API would not usually call this method directly.
148
+
/// Instead they would use the safe [`HrTimerHandle::cancel`] on the handle
149
+
/// returned when the timer was started.
150
+
///
151
+
/// This function is useful to get access to the value without creating
152
+
/// intermediate references.
153
+
///
154
+
/// # Safety
155
+
///
156
+
/// `this` must point to a valid `Self`.
157
+
pub(crate) unsafe fn raw_cancel(this: *const Self) -> bool {
158
+
// SAFETY: `this` points to an allocation of at least `HrTimer` size.
159
+
let c_timer_ptr = unsafe { HrTimer::raw_get(this) };
160
+
161
+
// If the handler is running, this will wait for the handler to return
162
+
// before returning.
163
+
// SAFETY: `c_timer_ptr` is initialized and valid. Synchronization is
164
+
// handled on the C side.
165
+
unsafe { bindings::hrtimer_cancel(c_timer_ptr) != 0 }
166
+
}
167
+
}
168
+
169
+
/// Implemented by pointer types that point to structs that contain a [`HrTimer`].
170
+
///
171
+
/// `Self` must be [`Sync`] because it is passed to timer callbacks in another
172
+
/// thread of execution (hard or soft interrupt context).
173
+
///
174
+
/// Starting a timer returns a [`HrTimerHandle`] that can be used to manipulate
175
+
/// the timer. Note that it is OK to call the start function repeatedly, and
176
+
/// that more than one [`HrTimerHandle`] associated with a [`HrTimerPointer`] may
177
+
/// exist. A timer can be manipulated through any of the handles, and a handle
178
+
/// may represent a cancelled timer.
179
+
pub trait HrTimerPointer: Sync + Sized {
180
+
/// A handle representing a started or restarted timer.
181
+
///
182
+
/// If the timer is running or if the timer callback is executing when the
183
+
/// handle is dropped, the drop method of [`HrTimerHandle`] should not return
184
+
/// until the timer is stopped and the callback has completed.
185
+
///
186
+
/// Note: When implementing this trait, consider that it is not unsafe to
187
+
/// leak the handle.
188
+
type TimerHandle: HrTimerHandle;
189
+
190
+
/// Start the timer with expiry after `expires` time units. If the timer was
191
+
/// already running, it is restarted with the new expiry time.
192
+
fn start(self, expires: Ktime) -> Self::TimerHandle;
193
+
}
194
+
195
+
/// Unsafe version of [`HrTimerPointer`] for situations where leaking the
196
+
/// [`HrTimerHandle`] returned by `start` would be unsound. This is the case for
197
+
/// stack allocated timers.
198
+
///
199
+
/// Typical implementers are pinned references such as [`Pin<&T>`].
200
+
///
201
+
/// # Safety
202
+
///
203
+
/// Implementers of this trait must ensure that instances of types implementing
204
+
/// [`UnsafeHrTimerPointer`] outlives any associated [`HrTimerPointer::TimerHandle`]
205
+
/// instances.
206
+
pub unsafe trait UnsafeHrTimerPointer: Sync + Sized {
207
+
/// A handle representing a running timer.
208
+
///
209
+
/// # Safety
210
+
///
211
+
/// If the timer is running, or if the timer callback is executing when the
212
+
/// handle is dropped, the drop method of [`Self::TimerHandle`] must not return
213
+
/// until the timer is stopped and the callback has completed.
214
+
type TimerHandle: HrTimerHandle;
215
+
216
+
/// Start the timer after `expires` time units. If the timer was already
217
+
/// running, it is restarted at the new expiry time.
218
+
///
219
+
/// # Safety
220
+
///
221
+
/// Caller promises keep the timer structure alive until the timer is dead.
222
+
/// Caller can ensure this by not leaking the returned [`Self::TimerHandle`].
223
+
unsafe fn start(self, expires: Ktime) -> Self::TimerHandle;
224
+
}
225
+
226
+
/// A trait for stack allocated timers.
227
+
///
228
+
/// # Safety
229
+
///
230
+
/// Implementers must ensure that `start_scoped` does not return until the
231
+
/// timer is dead and the timer handler is not running.
232
+
pub unsafe trait ScopedHrTimerPointer {
233
+
/// Start the timer to run after `expires` time units and immediately
234
+
/// after call `f`. When `f` returns, the timer is cancelled.
235
+
fn start_scoped<T, F>(self, expires: Ktime, f: F) -> T
236
+
where
237
+
F: FnOnce() -> T;
238
+
}
239
+
240
+
// SAFETY: By the safety requirement of [`UnsafeHrTimerPointer`], dropping the
241
+
// handle returned by [`UnsafeHrTimerPointer::start`] ensures that the timer is
242
+
// killed.
243
+
unsafe impl<T> ScopedHrTimerPointer for T
244
+
where
245
+
T: UnsafeHrTimerPointer,
246
+
{
247
+
fn start_scoped<U, F>(self, expires: Ktime, f: F) -> U
248
+
where
249
+
F: FnOnce() -> U,
250
+
{
251
+
// SAFETY: We drop the timer handle below before returning.
252
+
let handle = unsafe { UnsafeHrTimerPointer::start(self, expires) };
253
+
let t = f();
254
+
drop(handle);
255
+
t
256
+
}
257
+
}
258
+
259
+
/// Implemented by [`HrTimerPointer`] implementers to give the C timer callback a
260
+
/// function to call.
261
+
// This is split from `HrTimerPointer` to make it easier to specify trait bounds.
262
+
pub trait RawHrTimerCallback {
263
+
/// Type of the parameter passed to [`HrTimerCallback::run`]. It may be
264
+
/// [`Self`], or a pointer type derived from [`Self`].
265
+
type CallbackTarget<'a>;
266
+
267
+
/// Callback to be called from C when timer fires.
268
+
///
269
+
/// # Safety
270
+
///
271
+
/// Only to be called by C code in the `hrtimer` subsystem. `this` must point
272
+
/// to the `bindings::hrtimer` structure that was used to start the timer.
273
+
unsafe extern "C" fn run(this: *mut bindings::hrtimer) -> bindings::hrtimer_restart;
274
+
}
275
+
276
+
/// Implemented by structs that can be the target of a timer callback.
277
+
pub trait HrTimerCallback {
278
+
/// The type whose [`RawHrTimerCallback::run`] method will be invoked when
279
+
/// the timer expires.
280
+
type Pointer<'a>: RawHrTimerCallback;
281
+
282
+
/// Called by the timer logic when the timer fires.
283
+
fn run(this: <Self::Pointer<'_> as RawHrTimerCallback>::CallbackTarget<'_>) -> HrTimerRestart
284
+
where
285
+
Self: Sized;
286
+
}
287
+
288
+
/// A handle representing a potentially running timer.
289
+
///
290
+
/// More than one handle representing the same timer might exist.
291
+
///
292
+
/// # Safety
293
+
///
294
+
/// When dropped, the timer represented by this handle must be cancelled, if it
295
+
/// is running. If the timer handler is running when the handle is dropped, the
296
+
/// drop method must wait for the handler to return before returning.
297
+
///
298
+
/// Note: One way to satisfy the safety requirement is to call `Self::cancel` in
299
+
/// the drop implementation for `Self.`
300
+
pub unsafe trait HrTimerHandle {
301
+
/// Cancel the timer. If the timer is in the running state, block till the
302
+
/// handler has returned.
303
+
///
304
+
/// Note that the timer might be started by a concurrent start operation. If
305
+
/// so, the timer might not be in the **stopped** state when this function
306
+
/// returns.
307
+
fn cancel(&mut self) -> bool;
308
+
}
309
+
310
+
/// Implemented by structs that contain timer nodes.
311
+
///
312
+
/// Clients of the timer API would usually safely implement this trait by using
313
+
/// the [`crate::impl_has_hr_timer`] macro.
314
+
///
315
+
/// # Safety
316
+
///
317
+
/// Implementers of this trait must ensure that the implementer has a
318
+
/// [`HrTimer`] field and that all trait methods are implemented according to
319
+
/// their documentation. All the methods of this trait must operate on the same
320
+
/// field.
321
+
pub unsafe trait HasHrTimer<T> {
322
+
/// Return a pointer to the [`HrTimer`] within `Self`.
323
+
///
324
+
/// This function is useful to get access to the value without creating
325
+
/// intermediate references.
326
+
///
327
+
/// # Safety
328
+
///
329
+
/// `this` must be a valid pointer.
330
+
unsafe fn raw_get_timer(this: *const Self) -> *const HrTimer<T>;
331
+
332
+
/// Return a pointer to the struct that is containing the [`HrTimer`] pointed
333
+
/// to by `ptr`.
334
+
///
335
+
/// This function is useful to get access to the value without creating
336
+
/// intermediate references.
337
+
///
338
+
/// # Safety
339
+
///
340
+
/// `ptr` must point to a [`HrTimer<T>`] field in a struct of type `Self`.
341
+
unsafe fn timer_container_of(ptr: *mut HrTimer<T>) -> *mut Self
342
+
where
343
+
Self: Sized;
344
+
345
+
/// Get pointer to the contained `bindings::hrtimer` struct.
346
+
///
347
+
/// This function is useful to get access to the value without creating
348
+
/// intermediate references.
349
+
///
350
+
/// # Safety
351
+
///
352
+
/// `this` must be a valid pointer.
353
+
unsafe fn c_timer_ptr(this: *const Self) -> *const bindings::hrtimer {
354
+
// SAFETY: `this` is a valid pointer to a `Self`.
355
+
let timer_ptr = unsafe { Self::raw_get_timer(this) };
356
+
357
+
// SAFETY: timer_ptr points to an allocation of at least `HrTimer` size.
358
+
unsafe { HrTimer::raw_get(timer_ptr) }
359
+
}
360
+
361
+
/// Start the timer contained in the `Self` pointed to by `self_ptr`. If
362
+
/// it is already running it is removed and inserted.
363
+
///
364
+
/// # Safety
365
+
///
366
+
/// - `this` must point to a valid `Self`.
367
+
/// - Caller must ensure that the pointee of `this` lives until the timer
368
+
/// fires or is canceled.
369
+
unsafe fn start(this: *const Self, expires: Ktime) {
370
+
// SAFETY: By function safety requirement, `this` is a valid `Self`.
371
+
unsafe {
372
+
bindings::hrtimer_start_range_ns(
373
+
Self::c_timer_ptr(this).cast_mut(),
374
+
expires.to_ns(),
375
+
0,
376
+
(*Self::raw_get_timer(this)).mode.into_c(),
377
+
);
378
+
}
379
+
}
380
+
}
381
+
382
+
/// Restart policy for timers.
383
+
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
384
+
#[repr(u32)]
385
+
pub enum HrTimerRestart {
386
+
/// Timer should not be restarted.
387
+
#[allow(clippy::unnecessary_cast)]
388
+
NoRestart = bindings::hrtimer_restart_HRTIMER_NORESTART as u32,
389
+
/// Timer should be restarted.
390
+
#[allow(clippy::unnecessary_cast)]
391
+
Restart = bindings::hrtimer_restart_HRTIMER_RESTART as u32,
392
+
}
393
+
394
+
impl HrTimerRestart {
395
+
fn into_c(self) -> bindings::hrtimer_restart {
396
+
self as bindings::hrtimer_restart
397
+
}
398
+
}
399
+
400
+
/// Operational mode of [`HrTimer`].
401
+
// NOTE: Some of these have the same encoding on the C side, so we keep
402
+
// `repr(Rust)` and convert elsewhere.
403
+
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
404
+
pub enum HrTimerMode {
405
+
/// Timer expires at the given expiration time.
406
+
Absolute,
407
+
/// Timer expires after the given expiration time interpreted as a duration from now.
408
+
Relative,
409
+
/// Timer does not move between CPU cores.
410
+
Pinned,
411
+
/// Timer handler is executed in soft irq context.
412
+
Soft,
413
+
/// Timer handler is executed in hard irq context.
414
+
Hard,
415
+
/// Timer expires at the given expiration time.
416
+
/// Timer does not move between CPU cores.
417
+
AbsolutePinned,
418
+
/// Timer expires after the given expiration time interpreted as a duration from now.
419
+
/// Timer does not move between CPU cores.
420
+
RelativePinned,
421
+
/// Timer expires at the given expiration time.
422
+
/// Timer handler is executed in soft irq context.
423
+
AbsoluteSoft,
424
+
/// Timer expires after the given expiration time interpreted as a duration from now.
425
+
/// Timer handler is executed in soft irq context.
426
+
RelativeSoft,
427
+
/// Timer expires at the given expiration time.
428
+
/// Timer does not move between CPU cores.
429
+
/// Timer handler is executed in soft irq context.
430
+
AbsolutePinnedSoft,
431
+
/// Timer expires after the given expiration time interpreted as a duration from now.
432
+
/// Timer does not move between CPU cores.
433
+
/// Timer handler is executed in soft irq context.
434
+
RelativePinnedSoft,
435
+
/// Timer expires at the given expiration time.
436
+
/// Timer handler is executed in hard irq context.
437
+
AbsoluteHard,
438
+
/// Timer expires after the given expiration time interpreted as a duration from now.
439
+
/// Timer handler is executed in hard irq context.
440
+
RelativeHard,
441
+
/// Timer expires at the given expiration time.
442
+
/// Timer does not move between CPU cores.
443
+
/// Timer handler is executed in hard irq context.
444
+
AbsolutePinnedHard,
445
+
/// Timer expires after the given expiration time interpreted as a duration from now.
446
+
/// Timer does not move between CPU cores.
447
+
/// Timer handler is executed in hard irq context.
448
+
RelativePinnedHard,
449
+
}
450
+
451
+
impl HrTimerMode {
452
+
fn into_c(self) -> bindings::hrtimer_mode {
453
+
use bindings::*;
454
+
match self {
455
+
HrTimerMode::Absolute => hrtimer_mode_HRTIMER_MODE_ABS,
456
+
HrTimerMode::Relative => hrtimer_mode_HRTIMER_MODE_REL,
457
+
HrTimerMode::Pinned => hrtimer_mode_HRTIMER_MODE_PINNED,
458
+
HrTimerMode::Soft => hrtimer_mode_HRTIMER_MODE_SOFT,
459
+
HrTimerMode::Hard => hrtimer_mode_HRTIMER_MODE_HARD,
460
+
HrTimerMode::AbsolutePinned => hrtimer_mode_HRTIMER_MODE_ABS_PINNED,
461
+
HrTimerMode::RelativePinned => hrtimer_mode_HRTIMER_MODE_REL_PINNED,
462
+
HrTimerMode::AbsoluteSoft => hrtimer_mode_HRTIMER_MODE_ABS_SOFT,
463
+
HrTimerMode::RelativeSoft => hrtimer_mode_HRTIMER_MODE_REL_SOFT,
464
+
HrTimerMode::AbsolutePinnedSoft => hrtimer_mode_HRTIMER_MODE_ABS_PINNED_SOFT,
465
+
HrTimerMode::RelativePinnedSoft => hrtimer_mode_HRTIMER_MODE_REL_PINNED_SOFT,
466
+
HrTimerMode::AbsoluteHard => hrtimer_mode_HRTIMER_MODE_ABS_HARD,
467
+
HrTimerMode::RelativeHard => hrtimer_mode_HRTIMER_MODE_REL_HARD,
468
+
HrTimerMode::AbsolutePinnedHard => hrtimer_mode_HRTIMER_MODE_ABS_PINNED_HARD,
469
+
HrTimerMode::RelativePinnedHard => hrtimer_mode_HRTIMER_MODE_REL_PINNED_HARD,
470
+
}
471
+
}
472
+
}
473
+
474
+
/// Use to implement the [`HasHrTimer<T>`] trait.
475
+
///
476
+
/// See [`module`] documentation for an example.
477
+
///
478
+
/// [`module`]: crate::time::hrtimer
479
+
#[macro_export]
480
+
macro_rules! impl_has_hr_timer {
481
+
(
482
+
impl$({$($generics:tt)*})?
483
+
HasHrTimer<$timer_type:ty>
484
+
for $self:ty
485
+
{ self.$field:ident }
486
+
$($rest:tt)*
487
+
) => {
488
+
// SAFETY: This implementation of `raw_get_timer` only compiles if the
489
+
// field has the right type.
490
+
unsafe impl$(<$($generics)*>)? $crate::time::hrtimer::HasHrTimer<$timer_type> for $self {
491
+
492
+
#[inline]
493
+
unsafe fn raw_get_timer(
494
+
this: *const Self,
495
+
) -> *const $crate::time::hrtimer::HrTimer<$timer_type> {
496
+
// SAFETY: The caller promises that the pointer is not dangling.
497
+
unsafe { ::core::ptr::addr_of!((*this).$field) }
498
+
}
499
+
500
+
#[inline]
501
+
unsafe fn timer_container_of(
502
+
ptr: *mut $crate::time::hrtimer::HrTimer<$timer_type>,
503
+
) -> *mut Self {
504
+
// SAFETY: As per the safety requirement of this function, `ptr`
505
+
// is pointing inside a `$timer_type`.
506
+
unsafe { ::kernel::container_of!(ptr, $timer_type, $field).cast_mut() }
507
+
}
508
+
}
509
+
}
510
+
}
511
+
512
+
mod arc;
513
+
pub use arc::ArcHrTimerHandle;
514
+
mod pin;
515
+
pub use pin::PinHrTimerHandle;
516
+
mod pin_mut;
517
+
pub use pin_mut::PinMutHrTimerHandle;
518
+
// `box` is a reserved keyword, so prefix with `t` for timer
519
+
mod tbox;
520
+
pub use tbox::BoxHrTimerHandle;
+100
rust/kernel/time/hrtimer/arc.rs
+100
rust/kernel/time/hrtimer/arc.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
use super::HasHrTimer;
4
+
use super::HrTimer;
5
+
use super::HrTimerCallback;
6
+
use super::HrTimerHandle;
7
+
use super::HrTimerPointer;
8
+
use super::RawHrTimerCallback;
9
+
use crate::sync::Arc;
10
+
use crate::sync::ArcBorrow;
11
+
use crate::time::Ktime;
12
+
13
+
/// A handle for an `Arc<HasHrTimer<T>>` returned by a call to
14
+
/// [`HrTimerPointer::start`].
15
+
pub struct ArcHrTimerHandle<T>
16
+
where
17
+
T: HasHrTimer<T>,
18
+
{
19
+
pub(crate) inner: Arc<T>,
20
+
}
21
+
22
+
// SAFETY: We implement drop below, and we cancel the timer in the drop
23
+
// implementation.
24
+
unsafe impl<T> HrTimerHandle for ArcHrTimerHandle<T>
25
+
where
26
+
T: HasHrTimer<T>,
27
+
{
28
+
fn cancel(&mut self) -> bool {
29
+
let self_ptr = Arc::as_ptr(&self.inner);
30
+
31
+
// SAFETY: As we obtained `self_ptr` from a valid reference above, it
32
+
// must point to a valid `T`.
33
+
let timer_ptr = unsafe { <T as HasHrTimer<T>>::raw_get_timer(self_ptr) };
34
+
35
+
// SAFETY: As `timer_ptr` points into `T` and `T` is valid, `timer_ptr`
36
+
// must point to a valid `HrTimer` instance.
37
+
unsafe { HrTimer::<T>::raw_cancel(timer_ptr) }
38
+
}
39
+
}
40
+
41
+
impl<T> Drop for ArcHrTimerHandle<T>
42
+
where
43
+
T: HasHrTimer<T>,
44
+
{
45
+
fn drop(&mut self) {
46
+
self.cancel();
47
+
}
48
+
}
49
+
50
+
impl<T> HrTimerPointer for Arc<T>
51
+
where
52
+
T: 'static,
53
+
T: Send + Sync,
54
+
T: HasHrTimer<T>,
55
+
T: for<'a> HrTimerCallback<Pointer<'a> = Self>,
56
+
{
57
+
type TimerHandle = ArcHrTimerHandle<T>;
58
+
59
+
fn start(self, expires: Ktime) -> ArcHrTimerHandle<T> {
60
+
// SAFETY:
61
+
// - We keep `self` alive by wrapping it in a handle below.
62
+
// - Since we generate the pointer passed to `start` from a valid
63
+
// reference, it is a valid pointer.
64
+
unsafe { T::start(Arc::as_ptr(&self), expires) };
65
+
ArcHrTimerHandle { inner: self }
66
+
}
67
+
}
68
+
69
+
impl<T> RawHrTimerCallback for Arc<T>
70
+
where
71
+
T: 'static,
72
+
T: HasHrTimer<T>,
73
+
T: for<'a> HrTimerCallback<Pointer<'a> = Self>,
74
+
{
75
+
type CallbackTarget<'a> = ArcBorrow<'a, T>;
76
+
77
+
unsafe extern "C" fn run(ptr: *mut bindings::hrtimer) -> bindings::hrtimer_restart {
78
+
// `HrTimer` is `repr(C)`
79
+
let timer_ptr = ptr.cast::<super::HrTimer<T>>();
80
+
81
+
// SAFETY: By C API contract `ptr` is the pointer we passed when
82
+
// queuing the timer, so it is a `HrTimer<T>` embedded in a `T`.
83
+
let data_ptr = unsafe { T::timer_container_of(timer_ptr) };
84
+
85
+
// SAFETY:
86
+
// - `data_ptr` is derived form the pointer to the `T` that was used to
87
+
// queue the timer.
88
+
// - As per the safety requirements of the trait `HrTimerHandle`, the
89
+
// `ArcHrTimerHandle` associated with this timer is guaranteed to
90
+
// be alive until this method returns. That handle borrows the `T`
91
+
// behind `data_ptr` thus guaranteeing the validity of
92
+
// the `ArcBorrow` created below.
93
+
// - We own one refcount in the `ArcTimerHandle` associated with this
94
+
// timer, so it is not possible to get a `UniqueArc` to this
95
+
// allocation from other `Arc` clones.
96
+
let receiver = unsafe { ArcBorrow::from_raw(data_ptr) };
97
+
98
+
T::run(receiver).into_c()
99
+
}
100
+
}
+104
rust/kernel/time/hrtimer/pin.rs
+104
rust/kernel/time/hrtimer/pin.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
use super::HasHrTimer;
4
+
use super::HrTimer;
5
+
use super::HrTimerCallback;
6
+
use super::HrTimerHandle;
7
+
use super::RawHrTimerCallback;
8
+
use super::UnsafeHrTimerPointer;
9
+
use crate::time::Ktime;
10
+
use core::pin::Pin;
11
+
12
+
/// A handle for a `Pin<&HasHrTimer>`. When the handle exists, the timer might be
13
+
/// running.
14
+
pub struct PinHrTimerHandle<'a, T>
15
+
where
16
+
T: HasHrTimer<T>,
17
+
{
18
+
pub(crate) inner: Pin<&'a T>,
19
+
}
20
+
21
+
// SAFETY: We cancel the timer when the handle is dropped. The implementation of
22
+
// the `cancel` method will block if the timer handler is running.
23
+
unsafe impl<'a, T> HrTimerHandle for PinHrTimerHandle<'a, T>
24
+
where
25
+
T: HasHrTimer<T>,
26
+
{
27
+
fn cancel(&mut self) -> bool {
28
+
let self_ptr: *const T = self.inner.get_ref();
29
+
30
+
// SAFETY: As we got `self_ptr` from a reference above, it must point to
31
+
// a valid `T`.
32
+
let timer_ptr = unsafe { <T as HasHrTimer<T>>::raw_get_timer(self_ptr) };
33
+
34
+
// SAFETY: As `timer_ptr` is derived from a reference, it must point to
35
+
// a valid and initialized `HrTimer`.
36
+
unsafe { HrTimer::<T>::raw_cancel(timer_ptr) }
37
+
}
38
+
}
39
+
40
+
impl<'a, T> Drop for PinHrTimerHandle<'a, T>
41
+
where
42
+
T: HasHrTimer<T>,
43
+
{
44
+
fn drop(&mut self) {
45
+
self.cancel();
46
+
}
47
+
}
48
+
49
+
// SAFETY: We capture the lifetime of `Self` when we create a `PinHrTimerHandle`,
50
+
// so `Self` will outlive the handle.
51
+
unsafe impl<'a, T> UnsafeHrTimerPointer for Pin<&'a T>
52
+
where
53
+
T: Send + Sync,
54
+
T: HasHrTimer<T>,
55
+
T: HrTimerCallback<Pointer<'a> = Self>,
56
+
{
57
+
type TimerHandle = PinHrTimerHandle<'a, T>;
58
+
59
+
unsafe fn start(self, expires: Ktime) -> Self::TimerHandle {
60
+
// Cast to pointer
61
+
let self_ptr: *const T = self.get_ref();
62
+
63
+
// SAFETY:
64
+
// - As we derive `self_ptr` from a reference above, it must point to a
65
+
// valid `T`.
66
+
// - We keep `self` alive by wrapping it in a handle below.
67
+
unsafe { T::start(self_ptr, expires) };
68
+
69
+
PinHrTimerHandle { inner: self }
70
+
}
71
+
}
72
+
73
+
impl<'a, T> RawHrTimerCallback for Pin<&'a T>
74
+
where
75
+
T: HasHrTimer<T>,
76
+
T: HrTimerCallback<Pointer<'a> = Self>,
77
+
{
78
+
type CallbackTarget<'b> = Self;
79
+
80
+
unsafe extern "C" fn run(ptr: *mut bindings::hrtimer) -> bindings::hrtimer_restart {
81
+
// `HrTimer` is `repr(C)`
82
+
let timer_ptr = ptr as *mut HrTimer<T>;
83
+
84
+
// SAFETY: By the safety requirement of this function, `timer_ptr`
85
+
// points to a `HrTimer<T>` contained in an `T`.
86
+
let receiver_ptr = unsafe { T::timer_container_of(timer_ptr) };
87
+
88
+
// SAFETY:
89
+
// - By the safety requirement of this function, `timer_ptr`
90
+
// points to a `HrTimer<T>` contained in an `T`.
91
+
// - As per the safety requirements of the trait `HrTimerHandle`, the
92
+
// `PinHrTimerHandle` associated with this timer is guaranteed to
93
+
// be alive until this method returns. That handle borrows the `T`
94
+
// behind `receiver_ptr`, thus guaranteeing the validity of
95
+
// the reference created below.
96
+
let receiver_ref = unsafe { &*receiver_ptr };
97
+
98
+
// SAFETY: `receiver_ref` only exists as pinned, so it is safe to pin it
99
+
// here.
100
+
let receiver_pin = unsafe { Pin::new_unchecked(receiver_ref) };
101
+
102
+
T::run(receiver_pin).into_c()
103
+
}
104
+
}
+108
rust/kernel/time/hrtimer/pin_mut.rs
+108
rust/kernel/time/hrtimer/pin_mut.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
use super::{
4
+
HasHrTimer, HrTimer, HrTimerCallback, HrTimerHandle, RawHrTimerCallback, UnsafeHrTimerPointer,
5
+
};
6
+
use crate::time::Ktime;
7
+
use core::{marker::PhantomData, pin::Pin, ptr::NonNull};
8
+
9
+
/// A handle for a `Pin<&mut HasHrTimer>`. When the handle exists, the timer might
10
+
/// be running.
11
+
pub struct PinMutHrTimerHandle<'a, T>
12
+
where
13
+
T: HasHrTimer<T>,
14
+
{
15
+
pub(crate) inner: NonNull<T>,
16
+
_p: PhantomData<&'a mut T>,
17
+
}
18
+
19
+
// SAFETY: We cancel the timer when the handle is dropped. The implementation of
20
+
// the `cancel` method will block if the timer handler is running.
21
+
unsafe impl<'a, T> HrTimerHandle for PinMutHrTimerHandle<'a, T>
22
+
where
23
+
T: HasHrTimer<T>,
24
+
{
25
+
fn cancel(&mut self) -> bool {
26
+
let self_ptr = self.inner.as_ptr();
27
+
28
+
// SAFETY: As we got `self_ptr` from a reference above, it must point to
29
+
// a valid `T`.
30
+
let timer_ptr = unsafe { <T as HasHrTimer<T>>::raw_get_timer(self_ptr) };
31
+
32
+
// SAFETY: As `timer_ptr` is derived from a reference, it must point to
33
+
// a valid and initialized `HrTimer`.
34
+
unsafe { HrTimer::<T>::raw_cancel(timer_ptr) }
35
+
}
36
+
}
37
+
38
+
impl<'a, T> Drop for PinMutHrTimerHandle<'a, T>
39
+
where
40
+
T: HasHrTimer<T>,
41
+
{
42
+
fn drop(&mut self) {
43
+
self.cancel();
44
+
}
45
+
}
46
+
47
+
// SAFETY: We capture the lifetime of `Self` when we create a
48
+
// `PinMutHrTimerHandle`, so `Self` will outlive the handle.
49
+
unsafe impl<'a, T> UnsafeHrTimerPointer for Pin<&'a mut T>
50
+
where
51
+
T: Send + Sync,
52
+
T: HasHrTimer<T>,
53
+
T: HrTimerCallback<Pointer<'a> = Self>,
54
+
{
55
+
type TimerHandle = PinMutHrTimerHandle<'a, T>;
56
+
57
+
unsafe fn start(mut self, expires: Ktime) -> Self::TimerHandle {
58
+
// SAFETY:
59
+
// - We promise not to move out of `self`. We only pass `self`
60
+
// back to the caller as a `Pin<&mut self>`.
61
+
// - The return value of `get_unchecked_mut` is guaranteed not to be null.
62
+
let self_ptr = unsafe { NonNull::new_unchecked(self.as_mut().get_unchecked_mut()) };
63
+
64
+
// SAFETY:
65
+
// - As we derive `self_ptr` from a reference above, it must point to a
66
+
// valid `T`.
67
+
// - We keep `self` alive by wrapping it in a handle below.
68
+
unsafe { T::start(self_ptr.as_ptr(), expires) };
69
+
70
+
PinMutHrTimerHandle {
71
+
inner: self_ptr,
72
+
_p: PhantomData,
73
+
}
74
+
}
75
+
}
76
+
77
+
impl<'a, T> RawHrTimerCallback for Pin<&'a mut T>
78
+
where
79
+
T: HasHrTimer<T>,
80
+
T: HrTimerCallback<Pointer<'a> = Self>,
81
+
{
82
+
type CallbackTarget<'b> = Self;
83
+
84
+
unsafe extern "C" fn run(ptr: *mut bindings::hrtimer) -> bindings::hrtimer_restart {
85
+
// `HrTimer` is `repr(C)`
86
+
let timer_ptr = ptr as *mut HrTimer<T>;
87
+
88
+
// SAFETY: By the safety requirement of this function, `timer_ptr`
89
+
// points to a `HrTimer<T>` contained in an `T`.
90
+
let receiver_ptr = unsafe { T::timer_container_of(timer_ptr) };
91
+
92
+
// SAFETY:
93
+
// - By the safety requirement of this function, `timer_ptr`
94
+
// points to a `HrTimer<T>` contained in an `T`.
95
+
// - As per the safety requirements of the trait `HrTimerHandle`, the
96
+
// `PinMutHrTimerHandle` associated with this timer is guaranteed to
97
+
// be alive until this method returns. That handle borrows the `T`
98
+
// behind `receiver_ptr` mutably thus guaranteeing the validity of
99
+
// the reference created below.
100
+
let receiver_ref = unsafe { &mut *receiver_ptr };
101
+
102
+
// SAFETY: `receiver_ref` only exists as pinned, so it is safe to pin it
103
+
// here.
104
+
let receiver_pin = unsafe { Pin::new_unchecked(receiver_ref) };
105
+
106
+
T::run(receiver_pin).into_c()
107
+
}
108
+
}
+120
rust/kernel/time/hrtimer/tbox.rs
+120
rust/kernel/time/hrtimer/tbox.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
use super::HasHrTimer;
4
+
use super::HrTimer;
5
+
use super::HrTimerCallback;
6
+
use super::HrTimerHandle;
7
+
use super::HrTimerPointer;
8
+
use super::RawHrTimerCallback;
9
+
use crate::prelude::*;
10
+
use crate::time::Ktime;
11
+
use core::ptr::NonNull;
12
+
13
+
/// A handle for a [`Box<HasHrTimer<T>>`] returned by a call to
14
+
/// [`HrTimerPointer::start`].
15
+
///
16
+
/// # Invariants
17
+
///
18
+
/// - `self.inner` comes from a `Box::into_raw` call.
19
+
pub struct BoxHrTimerHandle<T, A>
20
+
where
21
+
T: HasHrTimer<T>,
22
+
A: crate::alloc::Allocator,
23
+
{
24
+
pub(crate) inner: NonNull<T>,
25
+
_p: core::marker::PhantomData<A>,
26
+
}
27
+
28
+
// SAFETY: We implement drop below, and we cancel the timer in the drop
29
+
// implementation.
30
+
unsafe impl<T, A> HrTimerHandle for BoxHrTimerHandle<T, A>
31
+
where
32
+
T: HasHrTimer<T>,
33
+
A: crate::alloc::Allocator,
34
+
{
35
+
fn cancel(&mut self) -> bool {
36
+
// SAFETY: As we obtained `self.inner` from a valid reference when we
37
+
// created `self`, it must point to a valid `T`.
38
+
let timer_ptr = unsafe { <T as HasHrTimer<T>>::raw_get_timer(self.inner.as_ptr()) };
39
+
40
+
// SAFETY: As `timer_ptr` points into `T` and `T` is valid, `timer_ptr`
41
+
// must point to a valid `HrTimer` instance.
42
+
unsafe { HrTimer::<T>::raw_cancel(timer_ptr) }
43
+
}
44
+
}
45
+
46
+
impl<T, A> Drop for BoxHrTimerHandle<T, A>
47
+
where
48
+
T: HasHrTimer<T>,
49
+
A: crate::alloc::Allocator,
50
+
{
51
+
fn drop(&mut self) {
52
+
self.cancel();
53
+
// SAFETY: By type invariant, `self.inner` came from a `Box::into_raw`
54
+
// call.
55
+
drop(unsafe { Box::<T, A>::from_raw(self.inner.as_ptr()) })
56
+
}
57
+
}
58
+
59
+
impl<T, A> HrTimerPointer for Pin<Box<T, A>>
60
+
where
61
+
T: 'static,
62
+
T: Send + Sync,
63
+
T: HasHrTimer<T>,
64
+
T: for<'a> HrTimerCallback<Pointer<'a> = Pin<Box<T, A>>>,
65
+
A: crate::alloc::Allocator,
66
+
{
67
+
type TimerHandle = BoxHrTimerHandle<T, A>;
68
+
69
+
fn start(self, expires: Ktime) -> Self::TimerHandle {
70
+
// SAFETY:
71
+
// - We will not move out of this box during timer callback (we pass an
72
+
// immutable reference to the callback).
73
+
// - `Box::into_raw` is guaranteed to return a valid pointer.
74
+
let inner =
75
+
unsafe { NonNull::new_unchecked(Box::into_raw(Pin::into_inner_unchecked(self))) };
76
+
77
+
// SAFETY:
78
+
// - We keep `self` alive by wrapping it in a handle below.
79
+
// - Since we generate the pointer passed to `start` from a valid
80
+
// reference, it is a valid pointer.
81
+
unsafe { T::start(inner.as_ptr(), expires) };
82
+
83
+
// INVARIANT: `inner` came from `Box::into_raw` above.
84
+
BoxHrTimerHandle {
85
+
inner,
86
+
_p: core::marker::PhantomData,
87
+
}
88
+
}
89
+
}
90
+
91
+
impl<T, A> RawHrTimerCallback for Pin<Box<T, A>>
92
+
where
93
+
T: 'static,
94
+
T: HasHrTimer<T>,
95
+
T: for<'a> HrTimerCallback<Pointer<'a> = Pin<Box<T, A>>>,
96
+
A: crate::alloc::Allocator,
97
+
{
98
+
type CallbackTarget<'a> = Pin<&'a mut T>;
99
+
100
+
unsafe extern "C" fn run(ptr: *mut bindings::hrtimer) -> bindings::hrtimer_restart {
101
+
// `HrTimer` is `repr(C)`
102
+
let timer_ptr = ptr.cast::<super::HrTimer<T>>();
103
+
104
+
// SAFETY: By C API contract `ptr` is the pointer we passed when
105
+
// queuing the timer, so it is a `HrTimer<T>` embedded in a `T`.
106
+
let data_ptr = unsafe { T::timer_container_of(timer_ptr) };
107
+
108
+
// SAFETY:
109
+
// - As per the safety requirements of the trait `HrTimerHandle`, the
110
+
// `BoxHrTimerHandle` associated with this timer is guaranteed to
111
+
// be alive until this method returns. That handle owns the `T`
112
+
// behind `data_ptr` thus guaranteeing the validity of
113
+
// the reference created below.
114
+
// - As `data_ptr` comes from a `Pin<Box<T>>`, only pinned references to
115
+
// `data_ptr` exist.
116
+
let data_mut_ref = unsafe { Pin::new_unchecked(&mut *data_ptr) };
117
+
118
+
T::run(data_mut_ref).into_c()
119
+
}
120
+
}
+14
-9
rust/kernel/types.rs
+14
-9
rust/kernel/types.rs
···
2
2
3
3
//! Kernel types.
4
4
5
-
use crate::init::{self, PinInit};
6
5
use core::{
7
6
cell::UnsafeCell,
8
7
marker::{PhantomData, PhantomPinned},
···
9
10
ops::{Deref, DerefMut},
10
11
ptr::NonNull,
11
12
};
13
+
use pin_init::{PinInit, Zeroable};
12
14
13
15
/// Used to transfer ownership to and from foreign (non-Rust) languages.
14
16
///
···
77
77
///
78
78
/// The provided pointer must have been returned by a previous call to [`into_foreign`], and if
79
79
/// the pointer is ever passed to [`from_foreign`], then that call must happen after the end of
80
-
/// the lifetime 'a.
80
+
/// the lifetime `'a`.
81
81
///
82
82
/// [`into_foreign`]: Self::into_foreign
83
83
/// [`from_foreign`]: Self::from_foreign
···
100
100
///
101
101
/// The provided pointer must have been returned by a previous call to [`into_foreign`], and if
102
102
/// the pointer is ever passed to [`from_foreign`], then that call must happen after the end of
103
-
/// the lifetime 'a.
103
+
/// the lifetime `'a`.
104
104
///
105
-
/// The lifetime 'a must not overlap with the lifetime of any other call to [`borrow`] or
105
+
/// The lifetime `'a` must not overlap with the lifetime of any other call to [`borrow`] or
106
106
/// `borrow_mut` on the same object.
107
107
///
108
108
/// [`into_foreign`]: Self::into_foreign
···
251
251
252
252
/// Stores an opaque value.
253
253
///
254
-
/// `Opaque<T>` is meant to be used with FFI objects that are never interpreted by Rust code.
254
+
/// [`Opaque<T>`] is meant to be used with FFI objects that are never interpreted by Rust code.
255
255
///
256
256
/// It is used to wrap structs from the C side, like for example `Opaque<bindings::mutex>`.
257
257
/// It gets rid of all the usual assumptions that Rust has for a value:
···
266
266
/// This has to be used for all values that the C side has access to, because it can't be ensured
267
267
/// that the C side is adhering to the usual constraints that Rust needs.
268
268
///
269
-
/// Using `Opaque<T>` allows to continue to use references on the Rust side even for values shared
269
+
/// Using [`Opaque<T>`] allows to continue to use references on the Rust side even for values shared
270
270
/// with C.
271
271
///
272
272
/// # Examples
···
309
309
_pin: PhantomPinned,
310
310
}
311
311
312
+
// SAFETY: `Opaque<T>` allows the inner value to be any bit pattern, including all zeros.
313
+
unsafe impl<T> Zeroable for Opaque<T> {}
314
+
312
315
impl<T> Opaque<T> {
313
316
/// Creates a new opaque value.
314
317
pub const fn new(value: T) -> Self {
···
336
333
// - `ptr` is a valid pointer to uninitialized memory,
337
334
// - `slot` is not accessed on error; the call is infallible,
338
335
// - `slot` is pinned in memory.
339
-
let _ = unsafe { init::PinInit::<T>::__pinned_init(slot, ptr) };
336
+
let _ = unsafe { PinInit::<T>::__pinned_init(slot, ptr) };
340
337
})
341
338
}
342
339
···
352
349
// SAFETY: We contain a `MaybeUninit`, so it is OK for the `init_func` to not fully
353
350
// initialize the `T`.
354
351
unsafe {
355
-
init::pin_init_from_closure::<_, ::core::convert::Infallible>(move |slot| {
352
+
pin_init::pin_init_from_closure::<_, ::core::convert::Infallible>(move |slot| {
356
353
init_func(Self::raw_get(slot));
357
354
Ok(())
358
355
})
···
372
369
) -> impl PinInit<Self, E> {
373
370
// SAFETY: We contain a `MaybeUninit`, so it is OK for the `init_func` to not fully
374
371
// initialize the `T`.
375
-
unsafe { init::pin_init_from_closure::<_, E>(move |slot| init_func(Self::raw_get(slot))) }
372
+
unsafe {
373
+
pin_init::pin_init_from_closure::<_, E>(move |slot| init_func(Self::raw_get(slot)))
374
+
}
376
375
}
377
376
378
377
/// Returns a raw pointer to the opaque data.
+1
-2
rust/kernel/uaccess.rs
+1
-2
rust/kernel/uaccess.rs
···
285
285
let len = self.length;
286
286
buf.reserve(len, flags)?;
287
287
288
-
// The call to `try_reserve` was successful, so the spare capacity is at least `len` bytes
289
-
// long.
288
+
// The call to `reserve` was successful, so the spare capacity is at least `len` bytes long.
290
289
self.read_raw(&mut buf.spare_capacity_mut()[..len])?;
291
290
292
291
// SAFETY: Since the call to `read_raw` was successful, so the next `len` bytes of the
+29
rust/macros/export.rs
+29
rust/macros/export.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
use crate::helpers::function_name;
4
+
use proc_macro::TokenStream;
5
+
6
+
/// Please see [`crate::export`] for documentation.
7
+
pub(crate) fn export(_attr: TokenStream, ts: TokenStream) -> TokenStream {
8
+
let Some(name) = function_name(ts.clone()) else {
9
+
return "::core::compile_error!(\"The #[export] attribute must be used on a function.\");"
10
+
.parse::<TokenStream>()
11
+
.unwrap();
12
+
};
13
+
14
+
// This verifies that the function has the same signature as the declaration generated by
15
+
// bindgen. It makes use of the fact that all branches of an if/else must have the same type.
16
+
let signature_check = quote!(
17
+
const _: () = {
18
+
if true {
19
+
::kernel::bindings::#name
20
+
} else {
21
+
#name
22
+
};
23
+
};
24
+
);
25
+
26
+
let no_mangle = quote!(#[no_mangle]);
27
+
28
+
TokenStream::from_iter([signature_check, no_mangle, ts])
29
+
}
+12
-141
rust/macros/helpers.rs
+12
-141
rust/macros/helpers.rs
···
1
1
// SPDX-License-Identifier: GPL-2.0
2
2
3
-
use proc_macro::{token_stream, Group, TokenStream, TokenTree};
3
+
use proc_macro::{token_stream, Group, Ident, TokenStream, TokenTree};
4
4
5
5
pub(crate) fn try_ident(it: &mut token_stream::IntoIter) -> Option<String> {
6
6
if let Some(TokenTree::Ident(ident)) = it.next() {
···
70
70
}
71
71
}
72
72
73
-
/// Parsed generics.
74
-
///
75
-
/// See the field documentation for an explanation what each of the fields represents.
76
-
///
77
-
/// # Examples
78
-
///
79
-
/// ```rust,ignore
80
-
/// # let input = todo!();
81
-
/// let (Generics { decl_generics, impl_generics, ty_generics }, rest) = parse_generics(input);
82
-
/// quote! {
83
-
/// struct Foo<$($decl_generics)*> {
84
-
/// // ...
85
-
/// }
86
-
///
87
-
/// impl<$impl_generics> Foo<$ty_generics> {
88
-
/// fn foo() {
89
-
/// // ...
90
-
/// }
91
-
/// }
92
-
/// }
93
-
/// ```
94
-
pub(crate) struct Generics {
95
-
/// The generics with bounds and default values (e.g. `T: Clone, const N: usize = 0`).
96
-
///
97
-
/// Use this on type definitions e.g. `struct Foo<$decl_generics> ...` (or `union`/`enum`).
98
-
pub(crate) decl_generics: Vec<TokenTree>,
99
-
/// The generics with bounds (e.g. `T: Clone, const N: usize`).
100
-
///
101
-
/// Use this on `impl` blocks e.g. `impl<$impl_generics> Trait for ...`.
102
-
pub(crate) impl_generics: Vec<TokenTree>,
103
-
/// The generics without bounds and without default values (e.g. `T, N`).
104
-
///
105
-
/// Use this when you use the type that is declared with these generics e.g.
106
-
/// `Foo<$ty_generics>`.
107
-
pub(crate) ty_generics: Vec<TokenTree>,
108
-
}
109
-
110
-
/// Parses the given `TokenStream` into `Generics` and the rest.
111
-
///
112
-
/// The generics are not present in the rest, but a where clause might remain.
113
-
pub(crate) fn parse_generics(input: TokenStream) -> (Generics, Vec<TokenTree>) {
114
-
// The generics with bounds and default values.
115
-
let mut decl_generics = vec![];
116
-
// `impl_generics`, the declared generics with their bounds.
117
-
let mut impl_generics = vec![];
118
-
// Only the names of the generics, without any bounds.
119
-
let mut ty_generics = vec![];
120
-
// Tokens not related to the generics e.g. the `where` token and definition.
121
-
let mut rest = vec![];
122
-
// The current level of `<`.
123
-
let mut nesting = 0;
124
-
let mut toks = input.into_iter();
125
-
// If we are at the beginning of a generic parameter.
126
-
let mut at_start = true;
127
-
let mut skip_until_comma = false;
128
-
while let Some(tt) = toks.next() {
129
-
if nesting == 1 && matches!(&tt, TokenTree::Punct(p) if p.as_char() == '>') {
130
-
// Found the end of the generics.
131
-
break;
132
-
} else if nesting >= 1 {
133
-
decl_generics.push(tt.clone());
134
-
}
135
-
match tt.clone() {
136
-
TokenTree::Punct(p) if p.as_char() == '<' => {
137
-
if nesting >= 1 && !skip_until_comma {
138
-
// This is inside of the generics and part of some bound.
139
-
impl_generics.push(tt);
73
+
/// Given a function declaration, finds the name of the function.
74
+
pub(crate) fn function_name(input: TokenStream) -> Option<Ident> {
75
+
let mut input = input.into_iter();
76
+
while let Some(token) = input.next() {
77
+
match token {
78
+
TokenTree::Ident(i) if i.to_string() == "fn" => {
79
+
if let Some(TokenTree::Ident(i)) = input.next() {
80
+
return Some(i);
140
81
}
141
-
nesting += 1;
82
+
return None;
142
83
}
143
-
TokenTree::Punct(p) if p.as_char() == '>' => {
144
-
// This is a parsing error, so we just end it here.
145
-
if nesting == 0 {
146
-
break;
147
-
} else {
148
-
nesting -= 1;
149
-
if nesting >= 1 && !skip_until_comma {
150
-
// We are still inside of the generics and part of some bound.
151
-
impl_generics.push(tt);
152
-
}
153
-
}
154
-
}
155
-
TokenTree::Punct(p) if skip_until_comma && p.as_char() == ',' => {
156
-
if nesting == 1 {
157
-
impl_generics.push(tt.clone());
158
-
impl_generics.push(tt);
159
-
skip_until_comma = false;
160
-
}
161
-
}
162
-
_ if !skip_until_comma => {
163
-
match nesting {
164
-
// If we haven't entered the generics yet, we still want to keep these tokens.
165
-
0 => rest.push(tt),
166
-
1 => {
167
-
// Here depending on the token, it might be a generic variable name.
168
-
match tt.clone() {
169
-
TokenTree::Ident(i) if at_start && i.to_string() == "const" => {
170
-
let Some(name) = toks.next() else {
171
-
// Parsing error.
172
-
break;
173
-
};
174
-
impl_generics.push(tt);
175
-
impl_generics.push(name.clone());
176
-
ty_generics.push(name.clone());
177
-
decl_generics.push(name);
178
-
at_start = false;
179
-
}
180
-
TokenTree::Ident(_) if at_start => {
181
-
impl_generics.push(tt.clone());
182
-
ty_generics.push(tt);
183
-
at_start = false;
184
-
}
185
-
TokenTree::Punct(p) if p.as_char() == ',' => {
186
-
impl_generics.push(tt.clone());
187
-
ty_generics.push(tt);
188
-
at_start = true;
189
-
}
190
-
// Lifetimes begin with `'`.
191
-
TokenTree::Punct(p) if p.as_char() == '\'' && at_start => {
192
-
impl_generics.push(tt.clone());
193
-
ty_generics.push(tt);
194
-
}
195
-
// Generics can have default values, we skip these.
196
-
TokenTree::Punct(p) if p.as_char() == '=' => {
197
-
skip_until_comma = true;
198
-
}
199
-
_ => impl_generics.push(tt),
200
-
}
201
-
}
202
-
_ => impl_generics.push(tt),
203
-
}
204
-
}
205
-
_ => {}
84
+
_ => continue,
206
85
}
207
86
}
208
-
rest.extend(toks);
209
-
(
210
-
Generics {
211
-
impl_generics,
212
-
decl_generics,
213
-
ty_generics,
214
-
},
215
-
rest,
216
-
)
87
+
None
217
88
}
+161
rust/macros/kunit.rs
+161
rust/macros/kunit.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
//! Procedural macro to run KUnit tests using a user-space like syntax.
4
+
//!
5
+
//! Copyright (c) 2023 José Expósito <jose.exposito89@gmail.com>
6
+
7
+
use proc_macro::{Delimiter, Group, TokenStream, TokenTree};
8
+
use std::fmt::Write;
9
+
10
+
pub(crate) fn kunit_tests(attr: TokenStream, ts: TokenStream) -> TokenStream {
11
+
let attr = attr.to_string();
12
+
13
+
if attr.is_empty() {
14
+
panic!("Missing test name in `#[kunit_tests(test_name)]` macro")
15
+
}
16
+
17
+
if attr.len() > 255 {
18
+
panic!(
19
+
"The test suite name `{}` exceeds the maximum length of 255 bytes",
20
+
attr
21
+
)
22
+
}
23
+
24
+
let mut tokens: Vec<_> = ts.into_iter().collect();
25
+
26
+
// Scan for the `mod` keyword.
27
+
tokens
28
+
.iter()
29
+
.find_map(|token| match token {
30
+
TokenTree::Ident(ident) => match ident.to_string().as_str() {
31
+
"mod" => Some(true),
32
+
_ => None,
33
+
},
34
+
_ => None,
35
+
})
36
+
.expect("`#[kunit_tests(test_name)]` attribute should only be applied to modules");
37
+
38
+
// Retrieve the main body. The main body should be the last token tree.
39
+
let body = match tokens.pop() {
40
+
Some(TokenTree::Group(group)) if group.delimiter() == Delimiter::Brace => group,
41
+
_ => panic!("Cannot locate main body of module"),
42
+
};
43
+
44
+
// Get the functions set as tests. Search for `[test]` -> `fn`.
45
+
let mut body_it = body.stream().into_iter();
46
+
let mut tests = Vec::new();
47
+
while let Some(token) = body_it.next() {
48
+
match token {
49
+
TokenTree::Group(ident) if ident.to_string() == "[test]" => match body_it.next() {
50
+
Some(TokenTree::Ident(ident)) if ident.to_string() == "fn" => {
51
+
let test_name = match body_it.next() {
52
+
Some(TokenTree::Ident(ident)) => ident.to_string(),
53
+
_ => continue,
54
+
};
55
+
tests.push(test_name);
56
+
}
57
+
_ => continue,
58
+
},
59
+
_ => (),
60
+
}
61
+
}
62
+
63
+
// Add `#[cfg(CONFIG_KUNIT)]` before the module declaration.
64
+
let config_kunit = "#[cfg(CONFIG_KUNIT)]".to_owned().parse().unwrap();
65
+
tokens.insert(
66
+
0,
67
+
TokenTree::Group(Group::new(Delimiter::None, config_kunit)),
68
+
);
69
+
70
+
// Generate the test KUnit test suite and a test case for each `#[test]`.
71
+
// The code generated for the following test module:
72
+
//
73
+
// ```
74
+
// #[kunit_tests(kunit_test_suit_name)]
75
+
// mod tests {
76
+
// #[test]
77
+
// fn foo() {
78
+
// assert_eq!(1, 1);
79
+
// }
80
+
//
81
+
// #[test]
82
+
// fn bar() {
83
+
// assert_eq!(2, 2);
84
+
// }
85
+
// }
86
+
// ```
87
+
//
88
+
// Looks like:
89
+
//
90
+
// ```
91
+
// unsafe extern "C" fn kunit_rust_wrapper_foo(_test: *mut kernel::bindings::kunit) { foo(); }
92
+
// unsafe extern "C" fn kunit_rust_wrapper_bar(_test: *mut kernel::bindings::kunit) { bar(); }
93
+
//
94
+
// static mut TEST_CASES: [kernel::bindings::kunit_case; 3] = [
95
+
// kernel::kunit::kunit_case(kernel::c_str!("foo"), kunit_rust_wrapper_foo),
96
+
// kernel::kunit::kunit_case(kernel::c_str!("bar"), kunit_rust_wrapper_bar),
97
+
// kernel::kunit::kunit_case_null(),
98
+
// ];
99
+
//
100
+
// kernel::kunit_unsafe_test_suite!(kunit_test_suit_name, TEST_CASES);
101
+
// ```
102
+
let mut kunit_macros = "".to_owned();
103
+
let mut test_cases = "".to_owned();
104
+
for test in &tests {
105
+
let kunit_wrapper_fn_name = format!("kunit_rust_wrapper_{}", test);
106
+
let kunit_wrapper = format!(
107
+
"unsafe extern \"C\" fn {}(_test: *mut kernel::bindings::kunit) {{ {}(); }}",
108
+
kunit_wrapper_fn_name, test
109
+
);
110
+
writeln!(kunit_macros, "{kunit_wrapper}").unwrap();
111
+
writeln!(
112
+
test_cases,
113
+
" kernel::kunit::kunit_case(kernel::c_str!(\"{}\"), {}),",
114
+
test, kunit_wrapper_fn_name
115
+
)
116
+
.unwrap();
117
+
}
118
+
119
+
writeln!(kunit_macros).unwrap();
120
+
writeln!(
121
+
kunit_macros,
122
+
"static mut TEST_CASES: [kernel::bindings::kunit_case; {}] = [\n{test_cases} kernel::kunit::kunit_case_null(),\n];",
123
+
tests.len() + 1
124
+
)
125
+
.unwrap();
126
+
127
+
writeln!(
128
+
kunit_macros,
129
+
"kernel::kunit_unsafe_test_suite!({attr}, TEST_CASES);"
130
+
)
131
+
.unwrap();
132
+
133
+
// Remove the `#[test]` macros.
134
+
// We do this at a token level, in order to preserve span information.
135
+
let mut new_body = vec![];
136
+
let mut body_it = body.stream().into_iter();
137
+
138
+
while let Some(token) = body_it.next() {
139
+
match token {
140
+
TokenTree::Punct(ref c) if c.as_char() == '#' => match body_it.next() {
141
+
Some(TokenTree::Group(group)) if group.to_string() == "[test]" => (),
142
+
Some(next) => {
143
+
new_body.extend([token, next]);
144
+
}
145
+
_ => {
146
+
new_body.push(token);
147
+
}
148
+
},
149
+
_ => {
150
+
new_body.push(token);
151
+
}
152
+
}
153
+
}
154
+
155
+
let mut new_body = TokenStream::from_iter(new_body);
156
+
new_body.extend::<TokenStream>(kunit_macros.parse().unwrap());
157
+
158
+
tokens.push(TokenTree::Group(Group::new(Delimiter::Brace, new_body)));
159
+
160
+
tokens.into_iter().collect()
161
+
}
+46
-118
rust/macros/lib.rs
+46
-118
rust/macros/lib.rs
···
9
9
#[macro_use]
10
10
mod quote;
11
11
mod concat_idents;
12
+
mod export;
12
13
mod helpers;
14
+
mod kunit;
13
15
mod module;
14
16
mod paste;
15
-
mod pin_data;
16
-
mod pinned_drop;
17
17
mod vtable;
18
-
mod zeroable;
19
18
20
19
use proc_macro::TokenStream;
21
20
···
35
36
/// module!{
36
37
/// type: MyModule,
37
38
/// name: "my_kernel_module",
38
-
/// author: "Rust for Linux Contributors",
39
+
/// authors: ["Rust for Linux Contributors"],
39
40
/// description: "My very own kernel module!",
40
41
/// license: "GPL",
41
42
/// alias: ["alternate_module_name"],
···
68
69
/// module!{
69
70
/// type: MyDeviceDriverModule,
70
71
/// name: "my_device_driver_module",
71
-
/// author: "Rust for Linux Contributors",
72
+
/// authors: ["Rust for Linux Contributors"],
72
73
/// description: "My device driver requires firmware",
73
74
/// license: "GPL",
74
75
/// firmware: ["my_device_firmware1.bin", "my_device_firmware2.bin"],
···
87
88
/// # Supported argument types
88
89
/// - `type`: type which implements the [`Module`] trait (required).
89
90
/// - `name`: ASCII string literal of the name of the kernel module (required).
90
-
/// - `author`: string literal of the author of the kernel module.
91
+
/// - `authors`: array of ASCII string literals of the authors of the kernel module.
91
92
/// - `description`: string literal of the description of the kernel module.
92
93
/// - `license`: ASCII string literal of the license of the kernel module (required).
93
94
/// - `alias`: array of ASCII string literals of the alias names of the kernel module.
···
173
174
vtable::vtable(attr, ts)
174
175
}
175
176
177
+
/// Export a function so that C code can call it via a header file.
178
+
///
179
+
/// Functions exported using this macro can be called from C code using the declaration in the
180
+
/// appropriate header file. It should only be used in cases where C calls the function through a
181
+
/// header file; cases where C calls into Rust via a function pointer in a vtable (such as
182
+
/// `file_operations`) should not use this macro.
183
+
///
184
+
/// This macro has the following effect:
185
+
///
186
+
/// * Disables name mangling for this function.
187
+
/// * Verifies at compile-time that the function signature matches the declaration in the header
188
+
/// file.
189
+
///
190
+
/// You must declare the signature of the Rust function in a header file that is included by
191
+
/// `rust/bindings/bindings_helper.h`.
192
+
///
193
+
/// This macro is *not* the same as the C macros `EXPORT_SYMBOL_*`. All Rust symbols are currently
194
+
/// automatically exported with `EXPORT_SYMBOL_GPL`.
195
+
#[proc_macro_attribute]
196
+
pub fn export(attr: TokenStream, ts: TokenStream) -> TokenStream {
197
+
export::export(attr, ts)
198
+
}
199
+
176
200
/// Concatenate two identifiers.
177
201
///
178
202
/// This is useful in macros that need to declare or reference items with names
···
252
230
#[proc_macro]
253
231
pub fn concat_idents(ts: TokenStream) -> TokenStream {
254
232
concat_idents::concat_idents(ts)
255
-
}
256
-
257
-
/// Used to specify the pinning information of the fields of a struct.
258
-
///
259
-
/// This is somewhat similar in purpose as
260
-
/// [pin-project-lite](https://crates.io/crates/pin-project-lite).
261
-
/// Place this macro on a struct definition and then `#[pin]` in front of the attributes of each
262
-
/// field you want to structurally pin.
263
-
///
264
-
/// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`,
265
-
/// then `#[pin]` directs the type of initializer that is required.
266
-
///
267
-
/// If your `struct` implements `Drop`, then you need to add `PinnedDrop` as arguments to this
268
-
/// macro, and change your `Drop` implementation to `PinnedDrop` annotated with
269
-
/// `#[`[`macro@pinned_drop`]`]`, since dropping pinned values requires extra care.
270
-
///
271
-
/// # Examples
272
-
///
273
-
/// ```
274
-
/// # #![feature(lint_reasons)]
275
-
/// # use kernel::prelude::*;
276
-
/// # use std::{sync::Mutex, process::Command};
277
-
/// # use kernel::macros::pin_data;
278
-
/// #[pin_data]
279
-
/// struct DriverData {
280
-
/// #[pin]
281
-
/// queue: Mutex<KVec<Command>>,
282
-
/// buf: KBox<[u8; 1024 * 1024]>,
283
-
/// }
284
-
/// ```
285
-
///
286
-
/// ```
287
-
/// # #![feature(lint_reasons)]
288
-
/// # use kernel::prelude::*;
289
-
/// # use std::{sync::Mutex, process::Command};
290
-
/// # use core::pin::Pin;
291
-
/// # pub struct Info;
292
-
/// # mod bindings {
293
-
/// # pub unsafe fn destroy_info(_ptr: *mut super::Info) {}
294
-
/// # }
295
-
/// use kernel::macros::{pin_data, pinned_drop};
296
-
///
297
-
/// #[pin_data(PinnedDrop)]
298
-
/// struct DriverData {
299
-
/// #[pin]
300
-
/// queue: Mutex<KVec<Command>>,
301
-
/// buf: KBox<[u8; 1024 * 1024]>,
302
-
/// raw_info: *mut Info,
303
-
/// }
304
-
///
305
-
/// #[pinned_drop]
306
-
/// impl PinnedDrop for DriverData {
307
-
/// fn drop(self: Pin<&mut Self>) {
308
-
/// unsafe { bindings::destroy_info(self.raw_info) };
309
-
/// }
310
-
/// }
311
-
/// # fn main() {}
312
-
/// ```
313
-
///
314
-
/// [`pin_init!`]: ../kernel/macro.pin_init.html
315
-
// ^ cannot use direct link, since `kernel` is not a dependency of `macros`.
316
-
#[proc_macro_attribute]
317
-
pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
318
-
pin_data::pin_data(inner, item)
319
-
}
320
-
321
-
/// Used to implement `PinnedDrop` safely.
322
-
///
323
-
/// Only works on structs that are annotated via `#[`[`macro@pin_data`]`]`.
324
-
///
325
-
/// # Examples
326
-
///
327
-
/// ```
328
-
/// # #![feature(lint_reasons)]
329
-
/// # use kernel::prelude::*;
330
-
/// # use macros::{pin_data, pinned_drop};
331
-
/// # use std::{sync::Mutex, process::Command};
332
-
/// # use core::pin::Pin;
333
-
/// # mod bindings {
334
-
/// # pub struct Info;
335
-
/// # pub unsafe fn destroy_info(_ptr: *mut Info) {}
336
-
/// # }
337
-
/// #[pin_data(PinnedDrop)]
338
-
/// struct DriverData {
339
-
/// #[pin]
340
-
/// queue: Mutex<KVec<Command>>,
341
-
/// buf: KBox<[u8; 1024 * 1024]>,
342
-
/// raw_info: *mut bindings::Info,
343
-
/// }
344
-
///
345
-
/// #[pinned_drop]
346
-
/// impl PinnedDrop for DriverData {
347
-
/// fn drop(self: Pin<&mut Self>) {
348
-
/// unsafe { bindings::destroy_info(self.raw_info) };
349
-
/// }
350
-
/// }
351
-
/// ```
352
-
#[proc_macro_attribute]
353
-
pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
354
-
pinned_drop::pinned_drop(args, input)
355
233
}
356
234
357
235
/// Paste identifiers together.
···
394
472
tokens.into_iter().collect()
395
473
}
396
474
397
-
/// Derives the [`Zeroable`] trait for the given struct.
475
+
/// Registers a KUnit test suite and its test cases using a user-space like syntax.
398
476
///
399
-
/// This can only be used for structs where every field implements the [`Zeroable`] trait.
477
+
/// This macro should be used on modules. If `CONFIG_KUNIT` (in `.config`) is `n`, the target module
478
+
/// is ignored.
400
479
///
401
480
/// # Examples
402
481
///
403
-
/// ```
404
-
/// use kernel::macros::Zeroable;
482
+
/// ```ignore
483
+
/// # use macros::kunit_tests;
484
+
/// #[kunit_tests(kunit_test_suit_name)]
485
+
/// mod tests {
486
+
/// #[test]
487
+
/// fn foo() {
488
+
/// assert_eq!(1, 1);
489
+
/// }
405
490
///
406
-
/// #[derive(Zeroable)]
407
-
/// pub struct DriverData {
408
-
/// id: i64,
409
-
/// buf_ptr: *mut u8,
410
-
/// len: usize,
491
+
/// #[test]
492
+
/// fn bar() {
493
+
/// assert_eq!(2, 2);
494
+
/// }
411
495
/// }
412
496
/// ```
413
-
#[proc_macro_derive(Zeroable)]
414
-
pub fn derive_zeroable(input: TokenStream) -> TokenStream {
415
-
zeroable::derive(input)
497
+
#[proc_macro_attribute]
498
+
pub fn kunit_tests(attr: TokenStream, ts: TokenStream) -> TokenStream {
499
+
kunit::kunit_tests(attr, ts)
416
500
}
+10
-2
rust/macros/module.rs
+10
-2
rust/macros/module.rs
···
56
56
"
57
57
{cfg}
58
58
#[doc(hidden)]
59
-
#[link_section = \".modinfo\"]
59
+
#[cfg_attr(not(target_os = \"macos\"), link_section = \".modinfo\")]
60
60
#[used]
61
61
pub static __{module}_{counter}: [u8; {length}] = *{string};
62
62
",
···
95
95
license: String,
96
96
name: String,
97
97
author: Option<String>,
98
+
authors: Option<Vec<String>>,
98
99
description: Option<String>,
99
100
alias: Option<Vec<String>>,
100
101
firmware: Option<Vec<String>>,
···
109
108
"type",
110
109
"name",
111
110
"author",
111
+
"authors",
112
112
"description",
113
113
"license",
114
114
"alias",
···
138
136
"type" => info.type_ = expect_ident(it),
139
137
"name" => info.name = expect_string_ascii(it),
140
138
"author" => info.author = Some(expect_string(it)),
139
+
"authors" => info.authors = Some(expect_string_array(it)),
141
140
"description" => info.description = Some(expect_string(it)),
142
141
"license" => info.license = expect_string_ascii(it),
143
142
"alias" => info.alias = Some(expect_string_array(it)),
···
188
185
let mut modinfo = ModInfoBuilder::new(info.name.as_ref());
189
186
if let Some(author) = info.author {
190
187
modinfo.emit("author", &author);
188
+
}
189
+
if let Some(authors) = info.authors {
190
+
for author in authors {
191
+
modinfo.emit("author", &author);
192
+
}
191
193
}
192
194
if let Some(description) = info.description {
193
195
modinfo.emit("description", &description);
···
248
240
mod __module_init {{
249
241
mod __module_init {{
250
242
use super::super::{type_};
251
-
use kernel::init::PinInit;
243
+
use pin_init::PinInit;
252
244
253
245
/// The \"Rust loadable module\" mark.
254
246
//
+5
-2
rust/macros/pin_data.rs
rust/pin-init/internal/src/pin_data.rs
+5
-2
rust/macros/pin_data.rs
rust/pin-init/internal/src/pin_data.rs
···
1
1
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
2
3
+
#[cfg(not(kernel))]
4
+
use proc_macro2 as proc_macro;
5
+
3
6
use crate::helpers::{parse_generics, Generics};
4
7
use proc_macro::{Group, Punct, Spacing, TokenStream, TokenTree};
5
8
6
9
pub(crate) fn pin_data(args: TokenStream, input: TokenStream) -> TokenStream {
7
10
// This proc-macro only does some pre-parsing and then delegates the actual parsing to
8
-
// `kernel::__pin_data!`.
11
+
// `pin_init::__pin_data!`.
9
12
10
13
let (
11
14
Generics {
···
74
71
.collect::<Vec<_>>();
75
72
// This should be the body of the struct `{...}`.
76
73
let last = rest.pop();
77
-
let mut quoted = quote!(::kernel::__pin_data! {
74
+
let mut quoted = quote!(::pin_init::__pin_data! {
78
75
parse_input:
79
76
@args(#args),
80
77
@sig(#(#rest)*),
+5
-2
rust/macros/pinned_drop.rs
rust/pin-init/internal/src/pinned_drop.rs
+5
-2
rust/macros/pinned_drop.rs
rust/pin-init/internal/src/pinned_drop.rs
···
1
1
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
2
3
+
#[cfg(not(kernel))]
4
+
use proc_macro2 as proc_macro;
5
+
3
6
use proc_macro::{TokenStream, TokenTree};
4
7
5
8
pub(crate) fn pinned_drop(_args: TokenStream, input: TokenStream) -> TokenStream {
···
38
35
let idx = pinned_drop_idx
39
36
.unwrap_or_else(|| panic!("Expected an `impl` block implementing `PinnedDrop`."));
40
37
// Fully qualify the `PinnedDrop`, as to avoid any tampering.
41
-
toks.splice(idx..idx, quote!(::kernel::init::));
38
+
toks.splice(idx..idx, quote!(::pin_init::));
42
39
// Take the `{}` body and call the declarative macro.
43
40
if let Some(TokenTree::Group(last)) = toks.pop() {
44
41
let last = last.stream();
45
-
quote!(::kernel::__pinned_drop! {
42
+
quote!(::pin_init::__pinned_drop! {
46
43
@impl_sig(#(#toks)*),
47
44
@impl_body(#last),
48
45
})
+26
-2
rust/macros/quote.rs
+26
-2
rust/macros/quote.rs
···
2
2
3
3
use proc_macro::{TokenStream, TokenTree};
4
4
5
+
#[allow(dead_code)]
5
6
pub(crate) trait ToTokens {
6
7
fn to_tokens(&self, tokens: &mut TokenStream);
7
8
}
···
16
15
}
17
16
18
17
impl ToTokens for proc_macro::Group {
18
+
fn to_tokens(&self, tokens: &mut TokenStream) {
19
+
tokens.extend([TokenTree::from(self.clone())]);
20
+
}
21
+
}
22
+
23
+
impl ToTokens for proc_macro::Ident {
19
24
fn to_tokens(&self, tokens: &mut TokenStream) {
20
25
tokens.extend([TokenTree::from(self.clone())]);
21
26
}
···
47
40
/// `quote` crate but provides only just enough functionality needed by the current `macros` crate.
48
41
macro_rules! quote_spanned {
49
42
($span:expr => $($tt:tt)*) => {{
50
-
let mut tokens;
43
+
let mut tokens: ::std::vec::Vec<::proc_macro::TokenTree>;
51
44
#[allow(clippy::vec_init_then_push)]
52
45
{
53
46
tokens = ::std::vec::Vec::new();
···
72
65
quote_spanned!(@proc $v $span $($tt)*);
73
66
};
74
67
(@proc $v:ident $span:ident ( $($inner:tt)* ) $($tt:tt)*) => {
75
-
let mut tokens = ::std::vec::Vec::new();
68
+
#[allow(unused_mut)]
69
+
let mut tokens = ::std::vec::Vec::<::proc_macro::TokenTree>::new();
76
70
quote_spanned!(@proc tokens $span $($inner)*);
77
71
$v.push(::proc_macro::TokenTree::Group(::proc_macro::Group::new(
78
72
::proc_macro::Delimiter::Parenthesis,
···
142
134
$v.push(::proc_macro::TokenTree::Punct(
143
135
::proc_macro::Punct::new('+', ::proc_macro::Spacing::Alone)
144
136
));
137
+
quote_spanned!(@proc $v $span $($tt)*);
138
+
};
139
+
(@proc $v:ident $span:ident = $($tt:tt)*) => {
140
+
$v.push(::proc_macro::TokenTree::Punct(
141
+
::proc_macro::Punct::new('=', ::proc_macro::Spacing::Alone)
142
+
));
143
+
quote_spanned!(@proc $v $span $($tt)*);
144
+
};
145
+
(@proc $v:ident $span:ident # $($tt:tt)*) => {
146
+
$v.push(::proc_macro::TokenTree::Punct(
147
+
::proc_macro::Punct::new('#', ::proc_macro::Spacing::Alone)
148
+
));
149
+
quote_spanned!(@proc $v $span $($tt)*);
150
+
};
151
+
(@proc $v:ident $span:ident _ $($tt:tt)*) => {
152
+
$v.push(::proc_macro::TokenTree::Ident(::proc_macro::Ident::new("_", $span)));
145
153
quote_spanned!(@proc $v $span $($tt)*);
146
154
};
147
155
(@proc $v:ident $span:ident $id:ident $($tt:tt)*) => {
+7
-4
rust/macros/zeroable.rs
rust/pin-init/internal/src/zeroable.rs
+7
-4
rust/macros/zeroable.rs
rust/pin-init/internal/src/zeroable.rs
···
1
1
// SPDX-License-Identifier: GPL-2.0
2
2
3
+
#[cfg(not(kernel))]
4
+
use proc_macro2 as proc_macro;
5
+
3
6
use crate::helpers::{parse_generics, Generics};
4
7
use proc_macro::{TokenStream, TokenTree};
5
8
···
30
27
// If we find a `,`, then we have finished a generic/constant/lifetime parameter.
31
28
TokenTree::Punct(p) if nested == 0 && p.as_char() == ',' => {
32
29
if in_generic && !inserted {
33
-
new_impl_generics.extend(quote! { : ::kernel::init::Zeroable });
30
+
new_impl_generics.extend(quote! { : ::pin_init::Zeroable });
34
31
}
35
32
in_generic = true;
36
33
inserted = false;
···
44
41
TokenTree::Punct(p) if nested == 0 && p.as_char() == ':' => {
45
42
new_impl_generics.push(tt);
46
43
if in_generic {
47
-
new_impl_generics.extend(quote! { ::kernel::init::Zeroable + });
44
+
new_impl_generics.extend(quote! { ::pin_init::Zeroable + });
48
45
inserted = true;
49
46
}
50
47
}
···
62
59
}
63
60
assert_eq!(nested, 0);
64
61
if in_generic && !inserted {
65
-
new_impl_generics.extend(quote! { : ::kernel::init::Zeroable });
62
+
new_impl_generics.extend(quote! { : ::pin_init::Zeroable });
66
63
}
67
64
quote! {
68
-
::kernel::__derive_zeroable!(
65
+
::pin_init::__derive_zeroable!(
69
66
parse_input:
70
67
@sig(#(#rest)*),
71
68
@impl_generics(#(#new_impl_generics)*),
+72
rust/pin-init/CONTRIBUTING.md
+72
rust/pin-init/CONTRIBUTING.md
···
1
+
# Contributing to `pin-init`
2
+
3
+
Thanks for showing interest in contributing to `pin-init`! This document outlines the guidelines for
4
+
contributing to `pin-init`.
5
+
6
+
All contributions are double-licensed under Apache 2.0 and MIT. You can find the respective licenses
7
+
in the `LICENSE-APACHE` and `LICENSE-MIT` files.
8
+
9
+
## Non-Code Contributions
10
+
11
+
### Bug Reports
12
+
13
+
For any type of bug report, please submit an issue using the bug report issue template.
14
+
15
+
If the issue is a soundness issue, please privately report it as a security vulnerability via the
16
+
GitHub web interface.
17
+
18
+
### Feature Requests
19
+
20
+
If you have any feature requests, please submit an issue using the feature request issue template.
21
+
22
+
### Questions and Getting Help
23
+
24
+
You can ask questions in the Discussions page of the GitHub repository. If you're encountering
25
+
problems or just have questions related to `pin-init` in the Linux kernel, you can also ask your
26
+
questions in the [Rust-for-Linux Zulip](https://rust-for-linux.zulipchat.com/) or see
27
+
<https://rust-for-linux.com/contact>.
28
+
29
+
## Contributing Code
30
+
31
+
### Linux Kernel
32
+
33
+
`pin-init` is used by the Linux kernel and all commits are synchronized to it. For this reason, the
34
+
same requirements for commits apply to `pin-init`. See [the kernel's documentation] for details. The
35
+
rest of this document will also cover some of the rules listed there and additional ones.
36
+
37
+
[the kernel's documentation]: https://docs.kernel.org/process/submitting-patches.html
38
+
39
+
Contributions to `pin-init` ideally go through the [GitHub repository], because that repository runs
40
+
a CI with lots of tests not present in the kernel. However, patches are also accepted (though not
41
+
preferred). Do note that there are some files that are only present in the GitHub repository such as
42
+
tests, licenses and cargo related files. Making changes to them can only happen via GitHub.
43
+
44
+
[GitHub repository]: https://github.com/Rust-for-Linux/pin-init
45
+
46
+
### Commit Style
47
+
48
+
Everything must compile without errors or warnings and all tests must pass after **every commit**.
49
+
This is important for bisection and also required by the kernel.
50
+
51
+
Each commit should be a single, logically cohesive change. Of course it's best to keep the changes
52
+
small and digestible, but logically linked changes should be made in the same commit. For example,
53
+
when fixing typos, create a single commit that fixes all of them instead of one commit per typo.
54
+
55
+
Commits must have a meaningful commit title. Commits with changes to files in the `internal`
56
+
directory should have a title prefixed with `internal:`. The commit message should explain the
57
+
change and its rationale. You also have to add your `Signed-off-by` tag, see [Developer's
58
+
Certificate of Origin]. This has to be done for both mailing list submissions as well as GitHub
59
+
submissions.
60
+
61
+
[Developer's Certificate of Origin]: https://docs.kernel.org/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin
62
+
63
+
Any changes made to public APIs must be documented not only in the commit message, but also in the
64
+
`CHANGELOG.md` file. This is especially important for breaking changes, as those warrant a major
65
+
version bump.
66
+
67
+
If you make changes to the top-level crate documentation, you also need to update the `README.md`
68
+
via `cargo rdme`.
69
+
70
+
Some of these rules can be ignored if the change is done solely to files that are not present in the
71
+
kernel version of this library. Those files are documented in the `sync-kernel.sh` script at the
72
+
very bottom in the `--exclude` flag given to the `git am` command.
+228
rust/pin-init/README.md
+228
rust/pin-init/README.md
···
1
+
[](https://crates.io/crates/pin-init)
2
+
[](https://docs.rs/pin-init/)
3
+
[](https://deps.rs/repo/github/Rust-for-Linux/pin-init)
4
+
5
+
[](#nightly-only)
6
+
7
+
# `pin-init`
8
+
9
+
<!-- cargo-rdme start -->
10
+
11
+
Library to safely and fallibly initialize pinned `struct`s using in-place constructors.
12
+
13
+
[Pinning][pinning] is Rust's way of ensuring data does not move.
14
+
15
+
It also allows in-place initialization of big `struct`s that would otherwise produce a stack
16
+
overflow.
17
+
18
+
This library's main use-case is in [Rust-for-Linux]. Although this version can be used
19
+
standalone.
20
+
21
+
There are cases when you want to in-place initialize a struct. For example when it is very big
22
+
and moving it from the stack is not an option, because it is bigger than the stack itself.
23
+
Another reason would be that you need the address of the object to initialize it. This stands
24
+
in direct conflict with Rust's normal process of first initializing an object and then moving
25
+
it into it's final memory location. For more information, see
26
+
<https://rust-for-linux.com/the-safe-pinned-initialization-problem>.
27
+
28
+
This library allows you to do in-place initialization safely.
29
+
30
+
### Nightly Needed for `alloc` feature
31
+
32
+
This library requires the [`allocator_api` unstable feature] when the `alloc` feature is
33
+
enabled and thus this feature can only be used with a nightly compiler. When enabling the
34
+
`alloc` feature, the user will be required to activate `allocator_api` as well.
35
+
36
+
[`allocator_api` unstable feature]: https://doc.rust-lang.org/nightly/unstable-book/library-features/allocator-api.html
37
+
38
+
The feature is enabled by default, thus by default `pin-init` will require a nightly compiler.
39
+
However, using the crate on stable compilers is possible by disabling `alloc`. In practice this
40
+
will require the `std` feature, because stable compilers have neither `Box` nor `Arc` in no-std
41
+
mode.
42
+
43
+
## Overview
44
+
45
+
To initialize a `struct` with an in-place constructor you will need two things:
46
+
- an in-place constructor,
47
+
- a memory location that can hold your `struct` (this can be the [stack], an [`Arc<T>`],
48
+
[`Box<T>`] or any other smart pointer that supports this library).
49
+
50
+
To get an in-place constructor there are generally three options:
51
+
- directly creating an in-place constructor using the [`pin_init!`] macro,
52
+
- a custom function/macro returning an in-place constructor provided by someone else,
53
+
- using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
54
+
55
+
Aside from pinned initialization, this library also supports in-place construction without
56
+
pinning, the macros/types/functions are generally named like the pinned variants without the
57
+
`pin_` prefix.
58
+
59
+
## Examples
60
+
61
+
Throughout the examples we will often make use of the `CMutex` type which can be found in
62
+
`../examples/mutex.rs`. It is essentially a userland rebuild of the `struct mutex` type from
63
+
the Linux kernel. It also uses a wait list and a basic spinlock. Importantly the wait list
64
+
requires it to be pinned to be locked and thus is a prime candidate for using this library.
65
+
66
+
### Using the [`pin_init!`] macro
67
+
68
+
If you want to use [`PinInit`], then you will have to annotate your `struct` with
69
+
`#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
70
+
[structurally pinned fields]. After doing this, you can then create an in-place constructor via
71
+
[`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
72
+
that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
73
+
74
+
```rust
75
+
use pin_init::{pin_data, pin_init, InPlaceInit};
76
+
77
+
#[pin_data]
78
+
struct Foo {
79
+
#[pin]
80
+
a: CMutex<usize>,
81
+
b: u32,
82
+
}
83
+
84
+
let foo = pin_init!(Foo {
85
+
a <- CMutex::new(42),
86
+
b: 24,
87
+
});
88
+
```
89
+
90
+
`foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
91
+
(or just the stack) to actually initialize a `Foo`:
92
+
93
+
```rust
94
+
let foo: Result<Pin<Box<Foo>>, AllocError> = Box::pin_init(foo);
95
+
```
96
+
97
+
For more information see the [`pin_init!`] macro.
98
+
99
+
### Using a custom function/macro that returns an initializer
100
+
101
+
Many types that use this library supply a function/macro that returns an initializer, because
102
+
the above method only works for types where you can access the fields.
103
+
104
+
```rust
105
+
let mtx: Result<Pin<Arc<CMutex<usize>>>, _> = Arc::pin_init(CMutex::new(42));
106
+
```
107
+
108
+
To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
109
+
110
+
```rust
111
+
#[pin_data]
112
+
struct DriverData {
113
+
#[pin]
114
+
status: CMutex<i32>,
115
+
buffer: Box<[u8; 1_000_000]>,
116
+
}
117
+
118
+
impl DriverData {
119
+
fn new() -> impl PinInit<Self, Error> {
120
+
try_pin_init!(Self {
121
+
status <- CMutex::new(0),
122
+
buffer: Box::init(pin_init::zeroed())?,
123
+
}? Error)
124
+
}
125
+
}
126
+
```
127
+
128
+
### Manual creation of an initializer
129
+
130
+
Often when working with primitives the previous approaches are not sufficient. That is where
131
+
[`pin_init_from_closure()`] comes in. This `unsafe` function allows you to create a
132
+
[`impl PinInit<T, E>`] directly from a closure. Of course you have to ensure that the closure
133
+
actually does the initialization in the correct way. Here are the things to look out for
134
+
(we are calling the parameter to the closure `slot`):
135
+
- when the closure returns `Ok(())`, then it has completed the initialization successfully, so
136
+
`slot` now contains a valid bit pattern for the type `T`,
137
+
- when the closure returns `Err(e)`, then the caller may deallocate the memory at `slot`, so
138
+
you need to take care to clean up anything if your initialization fails mid-way,
139
+
- you may assume that `slot` will stay pinned even after the closure returns until `drop` of
140
+
`slot` gets called.
141
+
142
+
```rust
143
+
use pin_init::{pin_data, pinned_drop, PinInit, PinnedDrop, pin_init_from_closure};
144
+
use core::{
145
+
ptr::addr_of_mut,
146
+
marker::PhantomPinned,
147
+
cell::UnsafeCell,
148
+
pin::Pin,
149
+
mem::MaybeUninit,
150
+
};
151
+
mod bindings {
152
+
#[repr(C)]
153
+
pub struct foo {
154
+
/* fields from C ... */
155
+
}
156
+
extern "C" {
157
+
pub fn init_foo(ptr: *mut foo);
158
+
pub fn destroy_foo(ptr: *mut foo);
159
+
#[must_use = "you must check the error return code"]
160
+
pub fn enable_foo(ptr: *mut foo, flags: u32) -> i32;
161
+
}
162
+
}
163
+
164
+
/// # Invariants
165
+
///
166
+
/// `foo` is always initialized
167
+
#[pin_data(PinnedDrop)]
168
+
pub struct RawFoo {
169
+
#[pin]
170
+
_p: PhantomPinned,
171
+
#[pin]
172
+
foo: UnsafeCell<MaybeUninit<bindings::foo>>,
173
+
}
174
+
175
+
impl RawFoo {
176
+
pub fn new(flags: u32) -> impl PinInit<Self, i32> {
177
+
// SAFETY:
178
+
// - when the closure returns `Ok(())`, then it has successfully initialized and
179
+
// enabled `foo`,
180
+
// - when it returns `Err(e)`, then it has cleaned up before
181
+
unsafe {
182
+
pin_init_from_closure(move |slot: *mut Self| {
183
+
// `slot` contains uninit memory, avoid creating a reference.
184
+
let foo = addr_of_mut!((*slot).foo);
185
+
let foo = UnsafeCell::raw_get(foo).cast::<bindings::foo>();
186
+
187
+
// Initialize the `foo`
188
+
bindings::init_foo(foo);
189
+
190
+
// Try to enable it.
191
+
let err = bindings::enable_foo(foo, flags);
192
+
if err != 0 {
193
+
// Enabling has failed, first clean up the foo and then return the error.
194
+
bindings::destroy_foo(foo);
195
+
Err(err)
196
+
} else {
197
+
// All fields of `RawFoo` have been initialized, since `_p` is a ZST.
198
+
Ok(())
199
+
}
200
+
})
201
+
}
202
+
}
203
+
}
204
+
205
+
#[pinned_drop]
206
+
impl PinnedDrop for RawFoo {
207
+
fn drop(self: Pin<&mut Self>) {
208
+
// SAFETY: Since `foo` is initialized, destroying is safe.
209
+
unsafe { bindings::destroy_foo(self.foo.get().cast::<bindings::foo>()) };
210
+
}
211
+
}
212
+
```
213
+
214
+
For more information on how to use [`pin_init_from_closure()`], take a look at the uses inside
215
+
the `kernel` crate. The [`sync`] module is a good starting point.
216
+
217
+
[`sync`]: https://rust.docs.kernel.org/kernel/sync/index.html
218
+
[pinning]: https://doc.rust-lang.org/std/pin/index.html
219
+
[structurally pinned fields]: https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
220
+
[stack]: https://docs.rs/pin-init/latest/pin_init/macro.stack_pin_init.html
221
+
[`Arc<T>`]: https://doc.rust-lang.org/stable/alloc/sync/struct.Arc.html
222
+
[`Box<T>`]: https://doc.rust-lang.org/stable/alloc/boxed/struct.Box.html
223
+
[`impl PinInit<Foo>`]: https://docs.rs/pin-init/latest/pin_init/trait.PinInit.html
224
+
[`impl PinInit<T, E>`]: https://docs.rs/pin-init/latest/pin_init/trait.PinInit.html
225
+
[`impl Init<T, E>`]: https://docs.rs/pin-init/latest/pin_init/trait.Init.html
226
+
[Rust-for-Linux]: https://rust-for-linux.com/
227
+
228
+
<!-- cargo-rdme end -->
+39
rust/pin-init/examples/big_struct_in_place.rs
+39
rust/pin-init/examples/big_struct_in_place.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
use pin_init::*;
4
+
5
+
// Struct with size over 1GiB
6
+
#[derive(Debug)]
7
+
pub struct BigStruct {
8
+
buf: [u8; 1024 * 1024 * 1024],
9
+
a: u64,
10
+
b: u64,
11
+
c: u64,
12
+
d: u64,
13
+
managed_buf: ManagedBuf,
14
+
}
15
+
16
+
#[derive(Debug)]
17
+
pub struct ManagedBuf {
18
+
buf: [u8; 1024 * 1024],
19
+
}
20
+
21
+
impl ManagedBuf {
22
+
pub fn new() -> impl Init<Self> {
23
+
init!(ManagedBuf { buf <- zeroed() })
24
+
}
25
+
}
26
+
27
+
fn main() {
28
+
// we want to initialize the struct in-place, otherwise we would get a stackoverflow
29
+
let buf: Box<BigStruct> = Box::init(init!(BigStruct {
30
+
buf <- zeroed(),
31
+
a: 7,
32
+
b: 186,
33
+
c: 7789,
34
+
d: 34,
35
+
managed_buf <- ManagedBuf::new(),
36
+
}))
37
+
.unwrap();
38
+
println!("{}", core::mem::size_of_val(&*buf));
39
+
}
+27
rust/pin-init/examples/error.rs
+27
rust/pin-init/examples/error.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
4
+
5
+
use core::convert::Infallible;
6
+
7
+
#[cfg(feature = "alloc")]
8
+
use std::alloc::AllocError;
9
+
10
+
#[derive(Debug)]
11
+
pub struct Error;
12
+
13
+
impl From<Infallible> for Error {
14
+
fn from(e: Infallible) -> Self {
15
+
match e {}
16
+
}
17
+
}
18
+
19
+
#[cfg(feature = "alloc")]
20
+
impl From<AllocError> for Error {
21
+
fn from(_: AllocError) -> Self {
22
+
Self
23
+
}
24
+
}
25
+
26
+
#[allow(dead_code)]
27
+
fn main() {}
+161
rust/pin-init/examples/linked_list.rs
+161
rust/pin-init/examples/linked_list.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#![allow(clippy::undocumented_unsafe_blocks)]
4
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
5
+
6
+
use core::{
7
+
cell::Cell,
8
+
convert::Infallible,
9
+
marker::PhantomPinned,
10
+
pin::Pin,
11
+
ptr::{self, NonNull},
12
+
};
13
+
14
+
use pin_init::*;
15
+
16
+
#[expect(unused_attributes)]
17
+
mod error;
18
+
use error::Error;
19
+
20
+
#[pin_data(PinnedDrop)]
21
+
#[repr(C)]
22
+
#[derive(Debug)]
23
+
pub struct ListHead {
24
+
next: Link,
25
+
prev: Link,
26
+
#[pin]
27
+
pin: PhantomPinned,
28
+
}
29
+
30
+
impl ListHead {
31
+
#[inline]
32
+
pub fn new() -> impl PinInit<Self, Infallible> {
33
+
try_pin_init!(&this in Self {
34
+
next: unsafe { Link::new_unchecked(this) },
35
+
prev: unsafe { Link::new_unchecked(this) },
36
+
pin: PhantomPinned,
37
+
}? Infallible)
38
+
}
39
+
40
+
#[inline]
41
+
pub fn insert_next(list: &ListHead) -> impl PinInit<Self, Infallible> + '_ {
42
+
try_pin_init!(&this in Self {
43
+
prev: list.next.prev().replace(unsafe { Link::new_unchecked(this)}),
44
+
next: list.next.replace(unsafe { Link::new_unchecked(this)}),
45
+
pin: PhantomPinned,
46
+
}? Infallible)
47
+
}
48
+
49
+
#[inline]
50
+
pub fn insert_prev(list: &ListHead) -> impl PinInit<Self, Infallible> + '_ {
51
+
try_pin_init!(&this in Self {
52
+
next: list.prev.next().replace(unsafe { Link::new_unchecked(this)}),
53
+
prev: list.prev.replace(unsafe { Link::new_unchecked(this)}),
54
+
pin: PhantomPinned,
55
+
}? Infallible)
56
+
}
57
+
58
+
#[inline]
59
+
pub fn next(&self) -> Option<NonNull<Self>> {
60
+
if ptr::eq(self.next.as_ptr(), self) {
61
+
None
62
+
} else {
63
+
Some(unsafe { NonNull::new_unchecked(self.next.as_ptr() as *mut Self) })
64
+
}
65
+
}
66
+
67
+
#[allow(dead_code)]
68
+
pub fn size(&self) -> usize {
69
+
let mut size = 1;
70
+
let mut cur = self.next.clone();
71
+
while !ptr::eq(self, cur.cur()) {
72
+
cur = cur.next().clone();
73
+
size += 1;
74
+
}
75
+
size
76
+
}
77
+
}
78
+
79
+
#[pinned_drop]
80
+
impl PinnedDrop for ListHead {
81
+
//#[inline]
82
+
fn drop(self: Pin<&mut Self>) {
83
+
if !ptr::eq(self.next.as_ptr(), &*self) {
84
+
let next = unsafe { &*self.next.as_ptr() };
85
+
let prev = unsafe { &*self.prev.as_ptr() };
86
+
next.prev.set(&self.prev);
87
+
prev.next.set(&self.next);
88
+
}
89
+
}
90
+
}
91
+
92
+
#[repr(transparent)]
93
+
#[derive(Clone, Debug)]
94
+
struct Link(Cell<NonNull<ListHead>>);
95
+
96
+
impl Link {
97
+
/// # Safety
98
+
///
99
+
/// The contents of the pointer should form a consistent circular
100
+
/// linked list; for example, a "next" link should be pointed back
101
+
/// by the target `ListHead`'s "prev" link and a "prev" link should be
102
+
/// pointed back by the target `ListHead`'s "next" link.
103
+
#[inline]
104
+
unsafe fn new_unchecked(ptr: NonNull<ListHead>) -> Self {
105
+
Self(Cell::new(ptr))
106
+
}
107
+
108
+
#[inline]
109
+
fn next(&self) -> &Link {
110
+
unsafe { &(*self.0.get().as_ptr()).next }
111
+
}
112
+
113
+
#[inline]
114
+
fn prev(&self) -> &Link {
115
+
unsafe { &(*self.0.get().as_ptr()).prev }
116
+
}
117
+
118
+
#[allow(dead_code)]
119
+
fn cur(&self) -> &ListHead {
120
+
unsafe { &*self.0.get().as_ptr() }
121
+
}
122
+
123
+
#[inline]
124
+
fn replace(&self, other: Link) -> Link {
125
+
unsafe { Link::new_unchecked(self.0.replace(other.0.get())) }
126
+
}
127
+
128
+
#[inline]
129
+
fn as_ptr(&self) -> *const ListHead {
130
+
self.0.get().as_ptr()
131
+
}
132
+
133
+
#[inline]
134
+
fn set(&self, val: &Link) {
135
+
self.0.set(val.0.get());
136
+
}
137
+
}
138
+
139
+
#[allow(dead_code)]
140
+
#[cfg_attr(test, test)]
141
+
fn main() -> Result<(), Error> {
142
+
let a = Box::pin_init(ListHead::new())?;
143
+
stack_pin_init!(let b = ListHead::insert_next(&a));
144
+
stack_pin_init!(let c = ListHead::insert_next(&a));
145
+
stack_pin_init!(let d = ListHead::insert_next(&b));
146
+
let e = Box::pin_init(ListHead::insert_next(&b))?;
147
+
println!("a ({a:p}): {a:?}");
148
+
println!("b ({b:p}): {b:?}");
149
+
println!("c ({c:p}): {c:?}");
150
+
println!("d ({d:p}): {d:?}");
151
+
println!("e ({e:p}): {e:?}");
152
+
let mut inspect = &*a;
153
+
while let Some(next) = inspect.next() {
154
+
println!("({inspect:p}): {inspect:?}");
155
+
inspect = unsafe { &*next.as_ptr() };
156
+
if core::ptr::eq(inspect, &*a) {
157
+
break;
158
+
}
159
+
}
160
+
Ok(())
161
+
}
+209
rust/pin-init/examples/mutex.rs
+209
rust/pin-init/examples/mutex.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#![allow(clippy::undocumented_unsafe_blocks)]
4
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
5
+
#![allow(clippy::missing_safety_doc)]
6
+
7
+
use core::{
8
+
cell::{Cell, UnsafeCell},
9
+
marker::PhantomPinned,
10
+
ops::{Deref, DerefMut},
11
+
pin::Pin,
12
+
sync::atomic::{AtomicBool, Ordering},
13
+
};
14
+
use std::{
15
+
sync::Arc,
16
+
thread::{self, park, sleep, Builder, Thread},
17
+
time::Duration,
18
+
};
19
+
20
+
use pin_init::*;
21
+
#[expect(unused_attributes)]
22
+
#[path = "./linked_list.rs"]
23
+
pub mod linked_list;
24
+
use linked_list::*;
25
+
26
+
pub struct SpinLock {
27
+
inner: AtomicBool,
28
+
}
29
+
30
+
impl SpinLock {
31
+
#[inline]
32
+
pub fn acquire(&self) -> SpinLockGuard<'_> {
33
+
while self
34
+
.inner
35
+
.compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed)
36
+
.is_err()
37
+
{
38
+
while self.inner.load(Ordering::Relaxed) {
39
+
thread::yield_now();
40
+
}
41
+
}
42
+
SpinLockGuard(self)
43
+
}
44
+
45
+
#[inline]
46
+
#[allow(clippy::new_without_default)]
47
+
pub const fn new() -> Self {
48
+
Self {
49
+
inner: AtomicBool::new(false),
50
+
}
51
+
}
52
+
}
53
+
54
+
pub struct SpinLockGuard<'a>(&'a SpinLock);
55
+
56
+
impl Drop for SpinLockGuard<'_> {
57
+
#[inline]
58
+
fn drop(&mut self) {
59
+
self.0.inner.store(false, Ordering::Release);
60
+
}
61
+
}
62
+
63
+
#[pin_data]
64
+
pub struct CMutex<T> {
65
+
#[pin]
66
+
wait_list: ListHead,
67
+
spin_lock: SpinLock,
68
+
locked: Cell<bool>,
69
+
#[pin]
70
+
data: UnsafeCell<T>,
71
+
}
72
+
73
+
impl<T> CMutex<T> {
74
+
#[inline]
75
+
pub fn new(val: impl PinInit<T>) -> impl PinInit<Self> {
76
+
pin_init!(CMutex {
77
+
wait_list <- ListHead::new(),
78
+
spin_lock: SpinLock::new(),
79
+
locked: Cell::new(false),
80
+
data <- unsafe {
81
+
pin_init_from_closure(|slot: *mut UnsafeCell<T>| {
82
+
val.__pinned_init(slot.cast::<T>())
83
+
})
84
+
},
85
+
})
86
+
}
87
+
88
+
#[inline]
89
+
pub fn lock(&self) -> Pin<CMutexGuard<'_, T>> {
90
+
let mut sguard = self.spin_lock.acquire();
91
+
if self.locked.get() {
92
+
stack_pin_init!(let wait_entry = WaitEntry::insert_new(&self.wait_list));
93
+
// println!("wait list length: {}", self.wait_list.size());
94
+
while self.locked.get() {
95
+
drop(sguard);
96
+
park();
97
+
sguard = self.spin_lock.acquire();
98
+
}
99
+
// This does have an effect, as the ListHead inside wait_entry implements Drop!
100
+
#[expect(clippy::drop_non_drop)]
101
+
drop(wait_entry);
102
+
}
103
+
self.locked.set(true);
104
+
unsafe {
105
+
Pin::new_unchecked(CMutexGuard {
106
+
mtx: self,
107
+
_pin: PhantomPinned,
108
+
})
109
+
}
110
+
}
111
+
112
+
#[allow(dead_code)]
113
+
pub fn get_data_mut(self: Pin<&mut Self>) -> &mut T {
114
+
// SAFETY: we have an exclusive reference and thus nobody has access to data.
115
+
unsafe { &mut *self.data.get() }
116
+
}
117
+
}
118
+
119
+
unsafe impl<T: Send> Send for CMutex<T> {}
120
+
unsafe impl<T: Send> Sync for CMutex<T> {}
121
+
122
+
pub struct CMutexGuard<'a, T> {
123
+
mtx: &'a CMutex<T>,
124
+
_pin: PhantomPinned,
125
+
}
126
+
127
+
impl<T> Drop for CMutexGuard<'_, T> {
128
+
#[inline]
129
+
fn drop(&mut self) {
130
+
let sguard = self.mtx.spin_lock.acquire();
131
+
self.mtx.locked.set(false);
132
+
if let Some(list_field) = self.mtx.wait_list.next() {
133
+
let wait_entry = list_field.as_ptr().cast::<WaitEntry>();
134
+
unsafe { (*wait_entry).thread.unpark() };
135
+
}
136
+
drop(sguard);
137
+
}
138
+
}
139
+
140
+
impl<T> Deref for CMutexGuard<'_, T> {
141
+
type Target = T;
142
+
143
+
#[inline]
144
+
fn deref(&self) -> &Self::Target {
145
+
unsafe { &*self.mtx.data.get() }
146
+
}
147
+
}
148
+
149
+
impl<T> DerefMut for CMutexGuard<'_, T> {
150
+
#[inline]
151
+
fn deref_mut(&mut self) -> &mut Self::Target {
152
+
unsafe { &mut *self.mtx.data.get() }
153
+
}
154
+
}
155
+
156
+
#[pin_data]
157
+
#[repr(C)]
158
+
struct WaitEntry {
159
+
#[pin]
160
+
wait_list: ListHead,
161
+
thread: Thread,
162
+
}
163
+
164
+
impl WaitEntry {
165
+
#[inline]
166
+
fn insert_new(list: &ListHead) -> impl PinInit<Self> + '_ {
167
+
pin_init!(Self {
168
+
thread: thread::current(),
169
+
wait_list <- ListHead::insert_prev(list),
170
+
})
171
+
}
172
+
}
173
+
174
+
#[cfg(not(any(feature = "std", feature = "alloc")))]
175
+
fn main() {}
176
+
177
+
#[allow(dead_code)]
178
+
#[cfg_attr(test, test)]
179
+
#[cfg(any(feature = "std", feature = "alloc"))]
180
+
fn main() {
181
+
let mtx: Pin<Arc<CMutex<usize>>> = Arc::pin_init(CMutex::new(0)).unwrap();
182
+
let mut handles = vec![];
183
+
let thread_count = 20;
184
+
let workload = if cfg!(miri) { 100 } else { 1_000 };
185
+
for i in 0..thread_count {
186
+
let mtx = mtx.clone();
187
+
handles.push(
188
+
Builder::new()
189
+
.name(format!("worker #{i}"))
190
+
.spawn(move || {
191
+
for _ in 0..workload {
192
+
*mtx.lock() += 1;
193
+
}
194
+
println!("{i} halfway");
195
+
sleep(Duration::from_millis((i as u64) * 10));
196
+
for _ in 0..workload {
197
+
*mtx.lock() += 1;
198
+
}
199
+
println!("{i} finished");
200
+
})
201
+
.expect("should not fail"),
202
+
);
203
+
}
204
+
for h in handles {
205
+
h.join().expect("thread panicked");
206
+
}
207
+
println!("{:?}", &*mtx.lock());
208
+
assert_eq!(*mtx.lock(), workload * thread_count * 2);
209
+
}
+178
rust/pin-init/examples/pthread_mutex.rs
+178
rust/pin-init/examples/pthread_mutex.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
// inspired by https://github.com/nbdd0121/pin-init/blob/trunk/examples/pthread_mutex.rs
4
+
#![allow(clippy::undocumented_unsafe_blocks)]
5
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
6
+
#[cfg(not(windows))]
7
+
mod pthread_mtx {
8
+
#[cfg(feature = "alloc")]
9
+
use core::alloc::AllocError;
10
+
use core::{
11
+
cell::UnsafeCell,
12
+
marker::PhantomPinned,
13
+
mem::MaybeUninit,
14
+
ops::{Deref, DerefMut},
15
+
pin::Pin,
16
+
};
17
+
use pin_init::*;
18
+
use std::convert::Infallible;
19
+
20
+
#[pin_data(PinnedDrop)]
21
+
pub struct PThreadMutex<T> {
22
+
#[pin]
23
+
raw: UnsafeCell<libc::pthread_mutex_t>,
24
+
data: UnsafeCell<T>,
25
+
#[pin]
26
+
pin: PhantomPinned,
27
+
}
28
+
29
+
unsafe impl<T: Send> Send for PThreadMutex<T> {}
30
+
unsafe impl<T: Send> Sync for PThreadMutex<T> {}
31
+
32
+
#[pinned_drop]
33
+
impl<T> PinnedDrop for PThreadMutex<T> {
34
+
fn drop(self: Pin<&mut Self>) {
35
+
unsafe {
36
+
libc::pthread_mutex_destroy(self.raw.get());
37
+
}
38
+
}
39
+
}
40
+
41
+
#[derive(Debug)]
42
+
pub enum Error {
43
+
#[expect(dead_code)]
44
+
IO(std::io::Error),
45
+
Alloc,
46
+
}
47
+
48
+
impl From<Infallible> for Error {
49
+
fn from(e: Infallible) -> Self {
50
+
match e {}
51
+
}
52
+
}
53
+
54
+
#[cfg(feature = "alloc")]
55
+
impl From<AllocError> for Error {
56
+
fn from(_: AllocError) -> Self {
57
+
Self::Alloc
58
+
}
59
+
}
60
+
61
+
impl<T> PThreadMutex<T> {
62
+
pub fn new(data: T) -> impl PinInit<Self, Error> {
63
+
fn init_raw() -> impl PinInit<UnsafeCell<libc::pthread_mutex_t>, Error> {
64
+
let init = |slot: *mut UnsafeCell<libc::pthread_mutex_t>| {
65
+
// we can cast, because `UnsafeCell` has the same layout as T.
66
+
let slot: *mut libc::pthread_mutex_t = slot.cast();
67
+
let mut attr = MaybeUninit::uninit();
68
+
let attr = attr.as_mut_ptr();
69
+
// SAFETY: ptr is valid
70
+
let ret = unsafe { libc::pthread_mutexattr_init(attr) };
71
+
if ret != 0 {
72
+
return Err(Error::IO(std::io::Error::from_raw_os_error(ret)));
73
+
}
74
+
// SAFETY: attr is initialized
75
+
let ret = unsafe {
76
+
libc::pthread_mutexattr_settype(attr, libc::PTHREAD_MUTEX_NORMAL)
77
+
};
78
+
if ret != 0 {
79
+
// SAFETY: attr is initialized
80
+
unsafe { libc::pthread_mutexattr_destroy(attr) };
81
+
return Err(Error::IO(std::io::Error::from_raw_os_error(ret)));
82
+
}
83
+
// SAFETY: slot is valid
84
+
unsafe { slot.write(libc::PTHREAD_MUTEX_INITIALIZER) };
85
+
// SAFETY: attr and slot are valid ptrs and attr is initialized
86
+
let ret = unsafe { libc::pthread_mutex_init(slot, attr) };
87
+
// SAFETY: attr was initialized
88
+
unsafe { libc::pthread_mutexattr_destroy(attr) };
89
+
if ret != 0 {
90
+
return Err(Error::IO(std::io::Error::from_raw_os_error(ret)));
91
+
}
92
+
Ok(())
93
+
};
94
+
// SAFETY: mutex has been initialized
95
+
unsafe { pin_init_from_closure(init) }
96
+
}
97
+
try_pin_init!(Self {
98
+
data: UnsafeCell::new(data),
99
+
raw <- init_raw(),
100
+
pin: PhantomPinned,
101
+
}? Error)
102
+
}
103
+
104
+
pub fn lock(&self) -> PThreadMutexGuard<'_, T> {
105
+
// SAFETY: raw is always initialized
106
+
unsafe { libc::pthread_mutex_lock(self.raw.get()) };
107
+
PThreadMutexGuard { mtx: self }
108
+
}
109
+
}
110
+
111
+
pub struct PThreadMutexGuard<'a, T> {
112
+
mtx: &'a PThreadMutex<T>,
113
+
}
114
+
115
+
impl<T> Drop for PThreadMutexGuard<'_, T> {
116
+
fn drop(&mut self) {
117
+
// SAFETY: raw is always initialized
118
+
unsafe { libc::pthread_mutex_unlock(self.mtx.raw.get()) };
119
+
}
120
+
}
121
+
122
+
impl<T> Deref for PThreadMutexGuard<'_, T> {
123
+
type Target = T;
124
+
125
+
fn deref(&self) -> &Self::Target {
126
+
unsafe { &*self.mtx.data.get() }
127
+
}
128
+
}
129
+
130
+
impl<T> DerefMut for PThreadMutexGuard<'_, T> {
131
+
fn deref_mut(&mut self) -> &mut Self::Target {
132
+
unsafe { &mut *self.mtx.data.get() }
133
+
}
134
+
}
135
+
}
136
+
137
+
#[cfg_attr(test, test)]
138
+
fn main() {
139
+
#[cfg(all(any(feature = "std", feature = "alloc"), not(windows)))]
140
+
{
141
+
use core::pin::Pin;
142
+
use pin_init::*;
143
+
use pthread_mtx::*;
144
+
use std::{
145
+
sync::Arc,
146
+
thread::{sleep, Builder},
147
+
time::Duration,
148
+
};
149
+
let mtx: Pin<Arc<PThreadMutex<usize>>> = Arc::try_pin_init(PThreadMutex::new(0)).unwrap();
150
+
let mut handles = vec![];
151
+
let thread_count = 20;
152
+
let workload = 1_000_000;
153
+
for i in 0..thread_count {
154
+
let mtx = mtx.clone();
155
+
handles.push(
156
+
Builder::new()
157
+
.name(format!("worker #{i}"))
158
+
.spawn(move || {
159
+
for _ in 0..workload {
160
+
*mtx.lock() += 1;
161
+
}
162
+
println!("{i} halfway");
163
+
sleep(Duration::from_millis((i as u64) * 10));
164
+
for _ in 0..workload {
165
+
*mtx.lock() += 1;
166
+
}
167
+
println!("{i} finished");
168
+
})
169
+
.expect("should not fail"),
170
+
);
171
+
}
172
+
for h in handles {
173
+
h.join().expect("thread panicked");
174
+
}
175
+
println!("{:?}", &*mtx.lock());
176
+
assert_eq!(*mtx.lock(), workload * thread_count * 2);
177
+
}
178
+
}
+122
rust/pin-init/examples/static_init.rs
+122
rust/pin-init/examples/static_init.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#![allow(clippy::undocumented_unsafe_blocks)]
4
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
5
+
6
+
use core::{
7
+
cell::{Cell, UnsafeCell},
8
+
mem::MaybeUninit,
9
+
ops,
10
+
pin::Pin,
11
+
time::Duration,
12
+
};
13
+
use pin_init::*;
14
+
use std::{
15
+
sync::Arc,
16
+
thread::{sleep, Builder},
17
+
};
18
+
19
+
#[expect(unused_attributes)]
20
+
mod mutex;
21
+
use mutex::*;
22
+
23
+
pub struct StaticInit<T, I> {
24
+
cell: UnsafeCell<MaybeUninit<T>>,
25
+
init: Cell<Option<I>>,
26
+
lock: SpinLock,
27
+
present: Cell<bool>,
28
+
}
29
+
30
+
unsafe impl<T: Sync, I> Sync for StaticInit<T, I> {}
31
+
unsafe impl<T: Send, I> Send for StaticInit<T, I> {}
32
+
33
+
impl<T, I: PinInit<T>> StaticInit<T, I> {
34
+
pub const fn new(init: I) -> Self {
35
+
Self {
36
+
cell: UnsafeCell::new(MaybeUninit::uninit()),
37
+
init: Cell::new(Some(init)),
38
+
lock: SpinLock::new(),
39
+
present: Cell::new(false),
40
+
}
41
+
}
42
+
}
43
+
44
+
impl<T, I: PinInit<T>> ops::Deref for StaticInit<T, I> {
45
+
type Target = T;
46
+
fn deref(&self) -> &Self::Target {
47
+
if self.present.get() {
48
+
unsafe { (*self.cell.get()).assume_init_ref() }
49
+
} else {
50
+
println!("acquire spinlock on static init");
51
+
let _guard = self.lock.acquire();
52
+
println!("rechecking present...");
53
+
std::thread::sleep(std::time::Duration::from_millis(200));
54
+
if self.present.get() {
55
+
return unsafe { (*self.cell.get()).assume_init_ref() };
56
+
}
57
+
println!("doing init");
58
+
let ptr = self.cell.get().cast::<T>();
59
+
match self.init.take() {
60
+
Some(f) => unsafe { f.__pinned_init(ptr).unwrap() },
61
+
None => unsafe { core::hint::unreachable_unchecked() },
62
+
}
63
+
self.present.set(true);
64
+
unsafe { (*self.cell.get()).assume_init_ref() }
65
+
}
66
+
}
67
+
}
68
+
69
+
pub struct CountInit;
70
+
71
+
unsafe impl PinInit<CMutex<usize>> for CountInit {
72
+
unsafe fn __pinned_init(
73
+
self,
74
+
slot: *mut CMutex<usize>,
75
+
) -> Result<(), core::convert::Infallible> {
76
+
let init = CMutex::new(0);
77
+
std::thread::sleep(std::time::Duration::from_millis(1000));
78
+
unsafe { init.__pinned_init(slot) }
79
+
}
80
+
}
81
+
82
+
pub static COUNT: StaticInit<CMutex<usize>, CountInit> = StaticInit::new(CountInit);
83
+
84
+
#[cfg(not(any(feature = "std", feature = "alloc")))]
85
+
fn main() {}
86
+
87
+
#[cfg(any(feature = "std", feature = "alloc"))]
88
+
fn main() {
89
+
let mtx: Pin<Arc<CMutex<usize>>> = Arc::pin_init(CMutex::new(0)).unwrap();
90
+
let mut handles = vec![];
91
+
let thread_count = 20;
92
+
let workload = 1_000;
93
+
for i in 0..thread_count {
94
+
let mtx = mtx.clone();
95
+
handles.push(
96
+
Builder::new()
97
+
.name(format!("worker #{i}"))
98
+
.spawn(move || {
99
+
for _ in 0..workload {
100
+
*COUNT.lock() += 1;
101
+
std::thread::sleep(std::time::Duration::from_millis(10));
102
+
*mtx.lock() += 1;
103
+
std::thread::sleep(std::time::Duration::from_millis(10));
104
+
*COUNT.lock() += 1;
105
+
}
106
+
println!("{i} halfway");
107
+
sleep(Duration::from_millis((i as u64) * 10));
108
+
for _ in 0..workload {
109
+
std::thread::sleep(std::time::Duration::from_millis(10));
110
+
*mtx.lock() += 1;
111
+
}
112
+
println!("{i} finished");
113
+
})
114
+
.expect("should not fail"),
115
+
);
116
+
}
117
+
for h in handles {
118
+
h.join().expect("thread panicked");
119
+
}
120
+
println!("{:?}, {:?}", &*mtx.lock(), &*COUNT.lock());
121
+
assert_eq!(*mtx.lock(), workload * thread_count * 2);
122
+
}
+152
rust/pin-init/internal/src/helpers.rs
+152
rust/pin-init/internal/src/helpers.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#[cfg(not(kernel))]
4
+
use proc_macro2 as proc_macro;
5
+
6
+
use proc_macro::{TokenStream, TokenTree};
7
+
8
+
/// Parsed generics.
9
+
///
10
+
/// See the field documentation for an explanation what each of the fields represents.
11
+
///
12
+
/// # Examples
13
+
///
14
+
/// ```rust,ignore
15
+
/// # let input = todo!();
16
+
/// let (Generics { decl_generics, impl_generics, ty_generics }, rest) = parse_generics(input);
17
+
/// quote! {
18
+
/// struct Foo<$($decl_generics)*> {
19
+
/// // ...
20
+
/// }
21
+
///
22
+
/// impl<$impl_generics> Foo<$ty_generics> {
23
+
/// fn foo() {
24
+
/// // ...
25
+
/// }
26
+
/// }
27
+
/// }
28
+
/// ```
29
+
pub(crate) struct Generics {
30
+
/// The generics with bounds and default values (e.g. `T: Clone, const N: usize = 0`).
31
+
///
32
+
/// Use this on type definitions e.g. `struct Foo<$decl_generics> ...` (or `union`/`enum`).
33
+
pub(crate) decl_generics: Vec<TokenTree>,
34
+
/// The generics with bounds (e.g. `T: Clone, const N: usize`).
35
+
///
36
+
/// Use this on `impl` blocks e.g. `impl<$impl_generics> Trait for ...`.
37
+
pub(crate) impl_generics: Vec<TokenTree>,
38
+
/// The generics without bounds and without default values (e.g. `T, N`).
39
+
///
40
+
/// Use this when you use the type that is declared with these generics e.g.
41
+
/// `Foo<$ty_generics>`.
42
+
pub(crate) ty_generics: Vec<TokenTree>,
43
+
}
44
+
45
+
/// Parses the given `TokenStream` into `Generics` and the rest.
46
+
///
47
+
/// The generics are not present in the rest, but a where clause might remain.
48
+
pub(crate) fn parse_generics(input: TokenStream) -> (Generics, Vec<TokenTree>) {
49
+
// The generics with bounds and default values.
50
+
let mut decl_generics = vec![];
51
+
// `impl_generics`, the declared generics with their bounds.
52
+
let mut impl_generics = vec![];
53
+
// Only the names of the generics, without any bounds.
54
+
let mut ty_generics = vec![];
55
+
// Tokens not related to the generics e.g. the `where` token and definition.
56
+
let mut rest = vec![];
57
+
// The current level of `<`.
58
+
let mut nesting = 0;
59
+
let mut toks = input.into_iter();
60
+
// If we are at the beginning of a generic parameter.
61
+
let mut at_start = true;
62
+
let mut skip_until_comma = false;
63
+
while let Some(tt) = toks.next() {
64
+
if nesting == 1 && matches!(&tt, TokenTree::Punct(p) if p.as_char() == '>') {
65
+
// Found the end of the generics.
66
+
break;
67
+
} else if nesting >= 1 {
68
+
decl_generics.push(tt.clone());
69
+
}
70
+
match tt.clone() {
71
+
TokenTree::Punct(p) if p.as_char() == '<' => {
72
+
if nesting >= 1 && !skip_until_comma {
73
+
// This is inside of the generics and part of some bound.
74
+
impl_generics.push(tt);
75
+
}
76
+
nesting += 1;
77
+
}
78
+
TokenTree::Punct(p) if p.as_char() == '>' => {
79
+
// This is a parsing error, so we just end it here.
80
+
if nesting == 0 {
81
+
break;
82
+
} else {
83
+
nesting -= 1;
84
+
if nesting >= 1 && !skip_until_comma {
85
+
// We are still inside of the generics and part of some bound.
86
+
impl_generics.push(tt);
87
+
}
88
+
}
89
+
}
90
+
TokenTree::Punct(p) if skip_until_comma && p.as_char() == ',' => {
91
+
if nesting == 1 {
92
+
impl_generics.push(tt.clone());
93
+
impl_generics.push(tt);
94
+
skip_until_comma = false;
95
+
}
96
+
}
97
+
_ if !skip_until_comma => {
98
+
match nesting {
99
+
// If we haven't entered the generics yet, we still want to keep these tokens.
100
+
0 => rest.push(tt),
101
+
1 => {
102
+
// Here depending on the token, it might be a generic variable name.
103
+
match tt.clone() {
104
+
TokenTree::Ident(i) if at_start && i.to_string() == "const" => {
105
+
let Some(name) = toks.next() else {
106
+
// Parsing error.
107
+
break;
108
+
};
109
+
impl_generics.push(tt);
110
+
impl_generics.push(name.clone());
111
+
ty_generics.push(name.clone());
112
+
decl_generics.push(name);
113
+
at_start = false;
114
+
}
115
+
TokenTree::Ident(_) if at_start => {
116
+
impl_generics.push(tt.clone());
117
+
ty_generics.push(tt);
118
+
at_start = false;
119
+
}
120
+
TokenTree::Punct(p) if p.as_char() == ',' => {
121
+
impl_generics.push(tt.clone());
122
+
ty_generics.push(tt);
123
+
at_start = true;
124
+
}
125
+
// Lifetimes begin with `'`.
126
+
TokenTree::Punct(p) if p.as_char() == '\'' && at_start => {
127
+
impl_generics.push(tt.clone());
128
+
ty_generics.push(tt);
129
+
}
130
+
// Generics can have default values, we skip these.
131
+
TokenTree::Punct(p) if p.as_char() == '=' => {
132
+
skip_until_comma = true;
133
+
}
134
+
_ => impl_generics.push(tt),
135
+
}
136
+
}
137
+
_ => impl_generics.push(tt),
138
+
}
139
+
}
140
+
_ => {}
141
+
}
142
+
}
143
+
rest.extend(toks);
144
+
(
145
+
Generics {
146
+
impl_generics,
147
+
decl_generics,
148
+
ty_generics,
149
+
},
150
+
rest,
151
+
)
152
+
}
+48
rust/pin-init/internal/src/lib.rs
+48
rust/pin-init/internal/src/lib.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
// When fixdep scans this, it will find this string `CONFIG_RUSTC_VERSION_TEXT`
4
+
// and thus add a dependency on `include/config/RUSTC_VERSION_TEXT`, which is
5
+
// touched by Kconfig when the version string from the compiler changes.
6
+
7
+
//! `pin-init` proc macros.
8
+
9
+
#![cfg_attr(not(RUSTC_LINT_REASONS_IS_STABLE), feature(lint_reasons))]
10
+
// Allow `.into()` to convert
11
+
// - `proc_macro2::TokenStream` into `proc_macro::TokenStream` in the user-space version.
12
+
// - `proc_macro::TokenStream` into `proc_macro::TokenStream` in the kernel version.
13
+
// Clippy warns on this conversion, but it's required by the user-space version.
14
+
//
15
+
// Remove once we have `proc_macro2` in the kernel.
16
+
#![allow(clippy::useless_conversion)]
17
+
// Documentation is done in the pin-init crate instead.
18
+
#![allow(missing_docs)]
19
+
20
+
use proc_macro::TokenStream;
21
+
22
+
#[cfg(kernel)]
23
+
#[path = "../../../macros/quote.rs"]
24
+
#[macro_use]
25
+
mod quote;
26
+
#[cfg(not(kernel))]
27
+
#[macro_use]
28
+
extern crate quote;
29
+
30
+
mod helpers;
31
+
mod pin_data;
32
+
mod pinned_drop;
33
+
mod zeroable;
34
+
35
+
#[proc_macro_attribute]
36
+
pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
37
+
pin_data::pin_data(inner.into(), item.into()).into()
38
+
}
39
+
40
+
#[proc_macro_attribute]
41
+
pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
42
+
pinned_drop::pinned_drop(args.into(), input.into()).into()
43
+
}
44
+
45
+
#[proc_macro_derive(Zeroable)]
46
+
pub fn derive_zeroable(input: TokenStream) -> TokenStream {
47
+
zeroable::derive(input.into()).into()
48
+
}
+158
rust/pin-init/src/alloc.rs
+158
rust/pin-init/src/alloc.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
#[cfg(all(feature = "alloc", not(feature = "std")))]
4
+
use alloc::{boxed::Box, sync::Arc};
5
+
#[cfg(feature = "alloc")]
6
+
use core::alloc::AllocError;
7
+
use core::{mem::MaybeUninit, pin::Pin};
8
+
#[cfg(feature = "std")]
9
+
use std::sync::Arc;
10
+
11
+
#[cfg(not(feature = "alloc"))]
12
+
type AllocError = core::convert::Infallible;
13
+
14
+
use crate::{
15
+
init_from_closure, pin_init_from_closure, InPlaceWrite, Init, PinInit, ZeroableOption,
16
+
};
17
+
18
+
pub extern crate alloc;
19
+
20
+
// SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee).
21
+
//
22
+
// In this case we are allowed to use `T: ?Sized`, since all zeros is the `None` variant and there
23
+
// is no problem with a VTABLE pointer being null.
24
+
unsafe impl<T: ?Sized> ZeroableOption for Box<T> {}
25
+
26
+
/// Smart pointer that can initialize memory in-place.
27
+
pub trait InPlaceInit<T>: Sized {
28
+
/// Use the given pin-initializer to pin-initialize a `T` inside of a new smart pointer of this
29
+
/// type.
30
+
///
31
+
/// If `T: !Unpin` it will not be able to move afterwards.
32
+
fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
33
+
where
34
+
E: From<AllocError>;
35
+
36
+
/// Use the given pin-initializer to pin-initialize a `T` inside of a new smart pointer of this
37
+
/// type.
38
+
///
39
+
/// If `T: !Unpin` it will not be able to move afterwards.
40
+
fn pin_init(init: impl PinInit<T>) -> Result<Pin<Self>, AllocError> {
41
+
// SAFETY: We delegate to `init` and only change the error type.
42
+
let init = unsafe {
43
+
pin_init_from_closure(|slot| match init.__pinned_init(slot) {
44
+
Ok(()) => Ok(()),
45
+
Err(i) => match i {},
46
+
})
47
+
};
48
+
Self::try_pin_init(init)
49
+
}
50
+
51
+
/// Use the given initializer to in-place initialize a `T`.
52
+
fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
53
+
where
54
+
E: From<AllocError>;
55
+
56
+
/// Use the given initializer to in-place initialize a `T`.
57
+
fn init(init: impl Init<T>) -> Result<Self, AllocError> {
58
+
// SAFETY: We delegate to `init` and only change the error type.
59
+
let init = unsafe {
60
+
init_from_closure(|slot| match init.__init(slot) {
61
+
Ok(()) => Ok(()),
62
+
Err(i) => match i {},
63
+
})
64
+
};
65
+
Self::try_init(init)
66
+
}
67
+
}
68
+
69
+
#[cfg(feature = "alloc")]
70
+
macro_rules! try_new_uninit {
71
+
($type:ident) => {
72
+
$type::try_new_uninit()?
73
+
};
74
+
}
75
+
#[cfg(all(feature = "std", not(feature = "alloc")))]
76
+
macro_rules! try_new_uninit {
77
+
($type:ident) => {
78
+
$type::new_uninit()
79
+
};
80
+
}
81
+
82
+
impl<T> InPlaceInit<T> for Box<T> {
83
+
#[inline]
84
+
fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
85
+
where
86
+
E: From<AllocError>,
87
+
{
88
+
try_new_uninit!(Box).write_pin_init(init)
89
+
}
90
+
91
+
#[inline]
92
+
fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
93
+
where
94
+
E: From<AllocError>,
95
+
{
96
+
try_new_uninit!(Box).write_init(init)
97
+
}
98
+
}
99
+
100
+
impl<T> InPlaceInit<T> for Arc<T> {
101
+
#[inline]
102
+
fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
103
+
where
104
+
E: From<AllocError>,
105
+
{
106
+
let mut this = try_new_uninit!(Arc);
107
+
let Some(slot) = Arc::get_mut(&mut this) else {
108
+
// SAFETY: the Arc has just been created and has no external references
109
+
unsafe { core::hint::unreachable_unchecked() }
110
+
};
111
+
let slot = slot.as_mut_ptr();
112
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
113
+
// slot is valid and will not be moved, because we pin it later.
114
+
unsafe { init.__pinned_init(slot)? };
115
+
// SAFETY: All fields have been initialized and this is the only `Arc` to that data.
116
+
Ok(unsafe { Pin::new_unchecked(this.assume_init()) })
117
+
}
118
+
119
+
#[inline]
120
+
fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
121
+
where
122
+
E: From<AllocError>,
123
+
{
124
+
let mut this = try_new_uninit!(Arc);
125
+
let Some(slot) = Arc::get_mut(&mut this) else {
126
+
// SAFETY: the Arc has just been created and has no external references
127
+
unsafe { core::hint::unreachable_unchecked() }
128
+
};
129
+
let slot = slot.as_mut_ptr();
130
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
131
+
// slot is valid.
132
+
unsafe { init.__init(slot)? };
133
+
// SAFETY: All fields have been initialized.
134
+
Ok(unsafe { this.assume_init() })
135
+
}
136
+
}
137
+
138
+
impl<T> InPlaceWrite<T> for Box<MaybeUninit<T>> {
139
+
type Initialized = Box<T>;
140
+
141
+
fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
142
+
let slot = self.as_mut_ptr();
143
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
144
+
// slot is valid.
145
+
unsafe { init.__init(slot)? };
146
+
// SAFETY: All fields have been initialized.
147
+
Ok(unsafe { self.assume_init() })
148
+
}
149
+
150
+
fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
151
+
let slot = self.as_mut_ptr();
152
+
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
153
+
// slot is valid and will not be moved, because we pin it later.
154
+
unsafe { init.__pinned_init(slot)? };
155
+
// SAFETY: All fields have been initialized.
156
+
Ok(unsafe { self.assume_init() }.into())
157
+
}
158
+
}
+1483
rust/pin-init/src/lib.rs
+1483
rust/pin-init/src/lib.rs
···
1
+
// SPDX-License-Identifier: Apache-2.0 OR MIT
2
+
3
+
//! Library to safely and fallibly initialize pinned `struct`s using in-place constructors.
4
+
//!
5
+
//! [Pinning][pinning] is Rust's way of ensuring data does not move.
6
+
//!
7
+
//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
8
+
//! overflow.
9
+
//!
10
+
//! This library's main use-case is in [Rust-for-Linux]. Although this version can be used
11
+
//! standalone.
12
+
//!
13
+
//! There are cases when you want to in-place initialize a struct. For example when it is very big
14
+
//! and moving it from the stack is not an option, because it is bigger than the stack itself.
15
+
//! Another reason would be that you need the address of the object to initialize it. This stands
16
+
//! in direct conflict with Rust's normal process of first initializing an object and then moving
17
+
//! it into it's final memory location. For more information, see
18
+
//! <https://rust-for-linux.com/the-safe-pinned-initialization-problem>.
19
+
//!
20
+
//! This library allows you to do in-place initialization safely.
21
+
//!
22
+
//! ## Nightly Needed for `alloc` feature
23
+
//!
24
+
//! This library requires the [`allocator_api` unstable feature] when the `alloc` feature is
25
+
//! enabled and thus this feature can only be used with a nightly compiler. When enabling the
26
+
//! `alloc` feature, the user will be required to activate `allocator_api` as well.
27
+
//!
28
+
//! [`allocator_api` unstable feature]: https://doc.rust-lang.org/nightly/unstable-book/library-features/allocator-api.html
29
+
//!
30
+
//! The feature is enabled by default, thus by default `pin-init` will require a nightly compiler.
31
+
//! However, using the crate on stable compilers is possible by disabling `alloc`. In practice this
32
+
//! will require the `std` feature, because stable compilers have neither `Box` nor `Arc` in no-std
33
+
//! mode.
34
+
//!
35
+
//! # Overview
36
+
//!
37
+
//! To initialize a `struct` with an in-place constructor you will need two things:
38
+
//! - an in-place constructor,
39
+
//! - a memory location that can hold your `struct` (this can be the [stack], an [`Arc<T>`],
40
+
//! [`Box<T>`] or any other smart pointer that supports this library).
41
+
//!
42
+
//! To get an in-place constructor there are generally three options:
43
+
//! - directly creating an in-place constructor using the [`pin_init!`] macro,
44
+
//! - a custom function/macro returning an in-place constructor provided by someone else,
45
+
//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
46
+
//!
47
+
//! Aside from pinned initialization, this library also supports in-place construction without
48
+
//! pinning, the macros/types/functions are generally named like the pinned variants without the
49
+
//! `pin_` prefix.
50
+
//!
51
+
//! # Examples
52
+
//!
53
+
//! Throughout the examples we will often make use of the `CMutex` type which can be found in
54
+
//! `../examples/mutex.rs`. It is essentially a userland rebuild of the `struct mutex` type from
55
+
//! the Linux kernel. It also uses a wait list and a basic spinlock. Importantly the wait list
56
+
//! requires it to be pinned to be locked and thus is a prime candidate for using this library.
57
+
//!
58
+
//! ## Using the [`pin_init!`] macro
59
+
//!
60
+
//! If you want to use [`PinInit`], then you will have to annotate your `struct` with
61
+
//! `#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
62
+
//! [structurally pinned fields]. After doing this, you can then create an in-place constructor via
63
+
//! [`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
64
+
//! that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
65
+
//!
66
+
//! ```rust
67
+
//! # #![expect(clippy::disallowed_names)]
68
+
//! # #![feature(allocator_api)]
69
+
//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
70
+
//! # use core::pin::Pin;
71
+
//! use pin_init::{pin_data, pin_init, InPlaceInit};
72
+
//!
73
+
//! #[pin_data]
74
+
//! struct Foo {
75
+
//! #[pin]
76
+
//! a: CMutex<usize>,
77
+
//! b: u32,
78
+
//! }
79
+
//!
80
+
//! let foo = pin_init!(Foo {
81
+
//! a <- CMutex::new(42),
82
+
//! b: 24,
83
+
//! });
84
+
//! # let _ = Box::pin_init(foo);
85
+
//! ```
86
+
//!
87
+
//! `foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
88
+
//! (or just the stack) to actually initialize a `Foo`:
89
+
//!
90
+
//! ```rust
91
+
//! # #![expect(clippy::disallowed_names)]
92
+
//! # #![feature(allocator_api)]
93
+
//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
94
+
//! # use core::{alloc::AllocError, pin::Pin};
95
+
//! # use pin_init::*;
96
+
//! #
97
+
//! # #[pin_data]
98
+
//! # struct Foo {
99
+
//! # #[pin]
100
+
//! # a: CMutex<usize>,
101
+
//! # b: u32,
102
+
//! # }
103
+
//! #
104
+
//! # let foo = pin_init!(Foo {
105
+
//! # a <- CMutex::new(42),
106
+
//! # b: 24,
107
+
//! # });
108
+
//! let foo: Result<Pin<Box<Foo>>, AllocError> = Box::pin_init(foo);
109
+
//! ```
110
+
//!
111
+
//! For more information see the [`pin_init!`] macro.
112
+
//!
113
+
//! ## Using a custom function/macro that returns an initializer
114
+
//!
115
+
//! Many types that use this library supply a function/macro that returns an initializer, because
116
+
//! the above method only works for types where you can access the fields.
117
+
//!
118
+
//! ```rust
119
+
//! # #![feature(allocator_api)]
120
+
//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
121
+
//! # use pin_init::*;
122
+
//! # use std::sync::Arc;
123
+
//! # use core::pin::Pin;
124
+
//! let mtx: Result<Pin<Arc<CMutex<usize>>>, _> = Arc::pin_init(CMutex::new(42));
125
+
//! ```
126
+
//!
127
+
//! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
128
+
//!
129
+
//! ```rust
130
+
//! # #![feature(allocator_api)]
131
+
//! # use pin_init::*;
132
+
//! # #[path = "../examples/error.rs"] mod error; use error::Error;
133
+
//! # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
134
+
//! #[pin_data]
135
+
//! struct DriverData {
136
+
//! #[pin]
137
+
//! status: CMutex<i32>,
138
+
//! buffer: Box<[u8; 1_000_000]>,
139
+
//! }
140
+
//!
141
+
//! impl DriverData {
142
+
//! fn new() -> impl PinInit<Self, Error> {
143
+
//! try_pin_init!(Self {
144
+
//! status <- CMutex::new(0),
145
+
//! buffer: Box::init(pin_init::zeroed())?,
146
+
//! }? Error)
147
+
//! }
148
+
//! }
149
+
//! ```
150
+
//!
151
+
//! ## Manual creation of an initializer
152
+
//!
153
+
//! Often when working with primitives the previous approaches are not sufficient. That is where
154
+
//! [`pin_init_from_closure()`] comes in. This `unsafe` function allows you to create a
155
+
//! [`impl PinInit<T, E>`] directly from a closure. Of course you have to ensure that the closure
156
+
//! actually does the initialization in the correct way. Here are the things to look out for
157
+
//! (we are calling the parameter to the closure `slot`):
158
+
//! - when the closure returns `Ok(())`, then it has completed the initialization successfully, so
159
+
//! `slot` now contains a valid bit pattern for the type `T`,
160
+
//! - when the closure returns `Err(e)`, then the caller may deallocate the memory at `slot`, so
161
+
//! you need to take care to clean up anything if your initialization fails mid-way,
162
+
//! - you may assume that `slot` will stay pinned even after the closure returns until `drop` of
163
+
//! `slot` gets called.
164
+
//!
165
+
//! ```rust
166
+
//! # #![feature(extern_types)]
167
+
//! use pin_init::{pin_data, pinned_drop, PinInit, PinnedDrop, pin_init_from_closure};
168
+
//! use core::{
169
+
//! ptr::addr_of_mut,
170
+
//! marker::PhantomPinned,
171
+
//! cell::UnsafeCell,
172
+
//! pin::Pin,
173
+
//! mem::MaybeUninit,
174
+
//! };
175
+
//! mod bindings {
176
+
//! #[repr(C)]
177
+
//! pub struct foo {
178
+
//! /* fields from C ... */
179
+
//! }
180
+
//! extern "C" {
181
+
//! pub fn init_foo(ptr: *mut foo);
182
+
//! pub fn destroy_foo(ptr: *mut foo);
183
+
//! #[must_use = "you must check the error return code"]
184
+
//! pub fn enable_foo(ptr: *mut foo, flags: u32) -> i32;
185
+
//! }
186
+
//! }
187
+
//!
188
+
//! /// # Invariants
189
+
//! ///
190
+
//! /// `foo` is always initialized
191
+
//! #[pin_data(PinnedDrop)]
192
+
//! pub struct RawFoo {
193
+
//! #[pin]
194
+
//! _p: PhantomPinned,
195
+
//! #[pin]
196
+
//! foo: UnsafeCell<MaybeUninit<bindings::foo>>,
197
+
//! }
198
+
//!
199
+
//! impl RawFoo {
200
+
//! pub fn new(flags: u32) -> impl PinInit<Self, i32> {
201
+
//! // SAFETY:
202
+
//! // - when the closure returns `Ok(())`, then it has successfully initialized and
203
+
//! // enabled `foo`,
204
+
//! // - when it returns `Err(e)`, then it has cleaned up before
205
+
//! unsafe {
206
+
//! pin_init_from_closure(move |slot: *mut Self| {
207
+
//! // `slot` contains uninit memory, avoid creating a reference.
208
+
//! let foo = addr_of_mut!((*slot).foo);
209
+
//! let foo = UnsafeCell::raw_get(foo).cast::<bindings::foo>();
210
+
//!
211
+
//! // Initialize the `foo`
212
+
//! bindings::init_foo(foo);
213
+
//!
214
+
//! // Try to enable it.
215
+
//! let err = bindings::enable_foo(foo, flags);
216
+
//! if err != 0 {
217
+
//! // Enabling has failed, first clean up the foo and then return the error.
218
+
//! bindings::destroy_foo(foo);
219
+
//! Err(err)
220
+
//! } else {
221
+
//! // All fields of `RawFoo` have been initialized, since `_p` is a ZST.
222
+
//! Ok(())
223
+
//! }
224
+
//! })
225
+
//! }
226
+
//! }
227
+
//! }
228
+
//!
229
+
//! #[pinned_drop]
230
+
//! impl PinnedDrop for RawFoo {
231
+
//! fn drop(self: Pin<&mut Self>) {
232
+
//! // SAFETY: Since `foo` is initialized, destroying is safe.
233
+
//! unsafe { bindings::destroy_foo(self.foo.get().cast::<bindings::foo>()) };
234
+
//! }
235
+
//! }
236
+
//! ```
237
+
//!
238
+
//! For more information on how to use [`pin_init_from_closure()`], take a look at the uses inside
239
+
//! the `kernel` crate. The [`sync`] module is a good starting point.
240
+
//!
241
+
//! [`sync`]: https://rust.docs.kernel.org/kernel/sync/index.html
242
+
//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
243
+
//! [structurally pinned fields]:
244
+
//! https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
245
+
//! [stack]: crate::stack_pin_init
246
+
#![cfg_attr(
247
+
kernel,
248
+
doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
249
+
)]
250
+
#![cfg_attr(
251
+
kernel,
252
+
doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
253
+
)]
254
+
#![cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
255
+
#![cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
256
+
//! [`impl PinInit<Foo>`]: crate::PinInit
257
+
//! [`impl PinInit<T, E>`]: crate::PinInit
258
+
//! [`impl Init<T, E>`]: crate::Init
259
+
//! [Rust-for-Linux]: https://rust-for-linux.com/
260
+
261
+
#![cfg_attr(not(RUSTC_LINT_REASONS_IS_STABLE), feature(lint_reasons))]
262
+
#![cfg_attr(
263
+
all(
264
+
any(feature = "alloc", feature = "std"),
265
+
not(RUSTC_NEW_UNINIT_IS_STABLE)
266
+
),
267
+
feature(new_uninit)
268
+
)]
269
+
#![forbid(missing_docs, unsafe_op_in_unsafe_fn)]
270
+
#![cfg_attr(not(feature = "std"), no_std)]
271
+
#![cfg_attr(feature = "alloc", feature(allocator_api))]
272
+
273
+
use core::{
274
+
cell::UnsafeCell,
275
+
convert::Infallible,
276
+
marker::PhantomData,
277
+
mem::MaybeUninit,
278
+
num::*,
279
+
pin::Pin,
280
+
ptr::{self, NonNull},
281
+
};
282
+
283
+
#[doc(hidden)]
284
+
pub mod __internal;
285
+
#[doc(hidden)]
286
+
pub mod macros;
287
+
288
+
#[cfg(any(feature = "std", feature = "alloc"))]
289
+
mod alloc;
290
+
#[cfg(any(feature = "std", feature = "alloc"))]
291
+
pub use alloc::InPlaceInit;
292
+
293
+
/// Used to specify the pinning information of the fields of a struct.
294
+
///
295
+
/// This is somewhat similar in purpose as
296
+
/// [pin-project-lite](https://crates.io/crates/pin-project-lite).
297
+
/// Place this macro on a struct definition and then `#[pin]` in front of the attributes of each
298
+
/// field you want to structurally pin.
299
+
///
300
+
/// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`,
301
+
/// then `#[pin]` directs the type of initializer that is required.
302
+
///
303
+
/// If your `struct` implements `Drop`, then you need to add `PinnedDrop` as arguments to this
304
+
/// macro, and change your `Drop` implementation to `PinnedDrop` annotated with
305
+
/// `#[`[`macro@pinned_drop`]`]`, since dropping pinned values requires extra care.
306
+
///
307
+
/// # Examples
308
+
///
309
+
/// ```
310
+
/// # #![feature(allocator_api)]
311
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
312
+
/// use pin_init::pin_data;
313
+
///
314
+
/// enum Command {
315
+
/// /* ... */
316
+
/// }
317
+
///
318
+
/// #[pin_data]
319
+
/// struct DriverData {
320
+
/// #[pin]
321
+
/// queue: CMutex<Vec<Command>>,
322
+
/// buf: Box<[u8; 1024 * 1024]>,
323
+
/// }
324
+
/// ```
325
+
///
326
+
/// ```
327
+
/// # #![feature(allocator_api)]
328
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
329
+
/// # mod bindings { pub struct info; pub unsafe fn destroy_info(_: *mut info) {} }
330
+
/// use core::pin::Pin;
331
+
/// use pin_init::{pin_data, pinned_drop, PinnedDrop};
332
+
///
333
+
/// enum Command {
334
+
/// /* ... */
335
+
/// }
336
+
///
337
+
/// #[pin_data(PinnedDrop)]
338
+
/// struct DriverData {
339
+
/// #[pin]
340
+
/// queue: CMutex<Vec<Command>>,
341
+
/// buf: Box<[u8; 1024 * 1024]>,
342
+
/// raw_info: *mut bindings::info,
343
+
/// }
344
+
///
345
+
/// #[pinned_drop]
346
+
/// impl PinnedDrop for DriverData {
347
+
/// fn drop(self: Pin<&mut Self>) {
348
+
/// unsafe { bindings::destroy_info(self.raw_info) };
349
+
/// }
350
+
/// }
351
+
/// ```
352
+
pub use ::pin_init_internal::pin_data;
353
+
354
+
/// Used to implement `PinnedDrop` safely.
355
+
///
356
+
/// Only works on structs that are annotated via `#[`[`macro@pin_data`]`]`.
357
+
///
358
+
/// # Examples
359
+
///
360
+
/// ```
361
+
/// # #![feature(allocator_api)]
362
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
363
+
/// # mod bindings { pub struct info; pub unsafe fn destroy_info(_: *mut info) {} }
364
+
/// use core::pin::Pin;
365
+
/// use pin_init::{pin_data, pinned_drop, PinnedDrop};
366
+
///
367
+
/// enum Command {
368
+
/// /* ... */
369
+
/// }
370
+
///
371
+
/// #[pin_data(PinnedDrop)]
372
+
/// struct DriverData {
373
+
/// #[pin]
374
+
/// queue: CMutex<Vec<Command>>,
375
+
/// buf: Box<[u8; 1024 * 1024]>,
376
+
/// raw_info: *mut bindings::info,
377
+
/// }
378
+
///
379
+
/// #[pinned_drop]
380
+
/// impl PinnedDrop for DriverData {
381
+
/// fn drop(self: Pin<&mut Self>) {
382
+
/// unsafe { bindings::destroy_info(self.raw_info) };
383
+
/// }
384
+
/// }
385
+
/// ```
386
+
pub use ::pin_init_internal::pinned_drop;
387
+
388
+
/// Derives the [`Zeroable`] trait for the given struct.
389
+
///
390
+
/// This can only be used for structs where every field implements the [`Zeroable`] trait.
391
+
///
392
+
/// # Examples
393
+
///
394
+
/// ```
395
+
/// use pin_init::Zeroable;
396
+
///
397
+
/// #[derive(Zeroable)]
398
+
/// pub struct DriverData {
399
+
/// id: i64,
400
+
/// buf_ptr: *mut u8,
401
+
/// len: usize,
402
+
/// }
403
+
/// ```
404
+
pub use ::pin_init_internal::Zeroable;
405
+
406
+
/// Initialize and pin a type directly on the stack.
407
+
///
408
+
/// # Examples
409
+
///
410
+
/// ```rust
411
+
/// # #![expect(clippy::disallowed_names)]
412
+
/// # #![feature(allocator_api)]
413
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
414
+
/// # use pin_init::*;
415
+
/// # use core::pin::Pin;
416
+
/// #[pin_data]
417
+
/// struct Foo {
418
+
/// #[pin]
419
+
/// a: CMutex<usize>,
420
+
/// b: Bar,
421
+
/// }
422
+
///
423
+
/// #[pin_data]
424
+
/// struct Bar {
425
+
/// x: u32,
426
+
/// }
427
+
///
428
+
/// stack_pin_init!(let foo = pin_init!(Foo {
429
+
/// a <- CMutex::new(42),
430
+
/// b: Bar {
431
+
/// x: 64,
432
+
/// },
433
+
/// }));
434
+
/// let foo: Pin<&mut Foo> = foo;
435
+
/// println!("a: {}", &*foo.a.lock());
436
+
/// ```
437
+
///
438
+
/// # Syntax
439
+
///
440
+
/// A normal `let` binding with optional type annotation. The expression is expected to implement
441
+
/// [`PinInit`]/[`Init`] with the error type [`Infallible`]. If you want to use a different error
442
+
/// type, then use [`stack_try_pin_init!`].
443
+
#[macro_export]
444
+
macro_rules! stack_pin_init {
445
+
(let $var:ident $(: $t:ty)? = $val:expr) => {
446
+
let val = $val;
447
+
let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
448
+
let mut $var = match $crate::__internal::StackInit::init($var, val) {
449
+
Ok(res) => res,
450
+
Err(x) => {
451
+
let x: ::core::convert::Infallible = x;
452
+
match x {}
453
+
}
454
+
};
455
+
};
456
+
}
457
+
458
+
/// Initialize and pin a type directly on the stack.
459
+
///
460
+
/// # Examples
461
+
///
462
+
/// ```rust
463
+
/// # #![expect(clippy::disallowed_names)]
464
+
/// # #![feature(allocator_api)]
465
+
/// # #[path = "../examples/error.rs"] mod error; use error::Error;
466
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
467
+
/// # use pin_init::*;
468
+
/// #[pin_data]
469
+
/// struct Foo {
470
+
/// #[pin]
471
+
/// a: CMutex<usize>,
472
+
/// b: Box<Bar>,
473
+
/// }
474
+
///
475
+
/// struct Bar {
476
+
/// x: u32,
477
+
/// }
478
+
///
479
+
/// stack_try_pin_init!(let foo: Foo = try_pin_init!(Foo {
480
+
/// a <- CMutex::new(42),
481
+
/// b: Box::try_new(Bar {
482
+
/// x: 64,
483
+
/// })?,
484
+
/// }? Error));
485
+
/// let foo = foo.unwrap();
486
+
/// println!("a: {}", &*foo.a.lock());
487
+
/// ```
488
+
///
489
+
/// ```rust
490
+
/// # #![expect(clippy::disallowed_names)]
491
+
/// # #![feature(allocator_api)]
492
+
/// # #[path = "../examples/error.rs"] mod error; use error::Error;
493
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
494
+
/// # use pin_init::*;
495
+
/// #[pin_data]
496
+
/// struct Foo {
497
+
/// #[pin]
498
+
/// a: CMutex<usize>,
499
+
/// b: Box<Bar>,
500
+
/// }
501
+
///
502
+
/// struct Bar {
503
+
/// x: u32,
504
+
/// }
505
+
///
506
+
/// stack_try_pin_init!(let foo: Foo =? try_pin_init!(Foo {
507
+
/// a <- CMutex::new(42),
508
+
/// b: Box::try_new(Bar {
509
+
/// x: 64,
510
+
/// })?,
511
+
/// }? Error));
512
+
/// println!("a: {}", &*foo.a.lock());
513
+
/// # Ok::<_, Error>(())
514
+
/// ```
515
+
///
516
+
/// # Syntax
517
+
///
518
+
/// A normal `let` binding with optional type annotation. The expression is expected to implement
519
+
/// [`PinInit`]/[`Init`]. This macro assigns a result to the given variable, adding a `?` after the
520
+
/// `=` will propagate this error.
521
+
#[macro_export]
522
+
macro_rules! stack_try_pin_init {
523
+
(let $var:ident $(: $t:ty)? = $val:expr) => {
524
+
let val = $val;
525
+
let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
526
+
let mut $var = $crate::__internal::StackInit::init($var, val);
527
+
};
528
+
(let $var:ident $(: $t:ty)? =? $val:expr) => {
529
+
let val = $val;
530
+
let mut $var = ::core::pin::pin!($crate::__internal::StackInit$(::<$t>)?::uninit());
531
+
let mut $var = $crate::__internal::StackInit::init($var, val)?;
532
+
};
533
+
}
534
+
535
+
/// Construct an in-place, pinned initializer for `struct`s.
536
+
///
537
+
/// This macro defaults the error to [`Infallible`]. If you need a different error, then use
538
+
/// [`try_pin_init!`].
539
+
///
540
+
/// The syntax is almost identical to that of a normal `struct` initializer:
541
+
///
542
+
/// ```rust
543
+
/// # use pin_init::*;
544
+
/// # use core::pin::Pin;
545
+
/// #[pin_data]
546
+
/// struct Foo {
547
+
/// a: usize,
548
+
/// b: Bar,
549
+
/// }
550
+
///
551
+
/// #[pin_data]
552
+
/// struct Bar {
553
+
/// x: u32,
554
+
/// }
555
+
///
556
+
/// # fn demo() -> impl PinInit<Foo> {
557
+
/// let a = 42;
558
+
///
559
+
/// let initializer = pin_init!(Foo {
560
+
/// a,
561
+
/// b: Bar {
562
+
/// x: 64,
563
+
/// },
564
+
/// });
565
+
/// # initializer }
566
+
/// # Box::pin_init(demo()).unwrap();
567
+
/// ```
568
+
///
569
+
/// Arbitrary Rust expressions can be used to set the value of a variable.
570
+
///
571
+
/// The fields are initialized in the order that they appear in the initializer. So it is possible
572
+
/// to read already initialized fields using raw pointers.
573
+
///
574
+
/// IMPORTANT: You are not allowed to create references to fields of the struct inside of the
575
+
/// initializer.
576
+
///
577
+
/// # Init-functions
578
+
///
579
+
/// When working with this library it is often desired to let others construct your types without
580
+
/// giving access to all fields. This is where you would normally write a plain function `new` that
581
+
/// would return a new instance of your type. With this library that is also possible. However,
582
+
/// there are a few extra things to keep in mind.
583
+
///
584
+
/// To create an initializer function, simply declare it like this:
585
+
///
586
+
/// ```rust
587
+
/// # use pin_init::*;
588
+
/// # use core::pin::Pin;
589
+
/// # #[pin_data]
590
+
/// # struct Foo {
591
+
/// # a: usize,
592
+
/// # b: Bar,
593
+
/// # }
594
+
/// # #[pin_data]
595
+
/// # struct Bar {
596
+
/// # x: u32,
597
+
/// # }
598
+
/// impl Foo {
599
+
/// fn new() -> impl PinInit<Self> {
600
+
/// pin_init!(Self {
601
+
/// a: 42,
602
+
/// b: Bar {
603
+
/// x: 64,
604
+
/// },
605
+
/// })
606
+
/// }
607
+
/// }
608
+
/// ```
609
+
///
610
+
/// Users of `Foo` can now create it like this:
611
+
///
612
+
/// ```rust
613
+
/// # #![expect(clippy::disallowed_names)]
614
+
/// # use pin_init::*;
615
+
/// # use core::pin::Pin;
616
+
/// # #[pin_data]
617
+
/// # struct Foo {
618
+
/// # a: usize,
619
+
/// # b: Bar,
620
+
/// # }
621
+
/// # #[pin_data]
622
+
/// # struct Bar {
623
+
/// # x: u32,
624
+
/// # }
625
+
/// # impl Foo {
626
+
/// # fn new() -> impl PinInit<Self> {
627
+
/// # pin_init!(Self {
628
+
/// # a: 42,
629
+
/// # b: Bar {
630
+
/// # x: 64,
631
+
/// # },
632
+
/// # })
633
+
/// # }
634
+
/// # }
635
+
/// let foo = Box::pin_init(Foo::new());
636
+
/// ```
637
+
///
638
+
/// They can also easily embed it into their own `struct`s:
639
+
///
640
+
/// ```rust
641
+
/// # use pin_init::*;
642
+
/// # use core::pin::Pin;
643
+
/// # #[pin_data]
644
+
/// # struct Foo {
645
+
/// # a: usize,
646
+
/// # b: Bar,
647
+
/// # }
648
+
/// # #[pin_data]
649
+
/// # struct Bar {
650
+
/// # x: u32,
651
+
/// # }
652
+
/// # impl Foo {
653
+
/// # fn new() -> impl PinInit<Self> {
654
+
/// # pin_init!(Self {
655
+
/// # a: 42,
656
+
/// # b: Bar {
657
+
/// # x: 64,
658
+
/// # },
659
+
/// # })
660
+
/// # }
661
+
/// # }
662
+
/// #[pin_data]
663
+
/// struct FooContainer {
664
+
/// #[pin]
665
+
/// foo1: Foo,
666
+
/// #[pin]
667
+
/// foo2: Foo,
668
+
/// other: u32,
669
+
/// }
670
+
///
671
+
/// impl FooContainer {
672
+
/// fn new(other: u32) -> impl PinInit<Self> {
673
+
/// pin_init!(Self {
674
+
/// foo1 <- Foo::new(),
675
+
/// foo2 <- Foo::new(),
676
+
/// other,
677
+
/// })
678
+
/// }
679
+
/// }
680
+
/// ```
681
+
///
682
+
/// Here we see that when using `pin_init!` with `PinInit`, one needs to write `<-` instead of `:`.
683
+
/// This signifies that the given field is initialized in-place. As with `struct` initializers, just
684
+
/// writing the field (in this case `other`) without `:` or `<-` means `other: other,`.
685
+
///
686
+
/// # Syntax
687
+
///
688
+
/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
689
+
/// the following modifications is expected:
690
+
/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
691
+
/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
692
+
/// pointer named `this` inside of the initializer.
693
+
/// - Using struct update syntax one can place `..Zeroable::zeroed()` at the very end of the
694
+
/// struct, this initializes every field with 0 and then runs all initializers specified in the
695
+
/// body. This can only be done if [`Zeroable`] is implemented for the struct.
696
+
///
697
+
/// For instance:
698
+
///
699
+
/// ```rust
700
+
/// # use pin_init::*;
701
+
/// # use core::{ptr::addr_of_mut, marker::PhantomPinned};
702
+
/// #[pin_data]
703
+
/// #[derive(Zeroable)]
704
+
/// struct Buf {
705
+
/// // `ptr` points into `buf`.
706
+
/// ptr: *mut u8,
707
+
/// buf: [u8; 64],
708
+
/// #[pin]
709
+
/// pin: PhantomPinned,
710
+
/// }
711
+
///
712
+
/// let init = pin_init!(&this in Buf {
713
+
/// buf: [0; 64],
714
+
/// // SAFETY: TODO.
715
+
/// ptr: unsafe { addr_of_mut!((*this.as_ptr()).buf).cast() },
716
+
/// pin: PhantomPinned,
717
+
/// });
718
+
/// let init = pin_init!(Buf {
719
+
/// buf: [1; 64],
720
+
/// ..Zeroable::zeroed()
721
+
/// });
722
+
/// ```
723
+
///
724
+
/// [`NonNull<Self>`]: core::ptr::NonNull
725
+
// For a detailed example of how this macro works, see the module documentation of the hidden
726
+
// module `macros` inside of `macros.rs`.
727
+
#[macro_export]
728
+
macro_rules! pin_init {
729
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
730
+
$($fields:tt)*
731
+
}) => {
732
+
$crate::try_pin_init!($(&$this in)? $t $(::<$($generics),*>)? {
733
+
$($fields)*
734
+
}? ::core::convert::Infallible)
735
+
};
736
+
}
737
+
738
+
/// Construct an in-place, fallible pinned initializer for `struct`s.
739
+
///
740
+
/// If the initialization can complete without error (or [`Infallible`]), then use [`pin_init!`].
741
+
///
742
+
/// You can use the `?` operator or use `return Err(err)` inside the initializer to stop
743
+
/// initialization and return the error.
744
+
///
745
+
/// IMPORTANT: if you have `unsafe` code inside of the initializer you have to ensure that when
746
+
/// initialization fails, the memory can be safely deallocated without any further modifications.
747
+
///
748
+
/// The syntax is identical to [`pin_init!`] with the following exception: you must append `? $type`
749
+
/// after the `struct` initializer to specify the error type you want to use.
750
+
///
751
+
/// # Examples
752
+
///
753
+
/// ```rust
754
+
/// # #![feature(allocator_api)]
755
+
/// # #[path = "../examples/error.rs"] mod error; use error::Error;
756
+
/// use pin_init::{pin_data, try_pin_init, PinInit, InPlaceInit, zeroed};
757
+
///
758
+
/// #[pin_data]
759
+
/// struct BigBuf {
760
+
/// big: Box<[u8; 1024 * 1024 * 1024]>,
761
+
/// small: [u8; 1024 * 1024],
762
+
/// ptr: *mut u8,
763
+
/// }
764
+
///
765
+
/// impl BigBuf {
766
+
/// fn new() -> impl PinInit<Self, Error> {
767
+
/// try_pin_init!(Self {
768
+
/// big: Box::init(zeroed())?,
769
+
/// small: [0; 1024 * 1024],
770
+
/// ptr: core::ptr::null_mut(),
771
+
/// }? Error)
772
+
/// }
773
+
/// }
774
+
/// # let _ = Box::pin_init(BigBuf::new());
775
+
/// ```
776
+
// For a detailed example of how this macro works, see the module documentation of the hidden
777
+
// module `macros` inside of `macros.rs`.
778
+
#[macro_export]
779
+
macro_rules! try_pin_init {
780
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
781
+
$($fields:tt)*
782
+
}? $err:ty) => {
783
+
$crate::__init_internal!(
784
+
@this($($this)?),
785
+
@typ($t $(::<$($generics),*>)? ),
786
+
@fields($($fields)*),
787
+
@error($err),
788
+
@data(PinData, use_data),
789
+
@has_data(HasPinData, __pin_data),
790
+
@construct_closure(pin_init_from_closure),
791
+
@munch_fields($($fields)*),
792
+
)
793
+
}
794
+
}
795
+
796
+
/// Construct an in-place initializer for `struct`s.
797
+
///
798
+
/// This macro defaults the error to [`Infallible`]. If you need a different error, then use
799
+
/// [`try_init!`].
800
+
///
801
+
/// The syntax is identical to [`pin_init!`] and its safety caveats also apply:
802
+
/// - `unsafe` code must guarantee either full initialization or return an error and allow
803
+
/// deallocation of the memory.
804
+
/// - the fields are initialized in the order given in the initializer.
805
+
/// - no references to fields are allowed to be created inside of the initializer.
806
+
///
807
+
/// This initializer is for initializing data in-place that might later be moved. If you want to
808
+
/// pin-initialize, use [`pin_init!`].
809
+
///
810
+
/// # Examples
811
+
///
812
+
/// ```rust
813
+
/// # #![feature(allocator_api)]
814
+
/// # #[path = "../examples/error.rs"] mod error; use error::Error;
815
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
816
+
/// # use pin_init::InPlaceInit;
817
+
/// use pin_init::{init, Init, zeroed};
818
+
///
819
+
/// struct BigBuf {
820
+
/// small: [u8; 1024 * 1024],
821
+
/// }
822
+
///
823
+
/// impl BigBuf {
824
+
/// fn new() -> impl Init<Self> {
825
+
/// init!(Self {
826
+
/// small <- zeroed(),
827
+
/// })
828
+
/// }
829
+
/// }
830
+
/// # let _ = Box::init(BigBuf::new());
831
+
/// ```
832
+
// For a detailed example of how this macro works, see the module documentation of the hidden
833
+
// module `macros` inside of `macros.rs`.
834
+
#[macro_export]
835
+
macro_rules! init {
836
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
837
+
$($fields:tt)*
838
+
}) => {
839
+
$crate::try_init!($(&$this in)? $t $(::<$($generics),*>)? {
840
+
$($fields)*
841
+
}? ::core::convert::Infallible)
842
+
}
843
+
}
844
+
845
+
/// Construct an in-place fallible initializer for `struct`s.
846
+
///
847
+
/// If the initialization can complete without error (or [`Infallible`]), then use
848
+
/// [`init!`].
849
+
///
850
+
/// The syntax is identical to [`try_pin_init!`]. You need to specify a custom error
851
+
/// via `? $type` after the `struct` initializer.
852
+
/// The safety caveats from [`try_pin_init!`] also apply:
853
+
/// - `unsafe` code must guarantee either full initialization or return an error and allow
854
+
/// deallocation of the memory.
855
+
/// - the fields are initialized in the order given in the initializer.
856
+
/// - no references to fields are allowed to be created inside of the initializer.
857
+
///
858
+
/// # Examples
859
+
///
860
+
/// ```rust
861
+
/// # #![feature(allocator_api)]
862
+
/// # use core::alloc::AllocError;
863
+
/// # use pin_init::InPlaceInit;
864
+
/// use pin_init::{try_init, Init, zeroed};
865
+
///
866
+
/// struct BigBuf {
867
+
/// big: Box<[u8; 1024 * 1024 * 1024]>,
868
+
/// small: [u8; 1024 * 1024],
869
+
/// }
870
+
///
871
+
/// impl BigBuf {
872
+
/// fn new() -> impl Init<Self, AllocError> {
873
+
/// try_init!(Self {
874
+
/// big: Box::init(zeroed())?,
875
+
/// small: [0; 1024 * 1024],
876
+
/// }? AllocError)
877
+
/// }
878
+
/// }
879
+
/// # let _ = Box::init(BigBuf::new());
880
+
/// ```
881
+
// For a detailed example of how this macro works, see the module documentation of the hidden
882
+
// module `macros` inside of `macros.rs`.
883
+
#[macro_export]
884
+
macro_rules! try_init {
885
+
($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
886
+
$($fields:tt)*
887
+
}? $err:ty) => {
888
+
$crate::__init_internal!(
889
+
@this($($this)?),
890
+
@typ($t $(::<$($generics),*>)?),
891
+
@fields($($fields)*),
892
+
@error($err),
893
+
@data(InitData, /*no use_data*/),
894
+
@has_data(HasInitData, __init_data),
895
+
@construct_closure(init_from_closure),
896
+
@munch_fields($($fields)*),
897
+
)
898
+
}
899
+
}
900
+
901
+
/// Asserts that a field on a struct using `#[pin_data]` is marked with `#[pin]` ie. that it is
902
+
/// structurally pinned.
903
+
///
904
+
/// # Example
905
+
///
906
+
/// This will succeed:
907
+
/// ```
908
+
/// use pin_init::{pin_data, assert_pinned};
909
+
///
910
+
/// #[pin_data]
911
+
/// struct MyStruct {
912
+
/// #[pin]
913
+
/// some_field: u64,
914
+
/// }
915
+
///
916
+
/// assert_pinned!(MyStruct, some_field, u64);
917
+
/// ```
918
+
///
919
+
/// This will fail:
920
+
/// ```compile_fail
921
+
/// use pin_init::{pin_data, assert_pinned};
922
+
///
923
+
/// #[pin_data]
924
+
/// struct MyStruct {
925
+
/// some_field: u64,
926
+
/// }
927
+
///
928
+
/// assert_pinned!(MyStruct, some_field, u64);
929
+
/// ```
930
+
///
931
+
/// Some uses of the macro may trigger the `can't use generic parameters from outer item` error. To
932
+
/// work around this, you may pass the `inline` parameter to the macro. The `inline` parameter can
933
+
/// only be used when the macro is invoked from a function body.
934
+
/// ```
935
+
/// # use core::pin::Pin;
936
+
/// use pin_init::{pin_data, assert_pinned};
937
+
///
938
+
/// #[pin_data]
939
+
/// struct Foo<T> {
940
+
/// #[pin]
941
+
/// elem: T,
942
+
/// }
943
+
///
944
+
/// impl<T> Foo<T> {
945
+
/// fn project(self: Pin<&mut Self>) -> Pin<&mut T> {
946
+
/// assert_pinned!(Foo<T>, elem, T, inline);
947
+
///
948
+
/// // SAFETY: The field is structurally pinned.
949
+
/// unsafe { self.map_unchecked_mut(|me| &mut me.elem) }
950
+
/// }
951
+
/// }
952
+
/// ```
953
+
#[macro_export]
954
+
macro_rules! assert_pinned {
955
+
($ty:ty, $field:ident, $field_ty:ty, inline) => {
956
+
let _ = move |ptr: *mut $field_ty| {
957
+
// SAFETY: This code is unreachable.
958
+
let data = unsafe { <$ty as $crate::__internal::HasPinData>::__pin_data() };
959
+
let init = $crate::__internal::AlwaysFail::<$field_ty>::new();
960
+
// SAFETY: This code is unreachable.
961
+
unsafe { data.$field(ptr, init) }.ok();
962
+
};
963
+
};
964
+
965
+
($ty:ty, $field:ident, $field_ty:ty) => {
966
+
const _: () = {
967
+
$crate::assert_pinned!($ty, $field, $field_ty, inline);
968
+
};
969
+
};
970
+
}
971
+
972
+
/// A pin-initializer for the type `T`.
973
+
///
974
+
/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
975
+
/// be [`Box<T>`], [`Arc<T>`] or even the stack (see [`stack_pin_init!`]).
976
+
///
977
+
/// Also see the [module description](self).
978
+
///
979
+
/// # Safety
980
+
///
981
+
/// When implementing this trait you will need to take great care. Also there are probably very few
982
+
/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
983
+
///
984
+
/// The [`PinInit::__pinned_init`] function:
985
+
/// - returns `Ok(())` if it initialized every field of `slot`,
986
+
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
987
+
/// - `slot` can be deallocated without UB occurring,
988
+
/// - `slot` does not need to be dropped,
989
+
/// - `slot` is not partially initialized.
990
+
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
991
+
///
992
+
#[cfg_attr(
993
+
kernel,
994
+
doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
995
+
)]
996
+
#[cfg_attr(
997
+
kernel,
998
+
doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
999
+
)]
1000
+
#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
1001
+
#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
1002
+
#[must_use = "An initializer must be used in order to create its value."]
1003
+
pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
1004
+
/// Initializes `slot`.
1005
+
///
1006
+
/// # Safety
1007
+
///
1008
+
/// - `slot` is a valid pointer to uninitialized memory.
1009
+
/// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
1010
+
/// deallocate.
1011
+
/// - `slot` will not move until it is dropped, i.e. it will be pinned.
1012
+
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E>;
1013
+
1014
+
/// First initializes the value using `self` then calls the function `f` with the initialized
1015
+
/// value.
1016
+
///
1017
+
/// If `f` returns an error the value is dropped and the initializer will forward the error.
1018
+
///
1019
+
/// # Examples
1020
+
///
1021
+
/// ```rust
1022
+
/// # #![feature(allocator_api)]
1023
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
1024
+
/// # use pin_init::*;
1025
+
/// let mtx_init = CMutex::new(42);
1026
+
/// // Make the initializer print the value.
1027
+
/// let mtx_init = mtx_init.pin_chain(|mtx| {
1028
+
/// println!("{:?}", mtx.get_data_mut());
1029
+
/// Ok(())
1030
+
/// });
1031
+
/// ```
1032
+
fn pin_chain<F>(self, f: F) -> ChainPinInit<Self, F, T, E>
1033
+
where
1034
+
F: FnOnce(Pin<&mut T>) -> Result<(), E>,
1035
+
{
1036
+
ChainPinInit(self, f, PhantomData)
1037
+
}
1038
+
}
1039
+
1040
+
/// An initializer returned by [`PinInit::pin_chain`].
1041
+
pub struct ChainPinInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, T)>);
1042
+
1043
+
// SAFETY: The `__pinned_init` function is implemented such that it
1044
+
// - returns `Ok(())` on successful initialization,
1045
+
// - returns `Err(err)` on error and in this case `slot` will be dropped.
1046
+
// - considers `slot` pinned.
1047
+
unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainPinInit<I, F, T, E>
1048
+
where
1049
+
I: PinInit<T, E>,
1050
+
F: FnOnce(Pin<&mut T>) -> Result<(), E>,
1051
+
{
1052
+
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
1053
+
// SAFETY: All requirements fulfilled since this function is `__pinned_init`.
1054
+
unsafe { self.0.__pinned_init(slot)? };
1055
+
// SAFETY: The above call initialized `slot` and we still have unique access.
1056
+
let val = unsafe { &mut *slot };
1057
+
// SAFETY: `slot` is considered pinned.
1058
+
let val = unsafe { Pin::new_unchecked(val) };
1059
+
// SAFETY: `slot` was initialized above.
1060
+
(self.1)(val).inspect_err(|_| unsafe { core::ptr::drop_in_place(slot) })
1061
+
}
1062
+
}
1063
+
1064
+
/// An initializer for `T`.
1065
+
///
1066
+
/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
1067
+
/// be [`Box<T>`], [`Arc<T>`] or even the stack (see [`stack_pin_init!`]). Because
1068
+
/// [`PinInit<T, E>`] is a super trait, you can use every function that takes it as well.
1069
+
///
1070
+
/// Also see the [module description](self).
1071
+
///
1072
+
/// # Safety
1073
+
///
1074
+
/// When implementing this trait you will need to take great care. Also there are probably very few
1075
+
/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
1076
+
///
1077
+
/// The [`Init::__init`] function:
1078
+
/// - returns `Ok(())` if it initialized every field of `slot`,
1079
+
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1080
+
/// - `slot` can be deallocated without UB occurring,
1081
+
/// - `slot` does not need to be dropped,
1082
+
/// - `slot` is not partially initialized.
1083
+
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1084
+
///
1085
+
/// The `__pinned_init` function from the supertrait [`PinInit`] needs to execute the exact same
1086
+
/// code as `__init`.
1087
+
///
1088
+
/// Contrary to its supertype [`PinInit<T, E>`] the caller is allowed to
1089
+
/// move the pointee after initialization.
1090
+
///
1091
+
#[cfg_attr(
1092
+
kernel,
1093
+
doc = "[`Arc<T>`]: https://rust.docs.kernel.org/kernel/sync/struct.Arc.html"
1094
+
)]
1095
+
#[cfg_attr(
1096
+
kernel,
1097
+
doc = "[`Box<T>`]: https://rust.docs.kernel.org/kernel/alloc/kbox/struct.Box.html"
1098
+
)]
1099
+
#[cfg_attr(not(kernel), doc = "[`Arc<T>`]: alloc::alloc::sync::Arc")]
1100
+
#[cfg_attr(not(kernel), doc = "[`Box<T>`]: alloc::alloc::boxed::Box")]
1101
+
#[must_use = "An initializer must be used in order to create its value."]
1102
+
pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
1103
+
/// Initializes `slot`.
1104
+
///
1105
+
/// # Safety
1106
+
///
1107
+
/// - `slot` is a valid pointer to uninitialized memory.
1108
+
/// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
1109
+
/// deallocate.
1110
+
unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
1111
+
1112
+
/// First initializes the value using `self` then calls the function `f` with the initialized
1113
+
/// value.
1114
+
///
1115
+
/// If `f` returns an error the value is dropped and the initializer will forward the error.
1116
+
///
1117
+
/// # Examples
1118
+
///
1119
+
/// ```rust
1120
+
/// # #![expect(clippy::disallowed_names)]
1121
+
/// use pin_init::{init, zeroed, Init};
1122
+
///
1123
+
/// struct Foo {
1124
+
/// buf: [u8; 1_000_000],
1125
+
/// }
1126
+
///
1127
+
/// impl Foo {
1128
+
/// fn setup(&mut self) {
1129
+
/// println!("Setting up foo");
1130
+
/// }
1131
+
/// }
1132
+
///
1133
+
/// let foo = init!(Foo {
1134
+
/// buf <- zeroed()
1135
+
/// }).chain(|foo| {
1136
+
/// foo.setup();
1137
+
/// Ok(())
1138
+
/// });
1139
+
/// ```
1140
+
fn chain<F>(self, f: F) -> ChainInit<Self, F, T, E>
1141
+
where
1142
+
F: FnOnce(&mut T) -> Result<(), E>,
1143
+
{
1144
+
ChainInit(self, f, PhantomData)
1145
+
}
1146
+
}
1147
+
1148
+
/// An initializer returned by [`Init::chain`].
1149
+
pub struct ChainInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, T)>);
1150
+
1151
+
// SAFETY: The `__init` function is implemented such that it
1152
+
// - returns `Ok(())` on successful initialization,
1153
+
// - returns `Err(err)` on error and in this case `slot` will be dropped.
1154
+
unsafe impl<T: ?Sized, E, I, F> Init<T, E> for ChainInit<I, F, T, E>
1155
+
where
1156
+
I: Init<T, E>,
1157
+
F: FnOnce(&mut T) -> Result<(), E>,
1158
+
{
1159
+
unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1160
+
// SAFETY: All requirements fulfilled since this function is `__init`.
1161
+
unsafe { self.0.__pinned_init(slot)? };
1162
+
// SAFETY: The above call initialized `slot` and we still have unique access.
1163
+
(self.1)(unsafe { &mut *slot }).inspect_err(|_|
1164
+
// SAFETY: `slot` was initialized above.
1165
+
unsafe { core::ptr::drop_in_place(slot) })
1166
+
}
1167
+
}
1168
+
1169
+
// SAFETY: `__pinned_init` behaves exactly the same as `__init`.
1170
+
unsafe impl<T: ?Sized, E, I, F> PinInit<T, E> for ChainInit<I, F, T, E>
1171
+
where
1172
+
I: Init<T, E>,
1173
+
F: FnOnce(&mut T) -> Result<(), E>,
1174
+
{
1175
+
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
1176
+
// SAFETY: `__init` has less strict requirements compared to `__pinned_init`.
1177
+
unsafe { self.__init(slot) }
1178
+
}
1179
+
}
1180
+
1181
+
/// Creates a new [`PinInit<T, E>`] from the given closure.
1182
+
///
1183
+
/// # Safety
1184
+
///
1185
+
/// The closure:
1186
+
/// - returns `Ok(())` if it initialized every field of `slot`,
1187
+
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1188
+
/// - `slot` can be deallocated without UB occurring,
1189
+
/// - `slot` does not need to be dropped,
1190
+
/// - `slot` is not partially initialized.
1191
+
/// - may assume that the `slot` does not move if `T: !Unpin`,
1192
+
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1193
+
#[inline]
1194
+
pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
1195
+
f: impl FnOnce(*mut T) -> Result<(), E>,
1196
+
) -> impl PinInit<T, E> {
1197
+
__internal::InitClosure(f, PhantomData)
1198
+
}
1199
+
1200
+
/// Creates a new [`Init<T, E>`] from the given closure.
1201
+
///
1202
+
/// # Safety
1203
+
///
1204
+
/// The closure:
1205
+
/// - returns `Ok(())` if it initialized every field of `slot`,
1206
+
/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
1207
+
/// - `slot` can be deallocated without UB occurring,
1208
+
/// - `slot` does not need to be dropped,
1209
+
/// - `slot` is not partially initialized.
1210
+
/// - the `slot` may move after initialization.
1211
+
/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
1212
+
#[inline]
1213
+
pub const unsafe fn init_from_closure<T: ?Sized, E>(
1214
+
f: impl FnOnce(*mut T) -> Result<(), E>,
1215
+
) -> impl Init<T, E> {
1216
+
__internal::InitClosure(f, PhantomData)
1217
+
}
1218
+
1219
+
/// An initializer that leaves the memory uninitialized.
1220
+
///
1221
+
/// The initializer is a no-op. The `slot` memory is not changed.
1222
+
#[inline]
1223
+
pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
1224
+
// SAFETY: The memory is allowed to be uninitialized.
1225
+
unsafe { init_from_closure(|_| Ok(())) }
1226
+
}
1227
+
1228
+
/// Initializes an array by initializing each element via the provided initializer.
1229
+
///
1230
+
/// # Examples
1231
+
///
1232
+
/// ```rust
1233
+
/// # use pin_init::*;
1234
+
/// use pin_init::init_array_from_fn;
1235
+
/// let array: Box<[usize; 1_000]> = Box::init(init_array_from_fn(|i| i)).unwrap();
1236
+
/// assert_eq!(array.len(), 1_000);
1237
+
/// ```
1238
+
pub fn init_array_from_fn<I, const N: usize, T, E>(
1239
+
mut make_init: impl FnMut(usize) -> I,
1240
+
) -> impl Init<[T; N], E>
1241
+
where
1242
+
I: Init<T, E>,
1243
+
{
1244
+
let init = move |slot: *mut [T; N]| {
1245
+
let slot = slot.cast::<T>();
1246
+
for i in 0..N {
1247
+
let init = make_init(i);
1248
+
// SAFETY: Since 0 <= `i` < N, it is still in bounds of `[T; N]`.
1249
+
let ptr = unsafe { slot.add(i) };
1250
+
// SAFETY: The pointer is derived from `slot` and thus satisfies the `__init`
1251
+
// requirements.
1252
+
if let Err(e) = unsafe { init.__init(ptr) } {
1253
+
// SAFETY: The loop has initialized the elements `slot[0..i]` and since we return
1254
+
// `Err` below, `slot` will be considered uninitialized memory.
1255
+
unsafe { ptr::drop_in_place(ptr::slice_from_raw_parts_mut(slot, i)) };
1256
+
return Err(e);
1257
+
}
1258
+
}
1259
+
Ok(())
1260
+
};
1261
+
// SAFETY: The initializer above initializes every element of the array. On failure it drops
1262
+
// any initialized elements and returns `Err`.
1263
+
unsafe { init_from_closure(init) }
1264
+
}
1265
+
1266
+
/// Initializes an array by initializing each element via the provided initializer.
1267
+
///
1268
+
/// # Examples
1269
+
///
1270
+
/// ```rust
1271
+
/// # #![feature(allocator_api)]
1272
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
1273
+
/// # use pin_init::*;
1274
+
/// # use core::pin::Pin;
1275
+
/// use pin_init::pin_init_array_from_fn;
1276
+
/// use std::sync::Arc;
1277
+
/// let array: Pin<Arc<[CMutex<usize>; 1_000]>> =
1278
+
/// Arc::pin_init(pin_init_array_from_fn(|i| CMutex::new(i))).unwrap();
1279
+
/// assert_eq!(array.len(), 1_000);
1280
+
/// ```
1281
+
pub fn pin_init_array_from_fn<I, const N: usize, T, E>(
1282
+
mut make_init: impl FnMut(usize) -> I,
1283
+
) -> impl PinInit<[T; N], E>
1284
+
where
1285
+
I: PinInit<T, E>,
1286
+
{
1287
+
let init = move |slot: *mut [T; N]| {
1288
+
let slot = slot.cast::<T>();
1289
+
for i in 0..N {
1290
+
let init = make_init(i);
1291
+
// SAFETY: Since 0 <= `i` < N, it is still in bounds of `[T; N]`.
1292
+
let ptr = unsafe { slot.add(i) };
1293
+
// SAFETY: The pointer is derived from `slot` and thus satisfies the `__init`
1294
+
// requirements.
1295
+
if let Err(e) = unsafe { init.__pinned_init(ptr) } {
1296
+
// SAFETY: The loop has initialized the elements `slot[0..i]` and since we return
1297
+
// `Err` below, `slot` will be considered uninitialized memory.
1298
+
unsafe { ptr::drop_in_place(ptr::slice_from_raw_parts_mut(slot, i)) };
1299
+
return Err(e);
1300
+
}
1301
+
}
1302
+
Ok(())
1303
+
};
1304
+
// SAFETY: The initializer above initializes every element of the array. On failure it drops
1305
+
// any initialized elements and returns `Err`.
1306
+
unsafe { pin_init_from_closure(init) }
1307
+
}
1308
+
1309
+
// SAFETY: Every type can be initialized by-value.
1310
+
unsafe impl<T, E> Init<T, E> for T {
1311
+
unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
1312
+
// SAFETY: TODO.
1313
+
unsafe { slot.write(self) };
1314
+
Ok(())
1315
+
}
1316
+
}
1317
+
1318
+
// SAFETY: Every type can be initialized by-value. `__pinned_init` calls `__init`.
1319
+
unsafe impl<T, E> PinInit<T, E> for T {
1320
+
unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
1321
+
// SAFETY: TODO.
1322
+
unsafe { self.__init(slot) }
1323
+
}
1324
+
}
1325
+
1326
+
/// Smart pointer containing uninitialized memory and that can write a value.
1327
+
pub trait InPlaceWrite<T> {
1328
+
/// The type `Self` turns into when the contents are initialized.
1329
+
type Initialized;
1330
+
1331
+
/// Use the given initializer to write a value into `self`.
1332
+
///
1333
+
/// Does not drop the current value and considers it as uninitialized memory.
1334
+
fn write_init<E>(self, init: impl Init<T, E>) -> Result<Self::Initialized, E>;
1335
+
1336
+
/// Use the given pin-initializer to write a value into `self`.
1337
+
///
1338
+
/// Does not drop the current value and considers it as uninitialized memory.
1339
+
fn write_pin_init<E>(self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E>;
1340
+
}
1341
+
1342
+
/// Trait facilitating pinned destruction.
1343
+
///
1344
+
/// Use [`pinned_drop`] to implement this trait safely:
1345
+
///
1346
+
/// ```rust
1347
+
/// # #![feature(allocator_api)]
1348
+
/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*;
1349
+
/// # use pin_init::*;
1350
+
/// use core::pin::Pin;
1351
+
/// #[pin_data(PinnedDrop)]
1352
+
/// struct Foo {
1353
+
/// #[pin]
1354
+
/// mtx: CMutex<usize>,
1355
+
/// }
1356
+
///
1357
+
/// #[pinned_drop]
1358
+
/// impl PinnedDrop for Foo {
1359
+
/// fn drop(self: Pin<&mut Self>) {
1360
+
/// println!("Foo is being dropped!");
1361
+
/// }
1362
+
/// }
1363
+
/// ```
1364
+
///
1365
+
/// # Safety
1366
+
///
1367
+
/// This trait must be implemented via the [`pinned_drop`] proc-macro attribute on the impl.
1368
+
pub unsafe trait PinnedDrop: __internal::HasPinData {
1369
+
/// Executes the pinned destructor of this type.
1370
+
///
1371
+
/// While this function is marked safe, it is actually unsafe to call it manually. For this
1372
+
/// reason it takes an additional parameter. This type can only be constructed by `unsafe` code
1373
+
/// and thus prevents this function from being called where it should not.
1374
+
///
1375
+
/// This extra parameter will be generated by the `#[pinned_drop]` proc-macro attribute
1376
+
/// automatically.
1377
+
fn drop(self: Pin<&mut Self>, only_call_from_drop: __internal::OnlyCallFromDrop);
1378
+
}
1379
+
1380
+
/// Marker trait for types that can be initialized by writing just zeroes.
1381
+
///
1382
+
/// # Safety
1383
+
///
1384
+
/// The bit pattern consisting of only zeroes is a valid bit pattern for this type. In other words,
1385
+
/// this is not UB:
1386
+
///
1387
+
/// ```rust,ignore
1388
+
/// let val: Self = unsafe { core::mem::zeroed() };
1389
+
/// ```
1390
+
pub unsafe trait Zeroable {}
1391
+
1392
+
/// Marker trait for types that allow `Option<Self>` to be set to all zeroes in order to write
1393
+
/// `None` to that location.
1394
+
///
1395
+
/// # Safety
1396
+
///
1397
+
/// The implementer needs to ensure that `unsafe impl Zeroable for Option<Self> {}` is sound.
1398
+
pub unsafe trait ZeroableOption {}
1399
+
1400
+
// SAFETY: by the safety requirement of `ZeroableOption`, this is valid.
1401
+
unsafe impl<T: ZeroableOption> Zeroable for Option<T> {}
1402
+
1403
+
/// Create a new zeroed T.
1404
+
///
1405
+
/// The returned initializer will write `0x00` to every byte of the given `slot`.
1406
+
#[inline]
1407
+
pub fn zeroed<T: Zeroable>() -> impl Init<T> {
1408
+
// SAFETY: Because `T: Zeroable`, all bytes zero is a valid bit pattern for `T`
1409
+
// and because we write all zeroes, the memory is initialized.
1410
+
unsafe {
1411
+
init_from_closure(|slot: *mut T| {
1412
+
slot.write_bytes(0, 1);
1413
+
Ok(())
1414
+
})
1415
+
}
1416
+
}
1417
+
1418
+
macro_rules! impl_zeroable {
1419
+
($($({$($generics:tt)*})? $t:ty, )*) => {
1420
+
// SAFETY: Safety comments written in the macro invocation.
1421
+
$(unsafe impl$($($generics)*)? Zeroable for $t {})*
1422
+
};
1423
+
}
1424
+
1425
+
impl_zeroable! {
1426
+
// SAFETY: All primitives that are allowed to be zero.
1427
+
bool,
1428
+
char,
1429
+
u8, u16, u32, u64, u128, usize,
1430
+
i8, i16, i32, i64, i128, isize,
1431
+
f32, f64,
1432
+
1433
+
// Note: do not add uninhabited types (such as `!` or `core::convert::Infallible`) to this list;
1434
+
// creating an instance of an uninhabited type is immediate undefined behavior. For more on
1435
+
// uninhabited/empty types, consult The Rustonomicon:
1436
+
// <https://doc.rust-lang.org/stable/nomicon/exotic-sizes.html#empty-types>. The Rust Reference
1437
+
// also has information on undefined behavior:
1438
+
// <https://doc.rust-lang.org/stable/reference/behavior-considered-undefined.html>.
1439
+
//
1440
+
// SAFETY: These are inhabited ZSTs; there is nothing to zero and a valid value exists.
1441
+
{<T: ?Sized>} PhantomData<T>, core::marker::PhantomPinned, (),
1442
+
1443
+
// SAFETY: Type is allowed to take any value, including all zeros.
1444
+
{<T>} MaybeUninit<T>,
1445
+
1446
+
// SAFETY: `T: Zeroable` and `UnsafeCell` is `repr(transparent)`.
1447
+
{<T: ?Sized + Zeroable>} UnsafeCell<T>,
1448
+
1449
+
// SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee:
1450
+
// https://doc.rust-lang.org/stable/std/option/index.html#representation).
1451
+
Option<NonZeroU8>, Option<NonZeroU16>, Option<NonZeroU32>, Option<NonZeroU64>,
1452
+
Option<NonZeroU128>, Option<NonZeroUsize>,
1453
+
Option<NonZeroI8>, Option<NonZeroI16>, Option<NonZeroI32>, Option<NonZeroI64>,
1454
+
Option<NonZeroI128>, Option<NonZeroIsize>,
1455
+
{<T>} Option<NonNull<T>>,
1456
+
1457
+
// SAFETY: `null` pointer is valid.
1458
+
//
1459
+
// We cannot use `T: ?Sized`, since the VTABLE pointer part of fat pointers is not allowed to be
1460
+
// null.
1461
+
//
1462
+
// When `Pointee` gets stabilized, we could use
1463
+
// `T: ?Sized where <T as Pointee>::Metadata: Zeroable`
1464
+
{<T>} *mut T, {<T>} *const T,
1465
+
1466
+
// SAFETY: `null` pointer is valid and the metadata part of these fat pointers is allowed to be
1467
+
// zero.
1468
+
{<T>} *mut [T], {<T>} *const [T], *mut str, *const str,
1469
+
1470
+
// SAFETY: `T` is `Zeroable`.
1471
+
{<const N: usize, T: Zeroable>} [T; N], {<T: Zeroable>} Wrapping<T>,
1472
+
}
1473
+
1474
+
macro_rules! impl_tuple_zeroable {
1475
+
($(,)?) => {};
1476
+
($first:ident, $($t:ident),* $(,)?) => {
1477
+
// SAFETY: All elements are zeroable and padding can be zero.
1478
+
unsafe impl<$first: Zeroable, $($t: Zeroable),*> Zeroable for ($first, $($t),*) {}
1479
+
impl_tuple_zeroable!($($t),* ,);
1480
+
}
1481
+
}
1482
+
1483
+
impl_tuple_zeroable!(A, B, C, D, E, F, G, H, I, J);
+11
samples/rust/Kconfig
+11
samples/rust/Kconfig
···
40
40
41
41
If unsure, say N.
42
42
43
+
config SAMPLE_RUST_DMA
44
+
tristate "DMA Test Driver"
45
+
depends on PCI
46
+
help
47
+
This option builds the Rust DMA Test driver sample.
48
+
49
+
To compile this as a module, choose M here:
50
+
the module will be called rust_dma.
51
+
52
+
If unsure, say N.
53
+
43
54
config SAMPLE_RUST_DRIVER_PCI
44
55
tristate "PCI Driver"
45
56
depends on PCI
+1
samples/rust/Makefile
+1
samples/rust/Makefile
···
4
4
obj-$(CONFIG_SAMPLE_RUST_MINIMAL) += rust_minimal.o
5
5
obj-$(CONFIG_SAMPLE_RUST_MISC_DEVICE) += rust_misc_device.o
6
6
obj-$(CONFIG_SAMPLE_RUST_PRINT) += rust_print.o
7
+
obj-$(CONFIG_SAMPLE_RUST_DMA) += rust_dma.o
7
8
obj-$(CONFIG_SAMPLE_RUST_DRIVER_PCI) += rust_driver_pci.o
8
9
obj-$(CONFIG_SAMPLE_RUST_DRIVER_PLATFORM) += rust_driver_platform.o
9
10
obj-$(CONFIG_SAMPLE_RUST_DRIVER_FAUX) += rust_driver_faux.o
+97
samples/rust/rust_dma.rs
+97
samples/rust/rust_dma.rs
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
3
+
//! Rust DMA api test (based on QEMU's `pci-testdev`).
4
+
//!
5
+
//! To make this driver probe, QEMU must be run with `-device pci-testdev`.
6
+
7
+
use kernel::{bindings, dma::CoherentAllocation, pci, prelude::*};
8
+
9
+
struct DmaSampleDriver {
10
+
pdev: pci::Device,
11
+
ca: CoherentAllocation<MyStruct>,
12
+
}
13
+
14
+
const TEST_VALUES: [(u32, u32); 5] = [
15
+
(0xa, 0xb),
16
+
(0xc, 0xd),
17
+
(0xe, 0xf),
18
+
(0xab, 0xba),
19
+
(0xcd, 0xef),
20
+
];
21
+
22
+
struct MyStruct {
23
+
h: u32,
24
+
b: u32,
25
+
}
26
+
27
+
impl MyStruct {
28
+
fn new(h: u32, b: u32) -> Self {
29
+
Self { h, b }
30
+
}
31
+
}
32
+
// SAFETY: All bit patterns are acceptable values for `MyStruct`.
33
+
unsafe impl kernel::transmute::AsBytes for MyStruct {}
34
+
// SAFETY: Instances of `MyStruct` have no uninitialized portions.
35
+
unsafe impl kernel::transmute::FromBytes for MyStruct {}
36
+
37
+
kernel::pci_device_table!(
38
+
PCI_TABLE,
39
+
MODULE_PCI_TABLE,
40
+
<DmaSampleDriver as pci::Driver>::IdInfo,
41
+
[(
42
+
pci::DeviceId::from_id(bindings::PCI_VENDOR_ID_REDHAT, 0x5),
43
+
()
44
+
)]
45
+
);
46
+
47
+
impl pci::Driver for DmaSampleDriver {
48
+
type IdInfo = ();
49
+
const ID_TABLE: pci::IdTable<Self::IdInfo> = &PCI_TABLE;
50
+
51
+
fn probe(pdev: &mut pci::Device, _info: &Self::IdInfo) -> Result<Pin<KBox<Self>>> {
52
+
dev_info!(pdev.as_ref(), "Probe DMA test driver.\n");
53
+
54
+
let ca: CoherentAllocation<MyStruct> =
55
+
CoherentAllocation::alloc_coherent(pdev.as_ref(), TEST_VALUES.len(), GFP_KERNEL)?;
56
+
57
+
|| -> Result {
58
+
for (i, value) in TEST_VALUES.into_iter().enumerate() {
59
+
kernel::dma_write!(ca[i] = MyStruct::new(value.0, value.1));
60
+
}
61
+
62
+
Ok(())
63
+
}()?;
64
+
65
+
let drvdata = KBox::new(
66
+
Self {
67
+
pdev: pdev.clone(),
68
+
ca,
69
+
},
70
+
GFP_KERNEL,
71
+
)?;
72
+
73
+
Ok(drvdata.into())
74
+
}
75
+
}
76
+
77
+
impl Drop for DmaSampleDriver {
78
+
fn drop(&mut self) {
79
+
dev_info!(self.pdev.as_ref(), "Unload DMA test driver.\n");
80
+
81
+
let _ = || -> Result {
82
+
for (i, value) in TEST_VALUES.into_iter().enumerate() {
83
+
assert_eq!(kernel::dma_read!(self.ca[i].h), value.0);
84
+
assert_eq!(kernel::dma_read!(self.ca[i].b), value.1);
85
+
}
86
+
Ok(())
87
+
}();
88
+
}
89
+
}
90
+
91
+
kernel::module_pci_driver! {
92
+
type: DmaSampleDriver,
93
+
name: "rust_dma",
94
+
authors: ["Abdiel Janulgue"],
95
+
description: "Rust DMA test",
96
+
license: "GPL v2",
97
+
}
+1
-1
samples/rust/rust_driver_faux.rs
+1
-1
samples/rust/rust_driver_faux.rs
+1
-1
samples/rust/rust_driver_pci.rs
+1
-1
samples/rust/rust_driver_pci.rs
+1
-1
samples/rust/rust_driver_platform.rs
+1
-1
samples/rust/rust_driver_platform.rs
+1
-1
samples/rust/rust_minimal.rs
+1
-1
samples/rust/rust_minimal.rs
+1
-1
samples/rust/rust_misc_device.rs
+1
-1
samples/rust/rust_misc_device.rs
+1
-1
samples/rust/rust_print_main.rs
+1
-1
samples/rust/rust_print_main.rs
+2
-2
scripts/Makefile.build
+2
-2
scripts/Makefile.build
···
226
226
# Compile Rust sources (.rs)
227
227
# ---------------------------------------------------------------------------
228
228
229
-
rust_allowed_features := asm_const,asm_goto,arbitrary_self_types,lint_reasons
229
+
rust_allowed_features := asm_const,asm_goto,arbitrary_self_types,lint_reasons,raw_ref_op
230
230
231
231
# `--out-dir` is required to avoid temporaries being created by `rustc` in the
232
232
# current working directory, which may be not accessible in the out-of-tree
···
237
237
-Zallow-features=$(rust_allowed_features) \
238
238
-Zcrate-attr=no_std \
239
239
-Zcrate-attr='feature($(rust_allowed_features))' \
240
-
-Zunstable-options --extern kernel \
240
+
-Zunstable-options --extern pin_init --extern kernel \
241
241
--crate-type rlib -L $(objtree)/rust/ \
242
242
--crate-name $(basename $(notdir $@)) \
243
243
--sysroot=/dev/null \
+16
-1
scripts/generate_rust_analyzer.py
+16
-1
scripts/generate_rust_analyzer.py
···
97
97
["core", "compiler_builtins"],
98
98
)
99
99
100
+
append_crate(
101
+
"pin_init_internal",
102
+
srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
103
+
[],
104
+
cfg=["kernel"],
105
+
is_proc_macro=True,
106
+
)
107
+
108
+
append_crate(
109
+
"pin_init",
110
+
srctree / "rust" / "pin-init" / "src" / "lib.rs",
111
+
["core", "pin_init_internal", "macros"],
112
+
cfg=["kernel"],
113
+
)
114
+
100
115
def append_crate_with_generated(
101
116
display_name,
102
117
deps,
···
133
118
134
119
append_crate_with_generated("bindings", ["core"])
135
120
append_crate_with_generated("uapi", ["core"])
136
-
append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"])
121
+
append_crate_with_generated("kernel", ["core", "macros", "build_error", "pin-init", "bindings", "uapi"])
137
122
138
123
def is_root_crate(build_file, target):
139
124
try:
+4
-4
scripts/rustdoc_test_gen.rs
+4
-4
scripts/rustdoc_test_gen.rs
···
87
87
88
88
assert!(
89
89
valid_paths.len() > 0,
90
-
"No path candidates found. This is likely a bug in the build system, or some files went \
91
-
away while compiling."
90
+
"No path candidates found for `{file}`. This is likely a bug in the build system, or some \
91
+
files went away while compiling."
92
92
);
93
93
94
94
if valid_paths.len() > 1 {
···
97
97
eprintln!(" {path:?}");
98
98
}
99
99
panic!(
100
-
"Several path candidates found, please resolve the ambiguity by renaming a file or \
101
-
folder."
100
+
"Several path candidates found for `{file}`, please resolve the ambiguity by renaming \
101
+
a file or folder."
102
102
);
103
103
}
104
104