tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'bpf-kernel-functions-with-kf_implicit_args'

Ihor Solodrai says:

====================
bpf: Kernel functions with KF_IMPLICIT_ARGS

This series implements a generic "implicit arguments" feature for BPF
kernel functions. For context see prior work [1][2].

A mechanism is created for kfuncs to have arguments that are not
visible to the BPF programs, and are provided to the kernel function
implementation by the verifier.

This mechanism is then used in the kfuncs that have a parameter with
__prog annotation [3], which is the current way of passing struct
bpf_prog_aux pointer to kfuncs.

The function with implicit arguments is defined by KF_IMPLICIT_ARGS
flag in BTF_IDS_FLAGS set. In this series, only a pointer to struct
bpf_prog_aux can be implicit, although it is simple to extend this to
more types.

The verifier handles a kfunc with KF_IMPLICIT_ARGS by resolving it to
a different (actual) BTF prototype early in verification (patch #3).

A <kfunc>_impl function generated in BTF for a kfunc with implicit
args does not have a "bpf_kfunc" decl tag, and a kernel address. The
verifier will reject a program trying to call such an _impl kfunc.

The usage of <kfunc>_impl functions in BPF is only allowed for kfuncs
with an explicit kernel (or kmodule) declaration, that is in "legacy"
cases. As of this series, there are no legacy kernel functions, as all
__prog users are migrated to KF_IMPLICIT_ARGS. However the
implementation allows for legacy cases support in principle.

The series removes the following BPF kernel functions:
- bpf_stream_vprintk_impl
- bpf_task_work_schedule_resume_impl
- bpf_task_work_schedule_signal_impl
- bpf_wq_set_callback_impl

This will break existing BPF programs calling these functions (the
verifier will not load them) on new kernels.

To mitigate, BPF users are advised to use the following pattern [4]:

if (xxx_impl)
xxx_impl(..., NULL);
else
xxx(...);

Which can be wrapped in a macro.

The series consists of the following patches:
- patches #1 and #2 are non-functional refactoring in kernel/bpf
- patch #3 defines KF_IMPLICIT_ARGS flag and teaches the verifier
about it
- patches #4-#5 implement btf2btf transformation in resolve_btfids
- patch #6 adds selftests specific to KF_IMPLICIT_ARGS feature
- patches #7-#11 migrate the current users of __prog argument to
KF_IMPLICIT_ARGS
- patch #12 removes __prog arg suffix support from the kernel
- patch #13 updates the docs

[1] https://lore.kernel.org/bpf/20251029190113.3323406-1-ihor.solodrai@linux.dev/
[2] https://lore.kernel.org/bpf/20250924211716.1287715-1-ihor.solodrai@linux.dev/
[3] https://docs.kernel.org/bpf/kfuncs.html#prog-annotation
[4] https://lore.kernel.org/bpf/CAEf4BzbgPfRm9BX=TsZm-TsHFAHcwhPY4vTt=9OT-uhWqf8tqw@mail.gmail.com/
---

v2->v3:
- resolve_btfids: Use dynamic reallocation for btf2btf_context arrays (Andrii)
- resolve_btfids: Add missing free() for btf2btf_context arrays (AI)
- Other nits in resolve_btfids (Andrii, Eduard)

v2: https://lore.kernel.org/bpf/20260116201700.864797-1-ihor.solodrai@linux.dev/

v1->v2:
- Replace the following kernel functions with KF_IMPLICIT_ARGS version:
- bpf_stream_vprintk_impl -> bpf_stream_vprintk
- bpf_task_work_schedule_resume_impl -> bpf_task_work_schedule_resume
- bpf_task_work_schedule_signal_impl -> bpf_task_work_schedule_signal
- bpf_wq_set_callback_impl -> bpf_wq_set_callback_impl
- Remove __prog arg suffix support from the verifier
- Rework btf2btf implementation in resolve_btfids
- Do distill base and sort before BTF_ids patching
- Collect kfuncs based on BTF decl tags, before BTF_ids are patched
- resolve_btfids: use dynamic memory for intermediate data (Andrii)
- verifier: reset .subreg_def for caller saved registers on kfunc
call (Eduard)
- selftests/hid: remove Makefile changes (Benjamin)
- selftests/bpf: Add a patch (#11) migrating struct_ops_assoc test
to KF_IMPLICIT_ARGS
- Various nits across the series (Alexei, Andrii, Eduard)

v1: https://lore.kernel.org/bpf/20260109184852.1089786-1-ihor.solodrai@linux.dev/

---
====================

Link: https://patch.msgid.link/20260120222638.3976562-1-ihor.solodrai@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

Alexei Starovoitov b236134f 2e6690d4

+821 -227
+32 -17
Documentation/bpf/kfuncs.rst
··· 232 232 ... 233 233 } 234 234 235 - 2.3.6 __prog Annotation 236 - --------------------------- 237 - This annotation is used to indicate that the argument needs to be fixed up to 238 - the bpf_prog_aux of the caller BPF program. Any value passed into this argument 239 - is ignored, and rewritten by the verifier. 240 - 241 - An example is given below:: 242 - 243 - __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq, 244 - int (callback_fn)(void *map, int *key, void *value), 245 - unsigned int flags, 246 - void *aux__prog) 247 - { 248 - struct bpf_prog_aux *aux = aux__prog; 249 - ... 250 - } 251 - 252 235 .. _BPF_kfunc_nodef: 253 236 254 237 2.4 Using an existing kernel function ··· 363 380 encouraged to make their use-cases known as early as possible, and participate 364 381 in upstream discussions regarding whether to keep, change, deprecate, or remove 365 382 those kfuncs if and when such discussions occur. 383 + 384 + 2.5.9 KF_IMPLICIT_ARGS flag 385 + ------------------------------------ 386 + 387 + The KF_IMPLICIT_ARGS flag is used to indicate that the BPF signature 388 + of the kfunc is different from it's kernel signature, and the values 389 + for implicit arguments are provided at load time by the verifier. 390 + 391 + Only arguments of specific types are implicit. 392 + Currently only ``struct bpf_prog_aux *`` type is supported. 393 + 394 + A kfunc with KF_IMPLICIT_ARGS flag therefore has two types in BTF: one 395 + function matching the kernel declaration (with _impl suffix in the 396 + name by convention), and another matching the intended BPF API. 397 + 398 + Verifier only allows calls to the non-_impl version of a kfunc, that 399 + uses a signature without the implicit arguments. 400 + 401 + Example declaration: 402 + 403 + .. code-block:: c 404 + 405 + __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw, 406 + void *map__map, bpf_task_work_callback_t callback, 407 + struct bpf_prog_aux *aux) { ... } 408 + 409 + Example usage in BPF program: 410 + 411 + .. code-block:: c 412 + 413 + /* note that the last argument is omitted */ 414 + bpf_task_work_schedule_signal(task, &work->tw, &arrmap, task_work_callback); 366 415 367 416 2.6 Registering the kfuncs 368 417 --------------------------
+3 -5
drivers/hid/bpf/progs/hid_bpf_helpers.h
··· 33 33 /* bpf_wq implementation */ 34 34 extern int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) __weak __ksym; 35 35 extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym; 36 - extern int bpf_wq_set_callback_impl(struct bpf_wq *wq, 37 - int (callback_fn)(void *map, int *key, void *value), 38 - unsigned int flags__k, void *aux__ign) __ksym; 39 - #define bpf_wq_set_callback(wq, cb, flags) \ 40 - bpf_wq_set_callback_impl(wq, cb, flags, NULL) 36 + extern int bpf_wq_set_callback(struct bpf_wq *wq, 37 + int (*callback_fn)(void *, int *, void *), 38 + unsigned int flags) __weak __ksym; 41 39 42 40 #define HID_MAX_DESCRIPTOR_SIZE 4096 43 41 #define HID_IGNORE_EVENT -1
+3 -2
include/linux/btf.h
··· 78 78 #define KF_ARENA_RET (1 << 13) /* kfunc returns an arena pointer */ 79 79 #define KF_ARENA_ARG1 (1 << 14) /* kfunc takes an arena pointer as its first argument */ 80 80 #define KF_ARENA_ARG2 (1 << 15) /* kfunc takes an arena pointer as its second argument */ 81 + #define KF_IMPLICIT_ARGS (1 << 16) /* kfunc has implicit arguments supplied by the verifier */ 81 82 82 83 /* 83 84 * Tag marking a kernel function as a kfunc. This is meant to minimize the ··· 576 575 const char *btf_str_by_offset(const struct btf *btf, u32 offset); 577 576 struct btf *btf_parse_vmlinux(void); 578 577 struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog); 579 - u32 *btf_kfunc_id_set_contains(const struct btf *btf, u32 kfunc_btf_id, 580 - const struct bpf_prog *prog); 578 + u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog); 579 + bool btf_kfunc_is_allowed(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog); 581 580 u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id, 582 581 const struct bpf_prog *prog); 583 582 int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
+53 -17
kernel/bpf/btf.c
··· 8757 8757 return ret; 8758 8758 } 8759 8759 8760 - static u32 *__btf_kfunc_id_set_contains(const struct btf *btf, 8761 - enum btf_kfunc_hook hook, 8762 - u32 kfunc_btf_id, 8763 - const struct bpf_prog *prog) 8760 + static u32 *btf_kfunc_id_set_contains(const struct btf *btf, 8761 + enum btf_kfunc_hook hook, 8762 + u32 kfunc_btf_id) 8764 8763 { 8765 - struct btf_kfunc_hook_filter *hook_filter; 8766 8764 struct btf_id_set8 *set; 8767 - u32 *id, i; 8765 + u32 *id; 8768 8766 8769 8767 if (hook >= BTF_KFUNC_HOOK_MAX) 8770 8768 return NULL; 8771 8769 if (!btf->kfunc_set_tab) 8772 8770 return NULL; 8773 - hook_filter = &btf->kfunc_set_tab->hook_filters[hook]; 8774 - for (i = 0; i < hook_filter->nr_filters; i++) { 8775 - if (hook_filter->filters[i](prog, kfunc_btf_id)) 8776 - return NULL; 8777 - } 8778 8771 set = btf->kfunc_set_tab->sets[hook]; 8779 8772 if (!set) 8780 8773 return NULL; ··· 8776 8783 return NULL; 8777 8784 /* The flags for BTF ID are located next to it */ 8778 8785 return id + 1; 8786 + } 8787 + 8788 + static bool __btf_kfunc_is_allowed(const struct btf *btf, 8789 + enum btf_kfunc_hook hook, 8790 + u32 kfunc_btf_id, 8791 + const struct bpf_prog *prog) 8792 + { 8793 + struct btf_kfunc_hook_filter *hook_filter; 8794 + int i; 8795 + 8796 + if (hook >= BTF_KFUNC_HOOK_MAX) 8797 + return false; 8798 + if (!btf->kfunc_set_tab) 8799 + return false; 8800 + 8801 + hook_filter = &btf->kfunc_set_tab->hook_filters[hook]; 8802 + for (i = 0; i < hook_filter->nr_filters; i++) { 8803 + if (hook_filter->filters[i](prog, kfunc_btf_id)) 8804 + return false; 8805 + } 8806 + 8807 + return true; 8779 8808 } 8780 8809 8781 8810 static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type) ··· 8847 8832 } 8848 8833 } 8849 8834 8835 + bool btf_kfunc_is_allowed(const struct btf *btf, 8836 + u32 kfunc_btf_id, 8837 + const struct bpf_prog *prog) 8838 + { 8839 + enum bpf_prog_type prog_type = resolve_prog_type(prog); 8840 + enum btf_kfunc_hook hook; 8841 + u32 *kfunc_flags; 8842 + 8843 + kfunc_flags = btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id); 8844 + if (kfunc_flags && __btf_kfunc_is_allowed(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id, prog)) 8845 + return true; 8846 + 8847 + hook = bpf_prog_type_to_kfunc_hook(prog_type); 8848 + kfunc_flags = btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id); 8849 + if (kfunc_flags && __btf_kfunc_is_allowed(btf, hook, kfunc_btf_id, prog)) 8850 + return true; 8851 + 8852 + return false; 8853 + } 8854 + 8850 8855 /* Caution: 8851 8856 * Reference to the module (obtained using btf_try_get_module) corresponding to 8852 8857 * the struct btf *MUST* be held when calling this function from verifier ··· 8874 8839 * keeping the reference for the duration of the call provides the necessary 8875 8840 * protection for looking up a well-formed btf->kfunc_set_tab. 8876 8841 */ 8877 - u32 *btf_kfunc_id_set_contains(const struct btf *btf, 8878 - u32 kfunc_btf_id, 8879 - const struct bpf_prog *prog) 8842 + u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog) 8880 8843 { 8881 8844 enum bpf_prog_type prog_type = resolve_prog_type(prog); 8882 8845 enum btf_kfunc_hook hook; 8883 8846 u32 *kfunc_flags; 8884 8847 8885 - kfunc_flags = __btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id, prog); 8848 + kfunc_flags = btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id); 8886 8849 if (kfunc_flags) 8887 8850 return kfunc_flags; 8888 8851 8889 8852 hook = bpf_prog_type_to_kfunc_hook(prog_type); 8890 - return __btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id, prog); 8853 + return btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id); 8891 8854 } 8892 8855 8893 8856 u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id, 8894 8857 const struct bpf_prog *prog) 8895 8858 { 8896 - return __btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id, prog); 8859 + if (!__btf_kfunc_is_allowed(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id, prog)) 8860 + return NULL; 8861 + 8862 + return btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id); 8897 8863 } 8898 8864 8899 8865 static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
+20 -23
kernel/bpf/helpers.c
··· 3120 3120 return 0; 3121 3121 } 3122 3122 3123 - __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq, 3124 - int (callback_fn)(void *map, int *key, void *value), 3125 - unsigned int flags, 3126 - void *aux__prog) 3123 + __bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq, 3124 + int (callback_fn)(void *map, int *key, void *value), 3125 + unsigned int flags, 3126 + struct bpf_prog_aux *aux) 3127 3127 { 3128 - struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__prog; 3129 3128 struct bpf_async_kern *async = (struct bpf_async_kern *)wq; 3130 3129 3131 3130 if (flags) ··· 4274 4275 } 4275 4276 4276 4277 /** 4277 - * bpf_task_work_schedule_signal_impl - Schedule BPF callback using task_work_add with TWA_SIGNAL 4278 + * bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL 4278 4279 * mode 4279 4280 * @task: Task struct for which callback should be scheduled 4280 4281 * @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping 4281 4282 * @map__map: bpf_map that embeds struct bpf_task_work in the values 4282 4283 * @callback: pointer to BPF subprogram to call 4283 - * @aux__prog: user should pass NULL 4284 + * @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier 4284 4285 * 4285 4286 * Return: 0 if task work has been scheduled successfully, negative error code otherwise 4286 4287 */ 4287 - __bpf_kfunc int bpf_task_work_schedule_signal_impl(struct task_struct *task, 4288 - struct bpf_task_work *tw, void *map__map, 4289 - bpf_task_work_callback_t callback, 4290 - void *aux__prog) 4288 + __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw, 4289 + void *map__map, bpf_task_work_callback_t callback, 4290 + struct bpf_prog_aux *aux) 4291 4291 { 4292 - return bpf_task_work_schedule(task, tw, map__map, callback, aux__prog, TWA_SIGNAL); 4292 + return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL); 4293 4293 } 4294 4294 4295 4295 /** 4296 - * bpf_task_work_schedule_resume_impl - Schedule BPF callback using task_work_add with TWA_RESUME 4296 + * bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME 4297 4297 * mode 4298 4298 * @task: Task struct for which callback should be scheduled 4299 4299 * @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping 4300 4300 * @map__map: bpf_map that embeds struct bpf_task_work in the values 4301 4301 * @callback: pointer to BPF subprogram to call 4302 - * @aux__prog: user should pass NULL 4302 + * @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier 4303 4303 * 4304 4304 * Return: 0 if task work has been scheduled successfully, negative error code otherwise 4305 4305 */ 4306 - __bpf_kfunc int bpf_task_work_schedule_resume_impl(struct task_struct *task, 4307 - struct bpf_task_work *tw, void *map__map, 4308 - bpf_task_work_callback_t callback, 4309 - void *aux__prog) 4306 + __bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw, 4307 + void *map__map, bpf_task_work_callback_t callback, 4308 + struct bpf_prog_aux *aux) 4310 4309 { 4311 - return bpf_task_work_schedule(task, tw, map__map, callback, aux__prog, TWA_RESUME); 4310 + return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME); 4312 4311 } 4313 4312 4314 4313 static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep, ··· 4485 4488 BTF_ID_FLAGS(func, bpf_modify_return_test_tp) 4486 4489 #endif 4487 4490 BTF_ID_FLAGS(func, bpf_wq_init) 4488 - BTF_ID_FLAGS(func, bpf_wq_set_callback_impl) 4491 + BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS) 4489 4492 BTF_ID_FLAGS(func, bpf_wq_start) 4490 4493 BTF_ID_FLAGS(func, bpf_preempt_disable) 4491 4494 BTF_ID_FLAGS(func, bpf_preempt_enable) ··· 4533 4536 #if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS) 4534 4537 BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU) 4535 4538 #endif 4536 - BTF_ID_FLAGS(func, bpf_stream_vprintk_impl) 4537 - BTF_ID_FLAGS(func, bpf_task_work_schedule_signal_impl) 4538 - BTF_ID_FLAGS(func, bpf_task_work_schedule_resume_impl) 4539 + BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS) 4540 + BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) 4541 + BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) 4539 4542 BTF_ID_FLAGS(func, bpf_dynptr_from_file) 4540 4543 BTF_ID_FLAGS(func, bpf_dynptr_file_discard) 4541 4544 BTF_KFUNCS_END(common_btf_ids)
+2 -3
kernel/bpf/stream.c
··· 212 212 * Avoid using enum bpf_stream_id so that kfunc users don't have to pull in the 213 213 * enum in headers. 214 214 */ 215 - __bpf_kfunc int bpf_stream_vprintk_impl(int stream_id, const char *fmt__str, const void *args, 216 - u32 len__sz, void *aux__prog) 215 + __bpf_kfunc int bpf_stream_vprintk(int stream_id, const char *fmt__str, const void *args, 216 + u32 len__sz, struct bpf_prog_aux *aux) 217 217 { 218 218 struct bpf_bprintf_data data = { 219 219 .get_bin_args = true, 220 220 .get_buf = true, 221 221 }; 222 - struct bpf_prog_aux *aux = aux__prog; 223 222 u32 fmt_size = strlen(fmt__str) + 1; 224 223 struct bpf_stream *stream; 225 224 u32 data_len = len__sz;
+159 -90
kernel/bpf/verifier.c
··· 294 294 s64 const_map_key; 295 295 }; 296 296 297 + struct bpf_kfunc_meta { 298 + struct btf *btf; 299 + const struct btf_type *proto; 300 + const char *name; 301 + const u32 *flags; 302 + s32 id; 303 + }; 304 + 297 305 struct bpf_kfunc_call_arg_meta { 298 306 /* In parameters */ 299 307 struct btf *btf; ··· 520 512 static bool is_callback_calling_kfunc(u32 btf_id); 521 513 static bool is_bpf_throw_kfunc(struct bpf_insn *insn); 522 514 523 - static bool is_bpf_wq_set_callback_impl_kfunc(u32 btf_id); 515 + static bool is_bpf_wq_set_callback_kfunc(u32 btf_id); 524 516 static bool is_task_work_add_kfunc(u32 func_id); 525 517 526 518 static bool is_sync_callback_calling_function(enum bpf_func_id func_id) ··· 562 554 563 555 /* bpf_wq and bpf_task_work callbacks are always sleepable. */ 564 556 if (bpf_pseudo_kfunc_call(insn) && insn->off == 0 && 565 - (is_bpf_wq_set_callback_impl_kfunc(insn->imm) || is_task_work_add_kfunc(insn->imm))) 557 + (is_bpf_wq_set_callback_kfunc(insn->imm) || is_task_work_add_kfunc(insn->imm))) 566 558 return true; 567 559 568 560 verifier_bug(env, "unhandled async callback in is_async_cb_sleepable"); ··· 3271 3263 return btf_vmlinux ?: ERR_PTR(-ENOENT); 3272 3264 } 3273 3265 3274 - static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) 3266 + #define KF_IMPL_SUFFIX "_impl" 3267 + 3268 + static const struct btf_type *find_kfunc_impl_proto(struct bpf_verifier_env *env, 3269 + struct btf *btf, 3270 + const char *func_name) 3271 + { 3272 + char *buf = env->tmp_str_buf; 3273 + const struct btf_type *func; 3274 + s32 impl_id; 3275 + int len; 3276 + 3277 + len = snprintf(buf, TMP_STR_BUF_LEN, "%s%s", func_name, KF_IMPL_SUFFIX); 3278 + if (len < 0 || len >= TMP_STR_BUF_LEN) { 3279 + verbose(env, "function name %s%s is too long\n", func_name, KF_IMPL_SUFFIX); 3280 + return NULL; 3281 + } 3282 + 3283 + impl_id = btf_find_by_name_kind(btf, buf, BTF_KIND_FUNC); 3284 + if (impl_id <= 0) { 3285 + verbose(env, "cannot find function %s in BTF\n", buf); 3286 + return NULL; 3287 + } 3288 + 3289 + func = btf_type_by_id(btf, impl_id); 3290 + 3291 + return btf_type_by_id(btf, func->type); 3292 + } 3293 + 3294 + static int fetch_kfunc_meta(struct bpf_verifier_env *env, 3295 + s32 func_id, 3296 + s16 offset, 3297 + struct bpf_kfunc_meta *kfunc) 3275 3298 { 3276 3299 const struct btf_type *func, *func_proto; 3300 + const char *func_name; 3301 + u32 *kfunc_flags; 3302 + struct btf *btf; 3303 + 3304 + if (func_id <= 0) { 3305 + verbose(env, "invalid kernel function btf_id %d\n", func_id); 3306 + return -EINVAL; 3307 + } 3308 + 3309 + btf = find_kfunc_desc_btf(env, offset); 3310 + if (IS_ERR(btf)) { 3311 + verbose(env, "failed to find BTF for kernel function\n"); 3312 + return PTR_ERR(btf); 3313 + } 3314 + 3315 + /* 3316 + * Note that kfunc_flags may be NULL at this point, which 3317 + * means that we couldn't find func_id in any relevant 3318 + * kfunc_id_set. This most likely indicates an invalid kfunc 3319 + * call. However we don't fail with an error here, 3320 + * and let the caller decide what to do with NULL kfunc->flags. 3321 + */ 3322 + kfunc_flags = btf_kfunc_flags(btf, func_id, env->prog); 3323 + 3324 + func = btf_type_by_id(btf, func_id); 3325 + if (!func || !btf_type_is_func(func)) { 3326 + verbose(env, "kernel btf_id %d is not a function\n", func_id); 3327 + return -EINVAL; 3328 + } 3329 + 3330 + func_name = btf_name_by_offset(btf, func->name_off); 3331 + 3332 + /* 3333 + * An actual prototype of a kfunc with KF_IMPLICIT_ARGS flag 3334 + * can be found through the counterpart _impl kfunc. 3335 + */ 3336 + if (kfunc_flags && (*kfunc_flags & KF_IMPLICIT_ARGS)) 3337 + func_proto = find_kfunc_impl_proto(env, btf, func_name); 3338 + else 3339 + func_proto = btf_type_by_id(btf, func->type); 3340 + 3341 + if (!func_proto || !btf_type_is_func_proto(func_proto)) { 3342 + verbose(env, "kernel function btf_id %d does not have a valid func_proto\n", 3343 + func_id); 3344 + return -EINVAL; 3345 + } 3346 + 3347 + memset(kfunc, 0, sizeof(*kfunc)); 3348 + kfunc->btf = btf; 3349 + kfunc->id = func_id; 3350 + kfunc->name = func_name; 3351 + kfunc->proto = func_proto; 3352 + kfunc->flags = kfunc_flags; 3353 + 3354 + return 0; 3355 + } 3356 + 3357 + static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) 3358 + { 3277 3359 struct bpf_kfunc_btf_tab *btf_tab; 3278 3360 struct btf_func_model func_model; 3279 3361 struct bpf_kfunc_desc_tab *tab; 3280 3362 struct bpf_prog_aux *prog_aux; 3363 + struct bpf_kfunc_meta kfunc; 3281 3364 struct bpf_kfunc_desc *desc; 3282 - const char *func_name; 3283 - struct btf *desc_btf; 3284 3365 unsigned long addr; 3285 3366 int err; 3286 3367 ··· 3419 3322 prog_aux->kfunc_btf_tab = btf_tab; 3420 3323 } 3421 3324 3422 - desc_btf = find_kfunc_desc_btf(env, offset); 3423 - if (IS_ERR(desc_btf)) { 3424 - verbose(env, "failed to find BTF for kernel function\n"); 3425 - return PTR_ERR(desc_btf); 3426 - } 3427 - 3428 3325 if (find_kfunc_desc(env->prog, func_id, offset)) 3429 3326 return 0; 3430 3327 ··· 3427 3336 return -E2BIG; 3428 3337 } 3429 3338 3430 - func = btf_type_by_id(desc_btf, func_id); 3431 - if (!func || !btf_type_is_func(func)) { 3432 - verbose(env, "kernel btf_id %u is not a function\n", 3433 - func_id); 3434 - return -EINVAL; 3435 - } 3436 - func_proto = btf_type_by_id(desc_btf, func->type); 3437 - if (!func_proto || !btf_type_is_func_proto(func_proto)) { 3438 - verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", 3439 - func_id); 3440 - return -EINVAL; 3441 - } 3339 + err = fetch_kfunc_meta(env, func_id, offset, &kfunc); 3340 + if (err) 3341 + return err; 3442 3342 3443 - func_name = btf_name_by_offset(desc_btf, func->name_off); 3444 - addr = kallsyms_lookup_name(func_name); 3343 + addr = kallsyms_lookup_name(kfunc.name); 3445 3344 if (!addr) { 3446 - verbose(env, "cannot find address for kernel function %s\n", 3447 - func_name); 3345 + verbose(env, "cannot find address for kernel function %s\n", kfunc.name); 3448 3346 return -EINVAL; 3449 3347 } 3450 3348 ··· 3443 3363 return err; 3444 3364 } 3445 3365 3446 - err = btf_distill_func_proto(&env->log, desc_btf, 3447 - func_proto, func_name, 3448 - &func_model); 3366 + err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model); 3449 3367 if (err) 3450 3368 return err; 3451 3369 ··· 12211 12133 return btf_param_match_suffix(btf, arg, "__irq_flag"); 12212 12134 } 12213 12135 12214 - static bool is_kfunc_arg_prog(const struct btf *btf, const struct btf_param *arg) 12215 - { 12216 - return btf_param_match_suffix(btf, arg, "__prog"); 12217 - } 12218 - 12219 12136 static bool is_kfunc_arg_scalar_with_name(const struct btf *btf, 12220 12137 const struct btf_param *arg, 12221 12138 const char *name) ··· 12239 12166 KF_ARG_WORKQUEUE_ID, 12240 12167 KF_ARG_RES_SPIN_LOCK_ID, 12241 12168 KF_ARG_TASK_WORK_ID, 12169 + KF_ARG_PROG_AUX_ID 12242 12170 }; 12243 12171 12244 12172 BTF_ID_LIST(kf_arg_btf_ids) ··· 12251 12177 BTF_ID(struct, bpf_wq) 12252 12178 BTF_ID(struct, bpf_res_spin_lock) 12253 12179 BTF_ID(struct, bpf_task_work) 12180 + BTF_ID(struct, bpf_prog_aux) 12254 12181 12255 12182 static bool __is_kfunc_ptr_arg_type(const struct btf *btf, 12256 12183 const struct btf_param *arg, int type) ··· 12330 12255 return false; 12331 12256 12332 12257 return true; 12258 + } 12259 + 12260 + static bool is_kfunc_arg_prog_aux(const struct btf *btf, const struct btf_param *arg) 12261 + { 12262 + return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_PROG_AUX_ID); 12333 12263 } 12334 12264 12335 12265 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */ ··· 12430 12350 KF_bpf_percpu_obj_new_impl, 12431 12351 KF_bpf_percpu_obj_drop_impl, 12432 12352 KF_bpf_throw, 12433 - KF_bpf_wq_set_callback_impl, 12353 + KF_bpf_wq_set_callback, 12434 12354 KF_bpf_preempt_disable, 12435 12355 KF_bpf_preempt_enable, 12436 12356 KF_bpf_iter_css_task_new, ··· 12450 12370 KF_bpf_dynptr_from_file, 12451 12371 KF_bpf_dynptr_file_discard, 12452 12372 KF___bpf_trap, 12453 - KF_bpf_task_work_schedule_signal_impl, 12454 - KF_bpf_task_work_schedule_resume_impl, 12373 + KF_bpf_task_work_schedule_signal, 12374 + KF_bpf_task_work_schedule_resume, 12455 12375 KF_bpf_arena_alloc_pages, 12456 12376 KF_bpf_arena_free_pages, 12457 12377 KF_bpf_arena_reserve_pages, ··· 12494 12414 BTF_ID(func, bpf_percpu_obj_new_impl) 12495 12415 BTF_ID(func, bpf_percpu_obj_drop_impl) 12496 12416 BTF_ID(func, bpf_throw) 12497 - BTF_ID(func, bpf_wq_set_callback_impl) 12417 + BTF_ID(func, bpf_wq_set_callback) 12498 12418 BTF_ID(func, bpf_preempt_disable) 12499 12419 BTF_ID(func, bpf_preempt_enable) 12500 12420 #ifdef CONFIG_CGROUPS ··· 12527 12447 BTF_ID(func, bpf_dynptr_from_file) 12528 12448 BTF_ID(func, bpf_dynptr_file_discard) 12529 12449 BTF_ID(func, __bpf_trap) 12530 - BTF_ID(func, bpf_task_work_schedule_signal_impl) 12531 - BTF_ID(func, bpf_task_work_schedule_resume_impl) 12450 + BTF_ID(func, bpf_task_work_schedule_signal) 12451 + BTF_ID(func, bpf_task_work_schedule_resume) 12532 12452 BTF_ID(func, bpf_arena_alloc_pages) 12533 12453 BTF_ID(func, bpf_arena_free_pages) 12534 12454 BTF_ID(func, bpf_arena_reserve_pages) 12535 12455 12536 12456 static bool is_task_work_add_kfunc(u32 func_id) 12537 12457 { 12538 - return func_id == special_kfunc_list[KF_bpf_task_work_schedule_signal_impl] || 12539 - func_id == special_kfunc_list[KF_bpf_task_work_schedule_resume_impl]; 12458 + return func_id == special_kfunc_list[KF_bpf_task_work_schedule_signal] || 12459 + func_id == special_kfunc_list[KF_bpf_task_work_schedule_resume]; 12540 12460 } 12541 12461 12542 12462 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) ··· 12987 12907 12988 12908 static bool is_async_callback_calling_kfunc(u32 btf_id) 12989 12909 { 12990 - return btf_id == special_kfunc_list[KF_bpf_wq_set_callback_impl] || 12910 + return is_bpf_wq_set_callback_kfunc(btf_id) || 12991 12911 is_task_work_add_kfunc(btf_id); 12992 12912 } 12993 12913 ··· 12997 12917 insn->imm == special_kfunc_list[KF_bpf_throw]; 12998 12918 } 12999 12919 13000 - static bool is_bpf_wq_set_callback_impl_kfunc(u32 btf_id) 12920 + static bool is_bpf_wq_set_callback_kfunc(u32 btf_id) 13001 12921 { 13002 - return btf_id == special_kfunc_list[KF_bpf_wq_set_callback_impl]; 12922 + return btf_id == special_kfunc_list[KF_bpf_wq_set_callback]; 13003 12923 } 13004 12924 13005 12925 static bool is_callback_calling_kfunc(u32 btf_id) ··· 13273 13193 if (is_kfunc_arg_ignore(btf, &args[i])) 13274 13194 continue; 13275 13195 13276 - if (is_kfunc_arg_prog(btf, &args[i])) { 13277 - /* Used to reject repeated use of __prog. */ 13196 + if (is_kfunc_arg_prog_aux(btf, &args[i])) { 13197 + /* Reject repeated use bpf_prog_aux */ 13278 13198 if (meta->arg_prog) { 13279 13199 verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc"); 13280 13200 return -EFAULT; ··· 13776 13696 return 0; 13777 13697 } 13778 13698 13779 - static int fetch_kfunc_meta(struct bpf_verifier_env *env, 13780 - struct bpf_insn *insn, 13781 - struct bpf_kfunc_call_arg_meta *meta, 13782 - const char **kfunc_name) 13699 + static int fetch_kfunc_arg_meta(struct bpf_verifier_env *env, 13700 + s32 func_id, 13701 + s16 offset, 13702 + struct bpf_kfunc_call_arg_meta *meta) 13783 13703 { 13784 - const struct btf_type *func, *func_proto; 13785 - u32 func_id, *kfunc_flags; 13786 - const char *func_name; 13787 - struct btf *desc_btf; 13704 + struct bpf_kfunc_meta kfunc; 13705 + int err; 13788 13706 13789 - if (kfunc_name) 13790 - *kfunc_name = NULL; 13791 - 13792 - if (!insn->imm) 13793 - return -EINVAL; 13794 - 13795 - desc_btf = find_kfunc_desc_btf(env, insn->off); 13796 - if (IS_ERR(desc_btf)) 13797 - return PTR_ERR(desc_btf); 13798 - 13799 - func_id = insn->imm; 13800 - func = btf_type_by_id(desc_btf, func_id); 13801 - func_name = btf_name_by_offset(desc_btf, func->name_off); 13802 - if (kfunc_name) 13803 - *kfunc_name = func_name; 13804 - func_proto = btf_type_by_id(desc_btf, func->type); 13805 - 13806 - kfunc_flags = btf_kfunc_id_set_contains(desc_btf, func_id, env->prog); 13807 - if (!kfunc_flags) { 13808 - return -EACCES; 13809 - } 13707 + err = fetch_kfunc_meta(env, func_id, offset, &kfunc); 13708 + if (err) 13709 + return err; 13810 13710 13811 13711 memset(meta, 0, sizeof(*meta)); 13812 - meta->btf = desc_btf; 13813 - meta->func_id = func_id; 13814 - meta->kfunc_flags = *kfunc_flags; 13815 - meta->func_proto = func_proto; 13816 - meta->func_name = func_name; 13712 + meta->btf = kfunc.btf; 13713 + meta->func_id = kfunc.id; 13714 + meta->func_proto = kfunc.proto; 13715 + meta->func_name = kfunc.name; 13716 + 13717 + if (!kfunc.flags || !btf_kfunc_is_allowed(kfunc.btf, kfunc.id, env->prog)) 13718 + return -EACCES; 13719 + 13720 + meta->kfunc_flags = *kfunc.flags; 13817 13721 13818 13722 return 0; 13819 13723 } ··· 14002 13938 if (!insn->imm) 14003 13939 return 0; 14004 13940 14005 - err = fetch_kfunc_meta(env, insn, &meta, &func_name); 14006 - if (err == -EACCES && func_name) 14007 - verbose(env, "calling kernel function %s is not allowed\n", func_name); 13941 + err = fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta); 13942 + if (err == -EACCES && meta.func_name) 13943 + verbose(env, "calling kernel function %s is not allowed\n", meta.func_name); 14008 13944 if (err) 14009 13945 return err; 14010 13946 desc_btf = meta.btf; 13947 + func_name = meta.func_name; 14011 13948 insn_aux = &env->insn_aux_data[insn_idx]; 14012 13949 14013 13950 insn_aux->is_iter_next = is_iter_next_kfunc(&meta); ··· 14078 14013 meta.r0_rdonly = false; 14079 14014 } 14080 14015 14081 - if (is_bpf_wq_set_callback_impl_kfunc(meta.func_id)) { 14016 + if (is_bpf_wq_set_callback_kfunc(meta.func_id)) { 14082 14017 err = push_callback_call(env, insn, insn_idx, meta.subprogno, 14083 14018 set_timer_callback_state); 14084 14019 if (err) { ··· 14216 14151 } 14217 14152 } 14218 14153 14219 - for (i = 0; i < CALLER_SAVED_REGS; i++) 14220 - mark_reg_not_init(env, regs, caller_saved[i]); 14154 + for (i = 0; i < CALLER_SAVED_REGS; i++) { 14155 + u32 regno = caller_saved[i]; 14156 + 14157 + mark_reg_not_init(env, regs, regno); 14158 + regs[regno].subreg_def = DEF_NOT_SUBREG; 14159 + } 14221 14160 14222 14161 /* Check return type */ 14223 14162 t = btf_type_skip_modifiers(desc_btf, meta.func_proto->type, NULL); ··· 17858 17789 if (bpf_pseudo_kfunc_call(call)) { 17859 17790 int err; 17860 17791 17861 - err = fetch_kfunc_meta(env, call, &meta, NULL); 17792 + err = fetch_kfunc_arg_meta(env, call->imm, call->off, &meta); 17862 17793 if (err < 0) 17863 17794 /* error would be reported later */ 17864 17795 return false; ··· 18366 18297 } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { 18367 18298 struct bpf_kfunc_call_arg_meta meta; 18368 18299 18369 - ret = fetch_kfunc_meta(env, insn, &meta, NULL); 18300 + ret = fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta); 18370 18301 if (ret == 0 && is_iter_next_kfunc(&meta)) { 18371 18302 mark_prune_point(env, t); 18372 18303 /* Checking and saving state checkpoints at iter_next() call
+430 -21
tools/bpf/resolve_btfids/main.c
··· 152 152 int nr_typedefs; 153 153 }; 154 154 155 + #define KF_IMPLICIT_ARGS (1 << 16) 156 + #define KF_IMPL_SUFFIX "_impl" 157 + 158 + struct kfunc { 159 + const char *name; 160 + u32 btf_id; 161 + u32 flags; 162 + }; 163 + 164 + struct btf2btf_context { 165 + struct btf *btf; 166 + u32 *decl_tags; 167 + u32 nr_decl_tags; 168 + u32 max_decl_tags; 169 + struct kfunc *kfuncs; 170 + u32 nr_kfuncs; 171 + u32 max_kfuncs; 172 + }; 173 + 155 174 static int verbose; 156 175 static int warnings; 157 176 ··· 582 563 obj->base_btf = base_btf; 583 564 obj->btf = btf; 584 565 585 - if (obj->base_btf && obj->distill_base) { 586 - err = btf__distill_base(obj->btf, &base_btf, &btf); 587 - if (err) { 588 - pr_err("FAILED to distill base BTF: %s\n", strerror(errno)); 589 - goto out_err; 590 - } 591 - 592 - btf__free(obj->base_btf); 593 - btf__free(obj->btf); 594 - obj->base_btf = base_btf; 595 - obj->btf = btf; 596 - } 597 - 598 566 return 0; 599 567 600 568 out_err: ··· 856 850 return 0; 857 851 } 858 852 853 + static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf, s32 type_id) 854 + { 855 + const struct btf_type *t = btf__type_by_id(btf, type_id); 856 + 857 + while (btf_is_mod(t)) 858 + t = btf__type_by_id(btf, t->type); 859 + 860 + return t; 861 + } 862 + 863 + static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id) 864 + { 865 + u32 *arr = ctx->decl_tags; 866 + u32 cap = ctx->max_decl_tags; 867 + 868 + if (ctx->nr_decl_tags + 1 > cap) { 869 + cap = max(cap + 256, cap * 2); 870 + arr = realloc(arr, sizeof(u32) * cap); 871 + if (!arr) 872 + return -ENOMEM; 873 + ctx->max_decl_tags = cap; 874 + ctx->decl_tags = arr; 875 + } 876 + 877 + ctx->decl_tags[ctx->nr_decl_tags++] = decl_tag_id; 878 + 879 + return 0; 880 + } 881 + 882 + static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc) 883 + { 884 + struct kfunc *arr = ctx->kfuncs; 885 + u32 cap = ctx->max_kfuncs; 886 + 887 + if (ctx->nr_kfuncs + 1 > cap) { 888 + cap = max(cap + 256, cap * 2); 889 + arr = realloc(arr, sizeof(struct kfunc) * cap); 890 + if (!arr) 891 + return -ENOMEM; 892 + ctx->max_kfuncs = cap; 893 + ctx->kfuncs = arr; 894 + } 895 + 896 + ctx->kfuncs[ctx->nr_kfuncs++] = *kfunc; 897 + 898 + return 0; 899 + } 900 + 901 + static int collect_decl_tags(struct btf2btf_context *ctx) 902 + { 903 + const u32 type_cnt = btf__type_cnt(ctx->btf); 904 + struct btf *btf = ctx->btf; 905 + const struct btf_type *t; 906 + int err; 907 + 908 + for (u32 id = 1; id < type_cnt; id++) { 909 + t = btf__type_by_id(btf, id); 910 + if (!btf_is_decl_tag(t)) 911 + continue; 912 + err = push_decl_tag_id(ctx, id); 913 + if (err) 914 + return err; 915 + } 916 + 917 + return 0; 918 + } 919 + 920 + /* 921 + * To find the kfunc flags having its struct btf_id (with ELF addresses) 922 + * we need to find the address that is in range of a set8. 923 + * If a set8 is found, then the flags are located at addr + 4 bytes. 924 + * Return 0 (no flags!) if not found. 925 + */ 926 + static u32 find_kfunc_flags(struct object *obj, struct btf_id *kfunc_id) 927 + { 928 + const u32 *elf_data_ptr = obj->efile.idlist->d_buf; 929 + u64 set_lower_addr, set_upper_addr, addr; 930 + struct btf_id *set_id; 931 + struct rb_node *next; 932 + u32 flags; 933 + u64 idx; 934 + 935 + for (next = rb_first(&obj->sets); next; next = rb_next(next)) { 936 + set_id = rb_entry(next, struct btf_id, rb_node); 937 + if (set_id->kind != BTF_ID_KIND_SET8 || set_id->addr_cnt != 1) 938 + continue; 939 + 940 + set_lower_addr = set_id->addr[0]; 941 + set_upper_addr = set_lower_addr + set_id->cnt * sizeof(u64); 942 + 943 + for (u32 i = 0; i < kfunc_id->addr_cnt; i++) { 944 + addr = kfunc_id->addr[i]; 945 + /* 946 + * Lower bound is exclusive to skip the 8-byte header of the set. 947 + * Upper bound is inclusive to capture the last entry at offset 8*cnt. 948 + */ 949 + if (set_lower_addr < addr && addr <= set_upper_addr) { 950 + pr_debug("found kfunc %s in BTF_ID_FLAGS %s\n", 951 + kfunc_id->name, set_id->name); 952 + idx = addr - obj->efile.idlist_addr; 953 + idx = idx / sizeof(u32) + 1; 954 + flags = elf_data_ptr[idx]; 955 + 956 + return flags; 957 + } 958 + } 959 + } 960 + 961 + return 0; 962 + } 963 + 964 + static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx) 965 + { 966 + const char *tag_name, *func_name; 967 + struct btf *btf = ctx->btf; 968 + const struct btf_type *t; 969 + u32 flags, func_id; 970 + struct kfunc kfunc; 971 + struct btf_id *id; 972 + int err; 973 + 974 + if (ctx->nr_decl_tags == 0) 975 + return 0; 976 + 977 + for (u32 i = 0; i < ctx->nr_decl_tags; i++) { 978 + t = btf__type_by_id(btf, ctx->decl_tags[i]); 979 + if (btf_kflag(t) || btf_decl_tag(t)->component_idx != -1) 980 + continue; 981 + 982 + tag_name = btf__name_by_offset(btf, t->name_off); 983 + if (strcmp(tag_name, "bpf_kfunc") != 0) 984 + continue; 985 + 986 + func_id = t->type; 987 + t = btf__type_by_id(btf, func_id); 988 + if (!btf_is_func(t)) 989 + continue; 990 + 991 + func_name = btf__name_by_offset(btf, t->name_off); 992 + if (!func_name) 993 + continue; 994 + 995 + id = btf_id__find(&obj->funcs, func_name); 996 + if (!id || id->kind != BTF_ID_KIND_SYM) 997 + continue; 998 + 999 + flags = find_kfunc_flags(obj, id); 1000 + 1001 + kfunc.name = id->name; 1002 + kfunc.btf_id = func_id; 1003 + kfunc.flags = flags; 1004 + 1005 + err = push_kfunc(ctx, &kfunc); 1006 + if (err) 1007 + return err; 1008 + } 1009 + 1010 + return 0; 1011 + } 1012 + 1013 + static int build_btf2btf_context(struct object *obj, struct btf2btf_context *ctx) 1014 + { 1015 + int err; 1016 + 1017 + ctx->btf = obj->btf; 1018 + 1019 + err = collect_decl_tags(ctx); 1020 + if (err) { 1021 + pr_err("ERROR: resolve_btfids: failed to collect decl tags from BTF\n"); 1022 + return err; 1023 + } 1024 + 1025 + err = collect_kfuncs(obj, ctx); 1026 + if (err) { 1027 + pr_err("ERROR: resolve_btfids: failed to collect kfuncs from BTF\n"); 1028 + return err; 1029 + } 1030 + 1031 + return 0; 1032 + } 1033 + 1034 + 1035 + /* Implicit BPF kfunc arguments can only be of particular types */ 1036 + static bool is_kf_implicit_arg(const struct btf *btf, const struct btf_param *p) 1037 + { 1038 + static const char *const kf_implicit_arg_types[] = { 1039 + "bpf_prog_aux", 1040 + }; 1041 + const struct btf_type *t; 1042 + const char *name; 1043 + 1044 + t = btf_type_skip_qualifiers(btf, p->type); 1045 + if (!btf_is_ptr(t)) 1046 + return false; 1047 + 1048 + t = btf_type_skip_qualifiers(btf, t->type); 1049 + if (!btf_is_struct(t)) 1050 + return false; 1051 + 1052 + name = btf__name_by_offset(btf, t->name_off); 1053 + if (!name) 1054 + return false; 1055 + 1056 + for (int i = 0; i < ARRAY_SIZE(kf_implicit_arg_types); i++) 1057 + if (strcmp(name, kf_implicit_arg_types[i]) == 0) 1058 + return true; 1059 + 1060 + return false; 1061 + } 1062 + 1063 + /* 1064 + * For a kfunc with KF_IMPLICIT_ARGS we do the following: 1065 + * 1. Add a new function with _impl suffix in the name, with the prototype 1066 + * of the original kfunc. 1067 + * 2. Add all decl tags except "bpf_kfunc" for the _impl func. 1068 + * 3. Add a new function prototype with modified list of arguments: 1069 + * omitting implicit args. 1070 + * 4. Change the prototype of the original kfunc to the new one. 1071 + * 1072 + * This way we transform the BTF associated with the kfunc from 1073 + * __bpf_kfunc bpf_foo(int arg1, void *implicit_arg); 1074 + * into 1075 + * bpf_foo_impl(int arg1, void *implicit_arg); 1076 + * __bpf_kfunc bpf_foo(int arg1); 1077 + * 1078 + * If a kfunc with KF_IMPLICIT_ARGS already has an _impl counterpart 1079 + * in BTF, then it's a legacy case: an _impl function is declared in the 1080 + * source code. In this case, we can skip adding an _impl function, but we 1081 + * still have to add a func prototype that omits implicit args. 1082 + */ 1083 + static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct kfunc *kfunc) 1084 + { 1085 + s32 idx, new_proto_id, new_func_id, proto_id; 1086 + const char *param_name, *tag_name; 1087 + const struct btf_param *params; 1088 + enum btf_func_linkage linkage; 1089 + char tmp_name[KSYM_NAME_LEN]; 1090 + struct btf *btf = ctx->btf; 1091 + int err, len, nr_params; 1092 + struct btf_type *t; 1093 + 1094 + t = (struct btf_type *)btf__type_by_id(btf, kfunc->btf_id); 1095 + if (!t || !btf_is_func(t)) { 1096 + pr_err("ERROR: resolve_btfids: btf id %d is not a function\n", kfunc->btf_id); 1097 + return -EINVAL; 1098 + } 1099 + 1100 + linkage = btf_vlen(t); 1101 + 1102 + proto_id = t->type; 1103 + t = (struct btf_type *)btf__type_by_id(btf, proto_id); 1104 + if (!t || !btf_is_func_proto(t)) { 1105 + pr_err("ERROR: resolve_btfids: btf id %d is not a function prototype\n", proto_id); 1106 + return -EINVAL; 1107 + } 1108 + 1109 + len = snprintf(tmp_name, sizeof(tmp_name), "%s%s", kfunc->name, KF_IMPL_SUFFIX); 1110 + if (len < 0 || len >= sizeof(tmp_name)) { 1111 + pr_err("ERROR: function name is too long: %s%s\n", kfunc->name, KF_IMPL_SUFFIX); 1112 + return -E2BIG; 1113 + } 1114 + 1115 + if (btf__find_by_name_kind(btf, tmp_name, BTF_KIND_FUNC) > 0) { 1116 + pr_debug("resolve_btfids: function %s already exists in BTF\n", tmp_name); 1117 + goto add_new_proto; 1118 + } 1119 + 1120 + /* Add a new function with _impl suffix and original prototype */ 1121 + new_func_id = btf__add_func(btf, tmp_name, linkage, proto_id); 1122 + if (new_func_id < 0) { 1123 + pr_err("ERROR: resolve_btfids: failed to add func %s to BTF\n", tmp_name); 1124 + return new_func_id; 1125 + } 1126 + 1127 + /* Copy all decl tags except "bpf_kfunc" from the original kfunc to the new one */ 1128 + for (int i = 0; i < ctx->nr_decl_tags; i++) { 1129 + t = (struct btf_type *)btf__type_by_id(btf, ctx->decl_tags[i]); 1130 + if (t->type != kfunc->btf_id) 1131 + continue; 1132 + 1133 + tag_name = btf__name_by_offset(btf, t->name_off); 1134 + if (strcmp(tag_name, "bpf_kfunc") == 0) 1135 + continue; 1136 + 1137 + idx = btf_decl_tag(t)->component_idx; 1138 + 1139 + if (btf_kflag(t)) 1140 + err = btf__add_decl_attr(btf, tag_name, new_func_id, idx); 1141 + else 1142 + err = btf__add_decl_tag(btf, tag_name, new_func_id, idx); 1143 + 1144 + if (err < 0) { 1145 + pr_err("ERROR: resolve_btfids: failed to add decl tag %s for %s\n", 1146 + tag_name, tmp_name); 1147 + return -EINVAL; 1148 + } 1149 + } 1150 + 1151 + add_new_proto: 1152 + t = (struct btf_type *)btf__type_by_id(btf, proto_id); 1153 + new_proto_id = btf__add_func_proto(btf, t->type); 1154 + if (new_proto_id < 0) { 1155 + pr_err("ERROR: resolve_btfids: failed to add func proto for %s\n", kfunc->name); 1156 + return new_proto_id; 1157 + } 1158 + 1159 + /* Add non-implicit args to the new prototype */ 1160 + t = (struct btf_type *)btf__type_by_id(btf, proto_id); 1161 + nr_params = btf_vlen(t); 1162 + for (int i = 0; i < nr_params; i++) { 1163 + params = btf_params(t); 1164 + if (is_kf_implicit_arg(btf, &params[i])) 1165 + break; 1166 + param_name = btf__name_by_offset(btf, params[i].name_off); 1167 + err = btf__add_func_param(btf, param_name, params[i].type); 1168 + if (err < 0) { 1169 + pr_err("ERROR: resolve_btfids: failed to add param %s for %s\n", 1170 + param_name, kfunc->name); 1171 + return err; 1172 + } 1173 + t = (struct btf_type *)btf__type_by_id(btf, proto_id); 1174 + } 1175 + 1176 + /* Finally change the prototype of the original kfunc to the new one */ 1177 + t = (struct btf_type *)btf__type_by_id(btf, kfunc->btf_id); 1178 + t->type = new_proto_id; 1179 + 1180 + pr_debug("resolve_btfids: updated BTF for kfunc with implicit args %s\n", kfunc->name); 1181 + 1182 + return 0; 1183 + } 1184 + 1185 + static int btf2btf(struct object *obj) 1186 + { 1187 + struct btf2btf_context ctx = {}; 1188 + int err; 1189 + 1190 + err = build_btf2btf_context(obj, &ctx); 1191 + if (err) 1192 + goto out; 1193 + 1194 + for (u32 i = 0; i < ctx.nr_kfuncs; i++) { 1195 + struct kfunc *kfunc = &ctx.kfuncs[i]; 1196 + 1197 + if (!(kfunc->flags & KF_IMPLICIT_ARGS)) 1198 + continue; 1199 + 1200 + err = process_kfunc_with_implicit_args(&ctx, kfunc); 1201 + if (err) 1202 + goto out; 1203 + } 1204 + 1205 + err = 0; 1206 + out: 1207 + free(ctx.decl_tags); 1208 + free(ctx.kfuncs); 1209 + 1210 + return err; 1211 + } 1212 + 859 1213 /* 860 1214 * Sort types by name in ascending order resulting in all 861 1215 * anonymous types being placed before named types. ··· 1274 908 out: 1275 909 free(permute_ids); 1276 910 free(id_map); 911 + return err; 912 + } 913 + 914 + static int finalize_btf(struct object *obj) 915 + { 916 + struct btf *base_btf = obj->base_btf, *btf = obj->btf; 917 + int err; 918 + 919 + if (obj->base_btf && obj->distill_base) { 920 + err = btf__distill_base(obj->btf, &base_btf, &btf); 921 + if (err) { 922 + pr_err("FAILED to distill base BTF: %s\n", strerror(errno)); 923 + goto out_err; 924 + } 925 + 926 + btf__free(obj->base_btf); 927 + btf__free(obj->btf); 928 + obj->base_btf = base_btf; 929 + obj->btf = btf; 930 + } 931 + 932 + err = sort_btf_by_name(obj->btf); 933 + if (err) { 934 + pr_err("FAILED to sort BTF: %s\n", strerror(errno)); 935 + goto out_err; 936 + } 937 + 938 + return 0; 939 + 940 + out_err: 941 + btf__free(base_btf); 942 + btf__free(btf); 943 + obj->base_btf = NULL; 944 + obj->btf = NULL; 945 + 1277 946 return err; 1278 947 } 1279 948 ··· 1455 1054 }; 1456 1055 const char *btfids_path = NULL; 1457 1056 bool fatal_warnings = false; 1057 + bool resolve_btfids = true; 1458 1058 char out_path[PATH_MAX]; 1459 1059 1460 1060 struct option btfid_options[] = { ··· 1485 1083 if (btfids_path) 1486 1084 return patch_btfids(btfids_path, obj.path); 1487 1085 1488 - if (load_btf(&obj)) 1489 - goto out; 1490 - 1491 - if (sort_btf_by_name(obj.btf)) 1492 - goto out; 1493 - 1494 1086 if (elf_collect(&obj)) 1495 1087 goto out; 1496 1088 ··· 1495 1099 if (obj.efile.idlist_shndx == -1 || 1496 1100 obj.efile.symbols_shndx == -1) { 1497 1101 pr_debug("Cannot find .BTF_ids or symbols sections, skip symbols resolution\n"); 1498 - goto dump_btf; 1102 + resolve_btfids = false; 1499 1103 } 1500 1104 1501 - if (symbols_collect(&obj)) 1105 + if (resolve_btfids) 1106 + if (symbols_collect(&obj)) 1107 + goto out; 1108 + 1109 + if (load_btf(&obj)) 1502 1110 goto out; 1111 + 1112 + if (btf2btf(&obj)) 1113 + goto out; 1114 + 1115 + if (finalize_btf(&obj)) 1116 + goto out; 1117 + 1118 + if (!resolve_btfids) 1119 + goto dump_btf; 1503 1120 1504 1121 if (symbols_resolve(&obj)) 1505 1122 goto out;
+3 -3
tools/lib/bpf/bpf_helpers.h
··· 315 315 ___param, sizeof(___param)); \ 316 316 }) 317 317 318 - extern int bpf_stream_vprintk_impl(int stream_id, const char *fmt__str, const void *args, 319 - __u32 len__sz, void *aux__prog) __weak __ksym; 318 + extern int bpf_stream_vprintk(int stream_id, const char *fmt__str, const void *args, 319 + __u32 len__sz) __weak __ksym; 320 320 321 321 #define bpf_stream_printk(stream_id, fmt, args...) \ 322 322 ({ \ ··· 328 328 ___bpf_fill(___param, args); \ 329 329 _Pragma("GCC diagnostic pop") \ 330 330 \ 331 - bpf_stream_vprintk_impl(stream_id, ___fmt, ___param, sizeof(___param), NULL); \ 331 + bpf_stream_vprintk(stream_id, ___fmt, ___param, sizeof(___param)); \ 332 332 }) 333 333 334 334 /* Use __bpf_printk when bpf_printk call has 3 or fewer fmt args
-5
tools/testing/selftests/bpf/bpf_experimental.h
··· 580 580 581 581 extern int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) __weak __ksym; 582 582 extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym; 583 - extern int bpf_wq_set_callback_impl(struct bpf_wq *wq, 584 - int (callback_fn)(void *map, int *key, void *value), 585 - unsigned int flags__k, void *aux__ign) __ksym; 586 - #define bpf_wq_set_callback(timer, cb, flags) \ 587 - bpf_wq_set_callback_impl(timer, cb, flags, NULL) 588 583 589 584 struct bpf_iter_kmem_cache; 590 585 extern int bpf_iter_kmem_cache_new(struct bpf_iter_kmem_cache *it) __weak __ksym;
+10
tools/testing/selftests/bpf/prog_tests/kfunc_implicit_args.c
··· 1 + // SPDX-License-Identifier: GPL-2.0 2 + /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ 3 + 4 + #include <test_progs.h> 5 + #include "kfunc_implicit_args.skel.h" 6 + 7 + void test_kfunc_implicit_args(void) 8 + { 9 + RUN_TESTS(kfunc_implicit_args); 10 + }
+1 -1
tools/testing/selftests/bpf/progs/file_reader.c
··· 77 77 err = 1; 78 78 return 0; 79 79 } 80 - bpf_task_work_schedule_signal_impl(task, &work->tw, &arrmap, task_work_callback, NULL); 80 + bpf_task_work_schedule_signal(task, &work->tw, &arrmap, task_work_callback); 81 81 return 0; 82 82 } 83 83
+41
tools/testing/selftests/bpf/progs/kfunc_implicit_args.c
··· 1 + // SPDX-License-Identifier: GPL-2.0 2 + /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ 3 + 4 + #include <vmlinux.h> 5 + #include <bpf/bpf_helpers.h> 6 + #include "bpf_misc.h" 7 + 8 + extern int bpf_kfunc_implicit_arg(int a) __weak __ksym; 9 + extern int bpf_kfunc_implicit_arg_impl(int a, struct bpf_prog_aux *aux) __weak __ksym; /* illegal */ 10 + extern int bpf_kfunc_implicit_arg_legacy(int a, int b) __weak __ksym; 11 + extern int bpf_kfunc_implicit_arg_legacy_impl(int a, int b, struct bpf_prog_aux *aux) __weak __ksym; 12 + 13 + char _license[] SEC("license") = "GPL"; 14 + 15 + SEC("syscall") 16 + __retval(5) 17 + int test_kfunc_implicit_arg(void *ctx) 18 + { 19 + return bpf_kfunc_implicit_arg(5); 20 + } 21 + 22 + SEC("syscall") 23 + __failure __msg("cannot find address for kernel function bpf_kfunc_implicit_arg_impl") 24 + int test_kfunc_implicit_arg_impl_illegal(void *ctx) 25 + { 26 + return bpf_kfunc_implicit_arg_impl(5, NULL); 27 + } 28 + 29 + SEC("syscall") 30 + __retval(7) 31 + int test_kfunc_implicit_arg_legacy(void *ctx) 32 + { 33 + return bpf_kfunc_implicit_arg_legacy(3, 4); 34 + } 35 + 36 + SEC("syscall") 37 + __retval(11) 38 + int test_kfunc_implicit_arg_legacy_impl(void *ctx) 39 + { 40 + return bpf_kfunc_implicit_arg_legacy_impl(5, 6, NULL); 41 + }
+3 -3
tools/testing/selftests/bpf/progs/stream_fail.c
··· 10 10 __failure __msg("Possibly NULL pointer passed") 11 11 int stream_vprintk_null_arg(void *ctx) 12 12 { 13 - bpf_stream_vprintk_impl(BPF_STDOUT, "", NULL, 0, NULL); 13 + bpf_stream_vprintk(BPF_STDOUT, "", NULL, 0); 14 14 return 0; 15 15 } 16 16 ··· 18 18 __failure __msg("R3 type=scalar expected=") 19 19 int stream_vprintk_scalar_arg(void *ctx) 20 20 { 21 - bpf_stream_vprintk_impl(BPF_STDOUT, "", (void *)46, 0, NULL); 21 + bpf_stream_vprintk(BPF_STDOUT, "", (void *)46, 0); 22 22 return 0; 23 23 } 24 24 ··· 26 26 __failure __msg("arg#1 doesn't point to a const string") 27 27 int stream_vprintk_string_arg(void *ctx) 28 28 { 29 - bpf_stream_vprintk_impl(BPF_STDOUT, ctx, NULL, 0, NULL); 29 + bpf_stream_vprintk(BPF_STDOUT, ctx, NULL, 0); 30 30 return 0; 31 31 } 32 32
+4 -4
tools/testing/selftests/bpf/progs/struct_ops_assoc.c
··· 32 32 if (!test_pid || task->pid != test_pid) 33 33 return 0; 34 34 35 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 35 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 36 36 if (ret != MAP_A_MAGIC) 37 37 test_err_a++; 38 38 ··· 45 45 struct st_ops_args args = {}; 46 46 int ret; 47 47 48 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 48 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 49 49 if (ret != MAP_A_MAGIC) 50 50 test_err_a++; 51 51 ··· 79 79 if (!test_pid || task->pid != test_pid) 80 80 return 0; 81 81 82 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 82 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 83 83 if (ret != MAP_B_MAGIC) 84 84 test_err_b++; 85 85 ··· 92 92 struct st_ops_args args = {}; 93 93 int ret; 94 94 95 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 95 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 96 96 if (ret != MAP_B_MAGIC) 97 97 test_err_b++; 98 98
+2 -2
tools/testing/selftests/bpf/progs/struct_ops_assoc_in_timer.c
··· 31 31 struct st_ops_args args = {}; 32 32 33 33 recur++; 34 - timer_test_1_ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 34 + timer_test_1_ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 35 35 recur--; 36 36 37 37 timer_cb_run++; ··· 64 64 struct st_ops_args args = {}; 65 65 int ret; 66 66 67 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 67 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 68 68 if (ret != MAP_MAGIC) 69 69 test_err++; 70 70
+3 -3
tools/testing/selftests/bpf/progs/struct_ops_assoc_reuse.c
··· 23 23 24 24 if (!recur) { 25 25 recur++; 26 - ret = bpf_kfunc_multi_st_ops_test_1_impl(args, NULL); 26 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(args); 27 27 if (ret != -1) 28 28 test_err_a++; 29 29 recur--; ··· 40 40 struct st_ops_args args = {}; 41 41 int ret; 42 42 43 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 43 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 44 44 if (ret != MAP_A_MAGIC) 45 45 test_err_a++; 46 46 ··· 62 62 struct st_ops_args args = {}; 63 63 int ret; 64 64 65 - ret = bpf_kfunc_multi_st_ops_test_1_impl(&args, NULL); 65 + ret = bpf_kfunc_multi_st_ops_test_1_assoc(&args); 66 66 if (ret != MAP_A_MAGIC) 67 67 test_err_b++; 68 68
+3 -4
tools/testing/selftests/bpf/progs/task_work.c
··· 65 65 work = bpf_map_lookup_elem(&hmap, &key); 66 66 if (!work) 67 67 return 0; 68 - 69 - bpf_task_work_schedule_resume_impl(task, &work->tw, &hmap, process_work, NULL); 68 + bpf_task_work_schedule_resume(task, &work->tw, &hmap, process_work); 70 69 return 0; 71 70 } 72 71 ··· 79 80 work = bpf_map_lookup_elem(&arrmap, &key); 80 81 if (!work) 81 82 return 0; 82 - bpf_task_work_schedule_signal_impl(task, &work->tw, &arrmap, process_work, NULL); 83 + bpf_task_work_schedule_signal(task, &work->tw, &arrmap, process_work); 83 84 return 0; 84 85 } 85 86 ··· 101 102 work = bpf_map_lookup_elem(&lrumap, &key); 102 103 if (!work || work->data[0]) 103 104 return 0; 104 - bpf_task_work_schedule_resume_impl(task, &work->tw, &lrumap, process_work, NULL); 105 + bpf_task_work_schedule_resume(task, &work->tw, &lrumap, process_work); 105 106 return 0; 106 107 }
+4 -4
tools/testing/selftests/bpf/progs/task_work_fail.c
··· 53 53 work = bpf_map_lookup_elem(&arrmap, &key); 54 54 if (!work) 55 55 return 0; 56 - bpf_task_work_schedule_resume_impl(task, &work->tw, &hmap, process_work, NULL); 56 + bpf_task_work_schedule_resume(task, &work->tw, &hmap, process_work); 57 57 return 0; 58 58 } 59 59 ··· 65 65 struct bpf_task_work tw; 66 66 67 67 task = bpf_get_current_task_btf(); 68 - bpf_task_work_schedule_resume_impl(task, &tw, &hmap, process_work, NULL); 68 + bpf_task_work_schedule_resume(task, &tw, &hmap, process_work); 69 69 return 0; 70 70 } 71 71 ··· 76 76 struct task_struct *task; 77 77 78 78 task = bpf_get_current_task_btf(); 79 - bpf_task_work_schedule_resume_impl(task, NULL, &hmap, process_work, NULL); 79 + bpf_task_work_schedule_resume(task, NULL, &hmap, process_work); 80 80 return 0; 81 81 } 82 82 ··· 91 91 work = bpf_map_lookup_elem(&arrmap, &key); 92 92 if (!work) 93 93 return 0; 94 - bpf_task_work_schedule_resume_impl(task, &work->tw, NULL, process_work, NULL); 94 + bpf_task_work_schedule_resume(task, &work->tw, NULL, process_work); 95 95 return 0; 96 96 }
+2 -2
tools/testing/selftests/bpf/progs/task_work_stress.c
··· 51 51 if (!work) 52 52 return 0; 53 53 } 54 - err = bpf_task_work_schedule_signal_impl(bpf_get_current_task_btf(), &work->tw, &hmap, 55 - process_work, NULL); 54 + err = bpf_task_work_schedule_signal(bpf_get_current_task_btf(), &work->tw, &hmap, 55 + process_work); 56 56 if (err) 57 57 __sync_fetch_and_add(&schedule_error, 1); 58 58 else
+4 -4
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c
··· 96 96 97 97 if (bpf_wq_init(&val->w, &wq_map, 0) != 0) 98 98 return 0; 99 - if (bpf_wq_set_callback_impl(&val->w, wq_cb, 0, NULL) != 0) 99 + if (bpf_wq_set_callback(&val->w, wq_cb, 0) != 0) 100 100 return 0; 101 101 return 0; 102 102 } ··· 114 114 115 115 if (bpf_wq_init(&val->w, &wq_map, 0) != 0) 116 116 return 0; 117 - if (bpf_wq_set_callback_impl(&val->w, wq_cb, 0, NULL) != 0) 117 + if (bpf_wq_set_callback(&val->w, wq_cb, 0) != 0) 118 118 return 0; 119 119 return 0; 120 120 } ··· 156 156 if (!task) 157 157 return 0; 158 158 159 - bpf_task_work_schedule_resume_impl(task, &val->tw, &task_work_map, task_work_cb, NULL); 159 + bpf_task_work_schedule_resume(task, &val->tw, &task_work_map, task_work_cb); 160 160 return 0; 161 161 } 162 162 ··· 176 176 if (!task) 177 177 return 0; 178 178 179 - bpf_task_work_schedule_resume_impl(task, &val->tw, &task_work_map, task_work_cb, NULL); 179 + bpf_task_work_schedule_resume(task, &val->tw, &task_work_map, task_work_cb); 180 180 return 0; 181 181 }
+2 -2
tools/testing/selftests/bpf/progs/wq_failures.c
··· 97 97 /* check that the first argument of bpf_wq_set_callback() 98 98 * is a correct bpf_wq pointer. 99 99 */ 100 - __msg(": (85) call bpf_wq_set_callback_impl#") /* anchor message */ 100 + __msg(": (85) call bpf_wq_set_callback#") /* anchor message */ 101 101 __msg("arg#0 doesn't point to a map value") 102 102 long test_wrong_wq_pointer(void *ctx) 103 103 { ··· 123 123 /* check that the first argument of bpf_wq_set_callback() 124 124 * is a correct bpf_wq pointer. 125 125 */ 126 - __msg(": (85) call bpf_wq_set_callback_impl#") /* anchor message */ 126 + __msg(": (85) call bpf_wq_set_callback#") /* anchor message */ 127 127 __msg("off 1 doesn't point to 'struct bpf_wq' that is at 0") 128 128 long test_wrong_wq_pointer_offset(void *ctx) 129 129 {
+30 -5
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
··· 1140 1140 } 1141 1141 1142 1142 __bpf_kfunc int bpf_kfunc_multi_st_ops_test_1(struct st_ops_args *args, u32 id); 1143 - __bpf_kfunc int bpf_kfunc_multi_st_ops_test_1_impl(struct st_ops_args *args, void *aux_prog); 1143 + __bpf_kfunc int bpf_kfunc_multi_st_ops_test_1_assoc(struct st_ops_args *args, struct bpf_prog_aux *aux); 1144 + 1145 + __bpf_kfunc int bpf_kfunc_implicit_arg(int a, struct bpf_prog_aux *aux); 1146 + __bpf_kfunc int bpf_kfunc_implicit_arg_legacy(int a, int b, struct bpf_prog_aux *aux); 1147 + __bpf_kfunc int bpf_kfunc_implicit_arg_legacy_impl(int a, int b, struct bpf_prog_aux *aux); 1144 1148 1145 1149 BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) 1146 1150 BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) ··· 1187 1183 BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_pro_epilogue, KF_SLEEPABLE) 1188 1184 BTF_ID_FLAGS(func, bpf_kfunc_st_ops_inc10) 1189 1185 BTF_ID_FLAGS(func, bpf_kfunc_multi_st_ops_test_1) 1190 - BTF_ID_FLAGS(func, bpf_kfunc_multi_st_ops_test_1_impl) 1186 + BTF_ID_FLAGS(func, bpf_kfunc_multi_st_ops_test_1_assoc, KF_IMPLICIT_ARGS) 1187 + BTF_ID_FLAGS(func, bpf_kfunc_implicit_arg, KF_IMPLICIT_ARGS) 1188 + BTF_ID_FLAGS(func, bpf_kfunc_implicit_arg_legacy, KF_IMPLICIT_ARGS) 1189 + BTF_ID_FLAGS(func, bpf_kfunc_implicit_arg_legacy_impl) 1191 1190 BTF_KFUNCS_END(bpf_testmod_check_kfunc_ids) 1192 1191 1193 1192 static int bpf_testmod_ops_init(struct btf *btf) ··· 1669 1662 } 1670 1663 1671 1664 /* Call test_1() of the associated struct_ops map */ 1672 - int bpf_kfunc_multi_st_ops_test_1_impl(struct st_ops_args *args, void *aux__prog) 1665 + int bpf_kfunc_multi_st_ops_test_1_assoc(struct st_ops_args *args, struct bpf_prog_aux *aux) 1673 1666 { 1674 - struct bpf_prog_aux *prog_aux = (struct bpf_prog_aux *)aux__prog; 1675 1667 struct bpf_testmod_multi_st_ops *st_ops; 1676 1668 int ret = -1; 1677 1669 1678 - st_ops = (struct bpf_testmod_multi_st_ops *)bpf_prog_get_assoc_struct_ops(prog_aux); 1670 + st_ops = (struct bpf_testmod_multi_st_ops *)bpf_prog_get_assoc_struct_ops(aux); 1679 1671 if (st_ops) 1680 1672 ret = st_ops->test_1(args); 1681 1673 1682 1674 return ret; 1675 + } 1676 + 1677 + int bpf_kfunc_implicit_arg(int a, struct bpf_prog_aux *aux) 1678 + { 1679 + if (aux && a > 0) 1680 + return a; 1681 + return -EINVAL; 1682 + } 1683 + 1684 + int bpf_kfunc_implicit_arg_legacy(int a, int b, struct bpf_prog_aux *aux) 1685 + { 1686 + if (aux) 1687 + return a + b; 1688 + return -EINVAL; 1689 + } 1690 + 1691 + int bpf_kfunc_implicit_arg_legacy_impl(int a, int b, struct bpf_prog_aux *aux) 1692 + { 1693 + return bpf_kfunc_implicit_arg_legacy(a, b, aux); 1683 1694 } 1684 1695 1685 1696 static int multi_st_ops_reg(void *kdata, struct bpf_link *link)
+4 -2
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
··· 161 161 struct task_struct *bpf_kfunc_ret_rcu_test(void) __ksym; 162 162 int *bpf_kfunc_ret_rcu_test_nostruct(int rdonly_buf_size) __ksym; 163 163 164 - int bpf_kfunc_multi_st_ops_test_1(struct st_ops_args *args, u32 id) __ksym; 165 - int bpf_kfunc_multi_st_ops_test_1_impl(struct st_ops_args *args, void *aux__prog) __ksym; 164 + #ifndef __KERNEL__ 165 + extern int bpf_kfunc_multi_st_ops_test_1(struct st_ops_args *args, u32 id) __weak __ksym; 166 + extern int bpf_kfunc_multi_st_ops_test_1_assoc(struct st_ops_args *args) __weak __ksym; 167 + #endif 166 168 167 169 #endif /* _BPF_TESTMOD_KFUNC_H */
+3 -5
tools/testing/selftests/hid/progs/hid_bpf_helpers.h
··· 116 116 /* bpf_wq implementation */ 117 117 extern int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) __weak __ksym; 118 118 extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym; 119 - extern int bpf_wq_set_callback_impl(struct bpf_wq *wq, 120 - int (callback_fn)(void *map, int *key, void *wq), 121 - unsigned int flags__k, void *aux__ign) __weak __ksym; 122 - #define bpf_wq_set_callback(timer, cb, flags) \ 123 - bpf_wq_set_callback_impl(timer, cb, flags, NULL) 119 + extern int bpf_wq_set_callback(struct bpf_wq *wq, 120 + int (*callback_fn)(void *, int *, void *), 121 + unsigned int flags) __weak __ksym; 124 122 125 123 #endif /* __HID_BPF_HELPERS_H */