bpf: Verifier support for KF_IMPLICIT_ARGS

A kernel function bpf_foo marked with KF_IMPLICIT_ARGS flag is
expected to have two associated types in BTF:
* `bpf_foo` with a function prototype that omits implicit arguments
* `bpf_foo_impl` with a function prototype that matches the kernel
declaration of `bpf_foo`, but doesn't have a ksym associated with
its name

In order to support kfuncs with implicit arguments, the verifier has
to know how to resolve a call of `bpf_foo` to the correct BTF function
prototype and address.

To implement this, in add_kfunc_call() kfunc flags are checked for
KF_IMPLICIT_ARGS. For such kfuncs a BTF func prototype is adjusted to
the one found for `bpf_foo_impl` (func_name + "_impl" suffix, by
convention) function in BTF.

This effectively changes the signature of the `bpf_foo` kfunc in the
context of verification: from one without implicit args to the one
with full argument list.

The values of implicit arguments by design are provided by the
verifier, and so they can only be of particular types. In this patch
the only allowed implicit arg type is a pointer to struct
bpf_prog_aux.

In order for the verifier to correctly set an implicit bpf_prog_aux
arg value at runtime, is_kfunc_arg_prog() is extended to check for the
arg type. At a point when prog arg is determined in check_kfunc_args()
the kfunc with implicit args already has a prototype with full
argument list, so the existing value patch mechanism just works.

If a new kfunc with KF_IMPLICIT_ARG is declared for an existing kfunc
that uses a __prog argument (a legacy case), the prototype
substitution works in exactly the same way, assuming the kfunc follows
the _impl naming convention. The difference is only in how _impl
prototype is added to the BTF, which is not the verifier's
concern. See a subsequent resolve_btfids patch for details.

__prog suffix is still supported at this point, but will be removed in
a subsequent patch, after current users are moved to KF_IMPLICIT_ARGS.

Introduction of KF_IMPLICIT_ARGS revealed an issue with zero-extension
tracking, because an explicit rX = 0 in place of the verifier-supplied
argument is now absent if the arg is implicit (the BPF prog doesn't
pass a dummy NULL anymore). To mitigate this, reset the subreg_def of
all caller saved registers in check_kfunc_call() [1].

[1] https://lore.kernel.org/bpf/b4a760ef828d40dac7ea6074d39452bb0dc82caa.camel@gmail.com/

Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/20260120222638.3976562-4-ihor.solodrai@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

authored by

Ihor Solodrai and committed by

Alexei Starovoitov 3 months ago 64e13605 08ca87d6

+55 -4

2 changed files

expand all

include

linux

btf.h

kernel

bpf

verifier.c

include/linux/btf.h

··· 78 78 #define KF_ARENA_RET (1 << 13) /* kfunc returns an arena pointer */ 79 79 #define KF_ARENA_ARG1 (1 << 14) /* kfunc takes an arena pointer as its first argument */ 80 80 #define KF_ARENA_ARG2 (1 << 15) /* kfunc takes an arena pointer as its second argument */ 81 + #define KF_IMPLICIT_ARGS (1 << 16) /* kfunc has implicit arguments supplied by the verifier */ 81 82 82 83 /* 83 84 * Tag marking a kernel function as a kfunc. This is meant to minimize the

+54 -4

kernel/bpf/verifier.c

··· 3271 3271 return btf_vmlinux ?: ERR_PTR(-ENOENT); 3272 3272 } 3273 3273 3274 + #define KF_IMPL_SUFFIX "_impl" 3275 + 3276 + static const struct btf_type *find_kfunc_impl_proto(struct bpf_verifier_env *env, 3277 + struct btf *btf, 3278 + const char *func_name) 3279 + { 3280 + char *buf = env->tmp_str_buf; 3281 + const struct btf_type *func; 3282 + s32 impl_id; 3283 + int len; 3284 + 3285 + len = snprintf(buf, TMP_STR_BUF_LEN, "%s%s", func_name, KF_IMPL_SUFFIX); 3286 + if (len < 0 || len >= TMP_STR_BUF_LEN) { 3287 + verbose(env, "function name %s%s is too long\n", func_name, KF_IMPL_SUFFIX); 3288 + return NULL; 3289 + } 3290 + 3291 + impl_id = btf_find_by_name_kind(btf, buf, BTF_KIND_FUNC); 3292 + if (impl_id <= 0) { 3293 + verbose(env, "cannot find function %s in BTF\n", buf); 3294 + return NULL; 3295 + } 3296 + 3297 + func = btf_type_by_id(btf, impl_id); 3298 + 3299 + return btf_type_by_id(btf, func->type); 3300 + } 3301 + 3274 3302 static int fetch_kfunc_meta(struct bpf_verifier_env *env, 3275 3303 s32 func_id, 3276 3304 s16 offset, ··· 3336 3308 } 3337 3309 3338 3310 func_name = btf_name_by_offset(btf, func->name_off); 3339 - func_proto = btf_type_by_id(btf, func->type); 3311 + 3312 + /* 3313 + * An actual prototype of a kfunc with KF_IMPLICIT_ARGS flag 3314 + * can be found through the counterpart _impl kfunc. 3315 + */ 3316 + if (kfunc_flags && (*kfunc_flags & KF_IMPLICIT_ARGS)) 3317 + func_proto = find_kfunc_impl_proto(env, btf, func_name); 3318 + else 3319 + func_proto = btf_type_by_id(btf, func->type); 3320 + 3340 3321 if (!func_proto || !btf_type_is_func_proto(func_proto)) { 3341 3322 verbose(env, "kernel function btf_id %d does not have a valid func_proto\n", 3342 3323 func_id); ··· 12211 12174 return btf_param_match_suffix(btf, arg, "__irq_flag"); 12212 12175 } 12213 12176 12177 + static bool is_kfunc_arg_prog_aux(const struct btf *btf, const struct btf_param *arg); 12178 + 12214 12179 static bool is_kfunc_arg_prog(const struct btf *btf, const struct btf_param *arg) 12215 12180 { 12216 - return btf_param_match_suffix(btf, arg, "__prog"); 12181 + return btf_param_match_suffix(btf, arg, "__prog") || is_kfunc_arg_prog_aux(btf, arg); 12217 12182 } 12218 12183 12219 12184 static bool is_kfunc_arg_scalar_with_name(const struct btf *btf, ··· 12246 12207 KF_ARG_WORKQUEUE_ID, 12247 12208 KF_ARG_RES_SPIN_LOCK_ID, 12248 12209 KF_ARG_TASK_WORK_ID, 12210 + KF_ARG_PROG_AUX_ID 12249 12211 }; 12250 12212 12251 12213 BTF_ID_LIST(kf_arg_btf_ids) ··· 12258 12218 BTF_ID(struct, bpf_wq) 12259 12219 BTF_ID(struct, bpf_res_spin_lock) 12260 12220 BTF_ID(struct, bpf_task_work) 12221 + BTF_ID(struct, bpf_prog_aux) 12261 12222 12262 12223 static bool __is_kfunc_ptr_arg_type(const struct btf *btf, 12263 12224 const struct btf_param *arg, int type) ··· 12337 12296 return false; 12338 12297 12339 12298 return true; 12299 + } 12300 + 12301 + static bool is_kfunc_arg_prog_aux(const struct btf *btf, const struct btf_param *arg) 12302 + { 12303 + return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_PROG_AUX_ID); 12340 12304 } 12341 12305 12342 12306 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */ ··· 14223 14177 } 14224 14178 } 14225 14179 14226 - for (i = 0; i < CALLER_SAVED_REGS; i++) 14227 - mark_reg_not_init(env, regs, caller_saved[i]); 14180 + for (i = 0; i < CALLER_SAVED_REGS; i++) { 14181 + u32 regno = caller_saved[i]; 14182 + 14183 + mark_reg_not_init(env, regs, regno); 14184 + regs[regno].subreg_def = DEF_NOT_SUBREG; 14185 + } 14228 14186 14229 14187 /* Check return type */ 14230 14188 t = btf_type_skip_modifiers(desc_btf, meta.func_proto->type, NULL);

Configure Feed

Configure Feed