tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'optimize-kprobe-session-attachment-for-exact-function-names'

Andrey Grodzovsky says:

====================
Optimize kprobe.session attachment for exact function names

When libbpf attaches kprobe.session programs with exact function names
(the common case: SEC("kprobe.session/vfs_read")), the current code path
has two independent performance bottlenecks:

1. Userspace (libbpf): attach_kprobe_session() always parses
/proc/kallsyms to resolve function names, even when the name is exact
(no wildcards). This takes ~150ms per function.

2. Kernel (ftrace): ftrace_lookup_symbols() does a full O(N) linear scan
over ~200K kernel symbols via kallsyms_on_each_symbol(), decompressing
every symbol name, even when resolving a single symbol (cnt == 1).

This series optimizes libbpf side: libbpf detects exact function names
(no wildcards) in bpf_program__attach_kprobe_multi_opts() and bypasses
kallsyms parsing, passing the symbol directly to the kernel via syms[]
array. ESRCH is normalized to ENOENT for API consistency.

Selftests validates exact-name attachment via kprobe_multi_session.c
and error consistency between wildcard and exact paths
in test_attach_api_fails.

Changes since v3 [3]:
- Skip fast path when unique_match is set (Jiri Olsa, CI bot)

Changes since v2 [2]:
- Use if/else-if instead of goto (Jiri Olsa)
- Use syms = &pattern directly (Jiri Olsa)
- Drop unneeded pattern = NULL (Jiri Olsa)
- Revert cosmetic rename in attach_kprobe_session (Jiri Olsa)
- Remove "module symbols" from ftrace comment (CI bot)

Changes since v1 [1]:
- Move optimization into attach_kprobe_multi_opts (Jiri Olsa)
- Use ftrace_location as boolean check only (Jiri Olsa)
- Remove verbose perf rationale from comment (Steven Rostedt)
- Consolidate tests into existing subtests (Jiri Olsa)
- Delete standalone _syms.c and _errors.c files

[1] https://lore.kernel.org/bpf/20260223215113.924599-1-andrey.grodzovsky@crowdstrike.com/
[2] https://lore.kernel.org/bpf/20260226173342.3565919-1-andrey.grodzovsky@crowdstrike.com/
[3] https://lore.kernel.org/bpf/20260227204052.725813-1-andrey.grodzovsky@crowdstrike.com/
====================

Link: https://patch.msgid.link/20260302200837.317907-1-andrey.grodzovsky@crowdstrike.com
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>

Andrii Nakryiko 6dd780f9 748f9c68

+59 -3
+18 -1
tools/lib/bpf/libbpf.c
··· 12041 12041 if (addrs && syms) 12042 12042 return libbpf_err_ptr(-EINVAL); 12043 12043 12044 - if (pattern) { 12044 + /* 12045 + * Exact function name (no wildcards) without unique_match: 12046 + * bypass kallsyms parsing and pass the symbol directly to the 12047 + * kernel via syms[] array. When unique_match is set, fall 12048 + * through to the slow path which detects duplicate symbols. 12049 + */ 12050 + if (pattern && !strpbrk(pattern, "*?") && !unique_match) { 12051 + syms = &pattern; 12052 + cnt = 1; 12053 + } else if (pattern) { 12045 12054 if (has_available_filter_functions_addrs()) 12046 12055 err = libbpf_available_kprobes_parse(&res); 12047 12056 else ··· 12093 12084 link_fd = bpf_link_create(prog_fd, 0, attach_type, &lopts); 12094 12085 if (link_fd < 0) { 12095 12086 err = -errno; 12087 + /* 12088 + * Normalize error code: when exact name bypasses kallsyms 12089 + * parsing, kernel returns ESRCH from ftrace_lookup_symbols(). 12090 + * Convert to ENOENT for API consistency with the pattern 12091 + * matching path which returns ENOENT from userspace. 12092 + */ 12093 + if (err == -ESRCH) 12094 + err = -ENOENT; 12096 12095 pr_warn("prog '%s': failed to attach: %s\n", 12097 12096 prog->name, errstr(err)); 12098 12097 goto error;
+31 -2
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
··· 327 327 if (!ASSERT_EQ(saved_error, -E2BIG, "fail_6_error")) 328 328 goto cleanup; 329 329 330 + /* fail_7 - non-existent wildcard pattern (slow path) */ 331 + LIBBPF_OPTS_RESET(opts); 332 + 333 + link = bpf_program__attach_kprobe_multi_opts(skel->progs.test_kprobe_manual, 334 + "__nonexistent_func_xyz_*", 335 + &opts); 336 + saved_error = -errno; 337 + if (!ASSERT_ERR_PTR(link, "fail_7")) 338 + goto cleanup; 339 + 340 + if (!ASSERT_EQ(saved_error, -ENOENT, "fail_7_error")) 341 + goto cleanup; 342 + 343 + /* fail_8 - non-existent exact name (fast path), same error as wildcard */ 344 + link = bpf_program__attach_kprobe_multi_opts(skel->progs.test_kprobe_manual, 345 + "__nonexistent_func_xyz_123", 346 + &opts); 347 + saved_error = -errno; 348 + if (!ASSERT_ERR_PTR(link, "fail_8")) 349 + goto cleanup; 350 + 351 + if (!ASSERT_EQ(saved_error, -ENOENT, "fail_8_error")) 352 + goto cleanup; 353 + 330 354 cleanup: 331 355 bpf_link__destroy(link); 332 356 kprobe_multi__destroy(skel); ··· 379 355 ASSERT_OK(err, "test_run"); 380 356 ASSERT_EQ(topts.retval, 0, "test_run"); 381 357 382 - /* bpf_fentry_test1-4 trigger return probe, result is 2 */ 383 - for (i = 0; i < 4; i++) 358 + /* 359 + * bpf_fentry_test1 is hit by both the wildcard probe and the exact 360 + * name probe (test_kprobe_syms), so entry + return fires twice: 4. 361 + * bpf_fentry_test2-4 are hit only by the wildcard probe: 2. 362 + */ 363 + ASSERT_EQ(skel->bss->kprobe_session_result[0], 4, "kprobe_session_result"); 364 + for (i = 1; i < 4; i++) 384 365 ASSERT_EQ(skel->bss->kprobe_session_result[i], 2, "kprobe_session_result"); 385 366 386 367 /* bpf_fentry_test5-8 trigger only entry probe, result is 1 */
+10
tools/testing/selftests/bpf/progs/kprobe_multi_session.c
··· 76 76 { 77 77 return session_check(ctx); 78 78 } 79 + 80 + /* 81 + * Exact function name (no wildcards) - exercises the fast syms[] path 82 + * in bpf_program__attach_kprobe_multi_opts() which bypasses kallsyms parsing. 83 + */ 84 + SEC("kprobe.session/bpf_fentry_test1") 85 + int test_kprobe_syms(struct pt_regs *ctx) 86 + { 87 + return session_check(ctx); 88 + }