Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge branch 'kfunc-annotation'
David Vernet says:
====================
This is v3 of the patchset [0]. v2 can be found at [1].
[0]: https://lore.kernel.org/bpf/Y7kCsjBZ%2FFrsWW%2Fe@maniforge.lan/T/
[1]: https://lore.kernel.org/lkml/20230123171506.71995-1-void@manifault.com/
Changelog:
----------
v2 -> v3:
- Go back to the __bpf_kfunc approach from v1. The BPF_KFUNC macro
received pushback as it didn't match the more typical EXPORT_SYMBOL*
APIs used elsewhere in the kernel. It's the longer term plan, but for
now we're proposing something less controversial to fix kfuncs and BTF
encoding.
- Add __bpf_kfunc macro to newly added cpumask kfuncs.
- Add __bpf_kfunc macro to newly added XDP metadata kfuncs, which were
failing to be BTF encoded in the thread in [2].
- Update patch description(s) to reference the discussions in [2].
- Add a selftest that validates that a static kfunc with unused args is
properly BTF encoded and can be invoked.
[2]: https://lore.kernel.org/all/fe5d42d1-faad-d05e-99ad-1c2c04776950@oracle.com/
v1 -> v2:
- Wrap entire function signature in BPF_KFUNC macro instead of using
__bpf_kfunc tag (Kumar)
- Update all kfunc definitions to use this macro.
- Update kfuncs.rst documentation to describe and illustrate the macro.
- Also clean up a few small parts of kfuncs.rst, e.g. some grammar, and
in general making it a bit tighter.
====================
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
+176
-131
+17
-3
Documentation/bpf/kfuncs.rst
+17
-3
Documentation/bpf/kfuncs.rst
···
41
41
__diag_ignore_all("-Wmissing-prototypes",
42
42
"Global kfuncs as their definitions will be in BTF");
43
43
44
-
struct task_struct *bpf_find_get_task_by_vpid(pid_t nr)
44
+
__bpf_kfunc struct task_struct *bpf_find_get_task_by_vpid(pid_t nr)
45
45
{
46
46
return find_get_task_by_vpid(nr);
47
47
}
···
66
66
This annotation is used to indicate a memory and size pair in the argument list.
67
67
An example is given below::
68
68
69
-
void bpf_memzero(void *mem, int mem__sz)
69
+
__bpf_kfunc void bpf_memzero(void *mem, int mem__sz)
70
70
{
71
71
...
72
72
}
···
86
86
87
87
An example is given below::
88
88
89
-
void *bpf_obj_new(u32 local_type_id__k, ...)
89
+
__bpf_kfunc void *bpf_obj_new(u32 local_type_id__k, ...)
90
90
{
91
91
...
92
92
}
···
124
124
125
125
This set encodes the BTF ID of each kfunc listed above, and encodes the flags
126
126
along with it. Ofcourse, it is also allowed to specify no flags.
127
+
128
+
kfunc definitions should also always be annotated with the ``__bpf_kfunc``
129
+
macro. This prevents issues such as the compiler inlining the kfunc if it's a
130
+
static kernel function, or the function being elided in an LTO build as it's
131
+
not used in the rest of the kernel. Developers should not manually add
132
+
annotations to their kfunc to prevent these issues. If an annotation is
133
+
required to prevent such an issue with your kfunc, it is a bug and should be
134
+
added to the definition of the macro so that other kfuncs are similarly
135
+
protected. An example is given below::
136
+
137
+
__bpf_kfunc struct task_struct *bpf_get_task_pid(s32 pid)
138
+
{
139
+
...
140
+
}
127
141
128
142
2.4.1 KF_ACQUIRE flag
129
143
---------------------
+3
Documentation/conf.py
+3
Documentation/conf.py
+8
include/linux/btf.h
+8
include/linux/btf.h
···
73
73
#define KF_RCU (1 << 7) /* kfunc only takes rcu pointer arguments */
74
74
75
75
/*
76
+
* Tag marking a kernel function as a kfunc. This is meant to minimize the
77
+
* amount of copy-paste that kfunc authors have to include for correctness so
78
+
* as to avoid issues such as the compiler inlining or eliding either a static
79
+
* kfunc, or a global kfunc in an LTO build.
80
+
*/
81
+
#define __bpf_kfunc __used noinline
82
+
83
+
/*
76
84
* Return the name of the passed struct, if exists, or halt the build if for
77
85
* example the structure gets renamed. In this way, developers have to revisit
78
86
* the code using that structure name, and update it accordingly.
+30
-30
kernel/bpf/cpumask.c
+30
-30
kernel/bpf/cpumask.c
···
48
48
* bpf_cpumask_create() allocates memory using the BPF memory allocator, and
49
49
* will not block. It may return NULL if no memory is available.
50
50
*/
51
-
struct bpf_cpumask *bpf_cpumask_create(void)
51
+
__bpf_kfunc struct bpf_cpumask *bpf_cpumask_create(void)
52
52
{
53
53
struct bpf_cpumask *cpumask;
54
54
···
74
74
* must either be embedded in a map as a kptr, or freed with
75
75
* bpf_cpumask_release().
76
76
*/
77
-
struct bpf_cpumask *bpf_cpumask_acquire(struct bpf_cpumask *cpumask)
77
+
__bpf_kfunc struct bpf_cpumask *bpf_cpumask_acquire(struct bpf_cpumask *cpumask)
78
78
{
79
79
refcount_inc(&cpumask->usage);
80
80
return cpumask;
···
90
90
* kptr, or freed with bpf_cpumask_release(). This function may return NULL if
91
91
* no BPF cpumask was found in the specified map value.
92
92
*/
93
-
struct bpf_cpumask *bpf_cpumask_kptr_get(struct bpf_cpumask **cpumaskp)
93
+
__bpf_kfunc struct bpf_cpumask *bpf_cpumask_kptr_get(struct bpf_cpumask **cpumaskp)
94
94
{
95
95
struct bpf_cpumask *cpumask;
96
96
···
116
116
* reference of the BPF cpumask has been released, it is subsequently freed in
117
117
* an RCU callback in the BPF memory allocator.
118
118
*/
119
-
void bpf_cpumask_release(struct bpf_cpumask *cpumask)
119
+
__bpf_kfunc void bpf_cpumask_release(struct bpf_cpumask *cpumask)
120
120
{
121
121
if (!cpumask)
122
122
return;
···
135
135
* Find the index of the first nonzero bit of the cpumask. A struct bpf_cpumask
136
136
* pointer may be safely passed to this function.
137
137
*/
138
-
u32 bpf_cpumask_first(const struct cpumask *cpumask)
138
+
__bpf_kfunc u32 bpf_cpumask_first(const struct cpumask *cpumask)
139
139
{
140
140
return cpumask_first(cpumask);
141
141
}
···
148
148
* Find the index of the first unset bit of the cpumask. A struct bpf_cpumask
149
149
* pointer may be safely passed to this function.
150
150
*/
151
-
u32 bpf_cpumask_first_zero(const struct cpumask *cpumask)
151
+
__bpf_kfunc u32 bpf_cpumask_first_zero(const struct cpumask *cpumask)
152
152
{
153
153
return cpumask_first_zero(cpumask);
154
154
}
···
158
158
* @cpu: The CPU to be set in the cpumask.
159
159
* @cpumask: The BPF cpumask in which a bit is being set.
160
160
*/
161
-
void bpf_cpumask_set_cpu(u32 cpu, struct bpf_cpumask *cpumask)
161
+
__bpf_kfunc void bpf_cpumask_set_cpu(u32 cpu, struct bpf_cpumask *cpumask)
162
162
{
163
163
if (!cpu_valid(cpu))
164
164
return;
···
171
171
* @cpu: The CPU to be cleared from the cpumask.
172
172
* @cpumask: The BPF cpumask in which a bit is being cleared.
173
173
*/
174
-
void bpf_cpumask_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask)
174
+
__bpf_kfunc void bpf_cpumask_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask)
175
175
{
176
176
if (!cpu_valid(cpu))
177
177
return;
···
188
188
* * true - @cpu is set in the cpumask
189
189
* * false - @cpu was not set in the cpumask, or @cpu is an invalid cpu.
190
190
*/
191
-
bool bpf_cpumask_test_cpu(u32 cpu, const struct cpumask *cpumask)
191
+
__bpf_kfunc bool bpf_cpumask_test_cpu(u32 cpu, const struct cpumask *cpumask)
192
192
{
193
193
if (!cpu_valid(cpu))
194
194
return false;
···
205
205
* * true - @cpu is set in the cpumask
206
206
* * false - @cpu was not set in the cpumask, or @cpu is invalid.
207
207
*/
208
-
bool bpf_cpumask_test_and_set_cpu(u32 cpu, struct bpf_cpumask *cpumask)
208
+
__bpf_kfunc bool bpf_cpumask_test_and_set_cpu(u32 cpu, struct bpf_cpumask *cpumask)
209
209
{
210
210
if (!cpu_valid(cpu))
211
211
return false;
···
223
223
* * true - @cpu is set in the cpumask
224
224
* * false - @cpu was not set in the cpumask, or @cpu is invalid.
225
225
*/
226
-
bool bpf_cpumask_test_and_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask)
226
+
__bpf_kfunc bool bpf_cpumask_test_and_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask)
227
227
{
228
228
if (!cpu_valid(cpu))
229
229
return false;
···
235
235
* bpf_cpumask_setall() - Set all of the bits in a BPF cpumask.
236
236
* @cpumask: The BPF cpumask having all of its bits set.
237
237
*/
238
-
void bpf_cpumask_setall(struct bpf_cpumask *cpumask)
238
+
__bpf_kfunc void bpf_cpumask_setall(struct bpf_cpumask *cpumask)
239
239
{
240
240
cpumask_setall((struct cpumask *)cpumask);
241
241
}
···
244
244
* bpf_cpumask_clear() - Clear all of the bits in a BPF cpumask.
245
245
* @cpumask: The BPF cpumask being cleared.
246
246
*/
247
-
void bpf_cpumask_clear(struct bpf_cpumask *cpumask)
247
+
__bpf_kfunc void bpf_cpumask_clear(struct bpf_cpumask *cpumask)
248
248
{
249
249
cpumask_clear((struct cpumask *)cpumask);
250
250
}
···
261
261
*
262
262
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
263
263
*/
264
-
bool bpf_cpumask_and(struct bpf_cpumask *dst,
265
-
const struct cpumask *src1,
266
-
const struct cpumask *src2)
264
+
__bpf_kfunc bool bpf_cpumask_and(struct bpf_cpumask *dst,
265
+
const struct cpumask *src1,
266
+
const struct cpumask *src2)
267
267
{
268
268
return cpumask_and((struct cpumask *)dst, src1, src2);
269
269
}
···
276
276
*
277
277
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
278
278
*/
279
-
void bpf_cpumask_or(struct bpf_cpumask *dst,
280
-
const struct cpumask *src1,
281
-
const struct cpumask *src2)
279
+
__bpf_kfunc void bpf_cpumask_or(struct bpf_cpumask *dst,
280
+
const struct cpumask *src1,
281
+
const struct cpumask *src2)
282
282
{
283
283
cpumask_or((struct cpumask *)dst, src1, src2);
284
284
}
···
291
291
*
292
292
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
293
293
*/
294
-
void bpf_cpumask_xor(struct bpf_cpumask *dst,
295
-
const struct cpumask *src1,
296
-
const struct cpumask *src2)
294
+
__bpf_kfunc void bpf_cpumask_xor(struct bpf_cpumask *dst,
295
+
const struct cpumask *src1,
296
+
const struct cpumask *src2)
297
297
{
298
298
cpumask_xor((struct cpumask *)dst, src1, src2);
299
299
}
···
309
309
*
310
310
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
311
311
*/
312
-
bool bpf_cpumask_equal(const struct cpumask *src1, const struct cpumask *src2)
312
+
__bpf_kfunc bool bpf_cpumask_equal(const struct cpumask *src1, const struct cpumask *src2)
313
313
{
314
314
return cpumask_equal(src1, src2);
315
315
}
···
325
325
*
326
326
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
327
327
*/
328
-
bool bpf_cpumask_intersects(const struct cpumask *src1, const struct cpumask *src2)
328
+
__bpf_kfunc bool bpf_cpumask_intersects(const struct cpumask *src1, const struct cpumask *src2)
329
329
{
330
330
return cpumask_intersects(src1, src2);
331
331
}
···
341
341
*
342
342
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
343
343
*/
344
-
bool bpf_cpumask_subset(const struct cpumask *src1, const struct cpumask *src2)
344
+
__bpf_kfunc bool bpf_cpumask_subset(const struct cpumask *src1, const struct cpumask *src2)
345
345
{
346
346
return cpumask_subset(src1, src2);
347
347
}
···
356
356
*
357
357
* A struct bpf_cpumask pointer may be safely passed to @cpumask.
358
358
*/
359
-
bool bpf_cpumask_empty(const struct cpumask *cpumask)
359
+
__bpf_kfunc bool bpf_cpumask_empty(const struct cpumask *cpumask)
360
360
{
361
361
return cpumask_empty(cpumask);
362
362
}
···
371
371
*
372
372
* A struct bpf_cpumask pointer may be safely passed to @cpumask.
373
373
*/
374
-
bool bpf_cpumask_full(const struct cpumask *cpumask)
374
+
__bpf_kfunc bool bpf_cpumask_full(const struct cpumask *cpumask)
375
375
{
376
376
return cpumask_full(cpumask);
377
377
}
···
383
383
*
384
384
* A struct bpf_cpumask pointer may be safely passed to @src.
385
385
*/
386
-
void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src)
386
+
__bpf_kfunc void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src)
387
387
{
388
388
cpumask_copy((struct cpumask *)dst, src);
389
389
}
···
398
398
*
399
399
* A struct bpf_cpumask pointer may be safely passed to @src.
400
400
*/
401
-
u32 bpf_cpumask_any(const struct cpumask *cpumask)
401
+
__bpf_kfunc u32 bpf_cpumask_any(const struct cpumask *cpumask)
402
402
{
403
403
return cpumask_any(cpumask);
404
404
}
···
415
415
*
416
416
* struct bpf_cpumask pointers may be safely passed to @src1 and @src2.
417
417
*/
418
-
u32 bpf_cpumask_any_and(const struct cpumask *src1, const struct cpumask *src2)
418
+
__bpf_kfunc u32 bpf_cpumask_any_and(const struct cpumask *src1, const struct cpumask *src2)
419
419
{
420
420
return cpumask_any_and(src1, src2);
421
421
}
+19
-19
kernel/bpf/helpers.c
+19
-19
kernel/bpf/helpers.c
···
1776
1776
__diag_ignore_all("-Wmissing-prototypes",
1777
1777
"Global functions as their definitions will be in vmlinux BTF");
1778
1778
1779
-
void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
1779
+
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
1780
1780
{
1781
1781
struct btf_struct_meta *meta = meta__ign;
1782
1782
u64 size = local_type_id__k;
···
1790
1790
return p;
1791
1791
}
1792
1792
1793
-
void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
1793
+
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
1794
1794
{
1795
1795
struct btf_struct_meta *meta = meta__ign;
1796
1796
void *p = p__alloc;
···
1811
1811
tail ? list_add_tail(n, h) : list_add(n, h);
1812
1812
}
1813
1813
1814
-
void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node)
1814
+
__bpf_kfunc void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node)
1815
1815
{
1816
1816
return __bpf_list_add(node, head, false);
1817
1817
}
1818
1818
1819
-
void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node)
1819
+
__bpf_kfunc void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node)
1820
1820
{
1821
1821
return __bpf_list_add(node, head, true);
1822
1822
}
···
1834
1834
return (struct bpf_list_node *)n;
1835
1835
}
1836
1836
1837
-
struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
1837
+
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
1838
1838
{
1839
1839
return __bpf_list_del(head, false);
1840
1840
}
1841
1841
1842
-
struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
1842
+
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
1843
1843
{
1844
1844
return __bpf_list_del(head, true);
1845
1845
}
···
1850
1850
* bpf_task_release().
1851
1851
* @p: The task on which a reference is being acquired.
1852
1852
*/
1853
-
struct task_struct *bpf_task_acquire(struct task_struct *p)
1853
+
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
1854
1854
{
1855
1855
return get_task_struct(p);
1856
1856
}
···
1861
1861
* released by calling bpf_task_release().
1862
1862
* @p: The task on which a reference is being acquired.
1863
1863
*/
1864
-
struct task_struct *bpf_task_acquire_not_zero(struct task_struct *p)
1864
+
__bpf_kfunc struct task_struct *bpf_task_acquire_not_zero(struct task_struct *p)
1865
1865
{
1866
1866
/* For the time being this function returns NULL, as it's not currently
1867
1867
* possible to safely acquire a reference to a task with RCU protection
···
1913
1913
* be released by calling bpf_task_release().
1914
1914
* @pp: A pointer to a task kptr on which a reference is being acquired.
1915
1915
*/
1916
-
struct task_struct *bpf_task_kptr_get(struct task_struct **pp)
1916
+
__bpf_kfunc struct task_struct *bpf_task_kptr_get(struct task_struct **pp)
1917
1917
{
1918
1918
/* We must return NULL here until we have clarity on how to properly
1919
1919
* leverage RCU for ensuring a task's lifetime. See the comment above
···
1926
1926
* bpf_task_release - Release the reference acquired on a task.
1927
1927
* @p: The task on which a reference is being released.
1928
1928
*/
1929
-
void bpf_task_release(struct task_struct *p)
1929
+
__bpf_kfunc void bpf_task_release(struct task_struct *p)
1930
1930
{
1931
1931
if (!p)
1932
1932
return;
···
1941
1941
* calling bpf_cgroup_release().
1942
1942
* @cgrp: The cgroup on which a reference is being acquired.
1943
1943
*/
1944
-
struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
1944
+
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
1945
1945
{
1946
1946
cgroup_get(cgrp);
1947
1947
return cgrp;
···
1953
1953
* be released by calling bpf_cgroup_release().
1954
1954
* @cgrpp: A pointer to a cgroup kptr on which a reference is being acquired.
1955
1955
*/
1956
-
struct cgroup *bpf_cgroup_kptr_get(struct cgroup **cgrpp)
1956
+
__bpf_kfunc struct cgroup *bpf_cgroup_kptr_get(struct cgroup **cgrpp)
1957
1957
{
1958
1958
struct cgroup *cgrp;
1959
1959
···
1985
1985
* drops to 0.
1986
1986
* @cgrp: The cgroup on which a reference is being released.
1987
1987
*/
1988
-
void bpf_cgroup_release(struct cgroup *cgrp)
1988
+
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
1989
1989
{
1990
1990
if (!cgrp)
1991
1991
return;
···
2000
2000
* @cgrp: The cgroup for which we're performing a lookup.
2001
2001
* @level: The level of ancestor to look up.
2002
2002
*/
2003
-
struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
2003
+
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
2004
2004
{
2005
2005
struct cgroup *ancestor;
2006
2006
···
2019
2019
* stored in a map, or released with bpf_task_release().
2020
2020
* @pid: The pid of the task being looked up.
2021
2021
*/
2022
-
struct task_struct *bpf_task_from_pid(s32 pid)
2022
+
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
2023
2023
{
2024
2024
struct task_struct *p;
2025
2025
···
2032
2032
return p;
2033
2033
}
2034
2034
2035
-
void *bpf_cast_to_kern_ctx(void *obj)
2035
+
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
2036
2036
{
2037
2037
return obj;
2038
2038
}
2039
2039
2040
-
void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)
2040
+
__bpf_kfunc void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)
2041
2041
{
2042
2042
return obj__ign;
2043
2043
}
2044
2044
2045
-
void bpf_rcu_read_lock(void)
2045
+
__bpf_kfunc void bpf_rcu_read_lock(void)
2046
2046
{
2047
2047
rcu_read_lock();
2048
2048
}
2049
2049
2050
-
void bpf_rcu_read_unlock(void)
2050
+
__bpf_kfunc void bpf_rcu_read_unlock(void)
2051
2051
{
2052
2052
rcu_read_unlock();
2053
2053
}
+2
-2
kernel/cgroup/rstat.c
+2
-2
kernel/cgroup/rstat.c
···
26
26
* rstat_cpu->updated_children list. See the comment on top of
27
27
* cgroup_rstat_cpu definition for details.
28
28
*/
29
-
void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
29
+
__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
30
30
{
31
31
raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu);
32
32
unsigned long flags;
···
231
231
*
232
232
* This function may block.
233
233
*/
234
-
void cgroup_rstat_flush(struct cgroup *cgrp)
234
+
__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp)
235
235
{
236
236
might_sleep();
237
237
+2
-1
kernel/kexec_core.c
+2
-1
kernel/kexec_core.c
···
6
6
7
7
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
8
8
9
+
#include <linux/btf.h>
9
10
#include <linux/capability.h>
10
11
#include <linux/mm.h>
11
12
#include <linux/file.h>
···
976
975
}
977
976
STACK_FRAME_NON_STANDARD(__crash_kexec);
978
977
979
-
void crash_kexec(struct pt_regs *regs)
978
+
__bpf_kfunc void crash_kexec(struct pt_regs *regs)
980
979
{
981
980
int old_cpu, this_cpu;
982
981
+4
-4
kernel/trace/bpf_trace.c
+4
-4
kernel/trace/bpf_trace.c
···
1236
1236
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
1237
1237
* NULL pointer otherwise.
1238
1238
*/
1239
-
struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
1239
+
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
1240
1240
{
1241
1241
key_ref_t key_ref;
1242
1242
struct bpf_key *bkey;
···
1285
1285
* Return: a bpf_key pointer with an invalid key pointer set from the
1286
1286
* pre-determined ID on success, a NULL pointer otherwise
1287
1287
*/
1288
-
struct bpf_key *bpf_lookup_system_key(u64 id)
1288
+
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
1289
1289
{
1290
1290
struct bpf_key *bkey;
1291
1291
···
1309
1309
* Decrement the reference count of the key inside *bkey*, if the pointer
1310
1310
* is valid, and free *bkey*.
1311
1311
*/
1312
-
void bpf_key_put(struct bpf_key *bkey)
1312
+
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
1313
1313
{
1314
1314
if (bkey->has_ref)
1315
1315
key_put(bkey->key);
···
1329
1329
*
1330
1330
* Return: 0 on success, a negative value on error.
1331
1331
*/
1332
-
int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr,
1332
+
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr,
1333
1333
struct bpf_dynptr_kern *sig_ptr,
1334
1334
struct bpf_key *trusted_keyring)
1335
1335
{
+35
-26
net/bpf/test_run.c
+35
-26
net/bpf/test_run.c
···
484
484
__diag_push();
485
485
__diag_ignore_all("-Wmissing-prototypes",
486
486
"Global functions as their definitions will be in vmlinux BTF");
487
-
int noinline bpf_fentry_test1(int a)
487
+
__bpf_kfunc int bpf_fentry_test1(int a)
488
488
{
489
489
return a + 1;
490
490
}
···
529
529
return (long)arg->a;
530
530
}
531
531
532
-
int noinline bpf_modify_return_test(int a, int *b)
532
+
__bpf_kfunc int bpf_modify_return_test(int a, int *b)
533
533
{
534
534
*b += 1;
535
535
return a + *b;
536
536
}
537
537
538
-
u64 noinline bpf_kfunc_call_test1(struct sock *sk, u32 a, u64 b, u32 c, u64 d)
538
+
__bpf_kfunc u64 bpf_kfunc_call_test1(struct sock *sk, u32 a, u64 b, u32 c, u64 d)
539
539
{
540
540
return a + b + c + d;
541
541
}
542
542
543
-
int noinline bpf_kfunc_call_test2(struct sock *sk, u32 a, u32 b)
543
+
__bpf_kfunc int bpf_kfunc_call_test2(struct sock *sk, u32 a, u32 b)
544
544
{
545
545
return a + b;
546
546
}
547
547
548
-
struct sock * noinline bpf_kfunc_call_test3(struct sock *sk)
548
+
__bpf_kfunc struct sock *bpf_kfunc_call_test3(struct sock *sk)
549
549
{
550
550
return sk;
551
551
}
···
582
582
.cnt = REFCOUNT_INIT(1),
583
583
};
584
584
585
-
noinline struct prog_test_ref_kfunc *
585
+
__bpf_kfunc struct prog_test_ref_kfunc *
586
586
bpf_kfunc_call_test_acquire(unsigned long *scalar_ptr)
587
587
{
588
588
refcount_inc(&prog_test_struct.cnt);
589
589
return &prog_test_struct;
590
590
}
591
591
592
-
noinline struct prog_test_member *
592
+
__bpf_kfunc struct prog_test_member *
593
593
bpf_kfunc_call_memb_acquire(void)
594
594
{
595
595
WARN_ON_ONCE(1);
596
596
return NULL;
597
597
}
598
598
599
-
noinline void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p)
599
+
__bpf_kfunc void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p)
600
600
{
601
601
if (!p)
602
602
return;
···
604
604
refcount_dec(&p->cnt);
605
605
}
606
606
607
-
noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p)
607
+
__bpf_kfunc void bpf_kfunc_call_memb_release(struct prog_test_member *p)
608
608
{
609
609
}
610
610
611
-
noinline void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p)
611
+
__bpf_kfunc void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p)
612
612
{
613
613
WARN_ON_ONCE(1);
614
614
}
···
621
621
return (int *)p;
622
622
}
623
623
624
-
noinline int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size)
624
+
__bpf_kfunc int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p,
625
+
const int rdwr_buf_size)
625
626
{
626
627
return __bpf_kfunc_call_test_get_mem(p, rdwr_buf_size);
627
628
}
628
629
629
-
noinline int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size)
630
+
__bpf_kfunc int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p,
631
+
const int rdonly_buf_size)
630
632
{
631
633
return __bpf_kfunc_call_test_get_mem(p, rdonly_buf_size);
632
634
}
···
638
636
* Acquire functions must return struct pointers, so these ones are
639
637
* failing.
640
638
*/
641
-
noinline int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size)
639
+
__bpf_kfunc int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p,
640
+
const int rdonly_buf_size)
642
641
{
643
642
return __bpf_kfunc_call_test_get_mem(p, rdonly_buf_size);
644
643
}
645
644
646
-
noinline void bpf_kfunc_call_int_mem_release(int *p)
645
+
__bpf_kfunc void bpf_kfunc_call_int_mem_release(int *p)
647
646
{
648
647
}
649
648
650
-
noinline struct prog_test_ref_kfunc *
649
+
__bpf_kfunc struct prog_test_ref_kfunc *
651
650
bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **pp, int a, int b)
652
651
{
653
652
struct prog_test_ref_kfunc *p = READ_ONCE(*pp);
···
697
694
char arr2[];
698
695
};
699
696
700
-
noinline void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb)
697
+
__bpf_kfunc void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb)
701
698
{
702
699
}
703
700
704
-
noinline void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p)
701
+
__bpf_kfunc void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p)
705
702
{
706
703
}
707
704
708
-
noinline void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p)
705
+
__bpf_kfunc void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p)
709
706
{
710
707
}
711
708
712
-
noinline void bpf_kfunc_call_test_fail1(struct prog_test_fail1 *p)
709
+
__bpf_kfunc void bpf_kfunc_call_test_fail1(struct prog_test_fail1 *p)
713
710
{
714
711
}
715
712
716
-
noinline void bpf_kfunc_call_test_fail2(struct prog_test_fail2 *p)
713
+
__bpf_kfunc void bpf_kfunc_call_test_fail2(struct prog_test_fail2 *p)
717
714
{
718
715
}
719
716
720
-
noinline void bpf_kfunc_call_test_fail3(struct prog_test_fail3 *p)
717
+
__bpf_kfunc void bpf_kfunc_call_test_fail3(struct prog_test_fail3 *p)
721
718
{
722
719
}
723
720
724
-
noinline void bpf_kfunc_call_test_mem_len_pass1(void *mem, int mem__sz)
721
+
__bpf_kfunc void bpf_kfunc_call_test_mem_len_pass1(void *mem, int mem__sz)
725
722
{
726
723
}
727
724
728
-
noinline void bpf_kfunc_call_test_mem_len_fail1(void *mem, int len)
725
+
__bpf_kfunc void bpf_kfunc_call_test_mem_len_fail1(void *mem, int len)
729
726
{
730
727
}
731
728
732
-
noinline void bpf_kfunc_call_test_mem_len_fail2(u64 *mem, int len)
729
+
__bpf_kfunc void bpf_kfunc_call_test_mem_len_fail2(u64 *mem, int len)
733
730
{
734
731
}
735
732
736
-
noinline void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p)
733
+
__bpf_kfunc void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p)
737
734
{
738
735
}
739
736
740
-
noinline void bpf_kfunc_call_test_destructive(void)
737
+
__bpf_kfunc void bpf_kfunc_call_test_destructive(void)
741
738
{
739
+
}
740
+
741
+
__bpf_kfunc static u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused)
742
+
{
743
+
return arg;
742
744
}
743
745
744
746
__diag_pop();
···
784
776
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
785
777
BTF_ID_FLAGS(func, bpf_kfunc_call_test_ref, KF_TRUSTED_ARGS)
786
778
BTF_ID_FLAGS(func, bpf_kfunc_call_test_destructive, KF_DESTRUCTIVE)
779
+
BTF_ID_FLAGS(func, bpf_kfunc_call_test_static_unused_arg)
787
780
BTF_SET8_END(test_sk_check_kfunc_ids)
788
781
789
782
static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size,
+3
-2
net/core/xdp.c
+3
-2
net/core/xdp.c
···
4
4
* Copyright (c) 2017 Jesper Dangaard Brouer, Red Hat Inc.
5
5
*/
6
6
#include <linux/bpf.h>
7
+
#include <linux/btf.h>
7
8
#include <linux/btf_ids.h>
8
9
#include <linux/filter.h>
9
10
#include <linux/types.h>
···
723
722
*
724
723
* Returns 0 on success or ``-errno`` on error.
725
724
*/
726
-
int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, u64 *timestamp)
725
+
__bpf_kfunc int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, u64 *timestamp)
727
726
{
728
727
return -EOPNOTSUPP;
729
728
}
···
735
734
*
736
735
* Returns 0 on success or ``-errno`` on error.
737
736
*/
738
-
int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash)
737
+
__bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash)
739
738
{
740
739
return -EOPNOTSUPP;
741
740
}
+8
-8
net/ipv4/tcp_bbr.c
+8
-8
net/ipv4/tcp_bbr.c
···
295
295
}
296
296
297
297
/* override sysctl_tcp_min_tso_segs */
298
-
static u32 bbr_min_tso_segs(struct sock *sk)
298
+
__bpf_kfunc static u32 bbr_min_tso_segs(struct sock *sk)
299
299
{
300
300
return sk->sk_pacing_rate < (bbr_min_tso_rate >> 3) ? 1 : 2;
301
301
}
···
328
328
bbr->prior_cwnd = max(bbr->prior_cwnd, tcp_snd_cwnd(tp));
329
329
}
330
330
331
-
static void bbr_cwnd_event(struct sock *sk, enum tcp_ca_event event)
331
+
__bpf_kfunc static void bbr_cwnd_event(struct sock *sk, enum tcp_ca_event event)
332
332
{
333
333
struct tcp_sock *tp = tcp_sk(sk);
334
334
struct bbr *bbr = inet_csk_ca(sk);
···
1023
1023
bbr_update_gains(sk);
1024
1024
}
1025
1025
1026
-
static void bbr_main(struct sock *sk, const struct rate_sample *rs)
1026
+
__bpf_kfunc static void bbr_main(struct sock *sk, const struct rate_sample *rs)
1027
1027
{
1028
1028
struct bbr *bbr = inet_csk_ca(sk);
1029
1029
u32 bw;
···
1035
1035
bbr_set_cwnd(sk, rs, rs->acked_sacked, bw, bbr->cwnd_gain);
1036
1036
}
1037
1037
1038
-
static void bbr_init(struct sock *sk)
1038
+
__bpf_kfunc static void bbr_init(struct sock *sk)
1039
1039
{
1040
1040
struct tcp_sock *tp = tcp_sk(sk);
1041
1041
struct bbr *bbr = inet_csk_ca(sk);
···
1077
1077
cmpxchg(&sk->sk_pacing_status, SK_PACING_NONE, SK_PACING_NEEDED);
1078
1078
}
1079
1079
1080
-
static u32 bbr_sndbuf_expand(struct sock *sk)
1080
+
__bpf_kfunc static u32 bbr_sndbuf_expand(struct sock *sk)
1081
1081
{
1082
1082
/* Provision 3 * cwnd since BBR may slow-start even during recovery. */
1083
1083
return 3;
···
1086
1086
/* In theory BBR does not need to undo the cwnd since it does not
1087
1087
* always reduce cwnd on losses (see bbr_main()). Keep it for now.
1088
1088
*/
1089
-
static u32 bbr_undo_cwnd(struct sock *sk)
1089
+
__bpf_kfunc static u32 bbr_undo_cwnd(struct sock *sk)
1090
1090
{
1091
1091
struct bbr *bbr = inet_csk_ca(sk);
1092
1092
···
1097
1097
}
1098
1098
1099
1099
/* Entering loss recovery, so save cwnd for when we exit or undo recovery. */
1100
-
static u32 bbr_ssthresh(struct sock *sk)
1100
+
__bpf_kfunc static u32 bbr_ssthresh(struct sock *sk)
1101
1101
{
1102
1102
bbr_save_cwnd(sk);
1103
1103
return tcp_sk(sk)->snd_ssthresh;
···
1125
1125
return 0;
1126
1126
}
1127
1127
1128
-
static void bbr_set_state(struct sock *sk, u8 new_state)
1128
+
__bpf_kfunc static void bbr_set_state(struct sock *sk, u8 new_state)
1129
1129
{
1130
1130
struct bbr *bbr = inet_csk_ca(sk);
1131
1131
+5
-5
net/ipv4/tcp_cong.c
+5
-5
net/ipv4/tcp_cong.c
···
403
403
* ABC caps N to 2. Slow start exits when cwnd grows over ssthresh and
404
404
* returns the leftover acks to adjust cwnd in congestion avoidance mode.
405
405
*/
406
-
u32 tcp_slow_start(struct tcp_sock *tp, u32 acked)
406
+
__bpf_kfunc u32 tcp_slow_start(struct tcp_sock *tp, u32 acked)
407
407
{
408
408
u32 cwnd = min(tcp_snd_cwnd(tp) + acked, tp->snd_ssthresh);
409
409
···
417
417
/* In theory this is tp->snd_cwnd += 1 / tp->snd_cwnd (or alternative w),
418
418
* for every packet that was ACKed.
419
419
*/
420
-
void tcp_cong_avoid_ai(struct tcp_sock *tp, u32 w, u32 acked)
420
+
__bpf_kfunc void tcp_cong_avoid_ai(struct tcp_sock *tp, u32 w, u32 acked)
421
421
{
422
422
/* If credits accumulated at a higher w, apply them gently now. */
423
423
if (tp->snd_cwnd_cnt >= w) {
···
443
443
/* This is Jacobson's slow start and congestion avoidance.
444
444
* SIGCOMM '88, p. 328.
445
445
*/
446
-
void tcp_reno_cong_avoid(struct sock *sk, u32 ack, u32 acked)
446
+
__bpf_kfunc void tcp_reno_cong_avoid(struct sock *sk, u32 ack, u32 acked)
447
447
{
448
448
struct tcp_sock *tp = tcp_sk(sk);
449
449
···
462
462
EXPORT_SYMBOL_GPL(tcp_reno_cong_avoid);
463
463
464
464
/* Slow start threshold is half the congestion window (min 2) */
465
-
u32 tcp_reno_ssthresh(struct sock *sk)
465
+
__bpf_kfunc u32 tcp_reno_ssthresh(struct sock *sk)
466
466
{
467
467
const struct tcp_sock *tp = tcp_sk(sk);
468
468
···
470
470
}
471
471
EXPORT_SYMBOL_GPL(tcp_reno_ssthresh);
472
472
473
-
u32 tcp_reno_undo_cwnd(struct sock *sk)
473
+
__bpf_kfunc u32 tcp_reno_undo_cwnd(struct sock *sk)
474
474
{
475
475
const struct tcp_sock *tp = tcp_sk(sk);
476
476
+6
-6
net/ipv4/tcp_cubic.c
+6
-6
net/ipv4/tcp_cubic.c
···
126
126
ca->sample_cnt = 0;
127
127
}
128
128
129
-
static void cubictcp_init(struct sock *sk)
129
+
__bpf_kfunc static void cubictcp_init(struct sock *sk)
130
130
{
131
131
struct bictcp *ca = inet_csk_ca(sk);
132
132
···
139
139
tcp_sk(sk)->snd_ssthresh = initial_ssthresh;
140
140
}
141
141
142
-
static void cubictcp_cwnd_event(struct sock *sk, enum tcp_ca_event event)
142
+
__bpf_kfunc static void cubictcp_cwnd_event(struct sock *sk, enum tcp_ca_event event)
143
143
{
144
144
if (event == CA_EVENT_TX_START) {
145
145
struct bictcp *ca = inet_csk_ca(sk);
···
321
321
ca->cnt = max(ca->cnt, 2U);
322
322
}
323
323
324
-
static void cubictcp_cong_avoid(struct sock *sk, u32 ack, u32 acked)
324
+
__bpf_kfunc static void cubictcp_cong_avoid(struct sock *sk, u32 ack, u32 acked)
325
325
{
326
326
struct tcp_sock *tp = tcp_sk(sk);
327
327
struct bictcp *ca = inet_csk_ca(sk);
···
338
338
tcp_cong_avoid_ai(tp, ca->cnt, acked);
339
339
}
340
340
341
-
static u32 cubictcp_recalc_ssthresh(struct sock *sk)
341
+
__bpf_kfunc static u32 cubictcp_recalc_ssthresh(struct sock *sk)
342
342
{
343
343
const struct tcp_sock *tp = tcp_sk(sk);
344
344
struct bictcp *ca = inet_csk_ca(sk);
···
355
355
return max((tcp_snd_cwnd(tp) * beta) / BICTCP_BETA_SCALE, 2U);
356
356
}
357
357
358
-
static void cubictcp_state(struct sock *sk, u8 new_state)
358
+
__bpf_kfunc static void cubictcp_state(struct sock *sk, u8 new_state)
359
359
{
360
360
if (new_state == TCP_CA_Loss) {
361
361
bictcp_reset(inet_csk_ca(sk));
···
445
445
}
446
446
}
447
447
448
-
static void cubictcp_acked(struct sock *sk, const struct ack_sample *sample)
448
+
__bpf_kfunc static void cubictcp_acked(struct sock *sk, const struct ack_sample *sample)
449
449
{
450
450
const struct tcp_sock *tp = tcp_sk(sk);
451
451
struct bictcp *ca = inet_csk_ca(sk);
+6
-6
net/ipv4/tcp_dctcp.c
+6
-6
net/ipv4/tcp_dctcp.c
···
75
75
ca->old_delivered_ce = tp->delivered_ce;
76
76
}
77
77
78
-
static void dctcp_init(struct sock *sk)
78
+
__bpf_kfunc static void dctcp_init(struct sock *sk)
79
79
{
80
80
const struct tcp_sock *tp = tcp_sk(sk);
81
81
···
104
104
INET_ECN_dontxmit(sk);
105
105
}
106
106
107
-
static u32 dctcp_ssthresh(struct sock *sk)
107
+
__bpf_kfunc static u32 dctcp_ssthresh(struct sock *sk)
108
108
{
109
109
struct dctcp *ca = inet_csk_ca(sk);
110
110
struct tcp_sock *tp = tcp_sk(sk);
···
113
113
return max(tcp_snd_cwnd(tp) - ((tcp_snd_cwnd(tp) * ca->dctcp_alpha) >> 11U), 2U);
114
114
}
115
115
116
-
static void dctcp_update_alpha(struct sock *sk, u32 flags)
116
+
__bpf_kfunc static void dctcp_update_alpha(struct sock *sk, u32 flags)
117
117
{
118
118
const struct tcp_sock *tp = tcp_sk(sk);
119
119
struct dctcp *ca = inet_csk_ca(sk);
···
169
169
tp->snd_ssthresh = max(tcp_snd_cwnd(tp) >> 1U, 2U);
170
170
}
171
171
172
-
static void dctcp_state(struct sock *sk, u8 new_state)
172
+
__bpf_kfunc static void dctcp_state(struct sock *sk, u8 new_state)
173
173
{
174
174
if (new_state == TCP_CA_Recovery &&
175
175
new_state != inet_csk(sk)->icsk_ca_state)
···
179
179
*/
180
180
}
181
181
182
-
static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev)
182
+
__bpf_kfunc static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev)
183
183
{
184
184
struct dctcp *ca = inet_csk_ca(sk);
185
185
···
229
229
return 0;
230
230
}
231
231
232
-
static u32 dctcp_cwnd_undo(struct sock *sk)
232
+
__bpf_kfunc static u32 dctcp_cwnd_undo(struct sock *sk)
233
233
{
234
234
const struct dctcp *ca = inet_csk_ca(sk);
235
235
struct tcp_sock *tp = tcp_sk(sk);
+10
-10
net/netfilter/nf_conntrack_bpf.c
+10
-10
net/netfilter/nf_conntrack_bpf.c
···
249
249
* @opts__sz - Length of the bpf_ct_opts structure
250
250
* Must be NF_BPF_CT_OPTS_SZ (12)
251
251
*/
252
-
struct nf_conn___init *
252
+
__bpf_kfunc struct nf_conn___init *
253
253
bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
254
254
u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)
255
255
{
···
283
283
* @opts__sz - Length of the bpf_ct_opts structure
284
284
* Must be NF_BPF_CT_OPTS_SZ (12)
285
285
*/
286
-
struct nf_conn *
286
+
__bpf_kfunc struct nf_conn *
287
287
bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
288
288
u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)
289
289
{
···
316
316
* @opts__sz - Length of the bpf_ct_opts structure
317
317
* Must be NF_BPF_CT_OPTS_SZ (12)
318
318
*/
319
-
struct nf_conn___init *
319
+
__bpf_kfunc struct nf_conn___init *
320
320
bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
321
321
u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)
322
322
{
···
351
351
* @opts__sz - Length of the bpf_ct_opts structure
352
352
* Must be NF_BPF_CT_OPTS_SZ (12)
353
353
*/
354
-
struct nf_conn *
354
+
__bpf_kfunc struct nf_conn *
355
355
bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
356
356
u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)
357
357
{
···
376
376
* @nfct - Pointer to referenced nf_conn___init object, obtained
377
377
* using bpf_xdp_ct_alloc or bpf_skb_ct_alloc.
378
378
*/
379
-
struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i)
379
+
__bpf_kfunc struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i)
380
380
{
381
381
struct nf_conn *nfct = (struct nf_conn *)nfct_i;
382
382
int err;
···
400
400
* @nf_conn - Pointer to referenced nf_conn object, obtained using
401
401
* bpf_xdp_ct_lookup or bpf_skb_ct_lookup.
402
402
*/
403
-
void bpf_ct_release(struct nf_conn *nfct)
403
+
__bpf_kfunc void bpf_ct_release(struct nf_conn *nfct)
404
404
{
405
405
if (!nfct)
406
406
return;
···
417
417
* bpf_xdp_ct_alloc or bpf_skb_ct_alloc.
418
418
* @timeout - Timeout in msecs.
419
419
*/
420
-
void bpf_ct_set_timeout(struct nf_conn___init *nfct, u32 timeout)
420
+
__bpf_kfunc void bpf_ct_set_timeout(struct nf_conn___init *nfct, u32 timeout)
421
421
{
422
422
__nf_ct_set_timeout((struct nf_conn *)nfct, msecs_to_jiffies(timeout));
423
423
}
···
432
432
* bpf_ct_insert_entry, bpf_xdp_ct_lookup, or bpf_skb_ct_lookup.
433
433
* @timeout - New timeout in msecs.
434
434
*/
435
-
int bpf_ct_change_timeout(struct nf_conn *nfct, u32 timeout)
435
+
__bpf_kfunc int bpf_ct_change_timeout(struct nf_conn *nfct, u32 timeout)
436
436
{
437
437
return __nf_ct_change_timeout(nfct, msecs_to_jiffies(timeout));
438
438
}
···
447
447
* bpf_xdp_ct_alloc or bpf_skb_ct_alloc.
448
448
* @status - New status value.
449
449
*/
450
-
int bpf_ct_set_status(const struct nf_conn___init *nfct, u32 status)
450
+
__bpf_kfunc int bpf_ct_set_status(const struct nf_conn___init *nfct, u32 status)
451
451
{
452
452
return nf_ct_change_status_common((struct nf_conn *)nfct, status);
453
453
}
···
462
462
* bpf_ct_insert_entry, bpf_xdp_ct_lookup or bpf_skb_ct_lookup.
463
463
* @status - New status value.
464
464
*/
465
-
int bpf_ct_change_status(struct nf_conn *nfct, u32 status)
465
+
__bpf_kfunc int bpf_ct_change_status(struct nf_conn *nfct, u32 status)
466
466
{
467
467
return nf_ct_change_status_common(nfct, status);
468
468
}
+3
-3
net/netfilter/nf_nat_bpf.c
+3
-3
net/netfilter/nf_nat_bpf.c
···
30
30
* interpreted as select a random port.
31
31
* @manip - NF_NAT_MANIP_SRC or NF_NAT_MANIP_DST
32
32
*/
33
-
int bpf_ct_set_nat_info(struct nf_conn___init *nfct,
34
-
union nf_inet_addr *addr, int port,
35
-
enum nf_nat_manip_type manip)
33
+
__bpf_kfunc int bpf_ct_set_nat_info(struct nf_conn___init *nfct,
34
+
union nf_inet_addr *addr, int port,
35
+
enum nf_nat_manip_type manip)
36
36
{
37
37
struct nf_conn *ct = (struct nf_conn *)nfct;
38
38
u16 proto = nf_ct_l3num(ct);
+2
-5
net/xfrm/xfrm_interface_bpf.c
+2
-5
net/xfrm/xfrm_interface_bpf.c
···
39
39
* @to - Pointer to memory to which the metadata will be copied
40
40
* Cannot be NULL
41
41
*/
42
-
__used noinline
43
-
int bpf_skb_get_xfrm_info(struct __sk_buff *skb_ctx, struct bpf_xfrm_info *to)
42
+
__bpf_kfunc int bpf_skb_get_xfrm_info(struct __sk_buff *skb_ctx, struct bpf_xfrm_info *to)
44
43
{
45
44
struct sk_buff *skb = (struct sk_buff *)skb_ctx;
46
45
struct xfrm_md_info *info;
···
61
62
* @from - Pointer to memory from which the metadata will be copied
62
63
* Cannot be NULL
63
64
*/
64
-
__used noinline
65
-
int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx,
66
-
const struct bpf_xfrm_info *from)
65
+
__bpf_kfunc int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx, const struct bpf_xfrm_info *from)
67
66
{
68
67
struct sk_buff *skb = (struct sk_buff *)skb_ctx;
69
68
struct metadata_dst *md_dst;
+1
-1
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
+1
-1
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
+1
tools/testing/selftests/bpf/prog_tests/kfunc_call.c
+1
tools/testing/selftests/bpf/prog_tests/kfunc_call.c
+11
tools/testing/selftests/bpf/progs/kfunc_call_test.c
+11
tools/testing/selftests/bpf/progs/kfunc_call_test.c
···
17
17
extern void bpf_kfunc_call_test_mem_len_fail2(__u64 *mem, int len) __ksym;
18
18
extern int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) __ksym;
19
19
extern int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
20
+
extern u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;
20
21
21
22
SEC("tc")
22
23
int kfunc_call_test4(struct __sk_buff *skb)
···
180
179
bpf_kfunc_call_test_release(pt);
181
180
}
182
181
return ret;
182
+
}
183
+
184
+
SEC("tc")
185
+
int kfunc_call_test_static_unused_arg(struct __sk_buff *skb)
186
+
{
187
+
188
+
u32 expected = 5, actual;
189
+
190
+
actual = bpf_kfunc_call_test_static_unused_arg(expected, 0xdeadbeef);
191
+
return actual != expected ? -1 : 0;
183
192
}
184
193
185
194
char _license[] SEC("license") = "GPL";