Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Daniel Borkmann says:
====================
pull-request: bpf-next 2023-10-16
We've added 90 non-merge commits during the last 25 day(s) which contain
a total of 120 files changed, 3519 insertions(+), 895 deletions(-).
The main changes are:
1) Add missed stats for kprobes to retrieve the number of missed kprobe
executions and subsequent executions of BPF programs, from Jiri Olsa.
2) Add cgroup BPF sockaddr hooks for unix sockets. The use case is
for systemd to reimplement the LogNamespace feature which allows
running multiple instances of systemd-journald to process the logs
of different services, from Daan De Meyer.
3) Implement BPF CPUv4 support for s390x BPF JIT, from Ilya Leoshkevich.
4) Improve BPF verifier log output for scalar registers to better
disambiguate their internal state wrt defaults vs min/max values
matching, from Andrii Nakryiko.
5) Extend the BPF fib lookup helpers for IPv4/IPv6 to support retrieving
the source IP address with a new BPF_FIB_LOOKUP_SRC flag,
from Martynas Pumputis.
6) Add support for open-coded task_vma iterator to help with symbolization
for BPF-collected user stacks, from Dave Marchevsky.
7) Add libbpf getters for accessing individual BPF ring buffers which
is useful for polling them individually, for example, from Martin Kelly.
8) Extend AF_XDP selftests to validate the SHARED_UMEM feature,
from Tushar Vyavahare.
9) Improve BPF selftests cross-building support for riscv arch,
from Björn Töpel.
10) Add the ability to pin a BPF timer to the same calling CPU,
from David Vernet.
11) Fix libbpf's bpf_tracing.h macros for riscv to use the generic
implementation of PT_REGS_SYSCALL_REGS() to access syscall arguments,
from Alexandre Ghiti.
12) Extend libbpf to support symbol versioning for uprobes, from Hengqi Chen.
13) Fix bpftool's skeleton code generation to guarantee that ELF data
is 8 byte aligned, from Ian Rogers.
14) Inherit system-wide cpu_mitigations_off() setting for Spectre v1/v4
security mitigations in BPF verifier, from Yafang Shao.
15) Annotate struct bpf_stack_map with __counted_by attribute to prepare
BPF side for upcoming __counted_by compiler support, from Kees Cook.
* tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (90 commits)
bpf: Ensure proper register state printing for cond jumps
bpf: Disambiguate SCALAR register state output in verifier logs
selftests/bpf: Make align selftests more robust
selftests/bpf: Improve missed_kprobe_recursion test robustness
selftests/bpf: Improve percpu_alloc test robustness
selftests/bpf: Add tests for open-coded task_vma iter
bpf: Introduce task_vma open-coded iterator kfuncs
selftests/bpf: Rename bpf_iter_task_vma.c to bpf_iter_task_vmas.c
bpf: Don't explicitly emit BTF for struct btf_iter_num
bpf: Change syscall_nr type to int in struct syscall_tp_t
net/bpf: Avoid unused "sin_addr_len" warning when CONFIG_CGROUP_BPF is not set
bpf: Avoid unnecessary audit log for CPU security mitigations
selftests/bpf: Add tests for cgroup unix socket address hooks
selftests/bpf: Make sure mount directory exists
documentation/bpf: Document cgroup unix socket address hooks
bpftool: Add support for cgroup unix socket address hooks
libbpf: Add support for cgroup unix socket address hooks
bpf: Implement cgroup sockaddr hooks for unix sockets
bpf: Add bpf_sock_addr_set_sun_path() to allow writing unix sockaddr from bpf
bpf: Propagate modified uaddrlen from cgroup sockaddr programs
...
====================
Link: https://lore.kernel.org/r/20231016204803.30153-1-daniel@iogearbox.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
+3534
-910
+1
Documentation/admin-guide/sysctl/net.rst
+1
Documentation/admin-guide/sysctl/net.rst
+10
Documentation/bpf/libbpf/program_types.rst
+10
Documentation/bpf/libbpf/program_types.rst
···
56
56
| | ``BPF_CGROUP_UDP6_RECVMSG`` | ``cgroup/recvmsg6`` | |
57
57
+ +----------------------------------------+----------------------------------+-----------+
58
58
| | ``BPF_CGROUP_UDP6_SENDMSG`` | ``cgroup/sendmsg6`` | |
59
+
| +----------------------------------------+----------------------------------+-----------+
60
+
| | ``BPF_CGROUP_UNIX_CONNECT`` | ``cgroup/connect_unix`` | |
61
+
| +----------------------------------------+----------------------------------+-----------+
62
+
| | ``BPF_CGROUP_UNIX_SENDMSG`` | ``cgroup/sendmsg_unix`` | |
63
+
| +----------------------------------------+----------------------------------+-----------+
64
+
| | ``BPF_CGROUP_UNIX_RECVMSG`` | ``cgroup/recvmsg_unix`` | |
65
+
| +----------------------------------------+----------------------------------+-----------+
66
+
| | ``BPF_CGROUP_UNIX_GETPEERNAME`` | ``cgroup/getpeername_unix`` | |
67
+
| +----------------------------------------+----------------------------------+-----------+
68
+
| | ``BPF_CGROUP_UNIX_GETSOCKNAME`` | ``cgroup/getsockname_unix`` | |
59
69
+-------------------------------------------+----------------------------------------+----------------------------------+-----------+
60
70
| ``BPF_PROG_TYPE_CGROUP_SOCK`` | ``BPF_CGROUP_INET4_POST_BIND`` | ``cgroup/post_bind4`` | |
61
71
+ +----------------------------------------+----------------------------------+-----------+
+2
-2
Documentation/networking/filter.rst
+2
-2
Documentation/networking/filter.rst
···
650
650
651
651
Currently, the classic BPF format is being used for JITing on most
652
652
32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64,
653
-
sparc64, arm32, riscv64, riscv32 perform JIT compilation from eBPF
654
-
instruction set.
653
+
sparc64, arm32, riscv64, riscv32, loongarch64 perform JIT compilation
654
+
from eBPF instruction set.
655
655
656
656
Testing
657
657
-------
+202
-63
arch/s390/net/bpf_jit_comp.c
+202
-63
arch/s390/net/bpf_jit_comp.c
···
670
670
static int get_probe_mem_regno(const u8 *insn)
671
671
{
672
672
/*
673
-
* insn must point to llgc, llgh, llgf or lg, which have destination
674
-
* register at the same position.
673
+
* insn must point to llgc, llgh, llgf, lg, lgb, lgh or lgf, which have
674
+
* destination register at the same position.
675
675
*/
676
-
if (insn[0] != 0xe3) /* common llgc, llgh, llgf and lg prefix */
676
+
if (insn[0] != 0xe3) /* common prefix */
677
677
return -1;
678
678
if (insn[5] != 0x90 && /* llgc */
679
679
insn[5] != 0x91 && /* llgh */
680
680
insn[5] != 0x16 && /* llgf */
681
-
insn[5] != 0x04) /* lg */
681
+
insn[5] != 0x04 && /* lg */
682
+
insn[5] != 0x77 && /* lgb */
683
+
insn[5] != 0x15 && /* lgh */
684
+
insn[5] != 0x14) /* lgf */
682
685
return -1;
683
686
return insn[1] >> 4;
684
687
}
···
779
776
int i, bool extra_pass, u32 stack_depth)
780
777
{
781
778
struct bpf_insn *insn = &fp->insnsi[i];
779
+
s16 branch_oc_off = insn->off;
782
780
u32 dst_reg = insn->dst_reg;
783
781
u32 src_reg = insn->src_reg;
784
782
int last, insn_count = 1;
···
792
788
int err;
793
789
794
790
if (BPF_CLASS(insn->code) == BPF_LDX &&
795
-
BPF_MODE(insn->code) == BPF_PROBE_MEM)
791
+
(BPF_MODE(insn->code) == BPF_PROBE_MEM ||
792
+
BPF_MODE(insn->code) == BPF_PROBE_MEMSX))
796
793
probe_prg = jit->prg;
797
794
798
795
switch (insn->code) {
799
796
/*
800
797
* BPF_MOV
801
798
*/
802
-
case BPF_ALU | BPF_MOV | BPF_X: /* dst = (u32) src */
803
-
/* llgfr %dst,%src */
804
-
EMIT4(0xb9160000, dst_reg, src_reg);
805
-
if (insn_is_zext(&insn[1]))
806
-
insn_count = 2;
799
+
case BPF_ALU | BPF_MOV | BPF_X:
800
+
switch (insn->off) {
801
+
case 0: /* DST = (u32) SRC */
802
+
/* llgfr %dst,%src */
803
+
EMIT4(0xb9160000, dst_reg, src_reg);
804
+
if (insn_is_zext(&insn[1]))
805
+
insn_count = 2;
806
+
break;
807
+
case 8: /* DST = (u32)(s8) SRC */
808
+
/* lbr %dst,%src */
809
+
EMIT4(0xb9260000, dst_reg, src_reg);
810
+
/* llgfr %dst,%dst */
811
+
EMIT4(0xb9160000, dst_reg, dst_reg);
812
+
break;
813
+
case 16: /* DST = (u32)(s16) SRC */
814
+
/* lhr %dst,%src */
815
+
EMIT4(0xb9270000, dst_reg, src_reg);
816
+
/* llgfr %dst,%dst */
817
+
EMIT4(0xb9160000, dst_reg, dst_reg);
818
+
break;
819
+
}
807
820
break;
808
-
case BPF_ALU64 | BPF_MOV | BPF_X: /* dst = src */
809
-
/* lgr %dst,%src */
810
-
EMIT4(0xb9040000, dst_reg, src_reg);
821
+
case BPF_ALU64 | BPF_MOV | BPF_X:
822
+
switch (insn->off) {
823
+
case 0: /* DST = SRC */
824
+
/* lgr %dst,%src */
825
+
EMIT4(0xb9040000, dst_reg, src_reg);
826
+
break;
827
+
case 8: /* DST = (s8) SRC */
828
+
/* lgbr %dst,%src */
829
+
EMIT4(0xb9060000, dst_reg, src_reg);
830
+
break;
831
+
case 16: /* DST = (s16) SRC */
832
+
/* lghr %dst,%src */
833
+
EMIT4(0xb9070000, dst_reg, src_reg);
834
+
break;
835
+
case 32: /* DST = (s32) SRC */
836
+
/* lgfr %dst,%src */
837
+
EMIT4(0xb9140000, dst_reg, src_reg);
838
+
break;
839
+
}
811
840
break;
812
841
case BPF_ALU | BPF_MOV | BPF_K: /* dst = (u32) imm */
813
842
/* llilf %dst,imm */
···
949
912
/*
950
913
* BPF_DIV / BPF_MOD
951
914
*/
952
-
case BPF_ALU | BPF_DIV | BPF_X: /* dst = (u32) dst / (u32) src */
953
-
case BPF_ALU | BPF_MOD | BPF_X: /* dst = (u32) dst % (u32) src */
915
+
case BPF_ALU | BPF_DIV | BPF_X:
916
+
case BPF_ALU | BPF_MOD | BPF_X:
954
917
{
955
918
int rc_reg = BPF_OP(insn->code) == BPF_DIV ? REG_W1 : REG_W0;
956
919
957
-
/* lhi %w0,0 */
958
-
EMIT4_IMM(0xa7080000, REG_W0, 0);
959
-
/* lr %w1,%dst */
960
-
EMIT2(0x1800, REG_W1, dst_reg);
961
-
/* dlr %w0,%src */
962
-
EMIT4(0xb9970000, REG_W0, src_reg);
920
+
switch (off) {
921
+
case 0: /* dst = (u32) dst {/,%} (u32) src */
922
+
/* xr %w0,%w0 */
923
+
EMIT2(0x1700, REG_W0, REG_W0);
924
+
/* lr %w1,%dst */
925
+
EMIT2(0x1800, REG_W1, dst_reg);
926
+
/* dlr %w0,%src */
927
+
EMIT4(0xb9970000, REG_W0, src_reg);
928
+
break;
929
+
case 1: /* dst = (u32) ((s32) dst {/,%} (s32) src) */
930
+
/* lgfr %r1,%dst */
931
+
EMIT4(0xb9140000, REG_W1, dst_reg);
932
+
/* dsgfr %r0,%src */
933
+
EMIT4(0xb91d0000, REG_W0, src_reg);
934
+
break;
935
+
}
963
936
/* llgfr %dst,%rc */
964
937
EMIT4(0xb9160000, dst_reg, rc_reg);
965
938
if (insn_is_zext(&insn[1]))
966
939
insn_count = 2;
967
940
break;
968
941
}
969
-
case BPF_ALU64 | BPF_DIV | BPF_X: /* dst = dst / src */
970
-
case BPF_ALU64 | BPF_MOD | BPF_X: /* dst = dst % src */
942
+
case BPF_ALU64 | BPF_DIV | BPF_X:
943
+
case BPF_ALU64 | BPF_MOD | BPF_X:
971
944
{
972
945
int rc_reg = BPF_OP(insn->code) == BPF_DIV ? REG_W1 : REG_W0;
973
946
974
-
/* lghi %w0,0 */
975
-
EMIT4_IMM(0xa7090000, REG_W0, 0);
976
-
/* lgr %w1,%dst */
977
-
EMIT4(0xb9040000, REG_W1, dst_reg);
978
-
/* dlgr %w0,%dst */
979
-
EMIT4(0xb9870000, REG_W0, src_reg);
947
+
switch (off) {
948
+
case 0: /* dst = dst {/,%} src */
949
+
/* lghi %w0,0 */
950
+
EMIT4_IMM(0xa7090000, REG_W0, 0);
951
+
/* lgr %w1,%dst */
952
+
EMIT4(0xb9040000, REG_W1, dst_reg);
953
+
/* dlgr %w0,%src */
954
+
EMIT4(0xb9870000, REG_W0, src_reg);
955
+
break;
956
+
case 1: /* dst = (s64) dst {/,%} (s64) src */
957
+
/* lgr %w1,%dst */
958
+
EMIT4(0xb9040000, REG_W1, dst_reg);
959
+
/* dsgr %w0,%src */
960
+
EMIT4(0xb90d0000, REG_W0, src_reg);
961
+
break;
962
+
}
980
963
/* lgr %dst,%rc */
981
964
EMIT4(0xb9040000, dst_reg, rc_reg);
982
965
break;
983
966
}
984
-
case BPF_ALU | BPF_DIV | BPF_K: /* dst = (u32) dst / (u32) imm */
985
-
case BPF_ALU | BPF_MOD | BPF_K: /* dst = (u32) dst % (u32) imm */
967
+
case BPF_ALU | BPF_DIV | BPF_K:
968
+
case BPF_ALU | BPF_MOD | BPF_K:
986
969
{
987
970
int rc_reg = BPF_OP(insn->code) == BPF_DIV ? REG_W1 : REG_W0;
988
971
989
972
if (imm == 1) {
990
973
if (BPF_OP(insn->code) == BPF_MOD)
991
-
/* lhgi %dst,0 */
974
+
/* lghi %dst,0 */
992
975
EMIT4_IMM(0xa7090000, dst_reg, 0);
993
976
else
994
977
EMIT_ZERO(dst_reg);
995
978
break;
996
979
}
997
-
/* lhi %w0,0 */
998
-
EMIT4_IMM(0xa7080000, REG_W0, 0);
999
-
/* lr %w1,%dst */
1000
-
EMIT2(0x1800, REG_W1, dst_reg);
1001
980
if (!is_first_pass(jit) && can_use_ldisp_for_lit32(jit)) {
1002
-
/* dl %w0,<d(imm)>(%l) */
1003
-
EMIT6_DISP_LH(0xe3000000, 0x0097, REG_W0, REG_0, REG_L,
1004
-
EMIT_CONST_U32(imm));
981
+
switch (off) {
982
+
case 0: /* dst = (u32) dst {/,%} (u32) imm */
983
+
/* xr %w0,%w0 */
984
+
EMIT2(0x1700, REG_W0, REG_W0);
985
+
/* lr %w1,%dst */
986
+
EMIT2(0x1800, REG_W1, dst_reg);
987
+
/* dl %w0,<d(imm)>(%l) */
988
+
EMIT6_DISP_LH(0xe3000000, 0x0097, REG_W0, REG_0,
989
+
REG_L, EMIT_CONST_U32(imm));
990
+
break;
991
+
case 1: /* dst = (s32) dst {/,%} (s32) imm */
992
+
/* lgfr %r1,%dst */
993
+
EMIT4(0xb9140000, REG_W1, dst_reg);
994
+
/* dsgf %r0,<d(imm)>(%l) */
995
+
EMIT6_DISP_LH(0xe3000000, 0x001d, REG_W0, REG_0,
996
+
REG_L, EMIT_CONST_U32(imm));
997
+
break;
998
+
}
1005
999
} else {
1006
-
/* lgfrl %dst,imm */
1007
-
EMIT6_PCREL_RILB(0xc40c0000, dst_reg,
1008
-
_EMIT_CONST_U32(imm));
1009
-
jit->seen |= SEEN_LITERAL;
1010
-
/* dlr %w0,%dst */
1011
-
EMIT4(0xb9970000, REG_W0, dst_reg);
1000
+
switch (off) {
1001
+
case 0: /* dst = (u32) dst {/,%} (u32) imm */
1002
+
/* xr %w0,%w0 */
1003
+
EMIT2(0x1700, REG_W0, REG_W0);
1004
+
/* lr %w1,%dst */
1005
+
EMIT2(0x1800, REG_W1, dst_reg);
1006
+
/* lrl %dst,imm */
1007
+
EMIT6_PCREL_RILB(0xc40d0000, dst_reg,
1008
+
_EMIT_CONST_U32(imm));
1009
+
jit->seen |= SEEN_LITERAL;
1010
+
/* dlr %w0,%dst */
1011
+
EMIT4(0xb9970000, REG_W0, dst_reg);
1012
+
break;
1013
+
case 1: /* dst = (s32) dst {/,%} (s32) imm */
1014
+
/* lgfr %w1,%dst */
1015
+
EMIT4(0xb9140000, REG_W1, dst_reg);
1016
+
/* lgfrl %dst,imm */
1017
+
EMIT6_PCREL_RILB(0xc40c0000, dst_reg,
1018
+
_EMIT_CONST_U32(imm));
1019
+
jit->seen |= SEEN_LITERAL;
1020
+
/* dsgr %w0,%dst */
1021
+
EMIT4(0xb90d0000, REG_W0, dst_reg);
1022
+
break;
1023
+
}
1012
1024
}
1013
1025
/* llgfr %dst,%rc */
1014
1026
EMIT4(0xb9160000, dst_reg, rc_reg);
···
1065
979
insn_count = 2;
1066
980
break;
1067
981
}
1068
-
case BPF_ALU64 | BPF_DIV | BPF_K: /* dst = dst / imm */
1069
-
case BPF_ALU64 | BPF_MOD | BPF_K: /* dst = dst % imm */
982
+
case BPF_ALU64 | BPF_DIV | BPF_K:
983
+
case BPF_ALU64 | BPF_MOD | BPF_K:
1070
984
{
1071
985
int rc_reg = BPF_OP(insn->code) == BPF_DIV ? REG_W1 : REG_W0;
1072
986
···
1076
990
EMIT4_IMM(0xa7090000, dst_reg, 0);
1077
991
break;
1078
992
}
1079
-
/* lghi %w0,0 */
1080
-
EMIT4_IMM(0xa7090000, REG_W0, 0);
1081
-
/* lgr %w1,%dst */
1082
-
EMIT4(0xb9040000, REG_W1, dst_reg);
1083
993
if (!is_first_pass(jit) && can_use_ldisp_for_lit64(jit)) {
1084
-
/* dlg %w0,<d(imm)>(%l) */
1085
-
EMIT6_DISP_LH(0xe3000000, 0x0087, REG_W0, REG_0, REG_L,
1086
-
EMIT_CONST_U64(imm));
994
+
switch (off) {
995
+
case 0: /* dst = dst {/,%} imm */
996
+
/* lghi %w0,0 */
997
+
EMIT4_IMM(0xa7090000, REG_W0, 0);
998
+
/* lgr %w1,%dst */
999
+
EMIT4(0xb9040000, REG_W1, dst_reg);
1000
+
/* dlg %w0,<d(imm)>(%l) */
1001
+
EMIT6_DISP_LH(0xe3000000, 0x0087, REG_W0, REG_0,
1002
+
REG_L, EMIT_CONST_U64(imm));
1003
+
break;
1004
+
case 1: /* dst = (s64) dst {/,%} (s64) imm */
1005
+
/* lgr %w1,%dst */
1006
+
EMIT4(0xb9040000, REG_W1, dst_reg);
1007
+
/* dsg %w0,<d(imm)>(%l) */
1008
+
EMIT6_DISP_LH(0xe3000000, 0x000d, REG_W0, REG_0,
1009
+
REG_L, EMIT_CONST_U64(imm));
1010
+
break;
1011
+
}
1087
1012
} else {
1088
-
/* lgrl %dst,imm */
1089
-
EMIT6_PCREL_RILB(0xc4080000, dst_reg,
1090
-
_EMIT_CONST_U64(imm));
1091
-
jit->seen |= SEEN_LITERAL;
1092
-
/* dlgr %w0,%dst */
1093
-
EMIT4(0xb9870000, REG_W0, dst_reg);
1013
+
switch (off) {
1014
+
case 0: /* dst = dst {/,%} imm */
1015
+
/* lghi %w0,0 */
1016
+
EMIT4_IMM(0xa7090000, REG_W0, 0);
1017
+
/* lgr %w1,%dst */
1018
+
EMIT4(0xb9040000, REG_W1, dst_reg);
1019
+
/* lgrl %dst,imm */
1020
+
EMIT6_PCREL_RILB(0xc4080000, dst_reg,
1021
+
_EMIT_CONST_U64(imm));
1022
+
jit->seen |= SEEN_LITERAL;
1023
+
/* dlgr %w0,%dst */
1024
+
EMIT4(0xb9870000, REG_W0, dst_reg);
1025
+
break;
1026
+
case 1: /* dst = (s64) dst {/,%} (s64) imm */
1027
+
/* lgr %w1,%dst */
1028
+
EMIT4(0xb9040000, REG_W1, dst_reg);
1029
+
/* lgrl %dst,imm */
1030
+
EMIT6_PCREL_RILB(0xc4080000, dst_reg,
1031
+
_EMIT_CONST_U64(imm));
1032
+
jit->seen |= SEEN_LITERAL;
1033
+
/* dsgr %w0,%dst */
1034
+
EMIT4(0xb90d0000, REG_W0, dst_reg);
1035
+
break;
1036
+
}
1094
1037
}
1095
1038
/* lgr %dst,%rc */
1096
1039
EMIT4(0xb9040000, dst_reg, rc_reg);
···
1332
1217
}
1333
1218
break;
1334
1219
case BPF_ALU | BPF_END | BPF_FROM_LE:
1220
+
case BPF_ALU64 | BPF_END | BPF_FROM_LE:
1335
1221
switch (imm) {
1336
1222
case 16: /* dst = (u16) cpu_to_le16(dst) */
1337
1223
/* lrvr %dst,%dst */
···
1490
1374
if (insn_is_zext(&insn[1]))
1491
1375
insn_count = 2;
1492
1376
break;
1377
+
case BPF_LDX | BPF_MEMSX | BPF_B: /* dst = *(s8 *)(ul) (src + off) */
1378
+
case BPF_LDX | BPF_PROBE_MEMSX | BPF_B:
1379
+
/* lgb %dst,0(off,%src) */
1380
+
EMIT6_DISP_LH(0xe3000000, 0x0077, dst_reg, src_reg, REG_0, off);
1381
+
jit->seen |= SEEN_MEM;
1382
+
break;
1493
1383
case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
1494
1384
case BPF_LDX | BPF_PROBE_MEM | BPF_H:
1495
1385
/* llgh %dst,0(off,%src) */
···
1504
1382
if (insn_is_zext(&insn[1]))
1505
1383
insn_count = 2;
1506
1384
break;
1385
+
case BPF_LDX | BPF_MEMSX | BPF_H: /* dst = *(s16 *)(ul) (src + off) */
1386
+
case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
1387
+
/* lgh %dst,0(off,%src) */
1388
+
EMIT6_DISP_LH(0xe3000000, 0x0015, dst_reg, src_reg, REG_0, off);
1389
+
jit->seen |= SEEN_MEM;
1390
+
break;
1507
1391
case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
1508
1392
case BPF_LDX | BPF_PROBE_MEM | BPF_W:
1509
1393
/* llgf %dst,off(%src) */
···
1517
1389
EMIT6_DISP_LH(0xe3000000, 0x0016, dst_reg, src_reg, REG_0, off);
1518
1390
if (insn_is_zext(&insn[1]))
1519
1391
insn_count = 2;
1392
+
break;
1393
+
case BPF_LDX | BPF_MEMSX | BPF_W: /* dst = *(s32 *)(ul) (src + off) */
1394
+
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
1395
+
/* lgf %dst,off(%src) */
1396
+
jit->seen |= SEEN_MEM;
1397
+
EMIT6_DISP_LH(0xe3000000, 0x0014, dst_reg, src_reg, REG_0, off);
1520
1398
break;
1521
1399
case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
1522
1400
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
···
1704
1570
* instruction itself (loop) and for BPF with offset 0 we
1705
1571
* branch to the instruction behind the branch.
1706
1572
*/
1573
+
case BPF_JMP32 | BPF_JA: /* if (true) */
1574
+
branch_oc_off = imm;
1575
+
fallthrough;
1707
1576
case BPF_JMP | BPF_JA: /* if (true) */
1708
1577
mask = 0xf000; /* j */
1709
1578
goto branch_oc;
···
1875
1738
break;
1876
1739
branch_oc:
1877
1740
if (!is_first_pass(jit) &&
1878
-
can_use_rel(jit, addrs[i + off + 1])) {
1741
+
can_use_rel(jit, addrs[i + branch_oc_off + 1])) {
1879
1742
/* brc mask,off */
1880
1743
EMIT4_PCREL_RIC(0xa7040000,
1881
-
mask >> 12, addrs[i + off + 1]);
1744
+
mask >> 12,
1745
+
addrs[i + branch_oc_off + 1]);
1882
1746
} else {
1883
1747
/* brcl mask,off */
1884
1748
EMIT6_PCREL_RILC(0xc0040000,
1885
-
mask >> 12, addrs[i + off + 1]);
1749
+
mask >> 12,
1750
+
addrs[i + branch_oc_off + 1]);
1886
1751
}
1887
1752
break;
1888
1753
}
+5
include/linux/bpf-cgroup-defs.h
+5
include/linux/bpf-cgroup-defs.h
···
28
28
CGROUP_INET6_BIND,
29
29
CGROUP_INET4_CONNECT,
30
30
CGROUP_INET6_CONNECT,
31
+
CGROUP_UNIX_CONNECT,
31
32
CGROUP_INET4_POST_BIND,
32
33
CGROUP_INET6_POST_BIND,
33
34
CGROUP_UDP4_SENDMSG,
34
35
CGROUP_UDP6_SENDMSG,
36
+
CGROUP_UNIX_SENDMSG,
35
37
CGROUP_SYSCTL,
36
38
CGROUP_UDP4_RECVMSG,
37
39
CGROUP_UDP6_RECVMSG,
40
+
CGROUP_UNIX_RECVMSG,
38
41
CGROUP_GETSOCKOPT,
39
42
CGROUP_SETSOCKOPT,
40
43
CGROUP_INET4_GETPEERNAME,
41
44
CGROUP_INET6_GETPEERNAME,
45
+
CGROUP_UNIX_GETPEERNAME,
42
46
CGROUP_INET4_GETSOCKNAME,
43
47
CGROUP_INET6_GETSOCKNAME,
48
+
CGROUP_UNIX_GETSOCKNAME,
44
49
CGROUP_INET_SOCK_RELEASE,
45
50
CGROUP_LSM_START,
46
51
CGROUP_LSM_END = CGROUP_LSM_START + CGROUP_LSM_NUM - 1,
+54
-36
include/linux/bpf-cgroup.h
+54
-36
include/linux/bpf-cgroup.h
···
48
48
CGROUP_ATYPE(CGROUP_INET6_BIND);
49
49
CGROUP_ATYPE(CGROUP_INET4_CONNECT);
50
50
CGROUP_ATYPE(CGROUP_INET6_CONNECT);
51
+
CGROUP_ATYPE(CGROUP_UNIX_CONNECT);
51
52
CGROUP_ATYPE(CGROUP_INET4_POST_BIND);
52
53
CGROUP_ATYPE(CGROUP_INET6_POST_BIND);
53
54
CGROUP_ATYPE(CGROUP_UDP4_SENDMSG);
54
55
CGROUP_ATYPE(CGROUP_UDP6_SENDMSG);
56
+
CGROUP_ATYPE(CGROUP_UNIX_SENDMSG);
55
57
CGROUP_ATYPE(CGROUP_SYSCTL);
56
58
CGROUP_ATYPE(CGROUP_UDP4_RECVMSG);
57
59
CGROUP_ATYPE(CGROUP_UDP6_RECVMSG);
60
+
CGROUP_ATYPE(CGROUP_UNIX_RECVMSG);
58
61
CGROUP_ATYPE(CGROUP_GETSOCKOPT);
59
62
CGROUP_ATYPE(CGROUP_SETSOCKOPT);
60
63
CGROUP_ATYPE(CGROUP_INET4_GETPEERNAME);
61
64
CGROUP_ATYPE(CGROUP_INET6_GETPEERNAME);
65
+
CGROUP_ATYPE(CGROUP_UNIX_GETPEERNAME);
62
66
CGROUP_ATYPE(CGROUP_INET4_GETSOCKNAME);
63
67
CGROUP_ATYPE(CGROUP_INET6_GETSOCKNAME);
68
+
CGROUP_ATYPE(CGROUP_UNIX_GETSOCKNAME);
64
69
CGROUP_ATYPE(CGROUP_INET_SOCK_RELEASE);
65
70
default:
66
71
return CGROUP_BPF_ATTACH_TYPE_INVALID;
···
125
120
126
121
int __cgroup_bpf_run_filter_sock_addr(struct sock *sk,
127
122
struct sockaddr *uaddr,
123
+
int *uaddrlen,
128
124
enum cgroup_bpf_attach_type atype,
129
125
void *t_ctx,
130
126
u32 *flags);
···
236
230
#define BPF_CGROUP_RUN_PROG_INET6_POST_BIND(sk) \
237
231
BPF_CGROUP_RUN_SK_PROG(sk, CGROUP_INET6_POST_BIND)
238
232
239
-
#define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, atype) \
233
+
#define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, atype) \
240
234
({ \
241
235
int __ret = 0; \
242
236
if (cgroup_bpf_enabled(atype)) \
243
-
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, atype, \
244
-
NULL, NULL); \
237
+
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, uaddrlen, \
238
+
atype, NULL, NULL); \
245
239
__ret; \
246
240
})
247
241
248
-
#define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, atype, t_ctx) \
242
+
#define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, atype, t_ctx) \
249
243
({ \
250
244
int __ret = 0; \
251
245
if (cgroup_bpf_enabled(atype)) { \
252
246
lock_sock(sk); \
253
-
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, atype, \
254
-
t_ctx, NULL); \
247
+
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, uaddrlen, \
248
+
atype, t_ctx, NULL); \
255
249
release_sock(sk); \
256
250
} \
257
251
__ret; \
···
262
256
* (at bit position 0) is to indicate CAP_NET_BIND_SERVICE capability check
263
257
* should be bypassed (BPF_RET_BIND_NO_CAP_NET_BIND_SERVICE).
264
258
*/
265
-
#define BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, atype, bind_flags) \
259
+
#define BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, uaddrlen, atype, bind_flags) \
266
260
({ \
267
261
u32 __flags = 0; \
268
262
int __ret = 0; \
269
263
if (cgroup_bpf_enabled(atype)) { \
270
264
lock_sock(sk); \
271
-
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, atype, \
272
-
NULL, &__flags); \
265
+
__ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, uaddrlen, \
266
+
atype, NULL, &__flags); \
273
267
release_sock(sk); \
274
268
if (__flags & BPF_RET_BIND_NO_CAP_NET_BIND_SERVICE) \
275
269
*bind_flags |= BIND_NO_CAP_NET_BIND_SERVICE; \
···
282
276
cgroup_bpf_enabled(CGROUP_INET6_CONNECT)) && \
283
277
(sk)->sk_prot->pre_connect)
284
278
285
-
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT(sk, uaddr) \
286
-
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, CGROUP_INET4_CONNECT)
279
+
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT(sk, uaddr, uaddrlen) \
280
+
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, CGROUP_INET4_CONNECT)
287
281
288
-
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT(sk, uaddr) \
289
-
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, CGROUP_INET6_CONNECT)
282
+
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT(sk, uaddr, uaddrlen) \
283
+
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, CGROUP_INET6_CONNECT)
290
284
291
-
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr) \
292
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_INET4_CONNECT, NULL)
285
+
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr, uaddrlen) \
286
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_INET4_CONNECT, NULL)
293
287
294
-
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr) \
295
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_INET6_CONNECT, NULL)
288
+
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr, uaddrlen) \
289
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_INET6_CONNECT, NULL)
296
290
297
-
#define BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK(sk, uaddr, t_ctx) \
298
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_UDP4_SENDMSG, t_ctx)
291
+
#define BPF_CGROUP_RUN_PROG_UNIX_CONNECT_LOCK(sk, uaddr, uaddrlen) \
292
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UNIX_CONNECT, NULL)
299
293
300
-
#define BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, uaddr, t_ctx) \
301
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_UDP6_SENDMSG, t_ctx)
294
+
#define BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) \
295
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UDP4_SENDMSG, t_ctx)
302
296
303
-
#define BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk, uaddr) \
304
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_UDP4_RECVMSG, NULL)
297
+
#define BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) \
298
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UDP6_SENDMSG, t_ctx)
305
299
306
-
#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr) \
307
-
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, CGROUP_UDP6_RECVMSG, NULL)
300
+
#define BPF_CGROUP_RUN_PROG_UNIX_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) \
301
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UNIX_SENDMSG, t_ctx)
302
+
303
+
#define BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk, uaddr, uaddrlen) \
304
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UDP4_RECVMSG, NULL)
305
+
306
+
#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr, uaddrlen) \
307
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UDP6_RECVMSG, NULL)
308
+
309
+
#define BPF_CGROUP_RUN_PROG_UNIX_RECVMSG_LOCK(sk, uaddr, uaddrlen) \
310
+
BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, CGROUP_UNIX_RECVMSG, NULL)
308
311
309
312
/* The SOCK_OPS"_SK" macro should be used when sock_ops->sk is not a
310
313
* fullsock and its parent fullsock cannot be traced by
···
492
477
}
493
478
494
479
#define cgroup_bpf_enabled(atype) (0)
495
-
#define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, atype, t_ctx) ({ 0; })
496
-
#define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, atype) ({ 0; })
480
+
#define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, atype, t_ctx) ({ 0; })
481
+
#define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, atype) ({ 0; })
497
482
#define BPF_CGROUP_PRE_CONNECT_ENABLED(sk) (0)
498
483
#define BPF_CGROUP_RUN_PROG_INET_INGRESS(sk,skb) ({ 0; })
499
484
#define BPF_CGROUP_RUN_PROG_INET_EGRESS(sk,skb) ({ 0; })
500
485
#define BPF_CGROUP_RUN_PROG_INET_SOCK(sk) ({ 0; })
501
486
#define BPF_CGROUP_RUN_PROG_INET_SOCK_RELEASE(sk) ({ 0; })
502
-
#define BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, atype, flags) ({ 0; })
487
+
#define BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, uaddrlen, atype, flags) ({ 0; })
503
488
#define BPF_CGROUP_RUN_PROG_INET4_POST_BIND(sk) ({ 0; })
504
489
#define BPF_CGROUP_RUN_PROG_INET6_POST_BIND(sk) ({ 0; })
505
-
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT(sk, uaddr) ({ 0; })
506
-
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr) ({ 0; })
507
-
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT(sk, uaddr) ({ 0; })
508
-
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr) ({ 0; })
509
-
#define BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK(sk, uaddr, t_ctx) ({ 0; })
510
-
#define BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, uaddr, t_ctx) ({ 0; })
511
-
#define BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk, uaddr) ({ 0; })
512
-
#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr) ({ 0; })
490
+
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT(sk, uaddr, uaddrlen) ({ 0; })
491
+
#define BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr, uaddrlen) ({ 0; })
492
+
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT(sk, uaddr, uaddrlen) ({ 0; })
493
+
#define BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr, uaddrlen) ({ 0; })
494
+
#define BPF_CGROUP_RUN_PROG_UNIX_CONNECT_LOCK(sk, uaddr, uaddrlen) ({ 0; })
495
+
#define BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) ({ 0; })
496
+
#define BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) ({ 0; })
497
+
#define BPF_CGROUP_RUN_PROG_UNIX_SENDMSG_LOCK(sk, uaddr, uaddrlen, t_ctx) ({ 0; })
498
+
#define BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
499
+
#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
500
+
#define BPF_CGROUP_RUN_PROG_UNIX_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
513
501
#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) ({ 0; })
514
502
#define BPF_CGROUP_RUN_PROG_DEVICE_CGROUP(atype, major, minor, access) ({ 0; })
515
503
#define BPF_CGROUP_RUN_PROG_SYSCTL(head,table,write,buf,count,pos) ({ 0; })
+18
-2
include/linux/bpf.h
+18
-2
include/linux/bpf.h
···
2164
2164
2165
2165
static inline bool bpf_bypass_spec_v1(void)
2166
2166
{
2167
-
return perfmon_capable();
2167
+
return cpu_mitigations_off() || perfmon_capable();
2168
2168
}
2169
2169
2170
2170
static inline bool bpf_bypass_spec_v4(void)
2171
2171
{
2172
-
return perfmon_capable();
2172
+
return cpu_mitigations_off() || perfmon_capable();
2173
2173
}
2174
2174
2175
2175
int bpf_map_new_fd(struct bpf_map *map, int flags);
···
2921
2921
}
2922
2922
#endif /* CONFIG_BPF_SYSCALL */
2923
2923
#endif /* CONFIG_NET && CONFIG_BPF_SYSCALL */
2924
+
2925
+
static __always_inline void
2926
+
bpf_prog_inc_misses_counters(const struct bpf_prog_array *array)
2927
+
{
2928
+
const struct bpf_prog_array_item *item;
2929
+
struct bpf_prog *prog;
2930
+
2931
+
if (unlikely(!array))
2932
+
return;
2933
+
2934
+
item = &array->items[0];
2935
+
while ((prog = READ_ONCE(item->prog))) {
2936
+
bpf_prog_inc_misses_counter(prog);
2937
+
item++;
2938
+
}
2939
+
}
2924
2940
2925
2941
#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
2926
2942
void bpf_sk_reuseport_detach(struct sock *sk);
+2
-1
include/linux/filter.h
+2
-1
include/linux/filter.h
···
736
736
cb->data_end = skb->data + skb_headlen(skb);
737
737
}
738
738
739
-
/* Restore data saved by bpf_compute_data_pointers(). */
739
+
/* Restore data saved by bpf_compute_and_save_data_end(). */
740
740
static inline void bpf_restore_data_end(
741
741
struct sk_buff *skb, void *saved_data_end)
742
742
{
···
1329
1329
*/
1330
1330
u64 tmp_reg;
1331
1331
void *t_ctx; /* Attach type specific context. */
1332
+
u32 uaddrlen;
1332
1333
};
1333
1334
1334
1335
struct bpf_sock_ops_kern {
+4
-2
include/linux/trace_events.h
+4
-2
include/linux/trace_events.h
···
761
761
void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp);
762
762
int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
763
763
u32 *fd_type, const char **buf,
764
-
u64 *probe_offset, u64 *probe_addr);
764
+
u64 *probe_offset, u64 *probe_addr,
765
+
unsigned long *missed);
765
766
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
766
767
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
767
768
#else
···
802
801
static inline int bpf_get_perf_event_info(const struct perf_event *event,
803
802
u32 *prog_id, u32 *fd_type,
804
803
const char **buf, u64 *probe_offset,
805
-
u64 *probe_addr)
804
+
u64 *probe_addr, unsigned long *missed)
806
805
{
807
806
return -EOPNOTSUPP;
808
807
}
···
878
877
extern int bpf_get_kprobe_info(const struct perf_event *event,
879
878
u32 *fd_type, const char **symbol,
880
879
u64 *probe_offset, u64 *probe_addr,
880
+
unsigned long *missed,
881
881
bool perf_type_tracepoint);
882
882
#endif
883
883
#ifdef CONFIG_UPROBE_EVENTS
+5
include/net/ipv6_stubs.h
+5
include/net/ipv6_stubs.h
···
85
85
sockptr_t optval, unsigned int optlen);
86
86
int (*ipv6_getsockopt)(struct sock *sk, int level, int optname,
87
87
sockptr_t optval, sockptr_t optlen);
88
+
int (*ipv6_dev_get_saddr)(struct net *net,
89
+
const struct net_device *dst_dev,
90
+
const struct in6_addr *daddr,
91
+
unsigned int prefs,
92
+
struct in6_addr *saddr);
88
93
};
89
94
extern const struct ipv6_bpf_stub *ipv6_bpf_stub __read_mostly;
90
95
+25
-4
include/uapi/linux/bpf.h
+25
-4
include/uapi/linux/bpf.h
···
1047
1047
BPF_TCX_INGRESS,
1048
1048
BPF_TCX_EGRESS,
1049
1049
BPF_TRACE_UPROBE_MULTI,
1050
+
BPF_CGROUP_UNIX_CONNECT,
1051
+
BPF_CGROUP_UNIX_SENDMSG,
1052
+
BPF_CGROUP_UNIX_RECVMSG,
1053
+
BPF_CGROUP_UNIX_GETPEERNAME,
1054
+
BPF_CGROUP_UNIX_GETSOCKNAME,
1050
1055
__MAX_BPF_ATTACH_TYPE
1051
1056
};
1052
1057
···
2709
2704
* *bpf_socket* should be one of the following:
2710
2705
*
2711
2706
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
2712
-
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**
2713
-
* and **BPF_CGROUP_INET6_CONNECT**.
2707
+
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
2708
+
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
2714
2709
*
2715
2710
* This helper actually implements a subset of **setsockopt()**.
2716
2711
* It supports the following *level*\ s:
···
2948
2943
* *bpf_socket* should be one of the following:
2949
2944
*
2950
2945
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
2951
-
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**
2952
-
* and **BPF_CGROUP_INET6_CONNECT**.
2946
+
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
2947
+
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
2953
2948
*
2954
2949
* This helper actually implements a subset of **getsockopt()**.
2955
2950
* It supports the same set of *optname*\ s that is supported by
···
3269
3264
* and *params*->smac will not be set as output. A common
3270
3265
* use case is to call **bpf_redirect_neigh**\ () after
3271
3266
* doing **bpf_fib_lookup**\ ().
3267
+
* **BPF_FIB_LOOKUP_SRC**
3268
+
* Derive and set source IP addr in *params*->ipv{4,6}_src
3269
+
* for the nexthop. If the src addr cannot be derived,
3270
+
* **BPF_FIB_LKUP_RET_NO_SRC_ADDR** is returned. In this
3271
+
* case, *params*->dmac and *params*->smac are not set either.
3272
3272
*
3273
3273
* *ctx* is either **struct xdp_md** for XDP programs or
3274
3274
* **struct sk_buff** tc cls_act programs.
···
5106
5096
* **BPF_F_TIMER_ABS**
5107
5097
* Start the timer in absolute expire value instead of the
5108
5098
* default relative one.
5099
+
* **BPF_F_TIMER_CPU_PIN**
5100
+
* Timer will be pinned to the CPU of the caller.
5109
5101
*
5110
5102
* Return
5111
5103
* 0 on success.
···
6544
6532
__aligned_u64 addrs;
6545
6533
__u32 count; /* in/out: kprobe_multi function count */
6546
6534
__u32 flags;
6535
+
__u64 missed;
6547
6536
} kprobe_multi;
6548
6537
struct {
6549
6538
__u32 type; /* enum bpf_perf_event_type */
···
6560
6547
__u32 name_len;
6561
6548
__u32 offset; /* offset from func_name */
6562
6549
__u64 addr;
6550
+
__u64 missed;
6563
6551
} kprobe; /* BPF_PERF_EVENT_KPROBE, BPF_PERF_EVENT_KRETPROBE */
6564
6552
struct {
6565
6553
__aligned_u64 tp_name; /* in/out */
···
6974
6960
BPF_FIB_LOOKUP_OUTPUT = (1U << 1),
6975
6961
BPF_FIB_LOOKUP_SKIP_NEIGH = (1U << 2),
6976
6962
BPF_FIB_LOOKUP_TBID = (1U << 3),
6963
+
BPF_FIB_LOOKUP_SRC = (1U << 4),
6977
6964
};
6978
6965
6979
6966
enum {
···
6987
6972
BPF_FIB_LKUP_RET_UNSUPP_LWT, /* fwd requires encapsulation */
6988
6973
BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */
6989
6974
BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */
6975
+
BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */
6990
6976
};
6991
6977
6992
6978
struct bpf_fib_lookup {
···
7022
7006
__u32 rt_metric;
7023
7007
};
7024
7008
7009
+
/* input: source address to consider for lookup
7010
+
* output: source address result from lookup
7011
+
*/
7025
7012
union {
7026
7013
__be32 ipv4_src;
7027
7014
__u32 ipv6_src[4]; /* in6_addr; network order */
···
7326
7307
* Flags to control bpf_timer_start() behaviour.
7327
7308
* - BPF_F_TIMER_ABS: Timeout passed is absolute time, by default it is
7328
7309
* relative to current time.
7310
+
* - BPF_F_TIMER_CPU_PIN: Timer will be pinned to the CPU of the caller.
7329
7311
*/
7330
7312
enum {
7331
7313
BPF_F_TIMER_ABS = (1ULL << 0),
7314
+
BPF_F_TIMER_CPU_PIN = (1ULL << 1),
7332
7315
};
7333
7316
7334
7317
/* BPF numbers iterator state */
-2
kernel/bpf/bpf_iter.c
-2
kernel/bpf/bpf_iter.c
···
793
793
BUILD_BUG_ON(sizeof(struct bpf_iter_num_kern) != sizeof(struct bpf_iter_num));
794
794
BUILD_BUG_ON(__alignof__(struct bpf_iter_num_kern) != __alignof__(struct bpf_iter_num));
795
795
796
-
BTF_TYPE_EMIT(struct btf_iter_num);
797
-
798
796
/* start == end is legit, it's an empty range and we'll just get NULL
799
797
* on first (and any subsequent) bpf_iter_num_next() call
800
798
*/
+1
kernel/bpf/btf.c
+1
kernel/bpf/btf.c
+24
-4
kernel/bpf/cgroup.c
+24
-4
kernel/bpf/cgroup.c
···
1450
1450
* provided by user sockaddr
1451
1451
* @sk: sock struct that will use sockaddr
1452
1452
* @uaddr: sockaddr struct provided by user
1453
+
* @uaddrlen: Pointer to the size of the sockaddr struct provided by user. It is
1454
+
* read-only for AF_INET[6] uaddr but can be modified for AF_UNIX
1455
+
* uaddr.
1453
1456
* @atype: The type of program to be executed
1454
1457
* @t_ctx: Pointer to attach type specific context
1455
1458
* @flags: Pointer to u32 which contains higher bits of BPF program
1456
1459
* return value (OR'ed together).
1457
1460
*
1458
-
* socket is expected to be of type INET or INET6.
1461
+
* socket is expected to be of type INET, INET6 or UNIX.
1459
1462
*
1460
1463
* This function will return %-EPERM if an attached program is found and
1461
1464
* returned value != 1 during execution. In all other cases, 0 is returned.
1462
1465
*/
1463
1466
int __cgroup_bpf_run_filter_sock_addr(struct sock *sk,
1464
1467
struct sockaddr *uaddr,
1468
+
int *uaddrlen,
1465
1469
enum cgroup_bpf_attach_type atype,
1466
1470
void *t_ctx,
1467
1471
u32 *flags)
···
1477
1473
};
1478
1474
struct sockaddr_storage unspec;
1479
1475
struct cgroup *cgrp;
1476
+
int ret;
1480
1477
1481
1478
/* Check socket family since not all sockets represent network
1482
1479
* endpoint (e.g. AF_UNIX).
1483
1480
*/
1484
-
if (sk->sk_family != AF_INET && sk->sk_family != AF_INET6)
1481
+
if (sk->sk_family != AF_INET && sk->sk_family != AF_INET6 &&
1482
+
sk->sk_family != AF_UNIX)
1485
1483
return 0;
1486
1484
1487
1485
if (!ctx.uaddr) {
1488
1486
memset(&unspec, 0, sizeof(unspec));
1489
1487
ctx.uaddr = (struct sockaddr *)&unspec;
1488
+
ctx.uaddrlen = 0;
1489
+
} else {
1490
+
ctx.uaddrlen = *uaddrlen;
1490
1491
}
1491
1492
1492
1493
cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data);
1493
-
return bpf_prog_run_array_cg(&cgrp->bpf, atype, &ctx, bpf_prog_run,
1494
-
0, flags);
1494
+
ret = bpf_prog_run_array_cg(&cgrp->bpf, atype, &ctx, bpf_prog_run,
1495
+
0, flags);
1496
+
1497
+
if (!ret && uaddr)
1498
+
*uaddrlen = ctx.uaddrlen;
1499
+
1500
+
return ret;
1495
1501
}
1496
1502
EXPORT_SYMBOL(__cgroup_bpf_run_filter_sock_addr);
1497
1503
···
2534
2520
case BPF_CGROUP_SOCK_OPS:
2535
2521
case BPF_CGROUP_UDP4_RECVMSG:
2536
2522
case BPF_CGROUP_UDP6_RECVMSG:
2523
+
case BPF_CGROUP_UNIX_RECVMSG:
2537
2524
case BPF_CGROUP_INET4_GETPEERNAME:
2538
2525
case BPF_CGROUP_INET6_GETPEERNAME:
2526
+
case BPF_CGROUP_UNIX_GETPEERNAME:
2539
2527
case BPF_CGROUP_INET4_GETSOCKNAME:
2540
2528
case BPF_CGROUP_INET6_GETSOCKNAME:
2529
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
2541
2530
return NULL;
2542
2531
default:
2543
2532
return &bpf_get_retval_proto;
···
2552
2535
case BPF_CGROUP_SOCK_OPS:
2553
2536
case BPF_CGROUP_UDP4_RECVMSG:
2554
2537
case BPF_CGROUP_UDP6_RECVMSG:
2538
+
case BPF_CGROUP_UNIX_RECVMSG:
2555
2539
case BPF_CGROUP_INET4_GETPEERNAME:
2556
2540
case BPF_CGROUP_INET6_GETPEERNAME:
2541
+
case BPF_CGROUP_UNIX_GETPEERNAME:
2557
2542
case BPF_CGROUP_INET4_GETSOCKNAME:
2558
2543
case BPF_CGROUP_INET6_GETSOCKNAME:
2544
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
2559
2545
return NULL;
2560
2546
default:
2561
2547
return &bpf_set_retval_proto;
+7
-1
kernel/bpf/helpers.c
+7
-1
kernel/bpf/helpers.c
···
1272
1272
1273
1273
if (in_nmi())
1274
1274
return -EOPNOTSUPP;
1275
-
if (flags > BPF_F_TIMER_ABS)
1275
+
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
1276
1276
return -EINVAL;
1277
1277
__bpf_spin_lock_irqsave(&timer->lock);
1278
1278
t = timer->timer;
···
1285
1285
mode = HRTIMER_MODE_ABS_SOFT;
1286
1286
else
1287
1287
mode = HRTIMER_MODE_REL_SOFT;
1288
+
1289
+
if (flags & BPF_F_TIMER_CPU_PIN)
1290
+
mode |= HRTIMER_MODE_PINNED;
1288
1291
1289
1292
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
1290
1293
out:
···
2552
2549
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
2553
2550
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
2554
2551
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
2552
+
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
2553
+
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
2554
+
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
2555
2555
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
2556
2556
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
2557
2557
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
+1
-1
kernel/bpf/stackmap.c
+1
-1
kernel/bpf/stackmap.c
+23
-6
kernel/bpf/syscall.c
+23
-6
kernel/bpf/syscall.c
···
2446
2446
case BPF_CGROUP_INET6_BIND:
2447
2447
case BPF_CGROUP_INET4_CONNECT:
2448
2448
case BPF_CGROUP_INET6_CONNECT:
2449
+
case BPF_CGROUP_UNIX_CONNECT:
2449
2450
case BPF_CGROUP_INET4_GETPEERNAME:
2450
2451
case BPF_CGROUP_INET6_GETPEERNAME:
2452
+
case BPF_CGROUP_UNIX_GETPEERNAME:
2451
2453
case BPF_CGROUP_INET4_GETSOCKNAME:
2452
2454
case BPF_CGROUP_INET6_GETSOCKNAME:
2455
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
2453
2456
case BPF_CGROUP_UDP4_SENDMSG:
2454
2457
case BPF_CGROUP_UDP6_SENDMSG:
2458
+
case BPF_CGROUP_UNIX_SENDMSG:
2455
2459
case BPF_CGROUP_UDP4_RECVMSG:
2456
2460
case BPF_CGROUP_UDP6_RECVMSG:
2461
+
case BPF_CGROUP_UNIX_RECVMSG:
2457
2462
return 0;
2458
2463
default:
2459
2464
return -EINVAL;
···
3379
3374
static int bpf_perf_link_fill_common(const struct perf_event *event,
3380
3375
char __user *uname, u32 ulen,
3381
3376
u64 *probe_offset, u64 *probe_addr,
3382
-
u32 *fd_type)
3377
+
u32 *fd_type, unsigned long *missed)
3383
3378
{
3384
3379
const char *buf;
3385
3380
u32 prog_id;
···
3390
3385
return -EINVAL;
3391
3386
3392
3387
err = bpf_get_perf_event_info(event, &prog_id, fd_type, &buf,
3393
-
probe_offset, probe_addr);
3388
+
probe_offset, probe_addr, missed);
3394
3389
if (err)
3395
3390
return err;
3396
3391
if (!uname)
···
3413
3408
static int bpf_perf_link_fill_kprobe(const struct perf_event *event,
3414
3409
struct bpf_link_info *info)
3415
3410
{
3411
+
unsigned long missed;
3416
3412
char __user *uname;
3417
3413
u64 addr, offset;
3418
3414
u32 ulen, type;
···
3422
3416
uname = u64_to_user_ptr(info->perf_event.kprobe.func_name);
3423
3417
ulen = info->perf_event.kprobe.name_len;
3424
3418
err = bpf_perf_link_fill_common(event, uname, ulen, &offset, &addr,
3425
-
&type);
3419
+
&type, &missed);
3426
3420
if (err)
3427
3421
return err;
3428
3422
if (type == BPF_FD_TYPE_KRETPROBE)
···
3431
3425
info->perf_event.type = BPF_PERF_EVENT_KPROBE;
3432
3426
3433
3427
info->perf_event.kprobe.offset = offset;
3428
+
info->perf_event.kprobe.missed = missed;
3434
3429
if (!kallsyms_show_value(current_cred()))
3435
3430
addr = 0;
3436
3431
info->perf_event.kprobe.addr = addr;
···
3451
3444
uname = u64_to_user_ptr(info->perf_event.uprobe.file_name);
3452
3445
ulen = info->perf_event.uprobe.name_len;
3453
3446
err = bpf_perf_link_fill_common(event, uname, ulen, &offset, &addr,
3454
-
&type);
3447
+
&type, NULL);
3455
3448
if (err)
3456
3449
return err;
3457
3450
···
3487
3480
uname = u64_to_user_ptr(info->perf_event.tracepoint.tp_name);
3488
3481
ulen = info->perf_event.tracepoint.name_len;
3489
3482
info->perf_event.type = BPF_PERF_EVENT_TRACEPOINT;
3490
-
return bpf_perf_link_fill_common(event, uname, ulen, NULL, NULL, NULL);
3483
+
return bpf_perf_link_fill_common(event, uname, ulen, NULL, NULL, NULL, NULL);
3491
3484
}
3492
3485
3493
3486
static int bpf_perf_link_fill_perf_event(const struct perf_event *event,
···
3683
3676
case BPF_CGROUP_INET6_BIND:
3684
3677
case BPF_CGROUP_INET4_CONNECT:
3685
3678
case BPF_CGROUP_INET6_CONNECT:
3679
+
case BPF_CGROUP_UNIX_CONNECT:
3686
3680
case BPF_CGROUP_INET4_GETPEERNAME:
3687
3681
case BPF_CGROUP_INET6_GETPEERNAME:
3682
+
case BPF_CGROUP_UNIX_GETPEERNAME:
3688
3683
case BPF_CGROUP_INET4_GETSOCKNAME:
3689
3684
case BPF_CGROUP_INET6_GETSOCKNAME:
3685
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
3690
3686
case BPF_CGROUP_UDP4_SENDMSG:
3691
3687
case BPF_CGROUP_UDP6_SENDMSG:
3688
+
case BPF_CGROUP_UNIX_SENDMSG:
3692
3689
case BPF_CGROUP_UDP4_RECVMSG:
3693
3690
case BPF_CGROUP_UDP6_RECVMSG:
3691
+
case BPF_CGROUP_UNIX_RECVMSG:
3694
3692
return BPF_PROG_TYPE_CGROUP_SOCK_ADDR;
3695
3693
case BPF_CGROUP_SOCK_OPS:
3696
3694
return BPF_PROG_TYPE_SOCK_OPS;
···
3961
3949
case BPF_CGROUP_INET6_POST_BIND:
3962
3950
case BPF_CGROUP_INET4_CONNECT:
3963
3951
case BPF_CGROUP_INET6_CONNECT:
3952
+
case BPF_CGROUP_UNIX_CONNECT:
3964
3953
case BPF_CGROUP_INET4_GETPEERNAME:
3965
3954
case BPF_CGROUP_INET6_GETPEERNAME:
3955
+
case BPF_CGROUP_UNIX_GETPEERNAME:
3966
3956
case BPF_CGROUP_INET4_GETSOCKNAME:
3967
3957
case BPF_CGROUP_INET6_GETSOCKNAME:
3958
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
3968
3959
case BPF_CGROUP_UDP4_SENDMSG:
3969
3960
case BPF_CGROUP_UDP6_SENDMSG:
3961
+
case BPF_CGROUP_UNIX_SENDMSG:
3970
3962
case BPF_CGROUP_UDP4_RECVMSG:
3971
3963
case BPF_CGROUP_UDP6_RECVMSG:
3964
+
case BPF_CGROUP_UNIX_RECVMSG:
3972
3965
case BPF_CGROUP_SOCK_OPS:
3973
3966
case BPF_CGROUP_DEVICE:
3974
3967
case BPF_CGROUP_SYSCTL:
···
4839
4822
4840
4823
err = bpf_get_perf_event_info(event, &prog_id, &fd_type,
4841
4824
&buf, &probe_offset,
4842
-
&probe_addr);
4825
+
&probe_addr, NULL);
4843
4826
if (!err)
4844
4827
err = bpf_task_fd_query_copy(attr, uattr, prog_id,
4845
4828
fd_type, buf,
+91
kernel/bpf/task_iter.c
+91
kernel/bpf/task_iter.c
···
7
7
#include <linux/fs.h>
8
8
#include <linux/fdtable.h>
9
9
#include <linux/filter.h>
10
+
#include <linux/bpf_mem_alloc.h>
10
11
#include <linux/btf_ids.h>
12
+
#include <linux/mm_types.h>
11
13
#include "mmap_unlock_work.h"
12
14
13
15
static const char * const iter_task_type_names[] = {
···
804
802
.arg4_type = ARG_PTR_TO_STACK_OR_NULL,
805
803
.arg5_type = ARG_ANYTHING,
806
804
};
805
+
806
+
struct bpf_iter_task_vma_kern_data {
807
+
struct task_struct *task;
808
+
struct mm_struct *mm;
809
+
struct mmap_unlock_irq_work *work;
810
+
struct vma_iterator vmi;
811
+
};
812
+
813
+
struct bpf_iter_task_vma {
814
+
/* opaque iterator state; having __u64 here allows to preserve correct
815
+
* alignment requirements in vmlinux.h, generated from BTF
816
+
*/
817
+
__u64 __opaque[1];
818
+
} __attribute__((aligned(8)));
819
+
820
+
/* Non-opaque version of bpf_iter_task_vma */
821
+
struct bpf_iter_task_vma_kern {
822
+
struct bpf_iter_task_vma_kern_data *data;
823
+
} __attribute__((aligned(8)));
824
+
825
+
__diag_push();
826
+
__diag_ignore_all("-Wmissing-prototypes",
827
+
"Global functions as their definitions will be in vmlinux BTF");
828
+
829
+
__bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
830
+
struct task_struct *task, u64 addr)
831
+
{
832
+
struct bpf_iter_task_vma_kern *kit = (void *)it;
833
+
bool irq_work_busy = false;
834
+
int err;
835
+
836
+
BUILD_BUG_ON(sizeof(struct bpf_iter_task_vma_kern) != sizeof(struct bpf_iter_task_vma));
837
+
BUILD_BUG_ON(__alignof__(struct bpf_iter_task_vma_kern) != __alignof__(struct bpf_iter_task_vma));
838
+
839
+
/* is_iter_reg_valid_uninit guarantees that kit hasn't been initialized
840
+
* before, so non-NULL kit->data doesn't point to previously
841
+
* bpf_mem_alloc'd bpf_iter_task_vma_kern_data
842
+
*/
843
+
kit->data = bpf_mem_alloc(&bpf_global_ma, sizeof(struct bpf_iter_task_vma_kern_data));
844
+
if (!kit->data)
845
+
return -ENOMEM;
846
+
847
+
kit->data->task = get_task_struct(task);
848
+
kit->data->mm = task->mm;
849
+
if (!kit->data->mm) {
850
+
err = -ENOENT;
851
+
goto err_cleanup_iter;
852
+
}
853
+
854
+
/* kit->data->work == NULL is valid after bpf_mmap_unlock_get_irq_work */
855
+
irq_work_busy = bpf_mmap_unlock_get_irq_work(&kit->data->work);
856
+
if (irq_work_busy || !mmap_read_trylock(kit->data->mm)) {
857
+
err = -EBUSY;
858
+
goto err_cleanup_iter;
859
+
}
860
+
861
+
vma_iter_init(&kit->data->vmi, kit->data->mm, addr);
862
+
return 0;
863
+
864
+
err_cleanup_iter:
865
+
if (kit->data->task)
866
+
put_task_struct(kit->data->task);
867
+
bpf_mem_free(&bpf_global_ma, kit->data);
868
+
/* NULL kit->data signals failed bpf_iter_task_vma initialization */
869
+
kit->data = NULL;
870
+
return err;
871
+
}
872
+
873
+
__bpf_kfunc struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it)
874
+
{
875
+
struct bpf_iter_task_vma_kern *kit = (void *)it;
876
+
877
+
if (!kit->data) /* bpf_iter_task_vma_new failed */
878
+
return NULL;
879
+
return vma_next(&kit->data->vmi);
880
+
}
881
+
882
+
__bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
883
+
{
884
+
struct bpf_iter_task_vma_kern *kit = (void *)it;
885
+
886
+
if (kit->data) {
887
+
bpf_mmap_unlock_mm(kit->data->work, kit->data->mm);
888
+
put_task_struct(kit->data->task);
889
+
bpf_mem_free(&bpf_global_ma, kit->data);
890
+
}
891
+
}
892
+
893
+
__diag_pop();
807
894
808
895
DEFINE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
809
896
+56
-25
kernel/bpf/verifier.c
+56
-25
kernel/bpf/verifier.c
···
1342
1342
*stype = STACK_MISC;
1343
1343
}
1344
1344
1345
+
static void print_scalar_ranges(struct bpf_verifier_env *env,
1346
+
const struct bpf_reg_state *reg,
1347
+
const char **sep)
1348
+
{
1349
+
struct {
1350
+
const char *name;
1351
+
u64 val;
1352
+
bool omit;
1353
+
} minmaxs[] = {
1354
+
{"smin", reg->smin_value, reg->smin_value == S64_MIN},
1355
+
{"smax", reg->smax_value, reg->smax_value == S64_MAX},
1356
+
{"umin", reg->umin_value, reg->umin_value == 0},
1357
+
{"umax", reg->umax_value, reg->umax_value == U64_MAX},
1358
+
{"smin32", (s64)reg->s32_min_value, reg->s32_min_value == S32_MIN},
1359
+
{"smax32", (s64)reg->s32_max_value, reg->s32_max_value == S32_MAX},
1360
+
{"umin32", reg->u32_min_value, reg->u32_min_value == 0},
1361
+
{"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX},
1362
+
}, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)];
1363
+
bool neg1, neg2;
1364
+
1365
+
for (m1 = &minmaxs[0]; m1 < mend; m1++) {
1366
+
if (m1->omit)
1367
+
continue;
1368
+
1369
+
neg1 = m1->name[0] == 's' && (s64)m1->val < 0;
1370
+
1371
+
verbose(env, "%s%s=", *sep, m1->name);
1372
+
*sep = ",";
1373
+
1374
+
for (m2 = m1 + 2; m2 < mend; m2 += 2) {
1375
+
if (m2->omit || m2->val != m1->val)
1376
+
continue;
1377
+
/* don't mix negatives with positives */
1378
+
neg2 = m2->name[0] == 's' && (s64)m2->val < 0;
1379
+
if (neg2 != neg1)
1380
+
continue;
1381
+
m2->omit = true;
1382
+
verbose(env, "%s=", m2->name);
1383
+
}
1384
+
1385
+
verbose(env, m1->name[0] == 's' ? "%lld" : "%llu", m1->val);
1386
+
}
1387
+
}
1388
+
1345
1389
static void print_verifier_state(struct bpf_verifier_env *env,
1346
1390
const struct bpf_func_state *state,
1347
1391
bool print_all)
···
1449
1405
*/
1450
1406
verbose_a("imm=%llx", reg->var_off.value);
1451
1407
} else {
1452
-
if (reg->smin_value != reg->umin_value &&
1453
-
reg->smin_value != S64_MIN)
1454
-
verbose_a("smin=%lld", (long long)reg->smin_value);
1455
-
if (reg->smax_value != reg->umax_value &&
1456
-
reg->smax_value != S64_MAX)
1457
-
verbose_a("smax=%lld", (long long)reg->smax_value);
1458
-
if (reg->umin_value != 0)
1459
-
verbose_a("umin=%llu", (unsigned long long)reg->umin_value);
1460
-
if (reg->umax_value != U64_MAX)
1461
-
verbose_a("umax=%llu", (unsigned long long)reg->umax_value);
1408
+
print_scalar_ranges(env, reg, &sep);
1462
1409
if (!tnum_is_unknown(reg->var_off)) {
1463
1410
char tn_buf[48];
1464
1411
1465
1412
tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
1466
1413
verbose_a("var_off=%s", tn_buf);
1467
1414
}
1468
-
if (reg->s32_min_value != reg->smin_value &&
1469
-
reg->s32_min_value != S32_MIN)
1470
-
verbose_a("s32_min=%d", (int)(reg->s32_min_value));
1471
-
if (reg->s32_max_value != reg->smax_value &&
1472
-
reg->s32_max_value != S32_MAX)
1473
-
verbose_a("s32_max=%d", (int)(reg->s32_max_value));
1474
-
if (reg->u32_min_value != reg->umin_value &&
1475
-
reg->u32_min_value != U32_MIN)
1476
-
verbose_a("u32_min=%d", (int)(reg->u32_min_value));
1477
-
if (reg->u32_max_value != reg->umax_value &&
1478
-
reg->u32_max_value != U32_MAX)
1479
-
verbose_a("u32_max=%d", (int)(reg->u32_max_value));
1480
1415
}
1481
1416
#undef verbose_a
1482
1417
···
1539
1516
if (state->in_async_callback_fn)
1540
1517
verbose(env, " async_cb");
1541
1518
verbose(env, "\n");
1542
-
mark_verifier_state_clean(env);
1519
+
if (!print_all)
1520
+
mark_verifier_state_clean(env);
1543
1521
}
1544
1522
1545
1523
static inline u32 vlog_alignment(u32 pos)
···
3138
3114
3139
3115
if (class == BPF_LDX) {
3140
3116
if (t != SRC_OP)
3141
-
return BPF_SIZE(code) == BPF_DW;
3117
+
return BPF_SIZE(code) == BPF_DW || BPF_MODE(code) == BPF_MEMSX;
3142
3118
/* LDX source must be ptr. */
3143
3119
return true;
3144
3120
}
···
14407
14383
!sanitize_speculative_path(env, insn, *insn_idx + 1,
14408
14384
*insn_idx))
14409
14385
return -EFAULT;
14386
+
if (env->log.level & BPF_LOG_LEVEL)
14387
+
print_insn_state(env, this_branch->frame[this_branch->curframe]);
14410
14388
*insn_idx += insn->off;
14411
14389
return 0;
14412
14390
} else if (pred == 0) {
···
14421
14395
*insn_idx + insn->off + 1,
14422
14396
*insn_idx))
14423
14397
return -EFAULT;
14398
+
if (env->log.level & BPF_LOG_LEVEL)
14399
+
print_insn_state(env, this_branch->frame[this_branch->curframe]);
14424
14400
return 0;
14425
14401
}
14426
14402
···
14823
14795
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
14824
14796
if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||
14825
14797
env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||
14798
+
env->prog->expected_attach_type == BPF_CGROUP_UNIX_RECVMSG ||
14826
14799
env->prog->expected_attach_type == BPF_CGROUP_INET4_GETPEERNAME ||
14827
14800
env->prog->expected_attach_type == BPF_CGROUP_INET6_GETPEERNAME ||
14801
+
env->prog->expected_attach_type == BPF_CGROUP_UNIX_GETPEERNAME ||
14828
14802
env->prog->expected_attach_type == BPF_CGROUP_INET4_GETSOCKNAME ||
14829
-
env->prog->expected_attach_type == BPF_CGROUP_INET6_GETSOCKNAME)
14803
+
env->prog->expected_attach_type == BPF_CGROUP_INET6_GETSOCKNAME ||
14804
+
env->prog->expected_attach_type == BPF_CGROUP_UNIX_GETSOCKNAME)
14830
14805
range = tnum_range(1, 1);
14831
14806
if (env->prog->expected_attach_type == BPF_CGROUP_INET4_BIND ||
14832
14807
env->prog->expected_attach_type == BPF_CGROUP_INET6_BIND)
+8
-2
kernel/trace/bpf_trace.c
+8
-2
kernel/trace/bpf_trace.c
···
117
117
* and don't send kprobe event into ring-buffer,
118
118
* so return zero here
119
119
*/
120
+
rcu_read_lock();
121
+
bpf_prog_inc_misses_counters(rcu_dereference(call->prog_array));
122
+
rcu_read_unlock();
120
123
ret = 0;
121
124
goto out;
122
125
}
···
2387
2384
2388
2385
int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
2389
2386
u32 *fd_type, const char **buf,
2390
-
u64 *probe_offset, u64 *probe_addr)
2387
+
u64 *probe_offset, u64 *probe_addr,
2388
+
unsigned long *missed)
2391
2389
{
2392
2390
bool is_tracepoint, is_syscall_tp;
2393
2391
struct bpf_prog *prog;
···
2423
2419
#ifdef CONFIG_KPROBE_EVENTS
2424
2420
if (flags & TRACE_EVENT_FL_KPROBE)
2425
2421
err = bpf_get_kprobe_info(event, fd_type, buf,
2426
-
probe_offset, probe_addr,
2422
+
probe_offset, probe_addr, missed,
2427
2423
event->attr.type == PERF_TYPE_TRACEPOINT);
2428
2424
#endif
2429
2425
#ifdef CONFIG_UPROBE_EVENTS
···
2618
2614
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
2619
2615
info->kprobe_multi.count = kmulti_link->cnt;
2620
2616
info->kprobe_multi.flags = kmulti_link->flags;
2617
+
info->kprobe_multi.missed = kmulti_link->fp.nmissed;
2621
2618
2622
2619
if (!uaddrs)
2623
2620
return 0;
···
2715
2710
int err;
2716
2711
2717
2712
if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
2713
+
bpf_prog_inc_misses_counter(link->link.prog);
2718
2714
err = 0;
2719
2715
goto out;
2720
2716
}
+11
-3
kernel/trace/trace_kprobe.c
+11
-3
kernel/trace/trace_kprobe.c
···
1189
1189
.write = probes_write,
1190
1190
};
1191
1191
1192
+
static unsigned long trace_kprobe_missed(struct trace_kprobe *tk)
1193
+
{
1194
+
return trace_kprobe_is_return(tk) ?
1195
+
tk->rp.kp.nmissed + tk->rp.nmissed : tk->rp.kp.nmissed;
1196
+
}
1197
+
1192
1198
/* Probes profiling interfaces */
1193
1199
static int probes_profile_seq_show(struct seq_file *m, void *v)
1194
1200
{
···
1206
1200
return 0;
1207
1201
1208
1202
tk = to_trace_kprobe(ev);
1209
-
nmissed = trace_kprobe_is_return(tk) ?
1210
-
tk->rp.kp.nmissed + tk->rp.nmissed : tk->rp.kp.nmissed;
1203
+
nmissed = trace_kprobe_missed(tk);
1211
1204
seq_printf(m, " %-44s %15lu %15lu\n",
1212
1205
trace_probe_name(&tk->tp),
1213
1206
trace_kprobe_nhit(tk),
···
1552
1547
1553
1548
int bpf_get_kprobe_info(const struct perf_event *event, u32 *fd_type,
1554
1549
const char **symbol, u64 *probe_offset,
1555
-
u64 *probe_addr, bool perf_type_tracepoint)
1550
+
u64 *probe_addr, unsigned long *missed,
1551
+
bool perf_type_tracepoint)
1556
1552
{
1557
1553
const char *pevent = trace_event_name(event->tp_event);
1558
1554
const char *group = event->tp_event->class->system;
···
1572
1566
*probe_addr = kallsyms_show_value(current_cred()) ?
1573
1567
(unsigned long)tk->rp.kp.addr : 0;
1574
1568
*symbol = tk->symbol;
1569
+
if (missed)
1570
+
*missed = trace_kprobe_missed(tk);
1575
1571
return 0;
1576
1572
}
1577
1573
#endif /* CONFIG_PERF_EVENTS */
+2
-2
kernel/trace/trace_syscalls.c
+2
-2
kernel/trace/trace_syscalls.c
···
556
556
{
557
557
struct syscall_tp_t {
558
558
struct trace_entry ent;
559
-
unsigned long syscall_nr;
559
+
int syscall_nr;
560
560
unsigned long args[SYSCALL_DEFINE_MAXARGS];
561
561
} __aligned(8) param;
562
562
int i;
···
661
661
{
662
662
struct syscall_tp_t {
663
663
struct trace_entry ent;
664
-
unsigned long syscall_nr;
664
+
int syscall_nr;
665
665
unsigned long ret;
666
666
} __aligned(8) param;
667
667
+63
-4
net/core/filter.c
+63
-4
net/core/filter.c
···
81
81
#include <net/xdp.h>
82
82
#include <net/mptcp.h>
83
83
#include <net/netfilter/nf_conntrack_bpf.h>
84
+
#include <linux/un.h>
84
85
85
86
static const struct bpf_func_proto *
86
87
bpf_sk_base_func_proto(enum bpf_func_id func_id);
···
5851
5850
params->rt_metric = res.fi->fib_priority;
5852
5851
params->ifindex = dev->ifindex;
5853
5852
5853
+
if (flags & BPF_FIB_LOOKUP_SRC)
5854
+
params->ipv4_src = fib_result_prefsrc(net, &res);
5855
+
5854
5856
/* xdp and cls_bpf programs are run in RCU-bh so
5855
5857
* rcu_read_lock_bh is not needed here
5856
5858
*/
···
5996
5992
params->rt_metric = res.f6i->fib6_metric;
5997
5993
params->ifindex = dev->ifindex;
5998
5994
5995
+
if (flags & BPF_FIB_LOOKUP_SRC) {
5996
+
if (res.f6i->fib6_prefsrc.plen) {
5997
+
*src = res.f6i->fib6_prefsrc.addr;
5998
+
} else {
5999
+
err = ipv6_bpf_stub->ipv6_dev_get_saddr(net, dev,
6000
+
&fl6.daddr, 0,
6001
+
src);
6002
+
if (err)
6003
+
return BPF_FIB_LKUP_RET_NO_SRC_ADDR;
6004
+
}
6005
+
}
6006
+
5999
6007
if (flags & BPF_FIB_LOOKUP_SKIP_NEIGH)
6000
6008
goto set_fwd_params;
6001
6009
···
6026
6010
#endif
6027
6011
6028
6012
#define BPF_FIB_LOOKUP_MASK (BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT | \
6029
-
BPF_FIB_LOOKUP_SKIP_NEIGH | BPF_FIB_LOOKUP_TBID)
6013
+
BPF_FIB_LOOKUP_SKIP_NEIGH | BPF_FIB_LOOKUP_TBID | \
6014
+
BPF_FIB_LOOKUP_SRC)
6030
6015
6031
6016
BPF_CALL_4(bpf_xdp_fib_lookup, struct xdp_buff *, ctx,
6032
6017
struct bpf_fib_lookup *, params, int, plen, u32, flags)
···
7875
7858
case BPF_CGROUP_INET6_BIND:
7876
7859
case BPF_CGROUP_INET4_CONNECT:
7877
7860
case BPF_CGROUP_INET6_CONNECT:
7861
+
case BPF_CGROUP_UNIX_CONNECT:
7878
7862
case BPF_CGROUP_UDP4_RECVMSG:
7879
7863
case BPF_CGROUP_UDP6_RECVMSG:
7864
+
case BPF_CGROUP_UNIX_RECVMSG:
7880
7865
case BPF_CGROUP_UDP4_SENDMSG:
7881
7866
case BPF_CGROUP_UDP6_SENDMSG:
7867
+
case BPF_CGROUP_UNIX_SENDMSG:
7882
7868
case BPF_CGROUP_INET4_GETPEERNAME:
7883
7869
case BPF_CGROUP_INET6_GETPEERNAME:
7870
+
case BPF_CGROUP_UNIX_GETPEERNAME:
7884
7871
case BPF_CGROUP_INET4_GETSOCKNAME:
7885
7872
case BPF_CGROUP_INET6_GETSOCKNAME:
7873
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
7886
7874
return &bpf_sock_addr_setsockopt_proto;
7887
7875
default:
7888
7876
return NULL;
···
7898
7876
case BPF_CGROUP_INET6_BIND:
7899
7877
case BPF_CGROUP_INET4_CONNECT:
7900
7878
case BPF_CGROUP_INET6_CONNECT:
7879
+
case BPF_CGROUP_UNIX_CONNECT:
7901
7880
case BPF_CGROUP_UDP4_RECVMSG:
7902
7881
case BPF_CGROUP_UDP6_RECVMSG:
7882
+
case BPF_CGROUP_UNIX_RECVMSG:
7903
7883
case BPF_CGROUP_UDP4_SENDMSG:
7904
7884
case BPF_CGROUP_UDP6_SENDMSG:
7885
+
case BPF_CGROUP_UNIX_SENDMSG:
7905
7886
case BPF_CGROUP_INET4_GETPEERNAME:
7906
7887
case BPF_CGROUP_INET6_GETPEERNAME:
7888
+
case BPF_CGROUP_UNIX_GETPEERNAME:
7907
7889
case BPF_CGROUP_INET4_GETSOCKNAME:
7908
7890
case BPF_CGROUP_INET6_GETSOCKNAME:
7891
+
case BPF_CGROUP_UNIX_GETSOCKNAME:
7909
7892
return &bpf_sock_addr_getsockopt_proto;
7910
7893
default:
7911
7894
return NULL;
···
8958
8931
if (off % size != 0)
8959
8932
return false;
8960
8933
8961
-
/* Disallow access to IPv6 fields from IPv4 contex and vise
8962
-
* versa.
8934
+
/* Disallow access to fields not belonging to the attach type's address
8935
+
* family.
8963
8936
*/
8964
8937
switch (off) {
8965
8938
case bpf_ctx_range(struct bpf_sock_addr, user_ip4):
···
11779
11752
11780
11753
return 0;
11781
11754
}
11755
+
11756
+
__bpf_kfunc int bpf_sock_addr_set_sun_path(struct bpf_sock_addr_kern *sa_kern,
11757
+
const u8 *sun_path, u32 sun_path__sz)
11758
+
{
11759
+
struct sockaddr_un *un;
11760
+
11761
+
if (sa_kern->sk->sk_family != AF_UNIX)
11762
+
return -EINVAL;
11763
+
11764
+
/* We do not allow changing the address to unnamed or larger than the
11765
+
* maximum allowed address size for a unix sockaddr.
11766
+
*/
11767
+
if (sun_path__sz == 0 || sun_path__sz > UNIX_PATH_MAX)
11768
+
return -EINVAL;
11769
+
11770
+
un = (struct sockaddr_un *)sa_kern->uaddr;
11771
+
memcpy(un->sun_path, sun_path, sun_path__sz);
11772
+
sa_kern->uaddrlen = offsetof(struct sockaddr_un, sun_path) + sun_path__sz;
11773
+
11774
+
return 0;
11775
+
}
11782
11776
__diag_pop();
11783
11777
11784
11778
int bpf_dynptr_from_skb_rdonly(struct sk_buff *skb, u64 flags,
···
11824
11776
BTF_ID_FLAGS(func, bpf_dynptr_from_xdp)
11825
11777
BTF_SET8_END(bpf_kfunc_check_set_xdp)
11826
11778
11779
+
BTF_SET8_START(bpf_kfunc_check_set_sock_addr)
11780
+
BTF_ID_FLAGS(func, bpf_sock_addr_set_sun_path)
11781
+
BTF_SET8_END(bpf_kfunc_check_set_sock_addr)
11782
+
11827
11783
static const struct btf_kfunc_id_set bpf_kfunc_set_skb = {
11828
11784
.owner = THIS_MODULE,
11829
11785
.set = &bpf_kfunc_check_set_skb,
···
11836
11784
static const struct btf_kfunc_id_set bpf_kfunc_set_xdp = {
11837
11785
.owner = THIS_MODULE,
11838
11786
.set = &bpf_kfunc_check_set_xdp,
11787
+
};
11788
+
11789
+
static const struct btf_kfunc_id_set bpf_kfunc_set_sock_addr = {
11790
+
.owner = THIS_MODULE,
11791
+
.set = &bpf_kfunc_check_set_sock_addr,
11839
11792
};
11840
11793
11841
11794
static int __init bpf_kfunc_init(void)
···
11857
11800
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LWT_XMIT, &bpf_kfunc_set_skb);
11858
11801
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LWT_SEG6LOCAL, &bpf_kfunc_set_skb);
11859
11802
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_NETFILTER, &bpf_kfunc_set_skb);
11860
-
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &bpf_kfunc_set_xdp);
11803
+
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &bpf_kfunc_set_xdp);
11804
+
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SOCK_ADDR,
11805
+
&bpf_kfunc_set_sock_addr);
11861
11806
}
11862
11807
late_initcall(bpf_kfunc_init);
11863
11808
+5
-4
net/ipv4/af_inet.c
+5
-4
net/ipv4/af_inet.c
···
452
452
/* BPF prog is run before any checks are done so that if the prog
453
453
* changes context in a wrong way it will be caught.
454
454
*/
455
-
err = BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr,
455
+
err = BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, &addr_len,
456
456
CGROUP_INET4_BIND, &flags);
457
457
if (err)
458
458
return err;
···
788
788
struct sock *sk = sock->sk;
789
789
struct inet_sock *inet = inet_sk(sk);
790
790
DECLARE_SOCKADDR(struct sockaddr_in *, sin, uaddr);
791
+
int sin_addr_len = sizeof(*sin);
791
792
792
793
sin->sin_family = AF_INET;
793
794
lock_sock(sk);
···
801
800
}
802
801
sin->sin_port = inet->inet_dport;
803
802
sin->sin_addr.s_addr = inet->inet_daddr;
804
-
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin,
803
+
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin, &sin_addr_len,
805
804
CGROUP_INET4_GETPEERNAME);
806
805
} else {
807
806
__be32 addr = inet->inet_rcv_saddr;
···
809
808
addr = inet->inet_saddr;
810
809
sin->sin_port = inet->inet_sport;
811
810
sin->sin_addr.s_addr = addr;
812
-
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin,
811
+
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin, &sin_addr_len,
813
812
CGROUP_INET4_GETSOCKNAME);
814
813
}
815
814
release_sock(sk);
816
815
memset(sin->sin_zero, 0, sizeof(sin->sin_zero));
817
-
return sizeof(*sin);
816
+
return sin_addr_len;
818
817
}
819
818
EXPORT_SYMBOL(inet_getname);
820
819
+1
-1
net/ipv4/ping.c
+1
-1
net/ipv4/ping.c
···
301
301
if (addr_len < sizeof(struct sockaddr_in))
302
302
return -EINVAL;
303
303
304
-
return BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr);
304
+
return BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr, &addr_len);
305
305
}
306
306
307
307
/* Checks the bind address and possibly modifies sk->sk_bound_dev_if. */
+1
-1
net/ipv4/tcp_ipv4.c
+1
-1
net/ipv4/tcp_ipv4.c
+6
-3
net/ipv4/udp.c
+6
-3
net/ipv4/udp.c
···
1144
1144
1145
1145
if (cgroup_bpf_enabled(CGROUP_UDP4_SENDMSG) && !connected) {
1146
1146
err = BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK(sk,
1147
-
(struct sockaddr *)usin, &ipc.addr);
1147
+
(struct sockaddr *)usin,
1148
+
&msg->msg_namelen,
1149
+
&ipc.addr);
1148
1150
if (err)
1149
1151
goto out_free;
1150
1152
if (usin) {
···
1869
1867
*addr_len = sizeof(*sin);
1870
1868
1871
1869
BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk,
1872
-
(struct sockaddr *)sin);
1870
+
(struct sockaddr *)sin,
1871
+
addr_len);
1873
1872
}
1874
1873
1875
1874
if (udp_test_bit(GRO_ENABLED, sk))
···
1909
1906
if (addr_len < sizeof(struct sockaddr_in))
1910
1907
return -EINVAL;
1911
1908
1912
-
return BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr);
1909
+
return BPF_CGROUP_RUN_PROG_INET4_CONNECT_LOCK(sk, uaddr, &addr_len);
1913
1910
}
1914
1911
EXPORT_SYMBOL(udp_pre_connect);
1915
1912
+6
-4
net/ipv6/af_inet6.c
+6
-4
net/ipv6/af_inet6.c
···
454
454
/* BPF prog is run before any checks are done so that if the prog
455
455
* changes context in a wrong way it will be caught.
456
456
*/
457
-
err = BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr,
457
+
err = BPF_CGROUP_RUN_PROG_INET_BIND_LOCK(sk, uaddr, &addr_len,
458
458
CGROUP_INET6_BIND, &flags);
459
459
if (err)
460
460
return err;
···
520
520
int peer)
521
521
{
522
522
struct sockaddr_in6 *sin = (struct sockaddr_in6 *)uaddr;
523
+
int sin_addr_len = sizeof(*sin);
523
524
struct sock *sk = sock->sk;
524
525
struct inet_sock *inet = inet_sk(sk);
525
526
struct ipv6_pinfo *np = inet6_sk(sk);
···
540
539
sin->sin6_addr = sk->sk_v6_daddr;
541
540
if (inet6_test_bit(SNDFLOW, sk))
542
541
sin->sin6_flowinfo = np->flow_label;
543
-
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin,
542
+
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin, &sin_addr_len,
544
543
CGROUP_INET6_GETPEERNAME);
545
544
} else {
546
545
if (ipv6_addr_any(&sk->sk_v6_rcv_saddr))
···
548
547
else
549
548
sin->sin6_addr = sk->sk_v6_rcv_saddr;
550
549
sin->sin6_port = inet->inet_sport;
551
-
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin,
550
+
BPF_CGROUP_RUN_SA_PROG(sk, (struct sockaddr *)sin, &sin_addr_len,
552
551
CGROUP_INET6_GETSOCKNAME);
553
552
}
554
553
sin->sin6_scope_id = ipv6_iface_scope_id(&sin->sin6_addr,
555
554
sk->sk_bound_dev_if);
556
555
release_sock(sk);
557
-
return sizeof(*sin);
556
+
return sin_addr_len;
558
557
}
559
558
EXPORT_SYMBOL(inet6_getname);
560
559
···
1062
1061
.udp6_lib_lookup = __udp6_lib_lookup,
1063
1062
.ipv6_setsockopt = do_ipv6_setsockopt,
1064
1063
.ipv6_getsockopt = do_ipv6_getsockopt,
1064
+
.ipv6_dev_get_saddr = ipv6_dev_get_saddr,
1065
1065
};
1066
1066
1067
1067
static int __init inet6_init(void)
+1
-1
net/ipv6/ping.c
+1
-1
net/ipv6/ping.c
···
56
56
if (addr_len < SIN6_LEN_RFC2133)
57
57
return -EINVAL;
58
58
59
-
return BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr);
59
+
return BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr, &addr_len);
60
60
}
61
61
62
62
static int ping_v6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+1
-1
net/ipv6/tcp_ipv6.c
+1
-1
net/ipv6/tcp_ipv6.c
+4
-2
net/ipv6/udp.c
+4
-2
net/ipv6/udp.c
···
410
410
*addr_len = sizeof(*sin6);
411
411
412
412
BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk,
413
-
(struct sockaddr *)sin6);
413
+
(struct sockaddr *)sin6,
414
+
addr_len);
414
415
}
415
416
416
417
if (udp_test_bit(GRO_ENABLED, sk))
···
1158
1157
if (addr_len < SIN6_LEN_RFC2133)
1159
1158
return -EINVAL;
1160
1159
1161
-
return BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr);
1160
+
return BPF_CGROUP_RUN_PROG_INET6_CONNECT_LOCK(sk, uaddr, &addr_len);
1162
1161
}
1163
1162
1164
1163
/**
···
1511
1510
if (cgroup_bpf_enabled(CGROUP_UDP6_SENDMSG) && !connected) {
1512
1511
err = BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk,
1513
1512
(struct sockaddr *)sin6,
1513
+
&addr_len,
1514
1514
&fl6->saddr);
1515
1515
if (err)
1516
1516
goto out_no_dst;
+34
-1
net/unix/af_unix.c
+34
-1
net/unix/af_unix.c
···
116
116
#include <linux/freezer.h>
117
117
#include <linux/file.h>
118
118
#include <linux/btf_ids.h>
119
+
#include <linux/bpf-cgroup.h>
119
120
120
121
#include "scm.h"
121
122
···
1382
1381
if (err)
1383
1382
goto out;
1384
1383
1384
+
err = BPF_CGROUP_RUN_PROG_UNIX_CONNECT_LOCK(sk, addr, &alen);
1385
+
if (err)
1386
+
goto out;
1387
+
1385
1388
if ((test_bit(SOCK_PASSCRED, &sock->flags) ||
1386
1389
test_bit(SOCK_PASSPIDFD, &sock->flags)) &&
1387
1390
!unix_sk(sk)->addr) {
···
1492
1487
int st;
1493
1488
1494
1489
err = unix_validate_addr(sunaddr, addr_len);
1490
+
if (err)
1491
+
goto out;
1492
+
1493
+
err = BPF_CGROUP_RUN_PROG_UNIX_CONNECT_LOCK(sk, uaddr, &addr_len);
1495
1494
if (err)
1496
1495
goto out;
1497
1496
···
1779
1770
} else {
1780
1771
err = addr->len;
1781
1772
memcpy(sunaddr, addr->name, addr->len);
1773
+
1774
+
if (peer)
1775
+
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, &err,
1776
+
CGROUP_UNIX_GETPEERNAME);
1777
+
else
1778
+
BPF_CGROUP_RUN_SA_PROG(sk, uaddr, &err,
1779
+
CGROUP_UNIX_GETSOCKNAME);
1782
1780
}
1783
1781
sock_put(sk);
1784
1782
out:
···
1936
1920
1937
1921
if (msg->msg_namelen) {
1938
1922
err = unix_validate_addr(sunaddr, msg->msg_namelen);
1923
+
if (err)
1924
+
goto out;
1925
+
1926
+
err = BPF_CGROUP_RUN_PROG_UNIX_SENDMSG_LOCK(sk,
1927
+
msg->msg_name,
1928
+
&msg->msg_namelen,
1929
+
NULL);
1939
1930
if (err)
1940
1931
goto out;
1941
1932
} else {
···
2413
2390
EPOLLOUT | EPOLLWRNORM |
2414
2391
EPOLLWRBAND);
2415
2392
2416
-
if (msg->msg_name)
2393
+
if (msg->msg_name) {
2417
2394
unix_copy_addr(msg, skb->sk);
2395
+
2396
+
BPF_CGROUP_RUN_PROG_UNIX_RECVMSG_LOCK(sk,
2397
+
msg->msg_name,
2398
+
&msg->msg_namelen);
2399
+
}
2418
2400
2419
2401
if (size > skb->len - skip)
2420
2402
size = skb->len - skip;
···
2772
2744
DECLARE_SOCKADDR(struct sockaddr_un *, sunaddr,
2773
2745
state->msg->msg_name);
2774
2746
unix_copy_addr(state->msg, skb->sk);
2747
+
2748
+
BPF_CGROUP_RUN_PROG_UNIX_RECVMSG_LOCK(sk,
2749
+
state->msg->msg_name,
2750
+
&state->msg->msg_namelen);
2751
+
2775
2752
sunaddr = NULL;
2776
2753
}
2777
2754
+3
samples/bpf/Makefile
+3
samples/bpf/Makefile
···
169
169
TPROGS_CFLAGS += -Wall -O2
170
170
TPROGS_CFLAGS += -Wmissing-prototypes
171
171
TPROGS_CFLAGS += -Wstrict-prototypes
172
+
TPROGS_CFLAGS += $(call try-run,\
173
+
printf "int main() { return 0; }" |\
174
+
$(CC) -Werror -fsanitize=bounds -x c - -o "$$TMP",-fsanitize=bounds,)
172
175
173
176
TPROGS_CFLAGS += -I$(objtree)/usr/include
174
177
TPROGS_CFLAGS += -I$(srctree)/tools/testing/selftests/bpf/
+31
-14
samples/bpf/syscall_tp_user.c
+31
-14
samples/bpf/syscall_tp_user.c
···
17
17
18
18
static void usage(const char *cmd)
19
19
{
20
-
printf("USAGE: %s [-i num_progs] [-h]\n", cmd);
21
-
printf(" -i num_progs # number of progs of the test\n");
22
-
printf(" -h # help\n");
20
+
printf("USAGE: %s [-i nr_tests] [-h]\n", cmd);
21
+
printf(" -i nr_tests # rounds of test to run\n");
22
+
printf(" -h # help\n");
23
23
}
24
24
25
25
static void verify_map(int map_id)
···
45
45
}
46
46
}
47
47
48
-
static int test(char *filename, int num_progs)
48
+
static int test(char *filename, int nr_tests)
49
49
{
50
-
int map0_fds[num_progs], map1_fds[num_progs], fd, i, j = 0;
51
-
struct bpf_link *links[num_progs * 4];
52
-
struct bpf_object *objs[num_progs];
50
+
int map0_fds[nr_tests], map1_fds[nr_tests], fd, i, j = 0;
51
+
struct bpf_link **links = NULL;
52
+
struct bpf_object *objs[nr_tests];
53
53
struct bpf_program *prog;
54
54
55
-
for (i = 0; i < num_progs; i++) {
55
+
for (i = 0; i < nr_tests; i++) {
56
56
objs[i] = bpf_object__open_file(filename, NULL);
57
57
if (libbpf_get_error(objs[i])) {
58
58
fprintf(stderr, "opening BPF object file failed\n");
59
59
objs[i] = NULL;
60
60
goto cleanup;
61
+
}
62
+
63
+
/* One-time initialization */
64
+
if (!links) {
65
+
int nr_progs = 0;
66
+
67
+
bpf_object__for_each_program(prog, objs[i])
68
+
nr_progs += 1;
69
+
70
+
links = calloc(nr_progs * nr_tests, sizeof(struct bpf_link *));
71
+
72
+
if (!links)
73
+
goto cleanup;
61
74
}
62
75
63
76
/* load BPF program */
···
114
101
close(fd);
115
102
116
103
/* verify the map */
117
-
for (i = 0; i < num_progs; i++) {
104
+
for (i = 0; i < nr_tests; i++) {
118
105
verify_map(map0_fds[i]);
119
106
verify_map(map1_fds[i]);
120
107
}
121
108
122
109
cleanup:
123
-
for (j--; j >= 0; j--)
124
-
bpf_link__destroy(links[j]);
110
+
if (links) {
111
+
for (j--; j >= 0; j--)
112
+
bpf_link__destroy(links[j]);
113
+
114
+
free(links);
115
+
}
125
116
126
117
for (i--; i >= 0; i--)
127
118
bpf_object__close(objs[i]);
···
134
117
135
118
int main(int argc, char **argv)
136
119
{
137
-
int opt, num_progs = 1;
120
+
int opt, nr_tests = 1;
138
121
char filename[256];
139
122
140
123
while ((opt = getopt(argc, argv, "i:h")) != -1) {
141
124
switch (opt) {
142
125
case 'i':
143
-
num_progs = atoi(optarg);
126
+
nr_tests = atoi(optarg);
144
127
break;
145
128
case 'h':
146
129
default:
···
151
134
152
135
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
153
136
154
-
return test(filename, num_progs);
137
+
return test(filename, nr_tests);
155
138
}
+13
-3
tools/bpf/bpftool/Documentation/bpftool-cgroup.rst
+13
-3
tools/bpf/bpftool/Documentation/bpftool-cgroup.rst
···
36
36
| **cgroup_device** | **cgroup_inet4_bind** | **cgroup_inet6_bind** |
37
37
| **cgroup_inet4_post_bind** | **cgroup_inet6_post_bind** |
38
38
| **cgroup_inet4_connect** | **cgroup_inet6_connect** |
39
-
| **cgroup_inet4_getpeername** | **cgroup_inet6_getpeername** |
39
+
| **cgroup_unix_connect** | **cgroup_inet4_getpeername** |
40
+
| **cgroup_inet6_getpeername** | **cgroup_unix_getpeername** |
40
41
| **cgroup_inet4_getsockname** | **cgroup_inet6_getsockname** |
41
-
| **cgroup_udp4_sendmsg** | **cgroup_udp6_sendmsg** |
42
+
| **cgroup_unix_getsockname** | **cgroup_udp4_sendmsg** |
43
+
| **cgroup_udp6_sendmsg** | **cgroup_unix_sendmsg** |
42
44
| **cgroup_udp4_recvmsg** | **cgroup_udp6_recvmsg** |
43
-
| **cgroup_sysctl** | **cgroup_getsockopt** | **cgroup_setsockopt** |
45
+
| **cgroup_unix_recvmsg** | **cgroup_sysctl** |
46
+
| **cgroup_getsockopt** | **cgroup_setsockopt** |
44
47
| **cgroup_inet_sock_release** }
45
48
| *ATTACH_FLAGS* := { **multi** | **override** }
46
49
···
105
102
**post_bind6** return from bind(2) for an inet6 socket (since 4.17);
106
103
**connect4** call to connect(2) for an inet4 socket (since 4.17);
107
104
**connect6** call to connect(2) for an inet6 socket (since 4.17);
105
+
**connect_unix** call to connect(2) for a unix socket (since 6.7);
108
106
**sendmsg4** call to sendto(2), sendmsg(2), sendmmsg(2) for an
109
107
unconnected udp4 socket (since 4.18);
110
108
**sendmsg6** call to sendto(2), sendmsg(2), sendmmsg(2) for an
111
109
unconnected udp6 socket (since 4.18);
110
+
**sendmsg_unix** call to sendto(2), sendmsg(2), sendmmsg(2) for
111
+
an unconnected unix socket (since 6.7);
112
112
**recvmsg4** call to recvfrom(2), recvmsg(2), recvmmsg(2) for
113
113
an unconnected udp4 socket (since 5.2);
114
114
**recvmsg6** call to recvfrom(2), recvmsg(2), recvmmsg(2) for
115
115
an unconnected udp6 socket (since 5.2);
116
+
**recvmsg_unix** call to recvfrom(2), recvmsg(2), recvmmsg(2) for
117
+
an unconnected unix socket (since 6.7);
116
118
**sysctl** sysctl access (since 5.2);
117
119
**getsockopt** call to getsockopt (since 5.3);
118
120
**setsockopt** call to setsockopt (since 5.3);
119
121
**getpeername4** call to getpeername(2) for an inet4 socket (since 5.8);
120
122
**getpeername6** call to getpeername(2) for an inet6 socket (since 5.8);
123
+
**getpeername_unix** call to getpeername(2) for a unix socket (since 6.7);
121
124
**getsockname4** call to getsockname(2) for an inet4 socket (since 5.8);
122
125
**getsockname6** call to getsockname(2) for an inet6 socket (since 5.8).
126
+
**getsockname_unix** call to getsockname(2) for a unix socket (since 6.7);
123
127
**sock_release** closing an userspace inet socket (since 5.9).
124
128
125
129
**bpftool cgroup detach** *CGROUP* *ATTACH_TYPE* *PROG*
+5
-3
tools/bpf/bpftool/Documentation/bpftool-prog.rst
+5
-3
tools/bpf/bpftool/Documentation/bpftool-prog.rst
···
47
47
| **cgroup/sock** | **cgroup/dev** | **lwt_in** | **lwt_out** | **lwt_xmit** |
48
48
| **lwt_seg6local** | **sockops** | **sk_skb** | **sk_msg** | **lirc_mode2** |
49
49
| **cgroup/bind4** | **cgroup/bind6** | **cgroup/post_bind4** | **cgroup/post_bind6** |
50
-
| **cgroup/connect4** | **cgroup/connect6** | **cgroup/getpeername4** | **cgroup/getpeername6** |
51
-
| **cgroup/getsockname4** | **cgroup/getsockname6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** |
52
-
| **cgroup/recvmsg4** | **cgroup/recvmsg6** | **cgroup/sysctl** |
50
+
| **cgroup/connect4** | **cgroup/connect6** | **cgroup/connect_unix** |
51
+
| **cgroup/getpeername4** | **cgroup/getpeername6** | **cgroup/getpeername_unix** |
52
+
| **cgroup/getsockname4** | **cgroup/getsockname6** | **cgroup/getsockname_unix** |
53
+
| **cgroup/sendmsg4** | **cgroup/sendmsg6** | **cgroup/sendmsg_unix** |
54
+
| **cgroup/recvmsg4** | **cgroup/recvmsg6** | **cgroup/recvmsg_unix** | **cgroup/sysctl** |
53
55
| **cgroup/getsockopt** | **cgroup/setsockopt** | **cgroup/sock_release** |
54
56
| **struct_ops** | **fentry** | **fexit** | **freplace** | **sk_lookup**
55
57
| }
+7
-7
tools/bpf/bpftool/bash-completion/bpftool
+7
-7
tools/bpf/bpftool/bash-completion/bpftool
···
480
480
action tracepoint raw_tracepoint \
481
481
xdp perf_event cgroup/skb cgroup/sock \
482
482
cgroup/dev lwt_in lwt_out lwt_xmit \
483
-
lwt_seg6local sockops sk_skb sk_msg \
484
-
lirc_mode2 cgroup/bind4 cgroup/bind6 \
485
-
cgroup/connect4 cgroup/connect6 \
486
-
cgroup/getpeername4 cgroup/getpeername6 \
487
-
cgroup/getsockname4 cgroup/getsockname6 \
488
-
cgroup/sendmsg4 cgroup/sendmsg6 \
489
-
cgroup/recvmsg4 cgroup/recvmsg6 \
483
+
lwt_seg6local sockops sk_skb sk_msg lirc_mode2 \
484
+
cgroup/bind4 cgroup/bind6 \
485
+
cgroup/connect4 cgroup/connect6 cgroup/connect_unix \
486
+
cgroup/getpeername4 cgroup/getpeername6 cgroup/getpeername_unix \
487
+
cgroup/getsockname4 cgroup/getsockname6 cgroup/getsockname_unix \
488
+
cgroup/sendmsg4 cgroup/sendmsg6 cgroup/sendmsg_unix \
489
+
cgroup/recvmsg4 cgroup/recvmsg6 cgroup/recvmsg_unix \
490
490
cgroup/post_bind4 cgroup/post_bind6 \
491
491
cgroup/sysctl cgroup/getsockopt \
492
492
cgroup/setsockopt cgroup/sock_release struct_ops \
+9
-7
tools/bpf/bpftool/cgroup.c
+9
-7
tools/bpf/bpftool/cgroup.c
···
28
28
" cgroup_device | cgroup_inet4_bind |\n" \
29
29
" cgroup_inet6_bind | cgroup_inet4_post_bind |\n" \
30
30
" cgroup_inet6_post_bind | cgroup_inet4_connect |\n" \
31
-
" cgroup_inet6_connect | cgroup_inet4_getpeername |\n" \
32
-
" cgroup_inet6_getpeername | cgroup_inet4_getsockname |\n" \
33
-
" cgroup_inet6_getsockname | cgroup_udp4_sendmsg |\n" \
34
-
" cgroup_udp6_sendmsg | cgroup_udp4_recvmsg |\n" \
35
-
" cgroup_udp6_recvmsg | cgroup_sysctl |\n" \
36
-
" cgroup_getsockopt | cgroup_setsockopt |\n" \
37
-
" cgroup_inet_sock_release }"
31
+
" cgroup_inet6_connect | cgroup_unix_connect |\n" \
32
+
" cgroup_inet4_getpeername | cgroup_inet6_getpeername |\n" \
33
+
" cgroup_unix_getpeername | cgroup_inet4_getsockname |\n" \
34
+
" cgroup_inet6_getsockname | cgroup_unix_getsockname |\n" \
35
+
" cgroup_udp4_sendmsg | cgroup_udp6_sendmsg |\n" \
36
+
" cgroup_unix_sendmsg | cgroup_udp4_recvmsg |\n" \
37
+
" cgroup_udp6_recvmsg | cgroup_unix_recvmsg |\n" \
38
+
" cgroup_sysctl | cgroup_getsockopt |\n" \
39
+
" cgroup_setsockopt | cgroup_inet_sock_release }"
38
40
39
41
static unsigned int query_flags;
40
42
static struct btf *btf_vmlinux;
+32
-26
tools/bpf/bpftool/gen.c
+32
-26
tools/bpf/bpftool/gen.c
···
708
708
709
709
codegen("\
710
710
\n\
711
-
skel->%1$s = skel_prep_map_data((void *)\"\\ \n\
712
-
", ident);
711
+
{ \n\
712
+
static const char data[] __attribute__((__aligned__(8))) = \"\\\n\
713
+
");
713
714
mmap_data = bpf_map__initial_value(map, &mmap_size);
714
715
print_hex(mmap_data, mmap_size);
715
716
codegen("\
716
717
\n\
717
-
\", %1$zd, %2$zd); \n\
718
-
if (!skel->%3$s) \n\
719
-
goto cleanup; \n\
720
-
skel->maps.%3$s.initial_value = (__u64) (long) skel->%3$s;\n\
721
-
", bpf_map_mmap_sz(map), mmap_size, ident);
718
+
\"; \n\
719
+
\n\
720
+
skel->%1$s = skel_prep_map_data((void *)data, %2$zd,\n\
721
+
sizeof(data) - 1);\n\
722
+
if (!skel->%1$s) \n\
723
+
goto cleanup; \n\
724
+
skel->maps.%1$s.initial_value = (__u64) (long) skel->%1$s;\n\
725
+
} \n\
726
+
", ident, bpf_map_mmap_sz(map));
722
727
}
723
728
codegen("\
724
729
\n\
···
738
733
{ \n\
739
734
struct bpf_load_and_run_opts opts = {}; \n\
740
735
int err; \n\
741
-
\n\
742
-
opts.ctx = (struct bpf_loader_ctx *)skel; \n\
743
-
opts.data_sz = %2$d; \n\
744
-
opts.data = (void *)\"\\ \n\
736
+
static const char opts_data[] __attribute__((__aligned__(8))) = \"\\\n\
745
737
",
746
-
obj_name, opts.data_sz);
738
+
obj_name);
747
739
print_hex(opts.data, opts.data_sz);
748
740
codegen("\
749
741
\n\
750
742
\"; \n\
743
+
static const char opts_insn[] __attribute__((__aligned__(8))) = \"\\\n\
751
744
");
752
-
753
-
codegen("\
754
-
\n\
755
-
opts.insns_sz = %d; \n\
756
-
opts.insns = (void *)\"\\ \n\
757
-
",
758
-
opts.insns_sz);
759
745
print_hex(opts.insns, opts.insns_sz);
760
746
codegen("\
761
747
\n\
762
748
\"; \n\
749
+
\n\
750
+
opts.ctx = (struct bpf_loader_ctx *)skel; \n\
751
+
opts.data_sz = sizeof(opts_data) - 1; \n\
752
+
opts.data = (void *)opts_data; \n\
753
+
opts.insns_sz = sizeof(opts_insn) - 1; \n\
754
+
opts.insns = (void *)opts_insn; \n\
755
+
\n\
763
756
err = bpf_load_and_run(&opts); \n\
764
757
if (err < 0) \n\
765
758
return err; \n\
766
-
", obj_name);
759
+
");
767
760
bpf_object__for_each_map(map, obj) {
768
761
const char *mmap_flags;
769
762
···
1212
1209
codegen("\
1213
1210
\n\
1214
1211
\n\
1215
-
s->data = %2$s__elf_bytes(&s->data_sz); \n\
1212
+
s->data = %1$s__elf_bytes(&s->data_sz); \n\
1216
1213
\n\
1217
1214
obj->skeleton = s; \n\
1218
1215
return 0; \n\
···
1221
1218
return err; \n\
1222
1219
} \n\
1223
1220
\n\
1224
-
static inline const void *%2$s__elf_bytes(size_t *sz) \n\
1221
+
static inline const void *%1$s__elf_bytes(size_t *sz) \n\
1225
1222
{ \n\
1226
-
*sz = %1$d; \n\
1227
-
return (const void *)\"\\ \n\
1228
-
"
1229
-
, file_sz, obj_name);
1223
+
static const char data[] __attribute__((__aligned__(8))) = \"\\\n\
1224
+
",
1225
+
obj_name
1226
+
);
1230
1227
1231
1228
/* embed contents of BPF object file */
1232
1229
print_hex(obj_data, file_sz);
···
1234
1231
codegen("\
1235
1232
\n\
1236
1233
\"; \n\
1234
+
\n\
1235
+
*sz = sizeof(data) - 1; \n\
1236
+
return (const void *)data; \n\
1237
1237
} \n\
1238
1238
\n\
1239
1239
#ifdef __cplusplus \n\
+6
tools/bpf/bpftool/link.c
+6
tools/bpf/bpftool/link.c
···
265
265
jsonw_bool_field(json_wtr, "retprobe",
266
266
info->kprobe_multi.flags & BPF_F_KPROBE_MULTI_RETURN);
267
267
jsonw_uint_field(json_wtr, "func_cnt", info->kprobe_multi.count);
268
+
jsonw_uint_field(json_wtr, "missed", info->kprobe_multi.missed);
268
269
jsonw_name(json_wtr, "funcs");
269
270
jsonw_start_array(json_wtr);
270
271
addrs = u64_to_ptr(info->kprobe_multi.addrs);
···
302
301
jsonw_string_field(wtr, "func",
303
302
u64_to_ptr(info->perf_event.kprobe.func_name));
304
303
jsonw_uint_field(wtr, "offset", info->perf_event.kprobe.offset);
304
+
jsonw_uint_field(wtr, "missed", info->perf_event.kprobe.missed);
305
305
}
306
306
307
307
static void
···
643
641
else
644
642
printf("\n\tkprobe.multi ");
645
643
printf("func_cnt %u ", info->kprobe_multi.count);
644
+
if (info->kprobe_multi.missed)
645
+
printf("missed %llu ", info->kprobe_multi.missed);
646
646
addrs = (__u64 *)u64_to_ptr(info->kprobe_multi.addrs);
647
647
qsort(addrs, info->kprobe_multi.count, sizeof(__u64), cmp_u64);
648
648
···
687
683
printf("%s", buf);
688
684
if (info->perf_event.kprobe.offset)
689
685
printf("+%#x", info->perf_event.kprobe.offset);
686
+
if (info->perf_event.kprobe.missed)
687
+
printf(" missed %llu", info->perf_event.kprobe.missed);
690
688
printf(" ");
691
689
}
692
690
+4
-3
tools/bpf/bpftool/prog.c
+4
-3
tools/bpf/bpftool/prog.c
···
2475
2475
" sk_reuseport | flow_dissector | cgroup/sysctl |\n"
2476
2476
" cgroup/bind4 | cgroup/bind6 | cgroup/post_bind4 |\n"
2477
2477
" cgroup/post_bind6 | cgroup/connect4 | cgroup/connect6 |\n"
2478
-
" cgroup/getpeername4 | cgroup/getpeername6 |\n"
2479
-
" cgroup/getsockname4 | cgroup/getsockname6 | cgroup/sendmsg4 |\n"
2480
-
" cgroup/sendmsg6 | cgroup/recvmsg4 | cgroup/recvmsg6 |\n"
2478
+
" cgroup/connect_unix | cgroup/getpeername4 | cgroup/getpeername6 |\n"
2479
+
" cgroup/getpeername_unix | cgroup/getsockname4 | cgroup/getsockname6 |\n"
2480
+
" cgroup/getsockname_unix | cgroup/sendmsg4 | cgroup/sendmsg6 |\n"
2481
+
" cgroup/sendmsg°unix | cgroup/recvmsg4 | cgroup/recvmsg6 | cgroup/recvmsg_unix |\n"
2481
2482
" cgroup/getsockopt | cgroup/setsockopt | cgroup/sock_release |\n"
2482
2483
" struct_ops | fentry | fexit | freplace | sk_lookup }\n"
2483
2484
" ATTACH_TYPE := { sk_msg_verdict | sk_skb_verdict | sk_skb_stream_verdict |\n"
+25
-4
tools/include/uapi/linux/bpf.h
+25
-4
tools/include/uapi/linux/bpf.h
···
1047
1047
BPF_TCX_INGRESS,
1048
1048
BPF_TCX_EGRESS,
1049
1049
BPF_TRACE_UPROBE_MULTI,
1050
+
BPF_CGROUP_UNIX_CONNECT,
1051
+
BPF_CGROUP_UNIX_SENDMSG,
1052
+
BPF_CGROUP_UNIX_RECVMSG,
1053
+
BPF_CGROUP_UNIX_GETPEERNAME,
1054
+
BPF_CGROUP_UNIX_GETSOCKNAME,
1050
1055
__MAX_BPF_ATTACH_TYPE
1051
1056
};
1052
1057
···
2709
2704
* *bpf_socket* should be one of the following:
2710
2705
*
2711
2706
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
2712
-
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**
2713
-
* and **BPF_CGROUP_INET6_CONNECT**.
2707
+
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
2708
+
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
2714
2709
*
2715
2710
* This helper actually implements a subset of **setsockopt()**.
2716
2711
* It supports the following *level*\ s:
···
2948
2943
* *bpf_socket* should be one of the following:
2949
2944
*
2950
2945
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
2951
-
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**
2952
-
* and **BPF_CGROUP_INET6_CONNECT**.
2946
+
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
2947
+
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
2953
2948
*
2954
2949
* This helper actually implements a subset of **getsockopt()**.
2955
2950
* It supports the same set of *optname*\ s that is supported by
···
3269
3264
* and *params*->smac will not be set as output. A common
3270
3265
* use case is to call **bpf_redirect_neigh**\ () after
3271
3266
* doing **bpf_fib_lookup**\ ().
3267
+
* **BPF_FIB_LOOKUP_SRC**
3268
+
* Derive and set source IP addr in *params*->ipv{4,6}_src
3269
+
* for the nexthop. If the src addr cannot be derived,
3270
+
* **BPF_FIB_LKUP_RET_NO_SRC_ADDR** is returned. In this
3271
+
* case, *params*->dmac and *params*->smac are not set either.
3272
3272
*
3273
3273
* *ctx* is either **struct xdp_md** for XDP programs or
3274
3274
* **struct sk_buff** tc cls_act programs.
···
5106
5096
* **BPF_F_TIMER_ABS**
5107
5097
* Start the timer in absolute expire value instead of the
5108
5098
* default relative one.
5099
+
* **BPF_F_TIMER_CPU_PIN**
5100
+
* Timer will be pinned to the CPU of the caller.
5109
5101
*
5110
5102
* Return
5111
5103
* 0 on success.
···
6544
6532
__aligned_u64 addrs;
6545
6533
__u32 count; /* in/out: kprobe_multi function count */
6546
6534
__u32 flags;
6535
+
__u64 missed;
6547
6536
} kprobe_multi;
6548
6537
struct {
6549
6538
__u32 type; /* enum bpf_perf_event_type */
···
6560
6547
__u32 name_len;
6561
6548
__u32 offset; /* offset from func_name */
6562
6549
__u64 addr;
6550
+
__u64 missed;
6563
6551
} kprobe; /* BPF_PERF_EVENT_KPROBE, BPF_PERF_EVENT_KRETPROBE */
6564
6552
struct {
6565
6553
__aligned_u64 tp_name; /* in/out */
···
6974
6960
BPF_FIB_LOOKUP_OUTPUT = (1U << 1),
6975
6961
BPF_FIB_LOOKUP_SKIP_NEIGH = (1U << 2),
6976
6962
BPF_FIB_LOOKUP_TBID = (1U << 3),
6963
+
BPF_FIB_LOOKUP_SRC = (1U << 4),
6977
6964
};
6978
6965
6979
6966
enum {
···
6987
6972
BPF_FIB_LKUP_RET_UNSUPP_LWT, /* fwd requires encapsulation */
6988
6973
BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */
6989
6974
BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */
6975
+
BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */
6990
6976
};
6991
6977
6992
6978
struct bpf_fib_lookup {
···
7022
7006
__u32 rt_metric;
7023
7007
};
7024
7008
7009
+
/* input: source address to consider for lookup
7010
+
* output: source address result from lookup
7011
+
*/
7025
7012
union {
7026
7013
__be32 ipv4_src;
7027
7014
__u32 ipv6_src[4]; /* in6_addr; network order */
···
7326
7307
* Flags to control bpf_timer_start() behaviour.
7327
7308
* - BPF_F_TIMER_ABS: Timeout passed is absolute time, by default it is
7328
7309
* relative to current time.
7310
+
* - BPF_F_TIMER_CPU_PIN: Timer will be pinned to the CPU of the caller.
7329
7311
*/
7330
7312
enum {
7331
7313
BPF_F_TIMER_ABS = (1ULL << 0),
7314
+
BPF_F_TIMER_CPU_PIN = (1ULL << 1),
7332
7315
};
7333
7316
7334
7317
/* BPF numbers iterator state */
-2
tools/lib/bpf/bpf_tracing.h
-2
tools/lib/bpf/bpf_tracing.h
···
362
362
#define __PT_PARM7_REG a6
363
363
#define __PT_PARM8_REG a7
364
364
365
-
/* riscv does not select ARCH_HAS_SYSCALL_WRAPPER. */
366
-
#define PT_REGS_SYSCALL_REGS(ctx) ctx
367
365
#define __PT_PARM1_SYSCALL_REG __PT_PARM1_REG
368
366
#define __PT_PARM2_SYSCALL_REG __PT_PARM2_REG
369
367
#define __PT_PARM3_SYSCALL_REG __PT_PARM3_REG
+127
-12
tools/lib/bpf/elf.c
+127
-12
tools/lib/bpf/elf.c
···
1
1
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
2
2
3
+
#ifndef _GNU_SOURCE
4
+
#define _GNU_SOURCE
5
+
#endif
3
6
#include <libelf.h>
4
7
#include <gelf.h>
5
8
#include <fcntl.h>
···
12
9
#include "str_error.h"
13
10
14
11
#define STRERR_BUFSIZE 128
12
+
13
+
/* A SHT_GNU_versym section holds 16-bit words. This bit is set if
14
+
* the symbol is hidden and can only be seen when referenced using an
15
+
* explicit version number. This is a GNU extension.
16
+
*/
17
+
#define VERSYM_HIDDEN 0x8000
18
+
19
+
/* This is the mask for the rest of the data in a word read from a
20
+
* SHT_GNU_versym section.
21
+
*/
22
+
#define VERSYM_VERSION 0x7fff
15
23
16
24
int elf_open(const char *binary_path, struct elf_fd *elf_fd)
17
25
{
···
78
64
const char *name;
79
65
GElf_Sym sym;
80
66
GElf_Shdr sh;
67
+
int ver;
68
+
bool hidden;
81
69
};
82
70
83
71
struct elf_sym_iter {
84
72
Elf *elf;
85
73
Elf_Data *syms;
74
+
Elf_Data *versyms;
75
+
Elf_Data *verdefs;
86
76
size_t nr_syms;
87
77
size_t strtabidx;
78
+
size_t verdef_strtabidx;
88
79
size_t next_sym_idx;
89
80
struct elf_sym sym;
90
81
int st_type;
···
130
111
iter->nr_syms = iter->syms->d_size / sh.sh_entsize;
131
112
iter->elf = elf;
132
113
iter->st_type = st_type;
114
+
115
+
/* Version symbol table is meaningful to dynsym only */
116
+
if (sh_type != SHT_DYNSYM)
117
+
return 0;
118
+
119
+
scn = elf_find_next_scn_by_type(elf, SHT_GNU_versym, NULL);
120
+
if (!scn)
121
+
return 0;
122
+
iter->versyms = elf_getdata(scn, 0);
123
+
124
+
scn = elf_find_next_scn_by_type(elf, SHT_GNU_verdef, NULL);
125
+
if (!scn) {
126
+
pr_debug("elf: failed to find verdef ELF sections in '%s'\n", binary_path);
127
+
return -ENOENT;
128
+
}
129
+
if (!gelf_getshdr(scn, &sh))
130
+
return -EINVAL;
131
+
iter->verdef_strtabidx = sh.sh_link;
132
+
iter->verdefs = elf_getdata(scn, 0);
133
+
133
134
return 0;
134
135
}
135
136
···
158
119
struct elf_sym *ret = &iter->sym;
159
120
GElf_Sym *sym = &ret->sym;
160
121
const char *name = NULL;
122
+
GElf_Versym versym;
161
123
Elf_Scn *sym_scn;
162
124
size_t idx;
163
125
···
178
138
179
139
iter->next_sym_idx = idx + 1;
180
140
ret->name = name;
141
+
ret->ver = 0;
142
+
ret->hidden = false;
143
+
144
+
if (iter->versyms) {
145
+
if (!gelf_getversym(iter->versyms, idx, &versym))
146
+
continue;
147
+
ret->ver = versym & VERSYM_VERSION;
148
+
ret->hidden = versym & VERSYM_HIDDEN;
149
+
}
181
150
return ret;
182
151
}
183
152
184
153
return NULL;
185
154
}
186
155
156
+
static const char *elf_get_vername(struct elf_sym_iter *iter, int ver)
157
+
{
158
+
GElf_Verdaux verdaux;
159
+
GElf_Verdef verdef;
160
+
int offset;
161
+
162
+
offset = 0;
163
+
while (gelf_getverdef(iter->verdefs, offset, &verdef)) {
164
+
if (verdef.vd_ndx != ver) {
165
+
if (!verdef.vd_next)
166
+
break;
167
+
168
+
offset += verdef.vd_next;
169
+
continue;
170
+
}
171
+
172
+
if (!gelf_getverdaux(iter->verdefs, offset + verdef.vd_aux, &verdaux))
173
+
break;
174
+
175
+
return elf_strptr(iter->elf, iter->verdef_strtabidx, verdaux.vda_name);
176
+
177
+
}
178
+
return NULL;
179
+
}
180
+
181
+
static bool symbol_match(struct elf_sym_iter *iter, int sh_type, struct elf_sym *sym,
182
+
const char *name, size_t name_len, const char *lib_ver)
183
+
{
184
+
const char *ver_name;
185
+
186
+
/* Symbols are in forms of func, func@LIB_VER or func@@LIB_VER
187
+
* make sure the func part matches the user specified name
188
+
*/
189
+
if (strncmp(sym->name, name, name_len) != 0)
190
+
return false;
191
+
192
+
/* ...but we don't want a search for "foo" to match 'foo2" also, so any
193
+
* additional characters in sname should be of the form "@@LIB".
194
+
*/
195
+
if (sym->name[name_len] != '\0' && sym->name[name_len] != '@')
196
+
return false;
197
+
198
+
/* If user does not specify symbol version, then we got a match */
199
+
if (!lib_ver)
200
+
return true;
201
+
202
+
/* If user specifies symbol version, for dynamic symbols,
203
+
* get version name from ELF verdef section for comparison.
204
+
*/
205
+
if (sh_type == SHT_DYNSYM) {
206
+
ver_name = elf_get_vername(iter, sym->ver);
207
+
if (!ver_name)
208
+
return false;
209
+
return strcmp(ver_name, lib_ver) == 0;
210
+
}
211
+
212
+
/* For normal symbols, it is already in form of func@LIB_VER */
213
+
return strcmp(sym->name, name) == 0;
214
+
}
187
215
188
216
/* Transform symbol's virtual address (absolute for binaries and relative
189
217
* for shared libs) into file offset, which is what kernel is expecting
···
274
166
long elf_find_func_offset(Elf *elf, const char *binary_path, const char *name)
275
167
{
276
168
int i, sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
277
-
bool is_shared_lib, is_name_qualified;
169
+
const char *at_symbol, *lib_ver;
170
+
bool is_shared_lib;
278
171
long ret = -ENOENT;
279
172
size_t name_len;
280
173
GElf_Ehdr ehdr;
···
288
179
/* for shared lib case, we do not need to calculate relative offset */
289
180
is_shared_lib = ehdr.e_type == ET_DYN;
290
181
291
-
name_len = strlen(name);
292
-
/* Does name specify "@@LIB"? */
293
-
is_name_qualified = strstr(name, "@@") != NULL;
182
+
/* Does name specify "@@LIB_VER" or "@LIB_VER" ? */
183
+
at_symbol = strchr(name, '@');
184
+
if (at_symbol) {
185
+
name_len = at_symbol - name;
186
+
/* skip second @ if it's @@LIB_VER case */
187
+
if (at_symbol[1] == '@')
188
+
at_symbol++;
189
+
lib_ver = at_symbol + 1;
190
+
} else {
191
+
name_len = strlen(name);
192
+
lib_ver = NULL;
193
+
}
294
194
295
195
/* Search SHT_DYNSYM, SHT_SYMTAB for symbol. This search order is used because if
296
196
* a binary is stripped, it may only have SHT_DYNSYM, and a fully-statically
···
319
201
goto out;
320
202
321
203
while ((sym = elf_sym_iter_next(&iter))) {
322
-
/* User can specify func, func@@LIB or func@@LIB_VERSION. */
323
-
if (strncmp(sym->name, name, name_len) != 0)
324
-
continue;
325
-
/* ...but we don't want a search for "foo" to match 'foo2" also, so any
326
-
* additional characters in sname should be of the form "@@LIB".
327
-
*/
328
-
if (!is_name_qualified && sym->name[name_len] != '\0' && sym->name[name_len] != '@')
204
+
if (!symbol_match(&iter, sh_types[i], sym, name, name_len, lib_ver))
329
205
continue;
330
206
331
207
cur_bind = GELF_ST_BIND(sym->sym.st_info);
332
208
333
209
if (ret > 0) {
334
210
/* handle multiple matches */
335
-
if (last_bind != STB_WEAK && cur_bind != STB_WEAK) {
211
+
if (elf_sym_offset(sym) == ret) {
212
+
/* same offset, no problem */
213
+
continue;
214
+
} else if (last_bind != STB_WEAK && cur_bind != STB_WEAK) {
336
215
/* Only accept one non-weak bind. */
337
216
pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
338
217
sym->name, name, binary_path);
+26
-6
tools/lib/bpf/libbpf.c
+26
-6
tools/lib/bpf/libbpf.c
···
82
82
[BPF_CGROUP_INET6_BIND] = "cgroup_inet6_bind",
83
83
[BPF_CGROUP_INET4_CONNECT] = "cgroup_inet4_connect",
84
84
[BPF_CGROUP_INET6_CONNECT] = "cgroup_inet6_connect",
85
+
[BPF_CGROUP_UNIX_CONNECT] = "cgroup_unix_connect",
85
86
[BPF_CGROUP_INET4_POST_BIND] = "cgroup_inet4_post_bind",
86
87
[BPF_CGROUP_INET6_POST_BIND] = "cgroup_inet6_post_bind",
87
88
[BPF_CGROUP_INET4_GETPEERNAME] = "cgroup_inet4_getpeername",
88
89
[BPF_CGROUP_INET6_GETPEERNAME] = "cgroup_inet6_getpeername",
90
+
[BPF_CGROUP_UNIX_GETPEERNAME] = "cgroup_unix_getpeername",
89
91
[BPF_CGROUP_INET4_GETSOCKNAME] = "cgroup_inet4_getsockname",
90
92
[BPF_CGROUP_INET6_GETSOCKNAME] = "cgroup_inet6_getsockname",
93
+
[BPF_CGROUP_UNIX_GETSOCKNAME] = "cgroup_unix_getsockname",
91
94
[BPF_CGROUP_UDP4_SENDMSG] = "cgroup_udp4_sendmsg",
92
95
[BPF_CGROUP_UDP6_SENDMSG] = "cgroup_udp6_sendmsg",
96
+
[BPF_CGROUP_UNIX_SENDMSG] = "cgroup_unix_sendmsg",
93
97
[BPF_CGROUP_SYSCTL] = "cgroup_sysctl",
94
98
[BPF_CGROUP_UDP4_RECVMSG] = "cgroup_udp4_recvmsg",
95
99
[BPF_CGROUP_UDP6_RECVMSG] = "cgroup_udp6_recvmsg",
100
+
[BPF_CGROUP_UNIX_RECVMSG] = "cgroup_unix_recvmsg",
96
101
[BPF_CGROUP_GETSOCKOPT] = "cgroup_getsockopt",
97
102
[BPF_CGROUP_SETSOCKOPT] = "cgroup_setsockopt",
98
103
[BPF_SK_SKB_STREAM_PARSER] = "sk_skb_stream_parser",
···
8965
8960
SEC_DEF("cgroup/bind6", CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_BIND, SEC_ATTACHABLE),
8966
8961
SEC_DEF("cgroup/connect4", CGROUP_SOCK_ADDR, BPF_CGROUP_INET4_CONNECT, SEC_ATTACHABLE),
8967
8962
SEC_DEF("cgroup/connect6", CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_CONNECT, SEC_ATTACHABLE),
8963
+
SEC_DEF("cgroup/connect_unix", CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_CONNECT, SEC_ATTACHABLE),
8968
8964
SEC_DEF("cgroup/sendmsg4", CGROUP_SOCK_ADDR, BPF_CGROUP_UDP4_SENDMSG, SEC_ATTACHABLE),
8969
8965
SEC_DEF("cgroup/sendmsg6", CGROUP_SOCK_ADDR, BPF_CGROUP_UDP6_SENDMSG, SEC_ATTACHABLE),
8966
+
SEC_DEF("cgroup/sendmsg_unix", CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_SENDMSG, SEC_ATTACHABLE),
8970
8967
SEC_DEF("cgroup/recvmsg4", CGROUP_SOCK_ADDR, BPF_CGROUP_UDP4_RECVMSG, SEC_ATTACHABLE),
8971
8968
SEC_DEF("cgroup/recvmsg6", CGROUP_SOCK_ADDR, BPF_CGROUP_UDP6_RECVMSG, SEC_ATTACHABLE),
8969
+
SEC_DEF("cgroup/recvmsg_unix", CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_RECVMSG, SEC_ATTACHABLE),
8972
8970
SEC_DEF("cgroup/getpeername4", CGROUP_SOCK_ADDR, BPF_CGROUP_INET4_GETPEERNAME, SEC_ATTACHABLE),
8973
8971
SEC_DEF("cgroup/getpeername6", CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_GETPEERNAME, SEC_ATTACHABLE),
8972
+
SEC_DEF("cgroup/getpeername_unix", CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_GETPEERNAME, SEC_ATTACHABLE),
8974
8973
SEC_DEF("cgroup/getsockname4", CGROUP_SOCK_ADDR, BPF_CGROUP_INET4_GETSOCKNAME, SEC_ATTACHABLE),
8975
8974
SEC_DEF("cgroup/getsockname6", CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_GETSOCKNAME, SEC_ATTACHABLE),
8975
+
SEC_DEF("cgroup/getsockname_unix", CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_GETSOCKNAME, SEC_ATTACHABLE),
8976
8976
SEC_DEF("cgroup/sysctl", CGROUP_SYSCTL, BPF_CGROUP_SYSCTL, SEC_ATTACHABLE),
8977
8977
SEC_DEF("cgroup/getsockopt", CGROUP_SOCKOPT, BPF_CGROUP_GETSOCKOPT, SEC_ATTACHABLE),
8978
8978
SEC_DEF("cgroup/setsockopt", CGROUP_SOCKOPT, BPF_CGROUP_SETSOCKOPT, SEC_ATTACHABLE),
···
11124
11114
11125
11115
*link = NULL;
11126
11116
11127
-
n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
11117
+
n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%m[^\n]",
11128
11118
&probe_type, &binary_path, &func_name);
11129
11119
switch (n) {
11130
11120
case 1:
···
11634
11624
static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
11635
11625
{
11636
11626
DECLARE_LIBBPF_OPTS(bpf_uprobe_opts, opts);
11637
-
char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
11638
-
int n, ret = -EINVAL;
11627
+
char *probe_type = NULL, *binary_path = NULL, *func_name = NULL, *func_off;
11628
+
int n, c, ret = -EINVAL;
11639
11629
long offset = 0;
11640
11630
11641
11631
*link = NULL;
11642
11632
11643
-
n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%m[a-zA-Z0-9_.]+%li",
11644
-
&probe_type, &binary_path, &func_name, &offset);
11633
+
n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%m[^\n]",
11634
+
&probe_type, &binary_path, &func_name);
11645
11635
switch (n) {
11646
11636
case 1:
11647
11637
/* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
···
11652
11642
prog->name, prog->sec_name);
11653
11643
break;
11654
11644
case 3:
11655
-
case 4:
11645
+
/* check if user specifies `+offset`, if yes, this should be
11646
+
* the last part of the string, make sure sscanf read to EOL
11647
+
*/
11648
+
func_off = strrchr(func_name, '+');
11649
+
if (func_off) {
11650
+
n = sscanf(func_off, "+%li%n", &offset, &c);
11651
+
if (n == 1 && *(func_off + c) == '\0')
11652
+
func_off[0] = '\0';
11653
+
else
11654
+
offset = 0;
11655
+
}
11656
11656
opts.retprobe = strcmp(probe_type, "uretprobe") == 0 ||
11657
11657
strcmp(probe_type, "uretprobe.s") == 0;
11658
11658
if (opts.retprobe && offset != 0) {
+73
tools/lib/bpf/libbpf.h
+73
tools/lib/bpf/libbpf.h
···
1229
1229
1230
1230
/* Ring buffer APIs */
1231
1231
struct ring_buffer;
1232
+
struct ring;
1232
1233
struct user_ring_buffer;
1233
1234
1234
1235
typedef int (*ring_buffer_sample_fn)(void *ctx, void *data, size_t size);
···
1249
1248
LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms);
1250
1249
LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb);
1251
1250
LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
1251
+
1252
+
/**
1253
+
* @brief **ring_buffer__ring()** returns the ringbuffer object inside a given
1254
+
* ringbuffer manager representing a single BPF_MAP_TYPE_RINGBUF map instance.
1255
+
*
1256
+
* @param rb A ringbuffer manager object.
1257
+
* @param idx An index into the ringbuffers contained within the ringbuffer
1258
+
* manager object. The index is 0-based and corresponds to the order in which
1259
+
* ring_buffer__add was called.
1260
+
* @return A ringbuffer object on success; NULL and errno set if the index is
1261
+
* invalid.
1262
+
*/
1263
+
LIBBPF_API struct ring *ring_buffer__ring(struct ring_buffer *rb,
1264
+
unsigned int idx);
1265
+
1266
+
/**
1267
+
* @brief **ring__consumer_pos()** returns the current consumer position in the
1268
+
* given ringbuffer.
1269
+
*
1270
+
* @param r A ringbuffer object.
1271
+
* @return The current consumer position.
1272
+
*/
1273
+
LIBBPF_API unsigned long ring__consumer_pos(const struct ring *r);
1274
+
1275
+
/**
1276
+
* @brief **ring__producer_pos()** returns the current producer position in the
1277
+
* given ringbuffer.
1278
+
*
1279
+
* @param r A ringbuffer object.
1280
+
* @return The current producer position.
1281
+
*/
1282
+
LIBBPF_API unsigned long ring__producer_pos(const struct ring *r);
1283
+
1284
+
/**
1285
+
* @brief **ring__avail_data_size()** returns the number of bytes in the
1286
+
* ringbuffer not yet consumed. This has no locking associated with it, so it
1287
+
* can be inaccurate if operations are ongoing while this is called. However, it
1288
+
* should still show the correct trend over the long-term.
1289
+
*
1290
+
* @param r A ringbuffer object.
1291
+
* @return The number of bytes not yet consumed.
1292
+
*/
1293
+
LIBBPF_API size_t ring__avail_data_size(const struct ring *r);
1294
+
1295
+
/**
1296
+
* @brief **ring__size()** returns the total size of the ringbuffer's map data
1297
+
* area (excluding special producer/consumer pages). Effectively this gives the
1298
+
* amount of usable bytes of data inside the ringbuffer.
1299
+
*
1300
+
* @param r A ringbuffer object.
1301
+
* @return The total size of the ringbuffer map data area.
1302
+
*/
1303
+
LIBBPF_API size_t ring__size(const struct ring *r);
1304
+
1305
+
/**
1306
+
* @brief **ring__map_fd()** returns the file descriptor underlying the given
1307
+
* ringbuffer.
1308
+
*
1309
+
* @param r A ringbuffer object.
1310
+
* @return The underlying ringbuffer file descriptor
1311
+
*/
1312
+
LIBBPF_API int ring__map_fd(const struct ring *r);
1313
+
1314
+
/**
1315
+
* @brief **ring__consume()** consumes available ringbuffer data without event
1316
+
* polling.
1317
+
*
1318
+
* @param r A ringbuffer object.
1319
+
* @return The number of records consumed (or INT_MAX, whichever is less), or
1320
+
* a negative number if any of the callbacks return an error.
1321
+
*/
1322
+
LIBBPF_API int ring__consume(struct ring *r);
1252
1323
1253
1324
struct user_ring_buffer_opts {
1254
1325
size_t sz; /* size of this struct, for forward/backward compatibility */
+7
tools/lib/bpf/libbpf.map
+7
tools/lib/bpf/libbpf.map
···
400
400
bpf_program__attach_netfilter;
401
401
bpf_program__attach_tcx;
402
402
bpf_program__attach_uprobe_multi;
403
+
ring__avail_data_size;
404
+
ring__consume;
405
+
ring__consumer_pos;
406
+
ring__map_fd;
407
+
ring__producer_pos;
408
+
ring__size;
409
+
ring_buffer__ring;
403
410
} LIBBPF_1.2.0;
+72
-13
tools/lib/bpf/ringbuf.c
+72
-13
tools/lib/bpf/ringbuf.c
···
34
34
35
35
struct ring_buffer {
36
36
struct epoll_event *events;
37
-
struct ring *rings;
37
+
struct ring **rings;
38
38
size_t page_size;
39
39
int epoll_fd;
40
40
int ring_cnt;
···
57
57
__u32 pad;
58
58
};
59
59
60
-
static void ringbuf_unmap_ring(struct ring_buffer *rb, struct ring *r)
60
+
static void ringbuf_free_ring(struct ring_buffer *rb, struct ring *r)
61
61
{
62
62
if (r->consumer_pos) {
63
63
munmap(r->consumer_pos, rb->page_size);
···
67
67
munmap(r->producer_pos, rb->page_size + 2 * (r->mask + 1));
68
68
r->producer_pos = NULL;
69
69
}
70
+
71
+
free(r);
70
72
}
71
73
72
74
/* Add extra RINGBUF maps to this ring buffer manager */
···
109
107
return libbpf_err(-ENOMEM);
110
108
rb->events = tmp;
111
109
112
-
r = &rb->rings[rb->ring_cnt];
113
-
memset(r, 0, sizeof(*r));
110
+
r = calloc(1, sizeof(*r));
111
+
if (!r)
112
+
return libbpf_err(-ENOMEM);
113
+
rb->rings[rb->ring_cnt] = r;
114
114
115
115
r->map_fd = map_fd;
116
116
r->sample_cb = sample_cb;
···
125
121
err = -errno;
126
122
pr_warn("ringbuf: failed to mmap consumer page for map fd=%d: %d\n",
127
123
map_fd, err);
128
-
return libbpf_err(err);
124
+
goto err_out;
129
125
}
130
126
r->consumer_pos = tmp;
131
127
···
135
131
*/
136
132
mmap_sz = rb->page_size + 2 * (__u64)info.max_entries;
137
133
if (mmap_sz != (__u64)(size_t)mmap_sz) {
134
+
err = -E2BIG;
138
135
pr_warn("ringbuf: ring buffer size (%u) is too big\n", info.max_entries);
139
-
return libbpf_err(-E2BIG);
136
+
goto err_out;
140
137
}
141
138
tmp = mmap(NULL, (size_t)mmap_sz, PROT_READ, MAP_SHARED, map_fd, rb->page_size);
142
139
if (tmp == MAP_FAILED) {
143
140
err = -errno;
144
-
ringbuf_unmap_ring(rb, r);
145
141
pr_warn("ringbuf: failed to mmap data pages for map fd=%d: %d\n",
146
142
map_fd, err);
147
-
return libbpf_err(err);
143
+
goto err_out;
148
144
}
149
145
r->producer_pos = tmp;
150
146
r->data = tmp + rb->page_size;
···
156
152
e->data.fd = rb->ring_cnt;
157
153
if (epoll_ctl(rb->epoll_fd, EPOLL_CTL_ADD, map_fd, e) < 0) {
158
154
err = -errno;
159
-
ringbuf_unmap_ring(rb, r);
160
155
pr_warn("ringbuf: failed to epoll add map fd=%d: %d\n",
161
156
map_fd, err);
162
-
return libbpf_err(err);
157
+
goto err_out;
163
158
}
164
159
165
160
rb->ring_cnt++;
166
161
return 0;
162
+
163
+
err_out:
164
+
ringbuf_free_ring(rb, r);
165
+
return libbpf_err(err);
167
166
}
168
167
169
168
void ring_buffer__free(struct ring_buffer *rb)
···
177
170
return;
178
171
179
172
for (i = 0; i < rb->ring_cnt; ++i)
180
-
ringbuf_unmap_ring(rb, &rb->rings[i]);
173
+
ringbuf_free_ring(rb, rb->rings[i]);
181
174
if (rb->epoll_fd >= 0)
182
175
close(rb->epoll_fd);
183
176
···
285
278
int i;
286
279
287
280
for (i = 0; i < rb->ring_cnt; i++) {
288
-
struct ring *ring = &rb->rings[i];
281
+
struct ring *ring = rb->rings[i];
289
282
290
283
err = ringbuf_process_ring(ring);
291
284
if (err < 0)
···
312
305
313
306
for (i = 0; i < cnt; i++) {
314
307
__u32 ring_id = rb->events[i].data.fd;
315
-
struct ring *ring = &rb->rings[ring_id];
308
+
struct ring *ring = rb->rings[ring_id];
316
309
317
310
err = ringbuf_process_ring(ring);
318
311
if (err < 0)
···
328
321
int ring_buffer__epoll_fd(const struct ring_buffer *rb)
329
322
{
330
323
return rb->epoll_fd;
324
+
}
325
+
326
+
struct ring *ring_buffer__ring(struct ring_buffer *rb, unsigned int idx)
327
+
{
328
+
if (idx >= rb->ring_cnt)
329
+
return errno = ERANGE, NULL;
330
+
331
+
return rb->rings[idx];
332
+
}
333
+
334
+
unsigned long ring__consumer_pos(const struct ring *r)
335
+
{
336
+
/* Synchronizes with smp_store_release() in ringbuf_process_ring(). */
337
+
return smp_load_acquire(r->consumer_pos);
338
+
}
339
+
340
+
unsigned long ring__producer_pos(const struct ring *r)
341
+
{
342
+
/* Synchronizes with smp_store_release() in __bpf_ringbuf_reserve() in
343
+
* the kernel.
344
+
*/
345
+
return smp_load_acquire(r->producer_pos);
346
+
}
347
+
348
+
size_t ring__avail_data_size(const struct ring *r)
349
+
{
350
+
unsigned long cons_pos, prod_pos;
351
+
352
+
cons_pos = ring__consumer_pos(r);
353
+
prod_pos = ring__producer_pos(r);
354
+
return prod_pos - cons_pos;
355
+
}
356
+
357
+
size_t ring__size(const struct ring *r)
358
+
{
359
+
return r->mask + 1;
360
+
}
361
+
362
+
int ring__map_fd(const struct ring *r)
363
+
{
364
+
return r->map_fd;
365
+
}
366
+
367
+
int ring__consume(struct ring *r)
368
+
{
369
+
int64_t res;
370
+
371
+
res = ringbuf_process_ring(r);
372
+
if (res < 0)
373
+
return libbpf_err(res);
374
+
375
+
return res > INT_MAX ? INT_MAX : res;
331
376
}
332
377
333
378
static void user_ringbuf_unmap_ring(struct user_ring_buffer *rb)
+1
tools/testing/selftests/bpf/DENYLIST.aarch64
+1
tools/testing/selftests/bpf/DENYLIST.aarch64
···
10
10
fill_link_info/kprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95
11
11
fill_link_info/kretprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95
12
12
fill_link_info/kprobe_multi_invalid_ubuff # bpf_program__attach_kprobe_multi_opts unexpected error: -95
13
+
missed/kprobe_recursion # missed_kprobe_recursion__attach unexpected error: -95 (errno 95)
-25
tools/testing/selftests/bpf/DENYLIST.s390x
-25
tools/testing/selftests/bpf/DENYLIST.s390x
···
1
1
# TEMPORARY
2
2
# Alphabetical order
3
-
bloom_filter_map # failed to find kernel BTF type ID of '__x64_sys_getpgid': -3 (?)
4
-
bpf_cookie # failed to open_and_load program: -524 (trampoline)
5
-
bpf_loop # attaches to __x64_sys_nanosleep
6
-
cgrp_local_storage # prog_attach unexpected error: -524 (trampoline)
7
-
dynptr/test_dynptr_skb_data
8
-
dynptr/test_skb_readonly
9
3
exceptions # JIT does not support calling kfunc bpf_throw (exceptions)
10
-
fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline)
11
4
get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace)
12
-
iters/testmod_seq* # s390x doesn't support kfuncs in modules yet
13
-
kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95
14
-
kprobe_multi_test # relies on fentry
15
-
ksyms_btf/weak_ksyms* # test_ksyms_weak__open_and_load unexpected error: -22 (kfunc)
16
-
ksyms_module # test_ksyms_module__open_and_load unexpected error: -9 (?)
17
-
ksyms_module_libbpf # JIT does not support calling kernel function (kfunc)
18
-
ksyms_module_lskel # test_ksyms_module_lskel__open_and_load unexpected error: -9 (?)
19
-
module_attach # skel_attach skeleton attach failed: -524 (trampoline)
20
-
ringbuf # skel_load skeleton load failed (?)
21
5
stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?)
22
-
test_lsm # attach unexpected error: -524 (trampoline)
23
-
trace_printk # trace_printk__load unexpected error: -2 (errno 2) (?)
24
-
trace_vprintk # trace_vprintk__open_and_load unexpected error: -9 (?)
25
-
unpriv_bpf_disabled # fentry
26
-
user_ringbuf # failed to find kernel BTF type ID of '__s390x_sys_prctl': -3 (?)
27
-
verif_stats # trace_vprintk__open_and_load unexpected error: -9 (?)
28
-
xdp_bonding # failed to auto-attach program 'trace_on_entry': -524 (trampoline)
29
-
xdp_metadata # JIT does not support calling kernel function (kfunc)
30
-
test_task_under_cgroup # JIT does not support calling kernel function (kfunc)
+23
-14
tools/testing/selftests/bpf/Makefile
+23
-14
tools/testing/selftests/bpf/Makefile
···
27
27
BPF_GCC ?= $(shell command -v bpf-gcc;)
28
28
SAN_CFLAGS ?=
29
29
SAN_LDFLAGS ?= $(SAN_CFLAGS)
30
-
CFLAGS += -g -O0 -rdynamic -Wall -Werror $(GENFLAGS) $(SAN_CFLAGS) \
30
+
RELEASE ?=
31
+
OPT_FLAGS ?= $(if $(RELEASE),-O2,-O0)
32
+
CFLAGS += -g $(OPT_FLAGS) -rdynamic \
33
+
-Wall -Werror \
34
+
$(GENFLAGS) $(SAN_CFLAGS) \
31
35
-I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
32
36
-I$(TOOLSINCDIR) -I$(APIDIR) -I$(OUTPUT)
33
37
LDFLAGS += $(SAN_LDFLAGS)
···
108
104
xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata \
109
105
xdp_features
110
106
111
-
TEST_GEN_FILES += liburandom_read.so urandom_read sign-file
107
+
TEST_GEN_FILES += liburandom_read.so urandom_read sign-file uprobe_multi
112
108
113
109
# Emit succinct information message describing current building step
114
110
# $1 - generic step name (e.g., CC, LINK, etc);
···
192
188
$(Q)$(LINK.c) $^ $(LDLIBS) -o $@
193
189
194
190
# LLVM's ld.lld doesn't support all the architectures, so use it only on x86
195
-
ifeq ($(SRCARCH),x86)
191
+
ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 riscv))
196
192
LLD := lld
197
193
else
198
194
LLD := ld
···
200
196
201
197
# Filter out -static for liburandom_read.so and its dependent targets so that static builds
202
198
# do not fail. Static builds leave urandom_read relying on system-wide shared libraries.
203
-
$(OUTPUT)/liburandom_read.so: urandom_read_lib1.c urandom_read_lib2.c
199
+
$(OUTPUT)/liburandom_read.so: urandom_read_lib1.c urandom_read_lib2.c liburandom_read.map
204
200
$(call msg,LIB,,$@)
205
-
$(Q)$(CLANG) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) \
206
-
$^ $(filter-out -static,$(LDLIBS)) \
201
+
$(Q)$(CLANG) $(CLANG_TARGET_ARCH) \
202
+
$(filter-out -static,$(CFLAGS) $(LDFLAGS)) \
203
+
$(filter %.c,$^) $(filter-out -static,$(LDLIBS)) \
207
204
-fuse-ld=$(LLD) -Wl,-znoseparate-code -Wl,--build-id=sha1 \
205
+
-Wl,--version-script=liburandom_read.map \
208
206
-fPIC -shared -o $@
209
207
210
208
$(OUTPUT)/urandom_read: urandom_read.c urandom_read_aux.c $(OUTPUT)/liburandom_read.so
211
209
$(call msg,BINARY,,$@)
212
-
$(Q)$(CLANG) $(filter-out -static,$(CFLAGS) $(LDFLAGS)) $(filter %.c,$^) \
213
-
-lurandom_read $(filter-out -static,$(LDLIBS)) -L$(OUTPUT) \
210
+
$(Q)$(CLANG) $(CLANG_TARGET_ARCH) \
211
+
$(filter-out -static,$(CFLAGS) $(LDFLAGS)) $(filter %.c,$^) \
212
+
-lurandom_read $(filter-out -static,$(LDLIBS)) -L$(OUTPUT) \
214
213
-fuse-ld=$(LLD) -Wl,-znoseparate-code -Wl,--build-id=sha1 \
215
214
-Wl,-rpath=. -o $@
216
215
···
245
238
BPFTOOL_OUTPUT=$(HOST_BUILD_DIR)/bpftool/ \
246
239
BPFOBJ_OUTPUT=$(BUILD_DIR)/libbpf \
247
240
BPFOBJ=$(BPFOBJ) BPF_INCLUDE=$(INCLUDE_DIR) \
248
-
EXTRA_CFLAGS='-g -O0 $(SAN_CFLAGS)' \
241
+
EXTRA_CFLAGS='-g $(OPT_FLAGS) $(SAN_CFLAGS)' \
249
242
EXTRA_LDFLAGS='$(SAN_LDFLAGS)' && \
250
243
cp $(RUNQSLOWER_OUTPUT)runqslower $@
251
244
···
283
276
$(HOST_BPFOBJ) | $(HOST_BUILD_DIR)/bpftool
284
277
$(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
285
278
ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" \
286
-
EXTRA_CFLAGS='-g -O0' \
279
+
EXTRA_CFLAGS='-g $(OPT_FLAGS)' \
287
280
OUTPUT=$(HOST_BUILD_DIR)/bpftool/ \
288
281
LIBBPF_OUTPUT=$(HOST_BUILD_DIR)/libbpf/ \
289
282
LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \
···
294
287
$(BPFOBJ) | $(BUILD_DIR)/bpftool
295
288
$(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
296
289
ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \
297
-
EXTRA_CFLAGS='-g -O0' \
290
+
EXTRA_CFLAGS='-g $(OPT_FLAGS)' \
298
291
OUTPUT=$(BUILD_DIR)/bpftool/ \
299
292
LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \
300
293
LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \
···
317
310
$(APIDIR)/linux/bpf.h \
318
311
| $(BUILD_DIR)/libbpf
319
312
$(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(BUILD_DIR)/libbpf/ \
320
-
EXTRA_CFLAGS='-g -O0 $(SAN_CFLAGS)' \
313
+
EXTRA_CFLAGS='-g $(OPT_FLAGS) $(SAN_CFLAGS)' \
321
314
EXTRA_LDFLAGS='$(SAN_LDFLAGS)' \
322
315
DESTDIR=$(SCRATCH_DIR) prefix= all install_headers
323
316
···
326
319
$(APIDIR)/linux/bpf.h \
327
320
| $(HOST_BUILD_DIR)/libbpf
328
321
$(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) \
329
-
EXTRA_CFLAGS='-g -O0' ARCH= CROSS_COMPILE= \
322
+
EXTRA_CFLAGS='-g $(OPT_FLAGS)' ARCH= CROSS_COMPILE= \
330
323
OUTPUT=$(HOST_BUILD_DIR)/libbpf/ \
331
324
CC="$(HOSTCC)" LD="$(HOSTLD)" \
332
325
DESTDIR=$(HOST_SCRATCH_DIR)/ prefix= all install_headers
···
647
640
$(call msg,BINARY,,$@)
648
641
$(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@
649
642
650
-
$(OUTPUT)/xskxceiver: xskxceiver.c xskxceiver.h $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel.h $(BPFOBJ) | $(OUTPUT)
643
+
# Include find_bit.c to compile xskxceiver.
644
+
EXTRA_SRC := $(TOOLSDIR)/lib/find_bit.c
645
+
$(OUTPUT)/xskxceiver: $(EXTRA_SRC) xskxceiver.c xskxceiver.h $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel.h $(BPFOBJ) | $(OUTPUT)
651
646
$(call msg,BINARY,,$@)
652
647
$(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@
653
648
+8
tools/testing/selftests/bpf/bpf_experimental.h
+8
tools/testing/selftests/bpf/bpf_experimental.h
···
159
159
*/
160
160
extern void bpf_percpu_obj_drop_impl(void *kptr, void *meta) __ksym;
161
161
162
+
struct bpf_iter_task_vma;
163
+
164
+
extern int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
165
+
struct task_struct *task,
166
+
unsigned long addr) __ksym;
167
+
extern struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it) __ksym;
168
+
extern void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it) __ksym;
169
+
162
170
/* Convenience macro to wrap over bpf_obj_drop_impl */
163
171
#define bpf_percpu_obj_drop(kptr) bpf_percpu_obj_drop_impl(kptr, NULL)
164
172
+14
tools/testing/selftests/bpf/bpf_kfuncs.h
+14
tools/testing/selftests/bpf/bpf_kfuncs.h
···
1
1
#ifndef __BPF_KFUNCS__
2
2
#define __BPF_KFUNCS__
3
3
4
+
struct bpf_sock_addr_kern;
5
+
4
6
/* Description
5
7
* Initializes an skb-type dynptr
6
8
* Returns
···
42
40
extern bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *ptr) __ksym;
43
41
extern __u32 bpf_dynptr_size(const struct bpf_dynptr *ptr) __ksym;
44
42
extern int bpf_dynptr_clone(const struct bpf_dynptr *ptr, struct bpf_dynptr *clone__init) __ksym;
43
+
44
+
/* Description
45
+
* Modify the address of a AF_UNIX sockaddr.
46
+
* Returns__bpf_kfunc
47
+
* -EINVAL if the address size is too big or, 0 if the sockaddr was successfully modified.
48
+
*/
49
+
extern int bpf_sock_addr_set_sun_path(struct bpf_sock_addr_kern *sa_kern,
50
+
const __u8 *sun_path, __u32 sun_path__sz) __ksym;
51
+
52
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
53
+
54
+
void *bpf_rdonly_cast(void *obj, __u32 btf_id) __ksym;
45
55
46
56
#endif
+5
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
+5
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
···
138
138
it->cnt = 0;
139
139
}
140
140
141
+
__bpf_kfunc void bpf_kfunc_common_test(void)
142
+
{
143
+
}
144
+
141
145
struct bpf_testmod_btf_type_tag_1 {
142
146
int a;
143
147
};
···
347
343
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_new, KF_ITER_NEW)
348
344
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL)
349
345
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY)
346
+
BTF_ID_FLAGS(func, bpf_kfunc_common_test)
350
347
BTF_SET8_END(bpf_testmod_common_kfunc_ids)
351
348
352
349
static const struct btf_kfunc_id_set bpf_testmod_common_kfunc_set = {
+2
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h
+2
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h
···
104
104
void bpf_kfunc_call_test_fail2(struct prog_test_fail2 *p);
105
105
void bpf_kfunc_call_test_fail3(struct prog_test_fail3 *p);
106
106
void bpf_kfunc_call_test_mem_len_fail1(void *mem, int len);
107
+
108
+
void bpf_kfunc_common_test(void) __ksym;
107
109
#endif /* _BPF_TESTMOD_KFUNC_H */
+31
-7
tools/testing/selftests/bpf/cgroup_helpers.c
+31
-7
tools/testing/selftests/bpf/cgroup_helpers.c
···
49
49
snprintf(buf, sizeof(buf), "%s%s", NETCLS_MOUNT_PATH, \
50
50
CGROUP_WORK_DIR)
51
51
52
+
static __thread bool cgroup_workdir_mounted;
53
+
54
+
static void __cleanup_cgroup_environment(void);
55
+
52
56
static int __enable_controllers(const char *cgroup_path, const char *controllers)
53
57
{
54
58
char path[PATH_MAX + 1];
···
199
195
200
196
format_cgroup_path(cgroup_workdir, "");
201
197
198
+
if (mkdir(CGROUP_MOUNT_PATH, 0777) && errno != EEXIST) {
199
+
log_err("mkdir mount");
200
+
return 1;
201
+
}
202
+
202
203
if (unshare(CLONE_NEWNS)) {
203
204
log_err("unshare");
204
205
return 1;
···
218
209
log_err("mount cgroup2");
219
210
return 1;
220
211
}
212
+
cgroup_workdir_mounted = true;
221
213
222
214
/* Cleanup existing failed runs, now that the environment is setup */
223
-
cleanup_cgroup_environment();
215
+
__cleanup_cgroup_environment();
224
216
225
217
if (mkdir(cgroup_workdir, 0777) && errno != EEXIST) {
226
218
log_err("mkdir cgroup work dir");
···
316
306
}
317
307
318
308
/**
309
+
* __cleanup_cgroup_environment() - Delete temporary cgroups
310
+
*
311
+
* This is a helper for cleanup_cgroup_environment() that is responsible for
312
+
* deletion of all temporary cgroups that have been created during the test.
313
+
*/
314
+
static void __cleanup_cgroup_environment(void)
315
+
{
316
+
char cgroup_workdir[PATH_MAX + 1];
317
+
318
+
format_cgroup_path(cgroup_workdir, "");
319
+
join_cgroup_from_top(CGROUP_MOUNT_PATH);
320
+
nftw(cgroup_workdir, nftwfunc, WALK_FD_LIMIT, FTW_DEPTH | FTW_MOUNT);
321
+
}
322
+
323
+
/**
319
324
* cleanup_cgroup_environment() - Cleanup Cgroup Testing Environment
320
325
*
321
326
* This is an idempotent function to delete all temporary cgroups that
322
-
* have been created during the test, including the cgroup testing work
327
+
* have been created during the test and unmount the cgroup testing work
323
328
* directory.
324
329
*
325
330
* At call time, it moves the calling process to the root cgroup, and then
···
345
320
*/
346
321
void cleanup_cgroup_environment(void)
347
322
{
348
-
char cgroup_workdir[PATH_MAX + 1];
349
-
350
-
format_cgroup_path(cgroup_workdir, "");
351
-
join_cgroup_from_top(CGROUP_MOUNT_PATH);
352
-
nftw(cgroup_workdir, nftwfunc, WALK_FD_LIMIT, FTW_DEPTH | FTW_MOUNT);
323
+
__cleanup_cgroup_environment();
324
+
if (cgroup_workdir_mounted && umount(CGROUP_MOUNT_PATH))
325
+
log_err("umount cgroup2");
326
+
cgroup_workdir_mounted = false;
353
327
}
354
328
355
329
/**
+1
tools/testing/selftests/bpf/config
+1
tools/testing/selftests/bpf/config
+15
tools/testing/selftests/bpf/liburandom_read.map
+15
tools/testing/selftests/bpf/liburandom_read.map
···
1
+
LIBURANDOM_READ_1.0.0 {
2
+
global:
3
+
urandlib_api;
4
+
urandlib_api_sameoffset;
5
+
urandlib_read_without_sema;
6
+
urandlib_read_with_sema;
7
+
urandlib_read_with_sema_semaphore;
8
+
local:
9
+
*;
10
+
};
11
+
12
+
LIBURANDOM_READ_2.0.0 {
13
+
global:
14
+
urandlib_api;
15
+
} LIBURANDOM_READ_1.0.0;
+2
-2
tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c
+2
-2
tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c
···
33
33
{
34
34
int map_fd, map_index, ret;
35
35
__u32 map_key = 0, map_id;
36
-
char map_name[15];
36
+
char map_name[16];
37
37
38
38
for (map_index = 0; map_index < OUTER_MAP_ENTRIES; map_index++) {
39
39
memset(map_name, 0, sizeof(map_name));
40
-
sprintf(map_name, "inner_map_fd_%d", map_index);
40
+
snprintf(map_name, sizeof(map_name), "inner_map_fd_%d", map_index);
41
41
map_fd = bpf_map_create(map_type, map_name, sizeof(__u32),
42
42
sizeof(__u32), 1, NULL);
43
43
CHECK(map_fd < 0,
+34
tools/testing/selftests/bpf/network_helpers.c
+34
tools/testing/selftests/bpf/network_helpers.c
···
11
11
#include <arpa/inet.h>
12
12
#include <sys/mount.h>
13
13
#include <sys/stat.h>
14
+
#include <sys/un.h>
14
15
15
16
#include <linux/err.h>
16
17
#include <linux/in.h>
···
258
257
return 0;
259
258
}
260
259
260
+
int connect_to_addr(const struct sockaddr_storage *addr, socklen_t addrlen, int type)
261
+
{
262
+
int fd;
263
+
264
+
fd = socket(addr->ss_family, type, 0);
265
+
if (fd < 0) {
266
+
log_err("Failed to create client socket");
267
+
return -1;
268
+
}
269
+
270
+
if (connect_fd_to_addr(fd, addr, addrlen, false))
271
+
goto error_close;
272
+
273
+
return fd;
274
+
275
+
error_close:
276
+
save_errno_close(fd);
277
+
return -1;
278
+
}
279
+
261
280
static const struct network_helper_opts default_opts;
262
281
263
282
int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts)
···
400
379
}
401
380
if (len)
402
381
*len = sizeof(*sin6);
382
+
return 0;
383
+
} else if (family == AF_UNIX) {
384
+
/* Note that we always use abstract unix sockets to avoid having
385
+
* to clean up leftover files.
386
+
*/
387
+
struct sockaddr_un *sun = (void *)addr;
388
+
389
+
memset(addr, 0, sizeof(*sun));
390
+
sun->sun_family = family;
391
+
sun->sun_path[0] = 0;
392
+
strcpy(sun->sun_path + 1, addr_str);
393
+
if (len)
394
+
*len = offsetof(struct sockaddr_un, sun_path) + 1 + strlen(addr_str);
403
395
return 0;
404
396
}
405
397
return -1;
+1
tools/testing/selftests/bpf/network_helpers.h
+1
tools/testing/selftests/bpf/network_helpers.h
···
51
51
__u16 port, int timeout_ms,
52
52
unsigned int nr_listens);
53
53
void free_fds(int *fds, unsigned int nr_close_fds);
54
+
int connect_to_addr(const struct sockaddr_storage *addr, socklen_t len, int type);
54
55
int connect_to_fd(int server_fd, int timeout_ms);
55
56
int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts);
56
57
int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms);
+121
-120
tools/testing/selftests/bpf/prog_tests/align.c
+121
-120
tools/testing/selftests/bpf/prog_tests/align.c
···
6
6
7
7
struct bpf_reg_match {
8
8
unsigned int line;
9
+
const char *reg;
9
10
const char *match;
10
11
};
11
12
···
40
39
},
41
40
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
42
41
.matches = {
43
-
{0, "R1=ctx(off=0,imm=0)"},
44
-
{0, "R10=fp0"},
45
-
{0, "R3_w=2"},
46
-
{1, "R3_w=4"},
47
-
{2, "R3_w=8"},
48
-
{3, "R3_w=16"},
49
-
{4, "R3_w=32"},
42
+
{0, "R1", "ctx(off=0,imm=0)"},
43
+
{0, "R10", "fp0"},
44
+
{0, "R3_w", "2"},
45
+
{1, "R3_w", "4"},
46
+
{2, "R3_w", "8"},
47
+
{3, "R3_w", "16"},
48
+
{4, "R3_w", "32"},
50
49
},
51
50
},
52
51
{
···
68
67
},
69
68
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
70
69
.matches = {
71
-
{0, "R1=ctx(off=0,imm=0)"},
72
-
{0, "R10=fp0"},
73
-
{0, "R3_w=1"},
74
-
{1, "R3_w=2"},
75
-
{2, "R3_w=4"},
76
-
{3, "R3_w=8"},
77
-
{4, "R3_w=16"},
78
-
{5, "R3_w=1"},
79
-
{6, "R4_w=32"},
80
-
{7, "R4_w=16"},
81
-
{8, "R4_w=8"},
82
-
{9, "R4_w=4"},
83
-
{10, "R4_w=2"},
70
+
{0, "R1", "ctx(off=0,imm=0)"},
71
+
{0, "R10", "fp0"},
72
+
{0, "R3_w", "1"},
73
+
{1, "R3_w", "2"},
74
+
{2, "R3_w", "4"},
75
+
{3, "R3_w", "8"},
76
+
{4, "R3_w", "16"},
77
+
{5, "R3_w", "1"},
78
+
{6, "R4_w", "32"},
79
+
{7, "R4_w", "16"},
80
+
{8, "R4_w", "8"},
81
+
{9, "R4_w", "4"},
82
+
{10, "R4_w", "2"},
84
83
},
85
84
},
86
85
{
···
97
96
},
98
97
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
99
98
.matches = {
100
-
{0, "R1=ctx(off=0,imm=0)"},
101
-
{0, "R10=fp0"},
102
-
{0, "R3_w=4"},
103
-
{1, "R3_w=8"},
104
-
{2, "R3_w=10"},
105
-
{3, "R4_w=8"},
106
-
{4, "R4_w=12"},
107
-
{5, "R4_w=14"},
99
+
{0, "R1", "ctx(off=0,imm=0)"},
100
+
{0, "R10", "fp0"},
101
+
{0, "R3_w", "4"},
102
+
{1, "R3_w", "8"},
103
+
{2, "R3_w", "10"},
104
+
{3, "R4_w", "8"},
105
+
{4, "R4_w", "12"},
106
+
{5, "R4_w", "14"},
108
107
},
109
108
},
110
109
{
···
119
118
},
120
119
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
121
120
.matches = {
122
-
{0, "R1=ctx(off=0,imm=0)"},
123
-
{0, "R10=fp0"},
124
-
{0, "R3_w=7"},
125
-
{1, "R3_w=7"},
126
-
{2, "R3_w=14"},
127
-
{3, "R3_w=56"},
121
+
{0, "R1", "ctx(off=0,imm=0)"},
122
+
{0, "R10", "fp0"},
123
+
{0, "R3_w", "7"},
124
+
{1, "R3_w", "7"},
125
+
{2, "R3_w", "14"},
126
+
{3, "R3_w", "56"},
128
127
},
129
128
},
130
129
···
162
161
},
163
162
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
164
163
.matches = {
165
-
{6, "R0_w=pkt(off=8,r=8,imm=0)"},
166
-
{6, "R3_w=scalar(umax=255,var_off=(0x0; 0xff))"},
167
-
{7, "R3_w=scalar(umax=510,var_off=(0x0; 0x1fe))"},
168
-
{8, "R3_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
169
-
{9, "R3_w=scalar(umax=2040,var_off=(0x0; 0x7f8))"},
170
-
{10, "R3_w=scalar(umax=4080,var_off=(0x0; 0xff0))"},
171
-
{12, "R3_w=pkt_end(off=0,imm=0)"},
172
-
{17, "R4_w=scalar(umax=255,var_off=(0x0; 0xff))"},
173
-
{18, "R4_w=scalar(umax=8160,var_off=(0x0; 0x1fe0))"},
174
-
{19, "R4_w=scalar(umax=4080,var_off=(0x0; 0xff0))"},
175
-
{20, "R4_w=scalar(umax=2040,var_off=(0x0; 0x7f8))"},
176
-
{21, "R4_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
177
-
{22, "R4_w=scalar(umax=510,var_off=(0x0; 0x1fe))"},
164
+
{6, "R0_w", "pkt(off=8,r=8,imm=0)"},
165
+
{6, "R3_w", "var_off=(0x0; 0xff)"},
166
+
{7, "R3_w", "var_off=(0x0; 0x1fe)"},
167
+
{8, "R3_w", "var_off=(0x0; 0x3fc)"},
168
+
{9, "R3_w", "var_off=(0x0; 0x7f8)"},
169
+
{10, "R3_w", "var_off=(0x0; 0xff0)"},
170
+
{12, "R3_w", "pkt_end(off=0,imm=0)"},
171
+
{17, "R4_w", "var_off=(0x0; 0xff)"},
172
+
{18, "R4_w", "var_off=(0x0; 0x1fe0)"},
173
+
{19, "R4_w", "var_off=(0x0; 0xff0)"},
174
+
{20, "R4_w", "var_off=(0x0; 0x7f8)"},
175
+
{21, "R4_w", "var_off=(0x0; 0x3fc)"},
176
+
{22, "R4_w", "var_off=(0x0; 0x1fe)"},
178
177
},
179
178
},
180
179
{
···
195
194
},
196
195
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
197
196
.matches = {
198
-
{6, "R3_w=scalar(umax=255,var_off=(0x0; 0xff))"},
199
-
{7, "R4_w=scalar(id=1,umax=255,var_off=(0x0; 0xff))"},
200
-
{8, "R4_w=scalar(umax=255,var_off=(0x0; 0xff))"},
201
-
{9, "R4_w=scalar(id=1,umax=255,var_off=(0x0; 0xff))"},
202
-
{10, "R4_w=scalar(umax=510,var_off=(0x0; 0x1fe))"},
203
-
{11, "R4_w=scalar(id=1,umax=255,var_off=(0x0; 0xff))"},
204
-
{12, "R4_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
205
-
{13, "R4_w=scalar(id=1,umax=255,var_off=(0x0; 0xff))"},
206
-
{14, "R4_w=scalar(umax=2040,var_off=(0x0; 0x7f8))"},
207
-
{15, "R4_w=scalar(umax=4080,var_off=(0x0; 0xff0))"},
197
+
{6, "R3_w", "var_off=(0x0; 0xff)"},
198
+
{7, "R4_w", "var_off=(0x0; 0xff)"},
199
+
{8, "R4_w", "var_off=(0x0; 0xff)"},
200
+
{9, "R4_w", "var_off=(0x0; 0xff)"},
201
+
{10, "R4_w", "var_off=(0x0; 0x1fe)"},
202
+
{11, "R4_w", "var_off=(0x0; 0xff)"},
203
+
{12, "R4_w", "var_off=(0x0; 0x3fc)"},
204
+
{13, "R4_w", "var_off=(0x0; 0xff)"},
205
+
{14, "R4_w", "var_off=(0x0; 0x7f8)"},
206
+
{15, "R4_w", "var_off=(0x0; 0xff0)"},
208
207
},
209
208
},
210
209
{
···
235
234
},
236
235
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
237
236
.matches = {
238
-
{2, "R5_w=pkt(off=0,r=0,imm=0)"},
239
-
{4, "R5_w=pkt(off=14,r=0,imm=0)"},
240
-
{5, "R4_w=pkt(off=14,r=0,imm=0)"},
241
-
{9, "R2=pkt(off=0,r=18,imm=0)"},
242
-
{10, "R5=pkt(off=14,r=18,imm=0)"},
243
-
{10, "R4_w=scalar(umax=255,var_off=(0x0; 0xff))"},
244
-
{13, "R4_w=scalar(umax=65535,var_off=(0x0; 0xffff))"},
245
-
{14, "R4_w=scalar(umax=65535,var_off=(0x0; 0xffff))"},
237
+
{2, "R5_w", "pkt(off=0,r=0,imm=0)"},
238
+
{4, "R5_w", "pkt(off=14,r=0,imm=0)"},
239
+
{5, "R4_w", "pkt(off=14,r=0,imm=0)"},
240
+
{9, "R2", "pkt(off=0,r=18,imm=0)"},
241
+
{10, "R5", "pkt(off=14,r=18,imm=0)"},
242
+
{10, "R4_w", "var_off=(0x0; 0xff)"},
243
+
{13, "R4_w", "var_off=(0x0; 0xffff)"},
244
+
{14, "R4_w", "var_off=(0x0; 0xffff)"},
246
245
},
247
246
},
248
247
{
···
299
298
/* Calculated offset in R6 has unknown value, but known
300
299
* alignment of 4.
301
300
*/
302
-
{6, "R2_w=pkt(off=0,r=8,imm=0)"},
303
-
{7, "R6_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
301
+
{6, "R2_w", "pkt(off=0,r=8,imm=0)"},
302
+
{7, "R6_w", "var_off=(0x0; 0x3fc)"},
304
303
/* Offset is added to packet pointer R5, resulting in
305
304
* known fixed offset, and variable offset from R6.
306
305
*/
307
-
{11, "R5_w=pkt(id=1,off=14,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
306
+
{11, "R5_w", "pkt(id=1,off=14,"},
308
307
/* At the time the word size load is performed from R5,
309
308
* it's total offset is NET_IP_ALIGN + reg->off (0) +
310
309
* reg->aux_off (14) which is 16. Then the variable
311
310
* offset is considered using reg->aux_off_align which
312
311
* is 4 and meets the load's requirements.
313
312
*/
314
-
{15, "R4=pkt(id=1,off=18,r=18,umax=1020,var_off=(0x0; 0x3fc))"},
315
-
{15, "R5=pkt(id=1,off=14,r=18,umax=1020,var_off=(0x0; 0x3fc))"},
313
+
{15, "R4", "var_off=(0x0; 0x3fc)"},
314
+
{15, "R5", "var_off=(0x0; 0x3fc)"},
316
315
/* Variable offset is added to R5 packet pointer,
317
316
* resulting in auxiliary alignment of 4. To avoid BPF
318
317
* verifier's precision backtracking logging
···
320
319
* instruction to validate R5 state. We also check
321
320
* that R4 is what it should be in such case.
322
321
*/
323
-
{18, "R4_w=pkt(id=2,off=0,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
324
-
{18, "R5_w=pkt(id=2,off=0,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
322
+
{18, "R4_w", "var_off=(0x0; 0x3fc)"},
323
+
{18, "R5_w", "var_off=(0x0; 0x3fc)"},
325
324
/* Constant offset is added to R5, resulting in
326
325
* reg->off of 14.
327
326
*/
328
-
{19, "R5_w=pkt(id=2,off=14,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
327
+
{19, "R5_w", "pkt(id=2,off=14,"},
329
328
/* At the time the word size load is performed from R5,
330
329
* its total fixed offset is NET_IP_ALIGN + reg->off
331
330
* (14) which is 16. Then the variable offset is 4-byte
332
331
* aligned, so the total offset is 4-byte aligned and
333
332
* meets the load's requirements.
334
333
*/
335
-
{24, "R4=pkt(id=2,off=18,r=18,umax=1020,var_off=(0x0; 0x3fc))"},
336
-
{24, "R5=pkt(id=2,off=14,r=18,umax=1020,var_off=(0x0; 0x3fc))"},
334
+
{24, "R4", "var_off=(0x0; 0x3fc)"},
335
+
{24, "R5", "var_off=(0x0; 0x3fc)"},
337
336
/* Constant offset is added to R5 packet pointer,
338
337
* resulting in reg->off value of 14.
339
338
*/
340
-
{26, "R5_w=pkt(off=14,r=8"},
339
+
{26, "R5_w", "pkt(off=14,r=8,"},
341
340
/* Variable offset is added to R5, resulting in a
342
341
* variable offset of (4n). See comment for insn #18
343
342
* for R4 = R5 trick.
344
343
*/
345
-
{28, "R4_w=pkt(id=3,off=14,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
346
-
{28, "R5_w=pkt(id=3,off=14,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
344
+
{28, "R4_w", "var_off=(0x0; 0x3fc)"},
345
+
{28, "R5_w", "var_off=(0x0; 0x3fc)"},
347
346
/* Constant is added to R5 again, setting reg->off to 18. */
348
-
{29, "R5_w=pkt(id=3,off=18,r=0,umax=1020,var_off=(0x0; 0x3fc))"},
347
+
{29, "R5_w", "pkt(id=3,off=18,"},
349
348
/* And once more we add a variable; resulting var_off
350
349
* is still (4n), fixed offset is not changed.
351
350
* Also, we create a new reg->id.
352
351
*/
353
-
{31, "R4_w=pkt(id=4,off=18,r=0,umax=2040,var_off=(0x0; 0x7fc)"},
354
-
{31, "R5_w=pkt(id=4,off=18,r=0,umax=2040,var_off=(0x0; 0x7fc)"},
352
+
{31, "R4_w", "var_off=(0x0; 0x7fc)"},
353
+
{31, "R5_w", "var_off=(0x0; 0x7fc)"},
355
354
/* At the time the word size load is performed from R5,
356
355
* its total fixed offset is NET_IP_ALIGN + reg->off (18)
357
356
* which is 20. Then the variable offset is (4n), so
358
357
* the total offset is 4-byte aligned and meets the
359
358
* load's requirements.
360
359
*/
361
-
{35, "R4=pkt(id=4,off=22,r=22,umax=2040,var_off=(0x0; 0x7fc)"},
362
-
{35, "R5=pkt(id=4,off=18,r=22,umax=2040,var_off=(0x0; 0x7fc)"},
360
+
{35, "R4", "var_off=(0x0; 0x7fc)"},
361
+
{35, "R5", "var_off=(0x0; 0x7fc)"},
363
362
},
364
363
},
365
364
{
···
397
396
/* Calculated offset in R6 has unknown value, but known
398
397
* alignment of 4.
399
398
*/
400
-
{6, "R2_w=pkt(off=0,r=8,imm=0)"},
401
-
{7, "R6_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
399
+
{6, "R2_w", "pkt(off=0,r=8,imm=0)"},
400
+
{7, "R6_w", "var_off=(0x0; 0x3fc)"},
402
401
/* Adding 14 makes R6 be (4n+2) */
403
-
{8, "R6_w=scalar(umin=14,umax=1034,var_off=(0x2; 0x7fc))"},
402
+
{8, "R6_w", "var_off=(0x2; 0x7fc)"},
404
403
/* Packet pointer has (4n+2) offset */
405
-
{11, "R5_w=pkt(id=1,off=0,r=0,umin=14,umax=1034,var_off=(0x2; 0x7fc)"},
406
-
{12, "R4=pkt(id=1,off=4,r=0,umin=14,umax=1034,var_off=(0x2; 0x7fc)"},
404
+
{11, "R5_w", "var_off=(0x2; 0x7fc)"},
405
+
{12, "R4", "var_off=(0x2; 0x7fc)"},
407
406
/* At the time the word size load is performed from R5,
408
407
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
409
408
* which is 2. Then the variable offset is (4n+2), so
410
409
* the total offset is 4-byte aligned and meets the
411
410
* load's requirements.
412
411
*/
413
-
{15, "R5=pkt(id=1,off=0,r=4,umin=14,umax=1034,var_off=(0x2; 0x7fc)"},
412
+
{15, "R5", "var_off=(0x2; 0x7fc)"},
414
413
/* Newly read value in R6 was shifted left by 2, so has
415
414
* known alignment of 4.
416
415
*/
417
-
{17, "R6_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
416
+
{17, "R6_w", "var_off=(0x0; 0x3fc)"},
418
417
/* Added (4n) to packet pointer's (4n+2) var_off, giving
419
418
* another (4n+2).
420
419
*/
421
-
{19, "R5_w=pkt(id=2,off=0,r=0,umin=14,umax=2054,var_off=(0x2; 0xffc)"},
422
-
{20, "R4=pkt(id=2,off=4,r=0,umin=14,umax=2054,var_off=(0x2; 0xffc)"},
420
+
{19, "R5_w", "var_off=(0x2; 0xffc)"},
421
+
{20, "R4", "var_off=(0x2; 0xffc)"},
423
422
/* At the time the word size load is performed from R5,
424
423
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
425
424
* which is 2. Then the variable offset is (4n+2), so
426
425
* the total offset is 4-byte aligned and meets the
427
426
* load's requirements.
428
427
*/
429
-
{23, "R5=pkt(id=2,off=0,r=4,umin=14,umax=2054,var_off=(0x2; 0xffc)"},
428
+
{23, "R5", "var_off=(0x2; 0xffc)"},
430
429
},
431
430
},
432
431
{
···
459
458
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
460
459
.result = REJECT,
461
460
.matches = {
462
-
{3, "R5_w=pkt_end(off=0,imm=0)"},
461
+
{3, "R5_w", "pkt_end(off=0,imm=0)"},
463
462
/* (ptr - ptr) << 2 == unknown, (4n) */
464
-
{5, "R5_w=scalar(smax=9223372036854775804,umax=18446744073709551612,var_off=(0x0; 0xfffffffffffffffc)"},
463
+
{5, "R5_w", "var_off=(0x0; 0xfffffffffffffffc)"},
465
464
/* (4n) + 14 == (4n+2). We blow our bounds, because
466
465
* the add could overflow.
467
466
*/
468
-
{6, "R5_w=scalar(smin=-9223372036854775806,smax=9223372036854775806,umin=2,umax=18446744073709551614,var_off=(0x2; 0xfffffffffffffffc)"},
467
+
{6, "R5_w", "var_off=(0x2; 0xfffffffffffffffc)"},
469
468
/* Checked s>=0 */
470
-
{9, "R5=scalar(umin=2,umax=9223372036854775806,var_off=(0x2; 0x7ffffffffffffffc)"},
469
+
{9, "R5", "var_off=(0x2; 0x7ffffffffffffffc)"},
471
470
/* packet pointer + nonnegative (4n+2) */
472
-
{11, "R6_w=pkt(id=1,off=0,r=0,umin=2,umax=9223372036854775806,var_off=(0x2; 0x7ffffffffffffffc)"},
473
-
{12, "R4_w=pkt(id=1,off=4,r=0,umin=2,umax=9223372036854775806,var_off=(0x2; 0x7ffffffffffffffc)"},
471
+
{11, "R6_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
472
+
{12, "R4_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
474
473
/* NET_IP_ALIGN + (4n+2) == (4n), alignment is fine.
475
474
* We checked the bounds, but it might have been able
476
475
* to overflow if the packet pointer started in the
···
478
477
* So we did not get a 'range' on R6, and the access
479
478
* attempt will fail.
480
479
*/
481
-
{15, "R6_w=pkt(id=1,off=0,r=0,umin=2,umax=9223372036854775806,var_off=(0x2; 0x7ffffffffffffffc)"},
480
+
{15, "R6_w", "var_off=(0x2; 0x7ffffffffffffffc)"},
482
481
}
483
482
},
484
483
{
···
513
512
/* Calculated offset in R6 has unknown value, but known
514
513
* alignment of 4.
515
514
*/
516
-
{6, "R2_w=pkt(off=0,r=8,imm=0)"},
517
-
{8, "R6_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
515
+
{6, "R2_w", "pkt(off=0,r=8,imm=0)"},
516
+
{8, "R6_w", "var_off=(0x0; 0x3fc)"},
518
517
/* Adding 14 makes R6 be (4n+2) */
519
-
{9, "R6_w=scalar(umin=14,umax=1034,var_off=(0x2; 0x7fc))"},
518
+
{9, "R6_w", "var_off=(0x2; 0x7fc)"},
520
519
/* New unknown value in R7 is (4n) */
521
-
{10, "R7_w=scalar(umax=1020,var_off=(0x0; 0x3fc))"},
520
+
{10, "R7_w", "var_off=(0x0; 0x3fc)"},
522
521
/* Subtracting it from R6 blows our unsigned bounds */
523
-
{11, "R6=scalar(smin=-1006,smax=1034,umin=2,umax=18446744073709551614,var_off=(0x2; 0xfffffffffffffffc)"},
522
+
{11, "R6", "var_off=(0x2; 0xfffffffffffffffc)"},
524
523
/* Checked s>= 0 */
525
-
{14, "R6=scalar(umin=2,umax=1034,var_off=(0x2; 0x7fc))"},
524
+
{14, "R6", "var_off=(0x2; 0x7fc)"},
526
525
/* At the time the word size load is performed from R5,
527
526
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
528
527
* which is 2. Then the variable offset is (4n+2), so
529
528
* the total offset is 4-byte aligned and meets the
530
529
* load's requirements.
531
530
*/
532
-
{20, "R5=pkt(id=2,off=0,r=4,umin=2,umax=1034,var_off=(0x2; 0x7fc)"},
533
-
531
+
{20, "R5", "var_off=(0x2; 0x7fc)"},
534
532
},
535
533
},
536
534
{
···
566
566
/* Calculated offset in R6 has unknown value, but known
567
567
* alignment of 4.
568
568
*/
569
-
{6, "R2_w=pkt(off=0,r=8,imm=0)"},
570
-
{9, "R6_w=scalar(umax=60,var_off=(0x0; 0x3c))"},
569
+
{6, "R2_w", "pkt(off=0,r=8,imm=0)"},
570
+
{9, "R6_w", "var_off=(0x0; 0x3c)"},
571
571
/* Adding 14 makes R6 be (4n+2) */
572
-
{10, "R6_w=scalar(umin=14,umax=74,var_off=(0x2; 0x7c))"},
572
+
{10, "R6_w", "var_off=(0x2; 0x7c)"},
573
573
/* Subtracting from packet pointer overflows ubounds */
574
-
{13, "R5_w=pkt(id=2,off=0,r=8,umin=18446744073709551542,umax=18446744073709551602,var_off=(0xffffffffffffff82; 0x7c)"},
574
+
{13, "R5_w", "var_off=(0xffffffffffffff82; 0x7c)"},
575
575
/* New unknown value in R7 is (4n), >= 76 */
576
-
{14, "R7_w=scalar(umin=76,umax=1096,var_off=(0x0; 0x7fc))"},
576
+
{14, "R7_w", "var_off=(0x0; 0x7fc)"},
577
577
/* Adding it to packet pointer gives nice bounds again */
578
-
{16, "R5_w=pkt(id=3,off=0,r=0,umin=2,umax=1082,var_off=(0x2; 0x7fc)"},
578
+
{16, "R5_w", "var_off=(0x2; 0x7fc)"},
579
579
/* At the time the word size load is performed from R5,
580
580
* its total fixed offset is NET_IP_ALIGN + reg->off (0)
581
581
* which is 2. Then the variable offset is (4n+2), so
582
582
* the total offset is 4-byte aligned and meets the
583
583
* load's requirements.
584
584
*/
585
-
{20, "R5=pkt(id=3,off=0,r=4,umin=2,umax=1082,var_off=(0x2; 0x7fc)"},
585
+
{20, "R5", "var_off=(0x2; 0x7fc)"},
586
586
},
587
587
},
588
588
};
···
635
635
line_ptr = strtok(bpf_vlog_copy, "\n");
636
636
for (i = 0; i < MAX_MATCHES; i++) {
637
637
struct bpf_reg_match m = test->matches[i];
638
+
const char *p;
638
639
int tmp;
639
640
640
641
if (!m.match)
···
650
649
line_ptr = strtok(NULL, "\n");
651
650
}
652
651
if (!line_ptr) {
653
-
printf("Failed to find line %u for match: %s\n",
654
-
m.line, m.match);
652
+
printf("Failed to find line %u for match: %s=%s\n",
653
+
m.line, m.reg, m.match);
655
654
ret = 1;
656
655
printf("%s", bpf_vlog);
657
656
break;
···
668
667
* 6: R0_w=pkt(off=8,r=8,imm=0) R1=ctx(off=0,imm=0) R2_w=pkt(off=0,r=8,imm=0) R3_w=pkt_end(off=0,imm=0) R10=fp0
669
668
* 6: (71) r3 = *(u8 *)(r2 +0) ; R2_w=pkt(off=0,r=8,imm=0) R3_w=scalar(umax=255,var_off=(0x0; 0xff))
670
669
*/
671
-
while (!strstr(line_ptr, m.match)) {
670
+
while (!(p = strstr(line_ptr, m.reg)) || !strstr(p, m.match)) {
672
671
cur_line = -1;
673
672
line_ptr = strtok(NULL, "\n");
674
673
sscanf(line_ptr ?: "", "%u: ", &cur_line);
675
674
if (!line_ptr || cur_line != m.line)
676
675
break;
677
676
}
678
-
if (cur_line != m.line || !line_ptr || !strstr(line_ptr, m.match)) {
679
-
printf("Failed to find match %u: %s\n", m.line, m.match);
677
+
if (cur_line != m.line || !line_ptr || !(p = strstr(line_ptr, m.reg)) || !strstr(p, m.match)) {
678
+
printf("Failed to find match %u: %s=%s\n", m.line, m.reg, m.match);
680
679
ret = 1;
681
680
printf("%s", bpf_vlog);
682
681
break;
+2
-2
tools/testing/selftests/bpf/prog_tests/bloom_filter_map.c
+2
-2
tools/testing/selftests/bpf/prog_tests/bloom_filter_map.c
+13
-13
tools/testing/selftests/bpf/prog_tests/bpf_iter.c
+13
-13
tools/testing/selftests/bpf/prog_tests/bpf_iter.c
···
10
10
#include "bpf_iter_task.skel.h"
11
11
#include "bpf_iter_task_stack.skel.h"
12
12
#include "bpf_iter_task_file.skel.h"
13
-
#include "bpf_iter_task_vma.skel.h"
13
+
#include "bpf_iter_task_vmas.skel.h"
14
14
#include "bpf_iter_task_btf.skel.h"
15
15
#include "bpf_iter_tcp4.skel.h"
16
16
#include "bpf_iter_tcp6.skel.h"
···
1399
1399
static void test_task_vma_common(struct bpf_iter_attach_opts *opts)
1400
1400
{
1401
1401
int err, iter_fd = -1, proc_maps_fd = -1;
1402
-
struct bpf_iter_task_vma *skel;
1402
+
struct bpf_iter_task_vmas *skel;
1403
1403
int len, read_size = 4;
1404
1404
char maps_path[64];
1405
1405
1406
-
skel = bpf_iter_task_vma__open();
1407
-
if (!ASSERT_OK_PTR(skel, "bpf_iter_task_vma__open"))
1406
+
skel = bpf_iter_task_vmas__open();
1407
+
if (!ASSERT_OK_PTR(skel, "bpf_iter_task_vmas__open"))
1408
1408
return;
1409
1409
1410
1410
skel->bss->pid = getpid();
1411
1411
skel->bss->one_task = opts ? 1 : 0;
1412
1412
1413
-
err = bpf_iter_task_vma__load(skel);
1414
-
if (!ASSERT_OK(err, "bpf_iter_task_vma__load"))
1413
+
err = bpf_iter_task_vmas__load(skel);
1414
+
if (!ASSERT_OK(err, "bpf_iter_task_vmas__load"))
1415
1415
goto out;
1416
1416
1417
1417
skel->links.proc_maps = bpf_program__attach_iter(
···
1462
1462
out:
1463
1463
close(proc_maps_fd);
1464
1464
close(iter_fd);
1465
-
bpf_iter_task_vma__destroy(skel);
1465
+
bpf_iter_task_vmas__destroy(skel);
1466
1466
}
1467
1467
1468
1468
static void test_task_vma_dead_task(void)
1469
1469
{
1470
-
struct bpf_iter_task_vma *skel;
1470
+
struct bpf_iter_task_vmas *skel;
1471
1471
int wstatus, child_pid = -1;
1472
1472
time_t start_tm, cur_tm;
1473
1473
int err, iter_fd = -1;
1474
1474
int wait_sec = 3;
1475
1475
1476
-
skel = bpf_iter_task_vma__open();
1477
-
if (!ASSERT_OK_PTR(skel, "bpf_iter_task_vma__open"))
1476
+
skel = bpf_iter_task_vmas__open();
1477
+
if (!ASSERT_OK_PTR(skel, "bpf_iter_task_vmas__open"))
1478
1478
return;
1479
1479
1480
1480
skel->bss->pid = getpid();
1481
1481
1482
-
err = bpf_iter_task_vma__load(skel);
1483
-
if (!ASSERT_OK(err, "bpf_iter_task_vma__load"))
1482
+
err = bpf_iter_task_vmas__load(skel);
1483
+
if (!ASSERT_OK(err, "bpf_iter_task_vmas__load"))
1484
1484
goto out;
1485
1485
1486
1486
skel->links.proc_maps = bpf_program__attach_iter(
···
1533
1533
out:
1534
1534
waitpid(child_pid, &wstatus, 0);
1535
1535
close(iter_fd);
1536
-
bpf_iter_task_vma__destroy(skel);
1536
+
bpf_iter_task_vmas__destroy(skel);
1537
1537
}
1538
1538
1539
1539
void test_bpf_sockmap_map_iter_fd(void)
+2
-2
tools/testing/selftests/bpf/prog_tests/connect_ping.c
+2
-2
tools/testing/selftests/bpf/prog_tests/connect_ping.c
+77
-6
tools/testing/selftests/bpf/prog_tests/fib_lookup.c
+77
-6
tools/testing/selftests/bpf/prog_tests/fib_lookup.c
···
11
11
12
12
#define NS_TEST "fib_lookup_ns"
13
13
#define IPV6_IFACE_ADDR "face::face"
14
+
#define IPV6_IFACE_ADDR_SEC "cafe::cafe"
15
+
#define IPV6_ADDR_DST "face::3"
14
16
#define IPV6_NUD_FAILED_ADDR "face::1"
15
17
#define IPV6_NUD_STALE_ADDR "face::2"
16
18
#define IPV4_IFACE_ADDR "10.0.0.254"
19
+
#define IPV4_IFACE_ADDR_SEC "10.1.0.254"
20
+
#define IPV4_ADDR_DST "10.2.0.254"
17
21
#define IPV4_NUD_FAILED_ADDR "10.0.0.1"
18
22
#define IPV4_NUD_STALE_ADDR "10.0.0.2"
19
23
#define IPV4_TBID_ADDR "172.0.0.254"
···
35
31
const char *desc;
36
32
const char *daddr;
37
33
int expected_ret;
34
+
const char *expected_src;
38
35
int lookup_flags;
39
36
__u32 tbid;
40
37
__u8 dmac[6];
···
74
69
.daddr = IPV6_TBID_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS,
75
70
.lookup_flags = BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID, .tbid = 100,
76
71
.dmac = DMAC_INIT2, },
72
+
{ .desc = "IPv4 set src addr from netdev",
73
+
.daddr = IPV4_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS,
74
+
.expected_src = IPV4_IFACE_ADDR,
75
+
.lookup_flags = BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_SKIP_NEIGH, },
76
+
{ .desc = "IPv6 set src addr from netdev",
77
+
.daddr = IPV6_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS,
78
+
.expected_src = IPV6_IFACE_ADDR,
79
+
.lookup_flags = BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_SKIP_NEIGH, },
80
+
{ .desc = "IPv4 set prefsrc addr from route",
81
+
.daddr = IPV4_ADDR_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS,
82
+
.expected_src = IPV4_IFACE_ADDR_SEC,
83
+
.lookup_flags = BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_SKIP_NEIGH, },
84
+
{ .desc = "IPv6 set prefsrc addr route",
85
+
.daddr = IPV6_ADDR_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS,
86
+
.expected_src = IPV6_IFACE_ADDR_SEC,
87
+
.lookup_flags = BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_SKIP_NEIGH, },
77
88
};
78
89
79
90
static int ifindex;
···
117
96
SYS(fail, "ip addr add %s/24 dev veth1", IPV4_IFACE_ADDR);
118
97
SYS(fail, "ip neigh add %s dev veth1 nud failed", IPV4_NUD_FAILED_ADDR);
119
98
SYS(fail, "ip neigh add %s dev veth1 lladdr %s nud stale", IPV4_NUD_STALE_ADDR, DMAC);
99
+
100
+
/* Setup for prefsrc IP addr selection */
101
+
SYS(fail, "ip addr add %s/24 dev veth1", IPV4_IFACE_ADDR_SEC);
102
+
SYS(fail, "ip route add %s/32 dev veth1 src %s", IPV4_ADDR_DST, IPV4_IFACE_ADDR_SEC);
103
+
104
+
SYS(fail, "ip addr add %s/64 dev veth1 nodad", IPV6_IFACE_ADDR_SEC);
105
+
SYS(fail, "ip route add %s/128 dev veth1 src %s", IPV6_ADDR_DST, IPV6_IFACE_ADDR_SEC);
120
106
121
107
/* Setup for tbid lookup tests */
122
108
SYS(fail, "ip addr add %s/24 dev veth2", IPV4_TBID_ADDR);
···
161
133
162
134
if (inet_pton(AF_INET6, test->daddr, params->ipv6_dst) == 1) {
163
135
params->family = AF_INET6;
164
-
ret = inet_pton(AF_INET6, IPV6_IFACE_ADDR, params->ipv6_src);
165
-
if (!ASSERT_EQ(ret, 1, "inet_pton(IPV6_IFACE_ADDR)"))
166
-
return -1;
136
+
if (!(test->lookup_flags & BPF_FIB_LOOKUP_SRC)) {
137
+
ret = inet_pton(AF_INET6, IPV6_IFACE_ADDR, params->ipv6_src);
138
+
if (!ASSERT_EQ(ret, 1, "inet_pton(IPV6_IFACE_ADDR)"))
139
+
return -1;
140
+
}
141
+
167
142
return 0;
168
143
}
169
144
···
174
143
if (!ASSERT_EQ(ret, 1, "convert IP[46] address"))
175
144
return -1;
176
145
params->family = AF_INET;
177
-
ret = inet_pton(AF_INET, IPV4_IFACE_ADDR, ¶ms->ipv4_src);
178
-
if (!ASSERT_EQ(ret, 1, "inet_pton(IPV4_IFACE_ADDR)"))
179
-
return -1;
146
+
147
+
if (!(test->lookup_flags & BPF_FIB_LOOKUP_SRC)) {
148
+
ret = inet_pton(AF_INET, IPV4_IFACE_ADDR, ¶ms->ipv4_src);
149
+
if (!ASSERT_EQ(ret, 1, "inet_pton(IPV4_IFACE_ADDR)"))
150
+
return -1;
151
+
}
180
152
181
153
return 0;
182
154
}
···
188
154
{
189
155
sprintf(b, "%02X:%02X:%02X:%02X:%02X:%02X",
190
156
mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
157
+
}
158
+
159
+
static void assert_src_ip(struct bpf_fib_lookup *fib_params, const char *expected_src)
160
+
{
161
+
int ret;
162
+
__u32 src6[4];
163
+
__be32 src4;
164
+
165
+
switch (fib_params->family) {
166
+
case AF_INET6:
167
+
ret = inet_pton(AF_INET6, expected_src, src6);
168
+
ASSERT_EQ(ret, 1, "inet_pton(expected_src)");
169
+
170
+
ret = memcmp(src6, fib_params->ipv6_src, sizeof(fib_params->ipv6_src));
171
+
if (!ASSERT_EQ(ret, 0, "fib_lookup ipv6 src")) {
172
+
char str_src6[64];
173
+
174
+
inet_ntop(AF_INET6, fib_params->ipv6_src, str_src6,
175
+
sizeof(str_src6));
176
+
printf("ipv6 expected %s actual %s ", expected_src,
177
+
str_src6);
178
+
}
179
+
180
+
break;
181
+
case AF_INET:
182
+
ret = inet_pton(AF_INET, expected_src, &src4);
183
+
ASSERT_EQ(ret, 1, "inet_pton(expected_src)");
184
+
185
+
ASSERT_EQ(fib_params->ipv4_src, src4, "fib_lookup ipv4 src");
186
+
187
+
break;
188
+
default:
189
+
PRINT_FAIL("invalid addr family: %d", fib_params->family);
190
+
}
191
191
}
192
192
193
193
void test_fib_lookup(void)
···
274
206
275
207
ASSERT_EQ(skel->bss->fib_lookup_ret, tests[i].expected_ret,
276
208
"fib_lookup_ret");
209
+
210
+
if (tests[i].expected_src)
211
+
assert_src_ip(fib_params, tests[i].expected_src);
277
212
278
213
ret = memcmp(tests[i].dmac, fib_params->dmac, sizeof(tests[i].dmac));
279
214
if (!ASSERT_EQ(ret, 0, "dmac not match")) {
+58
tools/testing/selftests/bpf/prog_tests/iters.c
+58
tools/testing/selftests/bpf/prog_tests/iters.c
···
8
8
#include "iters_looping.skel.h"
9
9
#include "iters_num.skel.h"
10
10
#include "iters_testmod_seq.skel.h"
11
+
#include "iters_task_vma.skel.h"
11
12
12
13
static void subtest_num_iters(void)
13
14
{
···
91
90
iters_testmod_seq__destroy(skel);
92
91
}
93
92
93
+
static void subtest_task_vma_iters(void)
94
+
{
95
+
unsigned long start, end, bpf_iter_start, bpf_iter_end;
96
+
struct iters_task_vma *skel;
97
+
char rest_of_line[1000];
98
+
unsigned int seen;
99
+
FILE *f = NULL;
100
+
int err;
101
+
102
+
skel = iters_task_vma__open_and_load();
103
+
if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
104
+
return;
105
+
106
+
skel->bss->target_pid = getpid();
107
+
108
+
err = iters_task_vma__attach(skel);
109
+
if (!ASSERT_OK(err, "skel_attach"))
110
+
goto cleanup;
111
+
112
+
getpgid(skel->bss->target_pid);
113
+
iters_task_vma__detach(skel);
114
+
115
+
if (!ASSERT_GT(skel->bss->vmas_seen, 0, "vmas_seen_gt_zero"))
116
+
goto cleanup;
117
+
118
+
f = fopen("/proc/self/maps", "r");
119
+
if (!ASSERT_OK_PTR(f, "proc_maps_fopen"))
120
+
goto cleanup;
121
+
122
+
seen = 0;
123
+
while (fscanf(f, "%lx-%lx %[^\n]\n", &start, &end, rest_of_line) == 3) {
124
+
/* [vsyscall] vma isn't _really_ part of task->mm vmas.
125
+
* /proc/PID/maps returns it when out of vmas - see get_gate_vma
126
+
* calls in fs/proc/task_mmu.c
127
+
*/
128
+
if (strstr(rest_of_line, "[vsyscall]"))
129
+
continue;
130
+
131
+
bpf_iter_start = skel->bss->vm_ranges[seen].vm_start;
132
+
bpf_iter_end = skel->bss->vm_ranges[seen].vm_end;
133
+
134
+
ASSERT_EQ(bpf_iter_start, start, "vma->vm_start match");
135
+
ASSERT_EQ(bpf_iter_end, end, "vma->vm_end match");
136
+
seen++;
137
+
}
138
+
139
+
if (!ASSERT_EQ(skel->bss->vmas_seen, seen, "vmas_seen_eq"))
140
+
goto cleanup;
141
+
142
+
cleanup:
143
+
if (f)
144
+
fclose(f);
145
+
iters_task_vma__destroy(skel);
146
+
}
147
+
94
148
void test_iters(void)
95
149
{
96
150
RUN_TESTS(iters_state_safety);
···
159
103
subtest_num_iters();
160
104
if (test__start_subtest("testmod_seq"))
161
105
subtest_testmod_seq_iters();
106
+
if (test__start_subtest("task_vma"))
107
+
subtest_task_vma_iters();
162
108
}
+1
-1
tools/testing/selftests/bpf/prog_tests/linked_list.c
+1
-1
tools/testing/selftests/bpf/prog_tests/linked_list.c
+2
-1
tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
+2
-1
tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
···
49
49
return -1;
50
50
51
51
ifr.ifr_flags = IFF_NO_PI | (need_mac ? IFF_TAP : IFF_TUN);
52
-
memcpy(ifr.ifr_name, dev_name, IFNAMSIZ);
52
+
strncpy(ifr.ifr_name, dev_name, IFNAMSIZ - 1);
53
+
ifr.ifr_name[IFNAMSIZ - 1] = '\0';
53
54
54
55
err = ioctl(fd, TUNSETIFF, &ifr);
55
56
if (!ASSERT_OK(err, "ioctl(TUNSETIFF)")) {
+138
tools/testing/selftests/bpf/prog_tests/missed.c
+138
tools/testing/selftests/bpf/prog_tests/missed.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#include <test_progs.h>
3
+
#include "missed_kprobe.skel.h"
4
+
#include "missed_kprobe_recursion.skel.h"
5
+
#include "missed_tp_recursion.skel.h"
6
+
7
+
/*
8
+
* Putting kprobe on bpf_fentry_test1 that calls bpf_kfunc_common_test
9
+
* kfunc, which has also kprobe on. The latter won't get triggered due
10
+
* to kprobe recursion check and kprobe missed counter is incremented.
11
+
*/
12
+
static void test_missed_perf_kprobe(void)
13
+
{
14
+
LIBBPF_OPTS(bpf_test_run_opts, topts);
15
+
struct bpf_link_info info = {};
16
+
struct missed_kprobe *skel;
17
+
__u32 len = sizeof(info);
18
+
int err, prog_fd;
19
+
20
+
skel = missed_kprobe__open_and_load();
21
+
if (!ASSERT_OK_PTR(skel, "missed_kprobe__open_and_load"))
22
+
goto cleanup;
23
+
24
+
err = missed_kprobe__attach(skel);
25
+
if (!ASSERT_OK(err, "missed_kprobe__attach"))
26
+
goto cleanup;
27
+
28
+
prog_fd = bpf_program__fd(skel->progs.trigger);
29
+
err = bpf_prog_test_run_opts(prog_fd, &topts);
30
+
ASSERT_OK(err, "test_run");
31
+
ASSERT_EQ(topts.retval, 0, "test_run");
32
+
33
+
err = bpf_link_get_info_by_fd(bpf_link__fd(skel->links.test2), &info, &len);
34
+
if (!ASSERT_OK(err, "bpf_link_get_info_by_fd"))
35
+
goto cleanup;
36
+
37
+
ASSERT_EQ(info.type, BPF_LINK_TYPE_PERF_EVENT, "info.type");
38
+
ASSERT_EQ(info.perf_event.type, BPF_PERF_EVENT_KPROBE, "info.perf_event.type");
39
+
ASSERT_EQ(info.perf_event.kprobe.missed, 1, "info.perf_event.kprobe.missed");
40
+
41
+
cleanup:
42
+
missed_kprobe__destroy(skel);
43
+
}
44
+
45
+
static __u64 get_missed_count(int fd)
46
+
{
47
+
struct bpf_prog_info info = {};
48
+
__u32 len = sizeof(info);
49
+
int err;
50
+
51
+
err = bpf_prog_get_info_by_fd(fd, &info, &len);
52
+
if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd"))
53
+
return (__u64) -1;
54
+
return info.recursion_misses;
55
+
}
56
+
57
+
/*
58
+
* Putting kprobe.multi on bpf_fentry_test1 that calls bpf_kfunc_common_test
59
+
* kfunc which has 3 perf event kprobes and 1 kprobe.multi attached.
60
+
*
61
+
* Because fprobe (kprobe.multi attach layear) does not have strict recursion
62
+
* check the kprobe's bpf_prog_active check is hit for test2-5.
63
+
*/
64
+
static void test_missed_kprobe_recursion(void)
65
+
{
66
+
LIBBPF_OPTS(bpf_test_run_opts, topts);
67
+
struct missed_kprobe_recursion *skel;
68
+
int err, prog_fd;
69
+
70
+
skel = missed_kprobe_recursion__open_and_load();
71
+
if (!ASSERT_OK_PTR(skel, "missed_kprobe_recursion__open_and_load"))
72
+
goto cleanup;
73
+
74
+
err = missed_kprobe_recursion__attach(skel);
75
+
if (!ASSERT_OK(err, "missed_kprobe_recursion__attach"))
76
+
goto cleanup;
77
+
78
+
prog_fd = bpf_program__fd(skel->progs.trigger);
79
+
err = bpf_prog_test_run_opts(prog_fd, &topts);
80
+
ASSERT_OK(err, "test_run");
81
+
ASSERT_EQ(topts.retval, 0, "test_run");
82
+
83
+
ASSERT_EQ(get_missed_count(bpf_program__fd(skel->progs.test1)), 0, "test1_recursion_misses");
84
+
ASSERT_GE(get_missed_count(bpf_program__fd(skel->progs.test2)), 1, "test2_recursion_misses");
85
+
ASSERT_GE(get_missed_count(bpf_program__fd(skel->progs.test3)), 1, "test3_recursion_misses");
86
+
ASSERT_GE(get_missed_count(bpf_program__fd(skel->progs.test4)), 1, "test4_recursion_misses");
87
+
ASSERT_GE(get_missed_count(bpf_program__fd(skel->progs.test5)), 1, "test5_recursion_misses");
88
+
89
+
cleanup:
90
+
missed_kprobe_recursion__destroy(skel);
91
+
}
92
+
93
+
/*
94
+
* Putting kprobe on bpf_fentry_test1 that calls bpf_printk and invokes
95
+
* bpf_trace_printk tracepoint. The bpf_trace_printk tracepoint has test[234]
96
+
* programs attached to it.
97
+
*
98
+
* Because kprobe execution goes through bpf_prog_active check, programs
99
+
* attached to the tracepoint will fail the recursion check and increment
100
+
* the recursion_misses stats.
101
+
*/
102
+
static void test_missed_tp_recursion(void)
103
+
{
104
+
LIBBPF_OPTS(bpf_test_run_opts, topts);
105
+
struct missed_tp_recursion *skel;
106
+
int err, prog_fd;
107
+
108
+
skel = missed_tp_recursion__open_and_load();
109
+
if (!ASSERT_OK_PTR(skel, "missed_tp_recursion__open_and_load"))
110
+
goto cleanup;
111
+
112
+
err = missed_tp_recursion__attach(skel);
113
+
if (!ASSERT_OK(err, "missed_tp_recursion__attach"))
114
+
goto cleanup;
115
+
116
+
prog_fd = bpf_program__fd(skel->progs.trigger);
117
+
err = bpf_prog_test_run_opts(prog_fd, &topts);
118
+
ASSERT_OK(err, "test_run");
119
+
ASSERT_EQ(topts.retval, 0, "test_run");
120
+
121
+
ASSERT_EQ(get_missed_count(bpf_program__fd(skel->progs.test1)), 0, "test1_recursion_misses");
122
+
ASSERT_EQ(get_missed_count(bpf_program__fd(skel->progs.test2)), 1, "test2_recursion_misses");
123
+
ASSERT_EQ(get_missed_count(bpf_program__fd(skel->progs.test3)), 1, "test3_recursion_misses");
124
+
ASSERT_EQ(get_missed_count(bpf_program__fd(skel->progs.test4)), 1, "test4_recursion_misses");
125
+
126
+
cleanup:
127
+
missed_tp_recursion__destroy(skel);
128
+
}
129
+
130
+
void test_missed(void)
131
+
{
132
+
if (test__start_subtest("perf_kprobe"))
133
+
test_missed_perf_kprobe();
134
+
if (test__start_subtest("kprobe_recursion"))
135
+
test_missed_kprobe_recursion();
136
+
if (test__start_subtest("tp_recursion"))
137
+
test_missed_tp_recursion();
138
+
}
+3
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c
+3
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c
···
19
19
bpf_program__set_autoload(skel->progs.test_array_map_3, true);
20
20
bpf_program__set_autoload(skel->progs.test_array_map_4, true);
21
21
22
+
skel->bss->my_pid = getpid();
22
23
skel->rodata->nr_cpus = libbpf_num_possible_cpus();
23
24
24
25
err = percpu_alloc_array__load(skel);
···
52
51
53
52
bpf_program__set_autoload(skel->progs.test_array_map_10, true);
54
53
54
+
skel->bss->my_pid = getpid();
55
55
skel->rodata->nr_cpus = libbpf_num_possible_cpus();
56
56
57
57
err = percpu_alloc_array__load(skel);
···
87
85
if (!ASSERT_OK_PTR(skel, "percpu_alloc_cgrp_local_storage__open"))
88
86
goto close_fd;
89
87
88
+
skel->bss->my_pid = getpid();
90
89
skel->rodata->nr_cpus = libbpf_num_possible_cpus();
91
90
92
91
err = percpu_alloc_cgrp_local_storage__load(skel);
+1
-1
tools/testing/selftests/bpf/prog_tests/queue_stack_map.c
+1
-1
tools/testing/selftests/bpf/prog_tests/queue_stack_map.c
+26
tools/testing/selftests/bpf/prog_tests/ringbuf.c
+26
tools/testing/selftests/bpf/prog_tests/ringbuf.c
···
91
91
int err, cnt, rb_fd;
92
92
int page_size = getpagesize();
93
93
void *mmap_ptr, *tmp_ptr;
94
+
struct ring *ring;
95
+
int map_fd;
96
+
unsigned long avail_data, ring_size, cons_pos, prod_pos;
94
97
95
98
skel = test_ringbuf_lskel__open();
96
99
if (CHECK(!skel, "skel_open", "skeleton open failed\n"))
···
165
162
166
163
trigger_samples();
167
164
165
+
ring = ring_buffer__ring(ringbuf, 0);
166
+
if (!ASSERT_OK_PTR(ring, "ring_buffer__ring_idx_0"))
167
+
goto cleanup;
168
+
169
+
map_fd = ring__map_fd(ring);
170
+
ASSERT_EQ(map_fd, skel->maps.ringbuf.map_fd, "ring_map_fd");
171
+
168
172
/* 2 submitted + 1 discarded records */
169
173
CHECK(skel->bss->avail_data != 3 * rec_sz,
170
174
"err_avail_size", "exp %ld, got %ld\n",
···
185
175
CHECK(skel->bss->prod_pos != 3 * rec_sz,
186
176
"err_prod_pos", "exp %ld, got %ld\n",
187
177
3L * rec_sz, skel->bss->prod_pos);
178
+
179
+
/* verify getting this data directly via the ring object yields the same
180
+
* results
181
+
*/
182
+
avail_data = ring__avail_data_size(ring);
183
+
ASSERT_EQ(avail_data, 3 * rec_sz, "ring_avail_size");
184
+
ring_size = ring__size(ring);
185
+
ASSERT_EQ(ring_size, page_size, "ring_ring_size");
186
+
cons_pos = ring__consumer_pos(ring);
187
+
ASSERT_EQ(cons_pos, 0, "ring_cons_pos");
188
+
prod_pos = ring__producer_pos(ring);
189
+
ASSERT_EQ(prod_pos, 3 * rec_sz, "ring_prod_pos");
188
190
189
191
/* poll for samples */
190
192
err = ring_buffer__poll(ringbuf, -1);
···
303
281
*/
304
282
err = ring_buffer__consume(ringbuf);
305
283
CHECK(err < 0, "rb_consume", "failed: %d\b", err);
284
+
285
+
/* also consume using ring__consume to make sure it works the same */
286
+
err = ring__consume(ring);
287
+
ASSERT_GE(err, 0, "ring_consume");
306
288
307
289
/* 3 rounds, 2 samples each */
308
290
cnt = atomic_xchg(&sample_cnt, 0);
+15
tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c
+15
tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c
···
42
42
{
43
43
struct test_ringbuf_multi *skel;
44
44
struct ring_buffer *ringbuf = NULL;
45
+
struct ring *ring_old;
46
+
struct ring *ring;
45
47
int err;
46
48
int page_size = getpagesize();
47
49
int proto_fd = -1;
···
86
84
if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
87
85
goto cleanup;
88
86
87
+
/* verify ring_buffer__ring returns expected results */
88
+
ring = ring_buffer__ring(ringbuf, 0);
89
+
if (!ASSERT_OK_PTR(ring, "ring_buffer__ring_idx_0"))
90
+
goto cleanup;
91
+
ring_old = ring;
92
+
ring = ring_buffer__ring(ringbuf, 1);
93
+
ASSERT_ERR_PTR(ring, "ring_buffer__ring_idx_1");
94
+
89
95
err = ring_buffer__add(ringbuf, bpf_map__fd(skel->maps.ringbuf2),
90
96
process_sample, (void *)(long)2);
91
97
if (CHECK(err, "ringbuf_add", "failed to add another ring\n"))
98
+
goto cleanup;
99
+
100
+
/* verify adding a new ring didn't invalidate our older pointer */
101
+
ring = ring_buffer__ring(ringbuf, 0);
102
+
if (!ASSERT_EQ(ring, ring_old, "ring_buffer__ring_again"))
92
103
goto cleanup;
93
104
94
105
err = test_ringbuf_multi__attach(skel);
+45
tools/testing/selftests/bpf/prog_tests/section_names.c
+45
tools/testing/selftests/bpf/prog_tests/section_names.c
···
124
124
{0, BPF_CGROUP_INET6_CONNECT},
125
125
},
126
126
{
127
+
"cgroup/connect_unix",
128
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_CONNECT},
129
+
{0, BPF_CGROUP_UNIX_CONNECT},
130
+
},
131
+
{
127
132
"cgroup/sendmsg4",
128
133
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UDP4_SENDMSG},
129
134
{0, BPF_CGROUP_UDP4_SENDMSG},
···
139
134
{0, BPF_CGROUP_UDP6_SENDMSG},
140
135
},
141
136
{
137
+
"cgroup/sendmsg_unix",
138
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_SENDMSG},
139
+
{0, BPF_CGROUP_UNIX_SENDMSG},
140
+
},
141
+
{
142
142
"cgroup/recvmsg4",
143
143
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UDP4_RECVMSG},
144
144
{0, BPF_CGROUP_UDP4_RECVMSG},
···
152
142
"cgroup/recvmsg6",
153
143
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UDP6_RECVMSG},
154
144
{0, BPF_CGROUP_UDP6_RECVMSG},
145
+
},
146
+
{
147
+
"cgroup/recvmsg_unix",
148
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_RECVMSG},
149
+
{0, BPF_CGROUP_UNIX_RECVMSG},
155
150
},
156
151
{
157
152
"cgroup/sysctl",
···
172
157
"cgroup/setsockopt",
173
158
{0, BPF_PROG_TYPE_CGROUP_SOCKOPT, BPF_CGROUP_SETSOCKOPT},
174
159
{0, BPF_CGROUP_SETSOCKOPT},
160
+
},
161
+
{
162
+
"cgroup/getpeername4",
163
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_INET4_GETPEERNAME},
164
+
{0, BPF_CGROUP_INET4_GETPEERNAME},
165
+
},
166
+
{
167
+
"cgroup/getpeername6",
168
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_GETPEERNAME},
169
+
{0, BPF_CGROUP_INET6_GETPEERNAME},
170
+
},
171
+
{
172
+
"cgroup/getpeername_unix",
173
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_GETPEERNAME},
174
+
{0, BPF_CGROUP_UNIX_GETPEERNAME},
175
+
},
176
+
{
177
+
"cgroup/getsockname4",
178
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_INET4_GETSOCKNAME},
179
+
{0, BPF_CGROUP_INET4_GETSOCKNAME},
180
+
},
181
+
{
182
+
"cgroup/getsockname6",
183
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_GETSOCKNAME},
184
+
{0, BPF_CGROUP_INET6_GETSOCKNAME},
185
+
},
186
+
{
187
+
"cgroup/getsockname_unix",
188
+
{0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UNIX_GETSOCKNAME},
189
+
{0, BPF_CGROUP_UNIX_GETSOCKNAME},
175
190
},
176
191
};
177
192
+612
tools/testing/selftests/bpf/prog_tests/sock_addr.c
+612
tools/testing/selftests/bpf/prog_tests/sock_addr.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#include <sys/un.h>
3
+
4
+
#include "test_progs.h"
5
+
6
+
#include "connect_unix_prog.skel.h"
7
+
#include "sendmsg_unix_prog.skel.h"
8
+
#include "recvmsg_unix_prog.skel.h"
9
+
#include "getsockname_unix_prog.skel.h"
10
+
#include "getpeername_unix_prog.skel.h"
11
+
#include "network_helpers.h"
12
+
13
+
#define SERVUN_ADDRESS "bpf_cgroup_unix_test"
14
+
#define SERVUN_REWRITE_ADDRESS "bpf_cgroup_unix_test_rewrite"
15
+
#define SRCUN_ADDRESS "bpf_cgroup_unix_test_src"
16
+
17
+
enum sock_addr_test_type {
18
+
SOCK_ADDR_TEST_BIND,
19
+
SOCK_ADDR_TEST_CONNECT,
20
+
SOCK_ADDR_TEST_SENDMSG,
21
+
SOCK_ADDR_TEST_RECVMSG,
22
+
SOCK_ADDR_TEST_GETSOCKNAME,
23
+
SOCK_ADDR_TEST_GETPEERNAME,
24
+
};
25
+
26
+
typedef void *(*load_fn)(int cgroup_fd);
27
+
typedef void (*destroy_fn)(void *skel);
28
+
29
+
struct sock_addr_test {
30
+
enum sock_addr_test_type type;
31
+
const char *name;
32
+
/* BPF prog properties */
33
+
load_fn loadfn;
34
+
destroy_fn destroyfn;
35
+
/* Socket properties */
36
+
int socket_family;
37
+
int socket_type;
38
+
/* IP:port pairs for BPF prog to override */
39
+
const char *requested_addr;
40
+
unsigned short requested_port;
41
+
const char *expected_addr;
42
+
unsigned short expected_port;
43
+
const char *expected_src_addr;
44
+
};
45
+
46
+
static void *connect_unix_prog_load(int cgroup_fd)
47
+
{
48
+
struct connect_unix_prog *skel;
49
+
50
+
skel = connect_unix_prog__open_and_load();
51
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
52
+
goto cleanup;
53
+
54
+
skel->links.connect_unix_prog = bpf_program__attach_cgroup(
55
+
skel->progs.connect_unix_prog, cgroup_fd);
56
+
if (!ASSERT_OK_PTR(skel->links.connect_unix_prog, "prog_attach"))
57
+
goto cleanup;
58
+
59
+
return skel;
60
+
cleanup:
61
+
connect_unix_prog__destroy(skel);
62
+
return NULL;
63
+
}
64
+
65
+
static void connect_unix_prog_destroy(void *skel)
66
+
{
67
+
connect_unix_prog__destroy(skel);
68
+
}
69
+
70
+
static void *sendmsg_unix_prog_load(int cgroup_fd)
71
+
{
72
+
struct sendmsg_unix_prog *skel;
73
+
74
+
skel = sendmsg_unix_prog__open_and_load();
75
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
76
+
goto cleanup;
77
+
78
+
skel->links.sendmsg_unix_prog = bpf_program__attach_cgroup(
79
+
skel->progs.sendmsg_unix_prog, cgroup_fd);
80
+
if (!ASSERT_OK_PTR(skel->links.sendmsg_unix_prog, "prog_attach"))
81
+
goto cleanup;
82
+
83
+
return skel;
84
+
cleanup:
85
+
sendmsg_unix_prog__destroy(skel);
86
+
return NULL;
87
+
}
88
+
89
+
static void sendmsg_unix_prog_destroy(void *skel)
90
+
{
91
+
sendmsg_unix_prog__destroy(skel);
92
+
}
93
+
94
+
static void *recvmsg_unix_prog_load(int cgroup_fd)
95
+
{
96
+
struct recvmsg_unix_prog *skel;
97
+
98
+
skel = recvmsg_unix_prog__open_and_load();
99
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
100
+
goto cleanup;
101
+
102
+
skel->links.recvmsg_unix_prog = bpf_program__attach_cgroup(
103
+
skel->progs.recvmsg_unix_prog, cgroup_fd);
104
+
if (!ASSERT_OK_PTR(skel->links.recvmsg_unix_prog, "prog_attach"))
105
+
goto cleanup;
106
+
107
+
return skel;
108
+
cleanup:
109
+
recvmsg_unix_prog__destroy(skel);
110
+
return NULL;
111
+
}
112
+
113
+
static void recvmsg_unix_prog_destroy(void *skel)
114
+
{
115
+
recvmsg_unix_prog__destroy(skel);
116
+
}
117
+
118
+
static void *getsockname_unix_prog_load(int cgroup_fd)
119
+
{
120
+
struct getsockname_unix_prog *skel;
121
+
122
+
skel = getsockname_unix_prog__open_and_load();
123
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
124
+
goto cleanup;
125
+
126
+
skel->links.getsockname_unix_prog = bpf_program__attach_cgroup(
127
+
skel->progs.getsockname_unix_prog, cgroup_fd);
128
+
if (!ASSERT_OK_PTR(skel->links.getsockname_unix_prog, "prog_attach"))
129
+
goto cleanup;
130
+
131
+
return skel;
132
+
cleanup:
133
+
getsockname_unix_prog__destroy(skel);
134
+
return NULL;
135
+
}
136
+
137
+
static void getsockname_unix_prog_destroy(void *skel)
138
+
{
139
+
getsockname_unix_prog__destroy(skel);
140
+
}
141
+
142
+
static void *getpeername_unix_prog_load(int cgroup_fd)
143
+
{
144
+
struct getpeername_unix_prog *skel;
145
+
146
+
skel = getpeername_unix_prog__open_and_load();
147
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
148
+
goto cleanup;
149
+
150
+
skel->links.getpeername_unix_prog = bpf_program__attach_cgroup(
151
+
skel->progs.getpeername_unix_prog, cgroup_fd);
152
+
if (!ASSERT_OK_PTR(skel->links.getpeername_unix_prog, "prog_attach"))
153
+
goto cleanup;
154
+
155
+
return skel;
156
+
cleanup:
157
+
getpeername_unix_prog__destroy(skel);
158
+
return NULL;
159
+
}
160
+
161
+
static void getpeername_unix_prog_destroy(void *skel)
162
+
{
163
+
getpeername_unix_prog__destroy(skel);
164
+
}
165
+
166
+
static struct sock_addr_test tests[] = {
167
+
{
168
+
SOCK_ADDR_TEST_CONNECT,
169
+
"connect_unix",
170
+
connect_unix_prog_load,
171
+
connect_unix_prog_destroy,
172
+
AF_UNIX,
173
+
SOCK_STREAM,
174
+
SERVUN_ADDRESS,
175
+
0,
176
+
SERVUN_REWRITE_ADDRESS,
177
+
0,
178
+
NULL,
179
+
},
180
+
{
181
+
SOCK_ADDR_TEST_SENDMSG,
182
+
"sendmsg_unix",
183
+
sendmsg_unix_prog_load,
184
+
sendmsg_unix_prog_destroy,
185
+
AF_UNIX,
186
+
SOCK_DGRAM,
187
+
SERVUN_ADDRESS,
188
+
0,
189
+
SERVUN_REWRITE_ADDRESS,
190
+
0,
191
+
NULL,
192
+
},
193
+
{
194
+
SOCK_ADDR_TEST_RECVMSG,
195
+
"recvmsg_unix-dgram",
196
+
recvmsg_unix_prog_load,
197
+
recvmsg_unix_prog_destroy,
198
+
AF_UNIX,
199
+
SOCK_DGRAM,
200
+
SERVUN_REWRITE_ADDRESS,
201
+
0,
202
+
SERVUN_REWRITE_ADDRESS,
203
+
0,
204
+
SERVUN_ADDRESS,
205
+
},
206
+
{
207
+
SOCK_ADDR_TEST_RECVMSG,
208
+
"recvmsg_unix-stream",
209
+
recvmsg_unix_prog_load,
210
+
recvmsg_unix_prog_destroy,
211
+
AF_UNIX,
212
+
SOCK_STREAM,
213
+
SERVUN_REWRITE_ADDRESS,
214
+
0,
215
+
SERVUN_REWRITE_ADDRESS,
216
+
0,
217
+
SERVUN_ADDRESS,
218
+
},
219
+
{
220
+
SOCK_ADDR_TEST_GETSOCKNAME,
221
+
"getsockname_unix",
222
+
getsockname_unix_prog_load,
223
+
getsockname_unix_prog_destroy,
224
+
AF_UNIX,
225
+
SOCK_STREAM,
226
+
SERVUN_ADDRESS,
227
+
0,
228
+
SERVUN_REWRITE_ADDRESS,
229
+
0,
230
+
NULL,
231
+
},
232
+
{
233
+
SOCK_ADDR_TEST_GETPEERNAME,
234
+
"getpeername_unix",
235
+
getpeername_unix_prog_load,
236
+
getpeername_unix_prog_destroy,
237
+
AF_UNIX,
238
+
SOCK_STREAM,
239
+
SERVUN_ADDRESS,
240
+
0,
241
+
SERVUN_REWRITE_ADDRESS,
242
+
0,
243
+
NULL,
244
+
},
245
+
};
246
+
247
+
typedef int (*info_fn)(int, struct sockaddr *, socklen_t *);
248
+
249
+
static int cmp_addr(const struct sockaddr_storage *addr1, socklen_t addr1_len,
250
+
const struct sockaddr_storage *addr2, socklen_t addr2_len,
251
+
bool cmp_port)
252
+
{
253
+
const struct sockaddr_in *four1, *four2;
254
+
const struct sockaddr_in6 *six1, *six2;
255
+
const struct sockaddr_un *un1, *un2;
256
+
257
+
if (addr1->ss_family != addr2->ss_family)
258
+
return -1;
259
+
260
+
if (addr1_len != addr2_len)
261
+
return -1;
262
+
263
+
if (addr1->ss_family == AF_INET) {
264
+
four1 = (const struct sockaddr_in *)addr1;
265
+
four2 = (const struct sockaddr_in *)addr2;
266
+
return !((four1->sin_port == four2->sin_port || !cmp_port) &&
267
+
four1->sin_addr.s_addr == four2->sin_addr.s_addr);
268
+
} else if (addr1->ss_family == AF_INET6) {
269
+
six1 = (const struct sockaddr_in6 *)addr1;
270
+
six2 = (const struct sockaddr_in6 *)addr2;
271
+
return !((six1->sin6_port == six2->sin6_port || !cmp_port) &&
272
+
!memcmp(&six1->sin6_addr, &six2->sin6_addr,
273
+
sizeof(struct in6_addr)));
274
+
} else if (addr1->ss_family == AF_UNIX) {
275
+
un1 = (const struct sockaddr_un *)addr1;
276
+
un2 = (const struct sockaddr_un *)addr2;
277
+
return memcmp(un1, un2, addr1_len);
278
+
}
279
+
280
+
return -1;
281
+
}
282
+
283
+
static int cmp_sock_addr(info_fn fn, int sock1,
284
+
const struct sockaddr_storage *addr2,
285
+
socklen_t addr2_len, bool cmp_port)
286
+
{
287
+
struct sockaddr_storage addr1;
288
+
socklen_t len1 = sizeof(addr1);
289
+
290
+
memset(&addr1, 0, len1);
291
+
if (fn(sock1, (struct sockaddr *)&addr1, (socklen_t *)&len1) != 0)
292
+
return -1;
293
+
294
+
return cmp_addr(&addr1, len1, addr2, addr2_len, cmp_port);
295
+
}
296
+
297
+
static int cmp_local_addr(int sock1, const struct sockaddr_storage *addr2,
298
+
socklen_t addr2_len, bool cmp_port)
299
+
{
300
+
return cmp_sock_addr(getsockname, sock1, addr2, addr2_len, cmp_port);
301
+
}
302
+
303
+
static int cmp_peer_addr(int sock1, const struct sockaddr_storage *addr2,
304
+
socklen_t addr2_len, bool cmp_port)
305
+
{
306
+
return cmp_sock_addr(getpeername, sock1, addr2, addr2_len, cmp_port);
307
+
}
308
+
309
+
static void test_bind(struct sock_addr_test *test)
310
+
{
311
+
struct sockaddr_storage expected_addr;
312
+
socklen_t expected_addr_len = sizeof(struct sockaddr_storage);
313
+
int serv = -1, client = -1, err;
314
+
315
+
serv = start_server(test->socket_family, test->socket_type,
316
+
test->requested_addr, test->requested_port, 0);
317
+
if (!ASSERT_GE(serv, 0, "start_server"))
318
+
goto cleanup;
319
+
320
+
err = make_sockaddr(test->socket_family,
321
+
test->expected_addr, test->expected_port,
322
+
&expected_addr, &expected_addr_len);
323
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
324
+
goto cleanup;
325
+
326
+
err = cmp_local_addr(serv, &expected_addr, expected_addr_len, true);
327
+
if (!ASSERT_EQ(err, 0, "cmp_local_addr"))
328
+
goto cleanup;
329
+
330
+
/* Try to connect to server just in case */
331
+
client = connect_to_addr(&expected_addr, expected_addr_len, test->socket_type);
332
+
if (!ASSERT_GE(client, 0, "connect_to_addr"))
333
+
goto cleanup;
334
+
335
+
cleanup:
336
+
if (client != -1)
337
+
close(client);
338
+
if (serv != -1)
339
+
close(serv);
340
+
}
341
+
342
+
static void test_connect(struct sock_addr_test *test)
343
+
{
344
+
struct sockaddr_storage addr, expected_addr, expected_src_addr;
345
+
socklen_t addr_len = sizeof(struct sockaddr_storage),
346
+
expected_addr_len = sizeof(struct sockaddr_storage),
347
+
expected_src_addr_len = sizeof(struct sockaddr_storage);
348
+
int serv = -1, client = -1, err;
349
+
350
+
serv = start_server(test->socket_family, test->socket_type,
351
+
test->expected_addr, test->expected_port, 0);
352
+
if (!ASSERT_GE(serv, 0, "start_server"))
353
+
goto cleanup;
354
+
355
+
err = make_sockaddr(test->socket_family, test->requested_addr, test->requested_port,
356
+
&addr, &addr_len);
357
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
358
+
goto cleanup;
359
+
360
+
client = connect_to_addr(&addr, addr_len, test->socket_type);
361
+
if (!ASSERT_GE(client, 0, "connect_to_addr"))
362
+
goto cleanup;
363
+
364
+
err = make_sockaddr(test->socket_family, test->expected_addr, test->expected_port,
365
+
&expected_addr, &expected_addr_len);
366
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
367
+
goto cleanup;
368
+
369
+
if (test->expected_src_addr) {
370
+
err = make_sockaddr(test->socket_family, test->expected_src_addr, 0,
371
+
&expected_src_addr, &expected_src_addr_len);
372
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
373
+
goto cleanup;
374
+
}
375
+
376
+
err = cmp_peer_addr(client, &expected_addr, expected_addr_len, true);
377
+
if (!ASSERT_EQ(err, 0, "cmp_peer_addr"))
378
+
goto cleanup;
379
+
380
+
if (test->expected_src_addr) {
381
+
err = cmp_local_addr(client, &expected_src_addr, expected_src_addr_len, false);
382
+
if (!ASSERT_EQ(err, 0, "cmp_local_addr"))
383
+
goto cleanup;
384
+
}
385
+
cleanup:
386
+
if (client != -1)
387
+
close(client);
388
+
if (serv != -1)
389
+
close(serv);
390
+
}
391
+
392
+
static void test_xmsg(struct sock_addr_test *test)
393
+
{
394
+
struct sockaddr_storage addr, src_addr;
395
+
socklen_t addr_len = sizeof(struct sockaddr_storage),
396
+
src_addr_len = sizeof(struct sockaddr_storage);
397
+
struct msghdr hdr;
398
+
struct iovec iov;
399
+
char data = 'a';
400
+
int serv = -1, client = -1, err;
401
+
402
+
/* Unlike the other tests, here we test that we can rewrite the src addr
403
+
* with a recvmsg() hook.
404
+
*/
405
+
406
+
serv = start_server(test->socket_family, test->socket_type,
407
+
test->expected_addr, test->expected_port, 0);
408
+
if (!ASSERT_GE(serv, 0, "start_server"))
409
+
goto cleanup;
410
+
411
+
client = socket(test->socket_family, test->socket_type, 0);
412
+
if (!ASSERT_GE(client, 0, "socket"))
413
+
goto cleanup;
414
+
415
+
/* AF_UNIX sockets have to be bound to something to trigger the recvmsg bpf program. */
416
+
if (test->socket_family == AF_UNIX) {
417
+
err = make_sockaddr(AF_UNIX, SRCUN_ADDRESS, 0, &src_addr, &src_addr_len);
418
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
419
+
goto cleanup;
420
+
421
+
err = bind(client, (const struct sockaddr *) &src_addr, src_addr_len);
422
+
if (!ASSERT_OK(err, "bind"))
423
+
goto cleanup;
424
+
}
425
+
426
+
err = make_sockaddr(test->socket_family, test->requested_addr, test->requested_port,
427
+
&addr, &addr_len);
428
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
429
+
goto cleanup;
430
+
431
+
if (test->socket_type == SOCK_DGRAM) {
432
+
memset(&iov, 0, sizeof(iov));
433
+
iov.iov_base = &data;
434
+
iov.iov_len = sizeof(data);
435
+
436
+
memset(&hdr, 0, sizeof(hdr));
437
+
hdr.msg_name = (void *)&addr;
438
+
hdr.msg_namelen = addr_len;
439
+
hdr.msg_iov = &iov;
440
+
hdr.msg_iovlen = 1;
441
+
442
+
err = sendmsg(client, &hdr, 0);
443
+
if (!ASSERT_EQ(err, sizeof(data), "sendmsg"))
444
+
goto cleanup;
445
+
} else {
446
+
/* Testing with connection-oriented sockets is only valid for
447
+
* recvmsg() tests.
448
+
*/
449
+
if (!ASSERT_EQ(test->type, SOCK_ADDR_TEST_RECVMSG, "recvmsg"))
450
+
goto cleanup;
451
+
452
+
err = connect(client, (const struct sockaddr *)&addr, addr_len);
453
+
if (!ASSERT_OK(err, "connect"))
454
+
goto cleanup;
455
+
456
+
err = send(client, &data, sizeof(data), 0);
457
+
if (!ASSERT_EQ(err, sizeof(data), "send"))
458
+
goto cleanup;
459
+
460
+
err = listen(serv, 0);
461
+
if (!ASSERT_OK(err, "listen"))
462
+
goto cleanup;
463
+
464
+
err = accept(serv, NULL, NULL);
465
+
if (!ASSERT_GE(err, 0, "accept"))
466
+
goto cleanup;
467
+
468
+
close(serv);
469
+
serv = err;
470
+
}
471
+
472
+
addr_len = src_addr_len = sizeof(struct sockaddr_storage);
473
+
474
+
err = recvfrom(serv, &data, sizeof(data), 0, (struct sockaddr *) &src_addr, &src_addr_len);
475
+
if (!ASSERT_EQ(err, sizeof(data), "recvfrom"))
476
+
goto cleanup;
477
+
478
+
ASSERT_EQ(data, 'a', "data mismatch");
479
+
480
+
if (test->expected_src_addr) {
481
+
err = make_sockaddr(test->socket_family, test->expected_src_addr, 0,
482
+
&addr, &addr_len);
483
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
484
+
goto cleanup;
485
+
486
+
err = cmp_addr(&src_addr, src_addr_len, &addr, addr_len, false);
487
+
if (!ASSERT_EQ(err, 0, "cmp_addr"))
488
+
goto cleanup;
489
+
}
490
+
491
+
cleanup:
492
+
if (client != -1)
493
+
close(client);
494
+
if (serv != -1)
495
+
close(serv);
496
+
}
497
+
498
+
static void test_getsockname(struct sock_addr_test *test)
499
+
{
500
+
struct sockaddr_storage expected_addr;
501
+
socklen_t expected_addr_len = sizeof(struct sockaddr_storage);
502
+
int serv = -1, err;
503
+
504
+
serv = start_server(test->socket_family, test->socket_type,
505
+
test->requested_addr, test->requested_port, 0);
506
+
if (!ASSERT_GE(serv, 0, "start_server"))
507
+
goto cleanup;
508
+
509
+
err = make_sockaddr(test->socket_family,
510
+
test->expected_addr, test->expected_port,
511
+
&expected_addr, &expected_addr_len);
512
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
513
+
goto cleanup;
514
+
515
+
err = cmp_local_addr(serv, &expected_addr, expected_addr_len, true);
516
+
if (!ASSERT_EQ(err, 0, "cmp_local_addr"))
517
+
goto cleanup;
518
+
519
+
cleanup:
520
+
if (serv != -1)
521
+
close(serv);
522
+
}
523
+
524
+
static void test_getpeername(struct sock_addr_test *test)
525
+
{
526
+
struct sockaddr_storage addr, expected_addr;
527
+
socklen_t addr_len = sizeof(struct sockaddr_storage),
528
+
expected_addr_len = sizeof(struct sockaddr_storage);
529
+
int serv = -1, client = -1, err;
530
+
531
+
serv = start_server(test->socket_family, test->socket_type,
532
+
test->requested_addr, test->requested_port, 0);
533
+
if (!ASSERT_GE(serv, 0, "start_server"))
534
+
goto cleanup;
535
+
536
+
err = make_sockaddr(test->socket_family, test->requested_addr, test->requested_port,
537
+
&addr, &addr_len);
538
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
539
+
goto cleanup;
540
+
541
+
client = connect_to_addr(&addr, addr_len, test->socket_type);
542
+
if (!ASSERT_GE(client, 0, "connect_to_addr"))
543
+
goto cleanup;
544
+
545
+
err = make_sockaddr(test->socket_family, test->expected_addr, test->expected_port,
546
+
&expected_addr, &expected_addr_len);
547
+
if (!ASSERT_EQ(err, 0, "make_sockaddr"))
548
+
goto cleanup;
549
+
550
+
err = cmp_peer_addr(client, &expected_addr, expected_addr_len, true);
551
+
if (!ASSERT_EQ(err, 0, "cmp_peer_addr"))
552
+
goto cleanup;
553
+
554
+
cleanup:
555
+
if (client != -1)
556
+
close(client);
557
+
if (serv != -1)
558
+
close(serv);
559
+
}
560
+
561
+
void test_sock_addr(void)
562
+
{
563
+
int cgroup_fd = -1;
564
+
void *skel;
565
+
566
+
cgroup_fd = test__join_cgroup("/sock_addr");
567
+
if (!ASSERT_GE(cgroup_fd, 0, "join_cgroup"))
568
+
goto cleanup;
569
+
570
+
for (size_t i = 0; i < ARRAY_SIZE(tests); ++i) {
571
+
struct sock_addr_test *test = &tests[i];
572
+
573
+
if (!test__start_subtest(test->name))
574
+
continue;
575
+
576
+
skel = test->loadfn(cgroup_fd);
577
+
if (!skel)
578
+
continue;
579
+
580
+
switch (test->type) {
581
+
/* Not exercised yet but we leave this code here for when the
582
+
* INET and INET6 sockaddr tests are migrated to this file in
583
+
* the future.
584
+
*/
585
+
case SOCK_ADDR_TEST_BIND:
586
+
test_bind(test);
587
+
break;
588
+
case SOCK_ADDR_TEST_CONNECT:
589
+
test_connect(test);
590
+
break;
591
+
case SOCK_ADDR_TEST_SENDMSG:
592
+
case SOCK_ADDR_TEST_RECVMSG:
593
+
test_xmsg(test);
594
+
break;
595
+
case SOCK_ADDR_TEST_GETSOCKNAME:
596
+
test_getsockname(test);
597
+
break;
598
+
case SOCK_ADDR_TEST_GETPEERNAME:
599
+
test_getpeername(test);
600
+
break;
601
+
default:
602
+
ASSERT_TRUE(false, "Unknown sock addr test type");
603
+
break;
604
+
}
605
+
606
+
test->destroyfn(skel);
607
+
}
608
+
609
+
cleanup:
610
+
if (cgroup_fd >= 0)
611
+
close(cgroup_fd);
612
+
}
+4
-4
tools/testing/selftests/bpf/prog_tests/sockmap_basic.c
+4
-4
tools/testing/selftests/bpf/prog_tests/sockmap_basic.c
···
359
359
static void test_sockmap_skb_verdict_shutdown(void)
360
360
{
361
361
struct epoll_event ev, events[MAX_EVENTS];
362
-
int n, err, map, verdict, s, c1, p1;
362
+
int n, err, map, verdict, s, c1 = -1, p1 = -1;
363
363
struct test_sockmap_pass_prog *skel;
364
364
int epollfd;
365
365
int zero = 0;
···
414
414
static void test_sockmap_skb_verdict_fionread(bool pass_prog)
415
415
{
416
416
int expected, zero = 0, sent, recvd, avail;
417
-
int err, map, verdict, s, c0, c1, p0, p1;
418
-
struct test_sockmap_pass_prog *pass;
419
-
struct test_sockmap_drop_prog *drop;
417
+
int err, map, verdict, s, c0 = -1, c1 = -1, p0 = -1, p1 = -1;
418
+
struct test_sockmap_pass_prog *pass = NULL;
419
+
struct test_sockmap_drop_prog *drop = NULL;
420
420
char buf[256] = "0123456789";
421
421
422
422
if (pass_prog) {
+1
-1
tools/testing/selftests/bpf/prog_tests/sockmap_helpers.h
+1
-1
tools/testing/selftests/bpf/prog_tests/sockmap_helpers.h
+43
-127
tools/testing/selftests/bpf/prog_tests/sockmap_listen.c
+43
-127
tools/testing/selftests/bpf/prog_tests/sockmap_listen.c
···
73
73
int family, int sotype, int mapfd)
74
74
{
75
75
struct sockaddr_storage addr;
76
-
socklen_t len;
76
+
socklen_t len = 0;
77
77
u32 key = 0;
78
78
u64 value;
79
79
int err, s;
···
871
871
872
872
static void redir_partial(int family, int sotype, int sock_map, int parser_map)
873
873
{
874
-
int s, c0, c1, p0, p1;
874
+
int s, c0 = -1, c1 = -1, p0 = -1, p1 = -1;
875
875
int err, n, key, value;
876
876
char buf[] = "abc";
877
877
···
1336
1336
}
1337
1337
}
1338
1338
1339
-
static void unix_redir_to_connected(int sotype, int sock_mapfd,
1340
-
int verd_mapfd, enum redir_mode mode)
1339
+
static void pairs_redir_to_connected(int cli0, int peer0, int cli1, int peer1,
1340
+
int sock_mapfd, int verd_mapfd, enum redir_mode mode)
1341
1341
{
1342
1342
const char *log_prefix = redir_mode_str(mode);
1343
-
int c0, c1, p0, p1;
1344
1343
unsigned int pass;
1345
1344
int err, n;
1346
-
int sfd[2];
1347
1345
u32 key;
1348
1346
char b;
1349
1347
1350
1348
zero_verdict_count(verd_mapfd);
1349
+
1350
+
err = add_to_sockmap(sock_mapfd, peer0, peer1);
1351
+
if (err)
1352
+
return;
1353
+
1354
+
n = write(cli1, "a", 1);
1355
+
if (n < 0)
1356
+
FAIL_ERRNO("%s: write", log_prefix);
1357
+
if (n == 0)
1358
+
FAIL("%s: incomplete write", log_prefix);
1359
+
if (n < 1)
1360
+
return;
1361
+
1362
+
key = SK_PASS;
1363
+
err = xbpf_map_lookup_elem(verd_mapfd, &key, &pass);
1364
+
if (err)
1365
+
return;
1366
+
if (pass != 1)
1367
+
FAIL("%s: want pass count 1, have %d", log_prefix, pass);
1368
+
1369
+
n = recv_timeout(mode == REDIR_INGRESS ? peer0 : cli0, &b, 1, 0, IO_TIMEOUT_SEC);
1370
+
if (n < 0)
1371
+
FAIL_ERRNO("%s: recv_timeout", log_prefix);
1372
+
if (n == 0)
1373
+
FAIL("%s: incomplete recv", log_prefix);
1374
+
}
1375
+
1376
+
static void unix_redir_to_connected(int sotype, int sock_mapfd,
1377
+
int verd_mapfd, enum redir_mode mode)
1378
+
{
1379
+
int c0, c1, p0, p1;
1380
+
int sfd[2];
1351
1381
1352
1382
if (socketpair(AF_UNIX, sotype | SOCK_NONBLOCK, 0, sfd))
1353
1383
return;
···
1387
1357
goto close0;
1388
1358
c1 = sfd[0], p1 = sfd[1];
1389
1359
1390
-
err = add_to_sockmap(sock_mapfd, p0, p1);
1391
-
if (err)
1392
-
goto close;
1360
+
pairs_redir_to_connected(c0, p0, c1, p1, sock_mapfd, verd_mapfd, mode);
1393
1361
1394
-
n = write(c1, "a", 1);
1395
-
if (n < 0)
1396
-
FAIL_ERRNO("%s: write", log_prefix);
1397
-
if (n == 0)
1398
-
FAIL("%s: incomplete write", log_prefix);
1399
-
if (n < 1)
1400
-
goto close;
1401
-
1402
-
key = SK_PASS;
1403
-
err = xbpf_map_lookup_elem(verd_mapfd, &key, &pass);
1404
-
if (err)
1405
-
goto close;
1406
-
if (pass != 1)
1407
-
FAIL("%s: want pass count 1, have %d", log_prefix, pass);
1408
-
1409
-
n = recv_timeout(mode == REDIR_INGRESS ? p0 : c0, &b, 1, 0, IO_TIMEOUT_SEC);
1410
-
if (n < 0)
1411
-
FAIL_ERRNO("%s: recv_timeout", log_prefix);
1412
-
if (n == 0)
1413
-
FAIL("%s: incomplete recv", log_prefix);
1414
-
1415
-
close:
1416
1362
xclose(c1);
1417
1363
xclose(p1);
1418
1364
close0:
···
1667
1661
static void udp_redir_to_connected(int family, int sock_mapfd, int verd_mapfd,
1668
1662
enum redir_mode mode)
1669
1663
{
1670
-
const char *log_prefix = redir_mode_str(mode);
1671
1664
int c0, c1, p0, p1;
1672
-
unsigned int pass;
1673
-
int err, n;
1674
-
u32 key;
1675
-
char b;
1676
-
1677
-
zero_verdict_count(verd_mapfd);
1665
+
int err;
1678
1666
1679
1667
err = inet_socketpair(family, SOCK_DGRAM, &p0, &c0);
1680
1668
if (err)
···
1677
1677
if (err)
1678
1678
goto close_cli0;
1679
1679
1680
-
err = add_to_sockmap(sock_mapfd, p0, p1);
1681
-
if (err)
1682
-
goto close_cli1;
1680
+
pairs_redir_to_connected(c0, p0, c1, p1, sock_mapfd, verd_mapfd, mode);
1683
1681
1684
-
n = write(c1, "a", 1);
1685
-
if (n < 0)
1686
-
FAIL_ERRNO("%s: write", log_prefix);
1687
-
if (n == 0)
1688
-
FAIL("%s: incomplete write", log_prefix);
1689
-
if (n < 1)
1690
-
goto close_cli1;
1691
-
1692
-
key = SK_PASS;
1693
-
err = xbpf_map_lookup_elem(verd_mapfd, &key, &pass);
1694
-
if (err)
1695
-
goto close_cli1;
1696
-
if (pass != 1)
1697
-
FAIL("%s: want pass count 1, have %d", log_prefix, pass);
1698
-
1699
-
n = recv_timeout(mode == REDIR_INGRESS ? p0 : c0, &b, 1, 0, IO_TIMEOUT_SEC);
1700
-
if (n < 0)
1701
-
FAIL_ERRNO("%s: recv_timeout", log_prefix);
1702
-
if (n == 0)
1703
-
FAIL("%s: incomplete recv", log_prefix);
1704
-
1705
-
close_cli1:
1706
1682
xclose(c1);
1707
1683
xclose(p1);
1708
1684
close_cli0:
···
1723
1747
static void inet_unix_redir_to_connected(int family, int type, int sock_mapfd,
1724
1748
int verd_mapfd, enum redir_mode mode)
1725
1749
{
1726
-
const char *log_prefix = redir_mode_str(mode);
1727
1750
int c0, c1, p0, p1;
1728
-
unsigned int pass;
1729
-
int err, n;
1730
1751
int sfd[2];
1731
-
u32 key;
1732
-
char b;
1733
-
1734
-
zero_verdict_count(verd_mapfd);
1752
+
int err;
1735
1753
1736
1754
if (socketpair(AF_UNIX, SOCK_DGRAM | SOCK_NONBLOCK, 0, sfd))
1737
1755
return;
···
1735
1765
if (err)
1736
1766
goto close;
1737
1767
1738
-
err = add_to_sockmap(sock_mapfd, p0, p1);
1739
-
if (err)
1740
-
goto close_cli1;
1768
+
pairs_redir_to_connected(c0, p0, c1, p1, sock_mapfd, verd_mapfd, mode);
1741
1769
1742
-
n = write(c1, "a", 1);
1743
-
if (n < 0)
1744
-
FAIL_ERRNO("%s: write", log_prefix);
1745
-
if (n == 0)
1746
-
FAIL("%s: incomplete write", log_prefix);
1747
-
if (n < 1)
1748
-
goto close_cli1;
1749
-
1750
-
key = SK_PASS;
1751
-
err = xbpf_map_lookup_elem(verd_mapfd, &key, &pass);
1752
-
if (err)
1753
-
goto close_cli1;
1754
-
if (pass != 1)
1755
-
FAIL("%s: want pass count 1, have %d", log_prefix, pass);
1756
-
1757
-
n = recv_timeout(mode == REDIR_INGRESS ? p0 : c0, &b, 1, 0, IO_TIMEOUT_SEC);
1758
-
if (n < 0)
1759
-
FAIL_ERRNO("%s: recv_timeout", log_prefix);
1760
-
if (n == 0)
1761
-
FAIL("%s: incomplete recv", log_prefix);
1762
-
1763
-
close_cli1:
1764
1770
xclose(c1);
1765
1771
xclose(p1);
1766
1772
close:
···
1773
1827
static void unix_inet_redir_to_connected(int family, int type, int sock_mapfd,
1774
1828
int verd_mapfd, enum redir_mode mode)
1775
1829
{
1776
-
const char *log_prefix = redir_mode_str(mode);
1777
1830
int c0, c1, p0, p1;
1778
-
unsigned int pass;
1779
-
int err, n;
1780
1831
int sfd[2];
1781
-
u32 key;
1782
-
char b;
1783
-
1784
-
zero_verdict_count(verd_mapfd);
1832
+
int err;
1785
1833
1786
1834
err = inet_socketpair(family, SOCK_DGRAM, &p0, &c0);
1787
1835
if (err)
···
1785
1845
goto close_cli0;
1786
1846
c1 = sfd[0], p1 = sfd[1];
1787
1847
1788
-
err = add_to_sockmap(sock_mapfd, p0, p1);
1789
-
if (err)
1790
-
goto close;
1848
+
pairs_redir_to_connected(c0, p0, c1, p1, sock_mapfd, verd_mapfd, mode);
1791
1849
1792
-
n = write(c1, "a", 1);
1793
-
if (n < 0)
1794
-
FAIL_ERRNO("%s: write", log_prefix);
1795
-
if (n == 0)
1796
-
FAIL("%s: incomplete write", log_prefix);
1797
-
if (n < 1)
1798
-
goto close;
1799
-
1800
-
key = SK_PASS;
1801
-
err = xbpf_map_lookup_elem(verd_mapfd, &key, &pass);
1802
-
if (err)
1803
-
goto close;
1804
-
if (pass != 1)
1805
-
FAIL("%s: want pass count 1, have %d", log_prefix, pass);
1806
-
1807
-
n = recv_timeout(mode == REDIR_INGRESS ? p0 : c0, &b, 1, 0, IO_TIMEOUT_SEC);
1808
-
if (n < 0)
1809
-
FAIL_ERRNO("%s: recv_timeout", log_prefix);
1810
-
if (n == 0)
1811
-
FAIL("%s: incomplete recv", log_prefix);
1812
-
1813
-
close:
1814
1850
xclose(c1);
1815
1851
xclose(p1);
1816
1852
close_cli0:
+4
tools/testing/selftests/bpf/prog_tests/timer.c
+4
tools/testing/selftests/bpf/prog_tests/timer.c
···
15
15
16
16
ASSERT_EQ(timer_skel->data->callback_check, 52, "callback_check1");
17
17
ASSERT_EQ(timer_skel->data->callback2_check, 52, "callback2_check1");
18
+
ASSERT_EQ(timer_skel->bss->pinned_callback_check, 0, "pinned_callback_check1");
18
19
19
20
prog_fd = bpf_program__fd(timer_skel->progs.test1);
20
21
err = bpf_prog_test_run_opts(prog_fd, &topts);
···
33
32
34
33
/* check that timer_cb3() was executed twice */
35
34
ASSERT_EQ(timer_skel->bss->abs_data, 12, "abs_data");
35
+
36
+
/* check that timer_cb_pinned() was executed twice */
37
+
ASSERT_EQ(timer_skel->bss->pinned_callback_check, 2, "pinned_callback_check");
36
38
37
39
/* check that there were no errors in timer execution */
38
40
ASSERT_EQ(timer_skel->bss->err, 0, "err");
+95
tools/testing/selftests/bpf/prog_tests/uprobe.c
+95
tools/testing/selftests/bpf/prog_tests/uprobe.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Hengqi Chen */
3
+
4
+
#include <test_progs.h>
5
+
#include "test_uprobe.skel.h"
6
+
7
+
static FILE *urand_spawn(int *pid)
8
+
{
9
+
FILE *f;
10
+
11
+
/* urandom_read's stdout is wired into f */
12
+
f = popen("./urandom_read 1 report-pid", "r");
13
+
if (!f)
14
+
return NULL;
15
+
16
+
if (fscanf(f, "%d", pid) != 1) {
17
+
pclose(f);
18
+
errno = EINVAL;
19
+
return NULL;
20
+
}
21
+
22
+
return f;
23
+
}
24
+
25
+
static int urand_trigger(FILE **urand_pipe)
26
+
{
27
+
int exit_code;
28
+
29
+
/* pclose() waits for child process to exit and returns their exit code */
30
+
exit_code = pclose(*urand_pipe);
31
+
*urand_pipe = NULL;
32
+
33
+
return exit_code;
34
+
}
35
+
36
+
void test_uprobe(void)
37
+
{
38
+
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
39
+
struct test_uprobe *skel;
40
+
FILE *urand_pipe = NULL;
41
+
int urand_pid = 0, err;
42
+
43
+
skel = test_uprobe__open_and_load();
44
+
if (!ASSERT_OK_PTR(skel, "skel_open"))
45
+
return;
46
+
47
+
urand_pipe = urand_spawn(&urand_pid);
48
+
if (!ASSERT_OK_PTR(urand_pipe, "urand_spawn"))
49
+
goto cleanup;
50
+
51
+
skel->bss->my_pid = urand_pid;
52
+
53
+
/* Manual attach uprobe to urandlib_api
54
+
* There are two `urandlib_api` symbols in .dynsym section:
55
+
* - urandlib_api@LIBURANDOM_READ_1.0.0
56
+
* - urandlib_api@@LIBURANDOM_READ_2.0.0
57
+
* Both are global bind and would cause a conflict if user
58
+
* specify the symbol name without a version suffix
59
+
*/
60
+
uprobe_opts.func_name = "urandlib_api";
61
+
skel->links.test4 = bpf_program__attach_uprobe_opts(skel->progs.test4,
62
+
urand_pid,
63
+
"./liburandom_read.so",
64
+
0 /* offset */,
65
+
&uprobe_opts);
66
+
if (!ASSERT_ERR_PTR(skel->links.test4, "urandlib_api_attach_conflict"))
67
+
goto cleanup;
68
+
69
+
uprobe_opts.func_name = "urandlib_api@LIBURANDOM_READ_1.0.0";
70
+
skel->links.test4 = bpf_program__attach_uprobe_opts(skel->progs.test4,
71
+
urand_pid,
72
+
"./liburandom_read.so",
73
+
0 /* offset */,
74
+
&uprobe_opts);
75
+
if (!ASSERT_OK_PTR(skel->links.test4, "urandlib_api_attach_ok"))
76
+
goto cleanup;
77
+
78
+
/* Auto attach 3 u[ret]probes to urandlib_api_sameoffset */
79
+
err = test_uprobe__attach(skel);
80
+
if (!ASSERT_OK(err, "skel_attach"))
81
+
goto cleanup;
82
+
83
+
/* trigger urandom_read */
84
+
ASSERT_OK(urand_trigger(&urand_pipe), "urand_exit_code");
85
+
86
+
ASSERT_EQ(skel->bss->test1_result, 1, "urandlib_api_sameoffset");
87
+
ASSERT_EQ(skel->bss->test2_result, 1, "urandlib_api_sameoffset@v1");
88
+
ASSERT_EQ(skel->bss->test3_result, 3, "urandlib_api_sameoffset@@v2");
89
+
ASSERT_EQ(skel->bss->test4_result, 1, "urandlib_api");
90
+
91
+
cleanup:
92
+
if (urand_pipe)
93
+
pclose(urand_pipe);
94
+
test_uprobe__destroy(skel);
95
+
}
+1
-1
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c
+1
-1
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c
tools/testing/selftests/bpf/progs/bpf_iter_task_vma.c
tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c
tools/testing/selftests/bpf/progs/bpf_iter_task_vma.c
tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c
+3
tools/testing/selftests/bpf/progs/bpf_misc.h
+3
tools/testing/selftests/bpf/progs/bpf_misc.h
···
99
99
#elif defined(__TARGET_ARCH_arm64)
100
100
#define SYSCALL_WRAPPER 1
101
101
#define SYS_PREFIX "__arm64_"
102
+
#elif defined(__TARGET_ARCH_riscv)
103
+
#define SYSCALL_WRAPPER 1
104
+
#define SYS_PREFIX "__riscv_"
102
105
#else
103
106
#define SYSCALL_WRAPPER 0
104
107
#define SYS_PREFIX "__se_"
+40
tools/testing/selftests/bpf/progs/connect_unix_prog.c
+40
tools/testing/selftests/bpf/progs/connect_unix_prog.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
6
+
#include <string.h>
7
+
#include <bpf/bpf_helpers.h>
8
+
#include <bpf/bpf_core_read.h>
9
+
#include "bpf_kfuncs.h"
10
+
11
+
__u8 SERVUN_REWRITE_ADDRESS[] = "\0bpf_cgroup_unix_test_rewrite";
12
+
13
+
SEC("cgroup/connect_unix")
14
+
int connect_unix_prog(struct bpf_sock_addr *ctx)
15
+
{
16
+
struct bpf_sock_addr_kern *sa_kern = bpf_cast_to_kern_ctx(ctx);
17
+
struct sockaddr_un *sa_kern_unaddr;
18
+
__u32 unaddrlen = offsetof(struct sockaddr_un, sun_path) +
19
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1;
20
+
int ret;
21
+
22
+
/* Rewrite destination. */
23
+
ret = bpf_sock_addr_set_sun_path(sa_kern, SERVUN_REWRITE_ADDRESS,
24
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1);
25
+
if (ret)
26
+
return 0;
27
+
28
+
if (sa_kern->uaddrlen != unaddrlen)
29
+
return 0;
30
+
31
+
sa_kern_unaddr = bpf_rdonly_cast(sa_kern->uaddr,
32
+
bpf_core_type_id_kernel(struct sockaddr_un));
33
+
if (memcmp(sa_kern_unaddr->sun_path, SERVUN_REWRITE_ADDRESS,
34
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1) != 0)
35
+
return 0;
36
+
37
+
return 1;
38
+
}
39
+
40
+
char _license[] SEC("license") = "GPL";
+9
-9
tools/testing/selftests/bpf/progs/exceptions_assert.c
+9
-9
tools/testing/selftests/bpf/progs/exceptions_assert.c
···
31
31
32
32
__msg(": R0_w=scalar(smax=2147483646) R10=fp0")
33
33
check_assert(s64, lt, pos, INT_MAX);
34
-
__msg(": R0_w=scalar(umin=9223372036854775808,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
34
+
__msg(": R0_w=scalar(smax=-1,umin=9223372036854775808,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
35
35
check_assert(s64, lt, zero, 0);
36
-
__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067967,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
36
+
__msg(": R0_w=scalar(smax=-2147483649,umin=9223372036854775808,umax=18446744071562067967,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
37
37
check_assert(s64, lt, neg, INT_MIN);
38
38
39
39
__msg(": R0_w=scalar(smax=2147483647) R10=fp0")
40
40
check_assert(s64, le, pos, INT_MAX);
41
41
__msg(": R0_w=scalar(smax=0) R10=fp0")
42
42
check_assert(s64, le, zero, 0);
43
-
__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067968,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
43
+
__msg(": R0_w=scalar(smax=-2147483648,umin=9223372036854775808,umax=18446744071562067968,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
44
44
check_assert(s64, le, neg, INT_MIN);
45
45
46
-
__msg(": R0_w=scalar(umin=2147483648,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
46
+
__msg(": R0_w=scalar(smin=umin=2147483648,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
47
47
check_assert(s64, gt, pos, INT_MAX);
48
-
__msg(": R0_w=scalar(umin=1,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
48
+
__msg(": R0_w=scalar(smin=umin=1,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
49
49
check_assert(s64, gt, zero, 0);
50
50
__msg(": R0_w=scalar(smin=-2147483647) R10=fp0")
51
51
check_assert(s64, gt, neg, INT_MIN);
52
52
53
-
__msg(": R0_w=scalar(umin=2147483647,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
53
+
__msg(": R0_w=scalar(smin=umin=2147483647,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))")
54
54
check_assert(s64, ge, pos, INT_MAX);
55
-
__msg(": R0_w=scalar(umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff)) R10=fp0")
55
+
__msg(": R0_w=scalar(smin=0,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff)) R10=fp0")
56
56
check_assert(s64, ge, zero, 0);
57
57
__msg(": R0_w=scalar(smin=-2147483648) R10=fp0")
58
58
check_assert(s64, ge, neg, INT_MIN);
59
59
60
60
SEC("?tc")
61
61
__log_level(2) __failure
62
-
__msg(": R0=0 R1=ctx(off=0,imm=0) R2=scalar(smin=-2147483646,smax=2147483645) R10=fp0")
62
+
__msg(": R0=0 R1=ctx(off=0,imm=0) R2=scalar(smin=smin32=-2147483646,smax=smax32=2147483645) R10=fp0")
63
63
int check_assert_range_s64(struct __sk_buff *ctx)
64
64
{
65
65
struct bpf_sock *sk = ctx->sk;
···
75
75
76
76
SEC("?tc")
77
77
__log_level(2) __failure
78
-
__msg(": R1=ctx(off=0,imm=0) R2=scalar(umin=4096,umax=8192,var_off=(0x0; 0x3fff))")
78
+
__msg(": R1=ctx(off=0,imm=0) R2=scalar(smin=umin=smin32=umin32=4096,smax=umax=smax32=umax32=8192,var_off=(0x0; 0x3fff))")
79
79
int check_assert_range_u64(struct __sk_buff *ctx)
80
80
{
81
81
u64 num = ctx->len;
+39
tools/testing/selftests/bpf/progs/getpeername_unix_prog.c
+39
tools/testing/selftests/bpf/progs/getpeername_unix_prog.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
6
+
#include <string.h>
7
+
#include <bpf/bpf_helpers.h>
8
+
#include <bpf/bpf_core_read.h>
9
+
#include "bpf_kfuncs.h"
10
+
11
+
__u8 SERVUN_REWRITE_ADDRESS[] = "\0bpf_cgroup_unix_test_rewrite";
12
+
13
+
SEC("cgroup/getpeername_unix")
14
+
int getpeername_unix_prog(struct bpf_sock_addr *ctx)
15
+
{
16
+
struct bpf_sock_addr_kern *sa_kern = bpf_cast_to_kern_ctx(ctx);
17
+
struct sockaddr_un *sa_kern_unaddr;
18
+
__u32 unaddrlen = offsetof(struct sockaddr_un, sun_path) +
19
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1;
20
+
int ret;
21
+
22
+
ret = bpf_sock_addr_set_sun_path(sa_kern, SERVUN_REWRITE_ADDRESS,
23
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1);
24
+
if (ret)
25
+
return 1;
26
+
27
+
if (sa_kern->uaddrlen != unaddrlen)
28
+
return 1;
29
+
30
+
sa_kern_unaddr = bpf_rdonly_cast(sa_kern->uaddr,
31
+
bpf_core_type_id_kernel(struct sockaddr_un));
32
+
if (memcmp(sa_kern_unaddr->sun_path, SERVUN_REWRITE_ADDRESS,
33
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1) != 0)
34
+
return 1;
35
+
36
+
return 1;
37
+
}
38
+
39
+
char _license[] SEC("license") = "GPL";
+39
tools/testing/selftests/bpf/progs/getsockname_unix_prog.c
+39
tools/testing/selftests/bpf/progs/getsockname_unix_prog.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
6
+
#include <string.h>
7
+
#include <bpf/bpf_helpers.h>
8
+
#include <bpf/bpf_core_read.h>
9
+
#include "bpf_kfuncs.h"
10
+
11
+
__u8 SERVUN_REWRITE_ADDRESS[] = "\0bpf_cgroup_unix_test_rewrite";
12
+
13
+
SEC("cgroup/getsockname_unix")
14
+
int getsockname_unix_prog(struct bpf_sock_addr *ctx)
15
+
{
16
+
struct bpf_sock_addr_kern *sa_kern = bpf_cast_to_kern_ctx(ctx);
17
+
struct sockaddr_un *sa_kern_unaddr;
18
+
__u32 unaddrlen = offsetof(struct sockaddr_un, sun_path) +
19
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1;
20
+
int ret;
21
+
22
+
ret = bpf_sock_addr_set_sun_path(sa_kern, SERVUN_REWRITE_ADDRESS,
23
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1);
24
+
if (ret)
25
+
return 1;
26
+
27
+
if (sa_kern->uaddrlen != unaddrlen)
28
+
return 1;
29
+
30
+
sa_kern_unaddr = bpf_rdonly_cast(sa_kern->uaddr,
31
+
bpf_core_type_id_kernel(struct sockaddr_un));
32
+
if (memcmp(sa_kern_unaddr->sun_path, SERVUN_REWRITE_ADDRESS,
33
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1) != 0)
34
+
return 1;
35
+
36
+
return 1;
37
+
}
38
+
39
+
char _license[] SEC("license") = "GPL";
+43
tools/testing/selftests/bpf/progs/iters_task_vma.c
+43
tools/testing/selftests/bpf/progs/iters_task_vma.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
#include "bpf_experimental.h"
6
+
#include <bpf/bpf_helpers.h>
7
+
#include "bpf_misc.h"
8
+
9
+
pid_t target_pid = 0;
10
+
unsigned int vmas_seen = 0;
11
+
12
+
struct {
13
+
__u64 vm_start;
14
+
__u64 vm_end;
15
+
} vm_ranges[1000];
16
+
17
+
SEC("raw_tp/sys_enter")
18
+
int iter_task_vma_for_each(const void *ctx)
19
+
{
20
+
struct task_struct *task = bpf_get_current_task_btf();
21
+
struct vm_area_struct *vma;
22
+
unsigned int seen = 0;
23
+
24
+
if (task->pid != target_pid)
25
+
return 0;
26
+
27
+
if (vmas_seen)
28
+
return 0;
29
+
30
+
bpf_for_each(task_vma, vma, task, 0) {
31
+
if (seen >= 1000)
32
+
break;
33
+
34
+
vm_ranges[seen].vm_start = vma->vm_start;
35
+
vm_ranges[seen].vm_end = vma->vm_end;
36
+
seen++;
37
+
}
38
+
39
+
vmas_seen = seen;
40
+
return 0;
41
+
}
42
+
43
+
char _license[] SEC("license") = "GPL";
+30
tools/testing/selftests/bpf/progs/missed_kprobe.c
+30
tools/testing/selftests/bpf/progs/missed_kprobe.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#include "vmlinux.h"
3
+
#include <bpf/bpf_helpers.h>
4
+
#include <bpf/bpf_tracing.h>
5
+
#include "../bpf_testmod/bpf_testmod_kfunc.h"
6
+
7
+
char _license[] SEC("license") = "GPL";
8
+
9
+
/*
10
+
* No tests in here, just to trigger 'bpf_fentry_test*'
11
+
* through tracing test_run
12
+
*/
13
+
SEC("fentry/bpf_modify_return_test")
14
+
int BPF_PROG(trigger)
15
+
{
16
+
return 0;
17
+
}
18
+
19
+
SEC("kprobe/bpf_fentry_test1")
20
+
int test1(struct pt_regs *ctx)
21
+
{
22
+
bpf_kfunc_common_test();
23
+
return 0;
24
+
}
25
+
26
+
SEC("kprobe/bpf_kfunc_common_test")
27
+
int test2(struct pt_regs *ctx)
28
+
{
29
+
return 0;
30
+
}
+48
tools/testing/selftests/bpf/progs/missed_kprobe_recursion.c
+48
tools/testing/selftests/bpf/progs/missed_kprobe_recursion.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#include "vmlinux.h"
3
+
#include <bpf/bpf_helpers.h>
4
+
#include <bpf/bpf_tracing.h>
5
+
#include "../bpf_testmod/bpf_testmod_kfunc.h"
6
+
7
+
char _license[] SEC("license") = "GPL";
8
+
9
+
/*
10
+
* No tests in here, just to trigger 'bpf_fentry_test*'
11
+
* through tracing test_run
12
+
*/
13
+
SEC("fentry/bpf_modify_return_test")
14
+
int BPF_PROG(trigger)
15
+
{
16
+
return 0;
17
+
}
18
+
19
+
SEC("kprobe.multi/bpf_fentry_test1")
20
+
int test1(struct pt_regs *ctx)
21
+
{
22
+
bpf_kfunc_common_test();
23
+
return 0;
24
+
}
25
+
26
+
SEC("kprobe/bpf_kfunc_common_test")
27
+
int test2(struct pt_regs *ctx)
28
+
{
29
+
return 0;
30
+
}
31
+
32
+
SEC("kprobe/bpf_kfunc_common_test")
33
+
int test3(struct pt_regs *ctx)
34
+
{
35
+
return 0;
36
+
}
37
+
38
+
SEC("kprobe/bpf_kfunc_common_test")
39
+
int test4(struct pt_regs *ctx)
40
+
{
41
+
return 0;
42
+
}
43
+
44
+
SEC("kprobe.multi/bpf_kfunc_common_test")
45
+
int test5(struct pt_regs *ctx)
46
+
{
47
+
return 0;
48
+
}
+41
tools/testing/selftests/bpf/progs/missed_tp_recursion.c
+41
tools/testing/selftests/bpf/progs/missed_tp_recursion.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
#include "vmlinux.h"
3
+
#include <bpf/bpf_helpers.h>
4
+
#include <bpf/bpf_tracing.h>
5
+
6
+
char _license[] SEC("license") = "GPL";
7
+
8
+
/*
9
+
* No tests in here, just to trigger 'bpf_fentry_test*'
10
+
* through tracing test_run
11
+
*/
12
+
SEC("fentry/bpf_modify_return_test")
13
+
int BPF_PROG(trigger)
14
+
{
15
+
return 0;
16
+
}
17
+
18
+
SEC("kprobe/bpf_fentry_test1")
19
+
int test1(struct pt_regs *ctx)
20
+
{
21
+
bpf_printk("test");
22
+
return 0;
23
+
}
24
+
25
+
SEC("tp/bpf_trace/bpf_trace_printk")
26
+
int test2(struct pt_regs *ctx)
27
+
{
28
+
return 0;
29
+
}
30
+
31
+
SEC("tp/bpf_trace/bpf_trace_printk")
32
+
int test3(struct pt_regs *ctx)
33
+
{
34
+
return 0;
35
+
}
36
+
37
+
SEC("tp/bpf_trace/bpf_trace_printk")
38
+
int test4(struct pt_regs *ctx)
39
+
{
40
+
return 0;
41
+
}
+7
tools/testing/selftests/bpf/progs/percpu_alloc_array.c
+7
tools/testing/selftests/bpf/progs/percpu_alloc_array.c
···
71
71
}
72
72
73
73
int cpu0_field_d, sum_field_c;
74
+
int my_pid;
74
75
75
76
/* Summarize percpu data */
76
77
SEC("?fentry/bpf_fentry_test3")
···
81
80
int i, index = 0;
82
81
struct val_t *v;
83
82
struct elem *e;
83
+
84
+
if ((bpf_get_current_pid_tgid() >> 32) != my_pid)
85
+
return 0;
84
86
85
87
e = bpf_map_lookup_elem(&array, &index);
86
88
if (!e)
···
133
129
int i, index = 0;
134
130
struct val_t *v;
135
131
struct elem *e;
132
+
133
+
if ((bpf_get_current_pid_tgid() >> 32) != my_pid)
134
+
return 0;
136
135
137
136
e = bpf_map_lookup_elem(&array, &index);
138
137
if (!e)
+4
tools/testing/selftests/bpf/progs/percpu_alloc_cgrp_local_storage.c
+4
tools/testing/selftests/bpf/progs/percpu_alloc_cgrp_local_storage.c
···
70
70
}
71
71
72
72
int cpu0_field_d, sum_field_c;
73
+
int my_pid;
73
74
74
75
/* Summarize percpu data collection */
75
76
SEC("fentry/bpf_fentry_test3")
···
81
80
struct val_t *v;
82
81
struct elem *e;
83
82
int i;
83
+
84
+
if ((bpf_get_current_pid_tgid() >> 32) != my_pid)
85
+
return 0;
84
86
85
87
task = bpf_get_current_task_btf();
86
88
e = bpf_cgrp_storage_get(&cgrp, task->cgroups->dfl_cgrp, 0, 0);
+1
-1
tools/testing/selftests/bpf/progs/profiler.inc.h
+1
-1
tools/testing/selftests/bpf/progs/profiler.inc.h
···
609
609
}
610
610
611
611
SEC("tracepoint/syscalls/sys_enter_kill")
612
-
int tracepoint__syscalls__sys_enter_kill(struct trace_event_raw_sys_enter* ctx)
612
+
int tracepoint__syscalls__sys_enter_kill(struct syscall_trace_enter* ctx)
613
613
{
614
614
struct bpf_func_stats_ctx stats_ctx;
615
615
+39
tools/testing/selftests/bpf/progs/recvmsg_unix_prog.c
+39
tools/testing/selftests/bpf/progs/recvmsg_unix_prog.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
6
+
#include <string.h>
7
+
#include <bpf/bpf_helpers.h>
8
+
#include <bpf/bpf_core_read.h>
9
+
#include "bpf_kfuncs.h"
10
+
11
+
__u8 SERVUN_ADDRESS[] = "\0bpf_cgroup_unix_test";
12
+
13
+
SEC("cgroup/recvmsg_unix")
14
+
int recvmsg_unix_prog(struct bpf_sock_addr *ctx)
15
+
{
16
+
struct bpf_sock_addr_kern *sa_kern = bpf_cast_to_kern_ctx(ctx);
17
+
struct sockaddr_un *sa_kern_unaddr;
18
+
__u32 unaddrlen = offsetof(struct sockaddr_un, sun_path) +
19
+
sizeof(SERVUN_ADDRESS) - 1;
20
+
int ret;
21
+
22
+
ret = bpf_sock_addr_set_sun_path(sa_kern, SERVUN_ADDRESS,
23
+
sizeof(SERVUN_ADDRESS) - 1);
24
+
if (ret)
25
+
return 1;
26
+
27
+
if (sa_kern->uaddrlen != unaddrlen)
28
+
return 1;
29
+
30
+
sa_kern_unaddr = bpf_rdonly_cast(sa_kern->uaddr,
31
+
bpf_core_type_id_kernel(struct sockaddr_un));
32
+
if (memcmp(sa_kern_unaddr->sun_path, SERVUN_ADDRESS,
33
+
sizeof(SERVUN_ADDRESS) - 1) != 0)
34
+
return 1;
35
+
36
+
return 1;
37
+
}
38
+
39
+
char _license[] SEC("license") = "GPL";
+40
tools/testing/selftests/bpf/progs/sendmsg_unix_prog.c
+40
tools/testing/selftests/bpf/progs/sendmsg_unix_prog.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
3
+
4
+
#include "vmlinux.h"
5
+
6
+
#include <string.h>
7
+
#include <bpf/bpf_helpers.h>
8
+
#include <bpf/bpf_core_read.h>
9
+
#include "bpf_kfuncs.h"
10
+
11
+
__u8 SERVUN_REWRITE_ADDRESS[] = "\0bpf_cgroup_unix_test_rewrite";
12
+
13
+
SEC("cgroup/sendmsg_unix")
14
+
int sendmsg_unix_prog(struct bpf_sock_addr *ctx)
15
+
{
16
+
struct bpf_sock_addr_kern *sa_kern = bpf_cast_to_kern_ctx(ctx);
17
+
struct sockaddr_un *sa_kern_unaddr;
18
+
__u32 unaddrlen = offsetof(struct sockaddr_un, sun_path) +
19
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1;
20
+
int ret;
21
+
22
+
/* Rewrite destination. */
23
+
ret = bpf_sock_addr_set_sun_path(sa_kern, SERVUN_REWRITE_ADDRESS,
24
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1);
25
+
if (ret)
26
+
return 0;
27
+
28
+
if (sa_kern->uaddrlen != unaddrlen)
29
+
return 0;
30
+
31
+
sa_kern_unaddr = bpf_rdonly_cast(sa_kern->uaddr,
32
+
bpf_core_type_id_kernel(struct sockaddr_un));
33
+
if (memcmp(sa_kern_unaddr->sun_path, SERVUN_REWRITE_ADDRESS,
34
+
sizeof(SERVUN_REWRITE_ADDRESS) - 1) != 0)
35
+
return 0;
36
+
37
+
return 1;
38
+
}
39
+
40
+
char _license[] SEC("license") = "GPL";
+7
-2
tools/testing/selftests/bpf/progs/test_ldsx_insn.c
+7
-2
tools/testing/selftests/bpf/progs/test_ldsx_insn.c
···
6
6
#include <bpf/bpf_tracing.h>
7
7
8
8
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
9
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && __clang_major__ >= 18
9
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
10
+
defined(__TARGET_ARCH_s390)) && __clang_major__ >= 18
10
11
const volatile int skip = 0;
11
12
#else
12
13
const volatile int skip = 1;
···
105
104
"%[tmp_mark] = r1"
106
105
: [tmp_mark]"=r"(tmp_mark)
107
106
: [ctx]"r"(skb),
108
-
[off_mark]"i"(offsetof(struct __sk_buff, mark))
107
+
[off_mark]"i"(offsetof(struct __sk_buff, mark)
108
+
#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
109
+
+ sizeof(skb->mark) - 1
110
+
#endif
111
+
)
109
112
: "r1");
110
113
#else
111
114
tmp_mark = (char)skb->mark;
+61
tools/testing/selftests/bpf/progs/test_uprobe.c
+61
tools/testing/selftests/bpf/progs/test_uprobe.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/* Copyright (c) 2023 Hengqi Chen */
3
+
4
+
#include "vmlinux.h"
5
+
#include <bpf/bpf_helpers.h>
6
+
#include <bpf/bpf_tracing.h>
7
+
8
+
pid_t my_pid = 0;
9
+
10
+
int test1_result = 0;
11
+
int test2_result = 0;
12
+
int test3_result = 0;
13
+
int test4_result = 0;
14
+
15
+
SEC("uprobe/./liburandom_read.so:urandlib_api_sameoffset")
16
+
int BPF_UPROBE(test1)
17
+
{
18
+
pid_t pid = bpf_get_current_pid_tgid() >> 32;
19
+
20
+
if (pid != my_pid)
21
+
return 0;
22
+
23
+
test1_result = 1;
24
+
return 0;
25
+
}
26
+
27
+
SEC("uprobe/./liburandom_read.so:urandlib_api_sameoffset@LIBURANDOM_READ_1.0.0")
28
+
int BPF_UPROBE(test2)
29
+
{
30
+
pid_t pid = bpf_get_current_pid_tgid() >> 32;
31
+
32
+
if (pid != my_pid)
33
+
return 0;
34
+
35
+
test2_result = 1;
36
+
return 0;
37
+
}
38
+
39
+
SEC("uretprobe/./liburandom_read.so:urandlib_api_sameoffset@@LIBURANDOM_READ_2.0.0")
40
+
int BPF_URETPROBE(test3, int ret)
41
+
{
42
+
pid_t pid = bpf_get_current_pid_tgid() >> 32;
43
+
44
+
if (pid != my_pid)
45
+
return 0;
46
+
47
+
test3_result = ret;
48
+
return 0;
49
+
}
50
+
51
+
SEC("uprobe")
52
+
int BPF_UPROBE(test4)
53
+
{
54
+
pid_t pid = bpf_get_current_pid_tgid() >> 32;
55
+
56
+
if (pid != my_pid)
57
+
return 0;
58
+
59
+
test4_result = 1;
60
+
return 0;
61
+
}
+2
-2
tools/testing/selftests/bpf/progs/test_vmlinux.c
+2
-2
tools/testing/selftests/bpf/progs/test_vmlinux.c
···
16
16
bool fentry_called = false;
17
17
18
18
SEC("tp/syscalls/sys_enter_nanosleep")
19
-
int handle__tp(struct trace_event_raw_sys_enter *args)
19
+
int handle__tp(struct syscall_trace_enter *args)
20
20
{
21
21
struct __kernel_timespec *ts;
22
22
long tv_nsec;
23
23
24
-
if (args->id != __NR_nanosleep)
24
+
if (args->nr != __NR_nanosleep)
25
25
return 0;
26
26
27
27
ts = (void *)args->args[0];
+62
-1
tools/testing/selftests/bpf/progs/timer.c
+62
-1
tools/testing/selftests/bpf/progs/timer.c
···
51
51
__uint(max_entries, 1);
52
52
__type(key, int);
53
53
__type(value, struct elem);
54
-
} abs_timer SEC(".maps");
54
+
} abs_timer SEC(".maps"), soft_timer_pinned SEC(".maps"), abs_timer_pinned SEC(".maps");
55
55
56
56
__u64 bss_data;
57
57
__u64 abs_data;
···
59
59
__u64 ok;
60
60
__u64 callback_check = 52;
61
61
__u64 callback2_check = 52;
62
+
__u64 pinned_callback_check;
63
+
__s32 pinned_cpu;
62
64
63
65
#define ARRAY 1
64
66
#define HTAB 2
···
328
326
bpf_timer_start(timer, bpf_ktime_get_boot_ns() + 1000,
329
327
BPF_F_TIMER_ABS);
330
328
}
329
+
330
+
return 0;
331
+
}
332
+
333
+
/* callback for pinned timer */
334
+
static int timer_cb_pinned(void *map, int *key, struct bpf_timer *timer)
335
+
{
336
+
__s32 cpu = bpf_get_smp_processor_id();
337
+
338
+
if (cpu != pinned_cpu)
339
+
err |= 16384;
340
+
341
+
pinned_callback_check++;
342
+
return 0;
343
+
}
344
+
345
+
static void test_pinned_timer(bool soft)
346
+
{
347
+
int key = 0;
348
+
void *map;
349
+
struct bpf_timer *timer;
350
+
__u64 flags = BPF_F_TIMER_CPU_PIN;
351
+
__u64 start_time;
352
+
353
+
if (soft) {
354
+
map = &soft_timer_pinned;
355
+
start_time = 0;
356
+
} else {
357
+
map = &abs_timer_pinned;
358
+
start_time = bpf_ktime_get_boot_ns();
359
+
flags |= BPF_F_TIMER_ABS;
360
+
}
361
+
362
+
timer = bpf_map_lookup_elem(map, &key);
363
+
if (timer) {
364
+
if (bpf_timer_init(timer, map, CLOCK_BOOTTIME) != 0)
365
+
err |= 4096;
366
+
bpf_timer_set_callback(timer, timer_cb_pinned);
367
+
pinned_cpu = bpf_get_smp_processor_id();
368
+
bpf_timer_start(timer, start_time + 1000, flags);
369
+
} else {
370
+
err |= 8192;
371
+
}
372
+
}
373
+
374
+
SEC("fentry/bpf_fentry_test4")
375
+
int BPF_PROG2(test4, int, a)
376
+
{
377
+
bpf_printk("test4");
378
+
test_pinned_timer(true);
379
+
380
+
return 0;
381
+
}
382
+
383
+
SEC("fentry/bpf_fentry_test5")
384
+
int BPF_PROG2(test5, int, a)
385
+
{
386
+
bpf_printk("test5");
387
+
test_pinned_timer(false);
331
388
332
389
return 0;
333
390
}
+2
-1
tools/testing/selftests/bpf/progs/verifier_bswap.c
+2
-1
tools/testing/selftests/bpf/progs/verifier_bswap.c
···
5
5
#include "bpf_misc.h"
6
6
7
7
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
8
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || defined(__TARGET_ARCH_arm)) && \
8
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
9
+
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \
9
10
__clang_major__ >= 18
10
11
11
12
SEC("socket")
+2
-1
tools/testing/selftests/bpf/progs/verifier_gotol.c
+2
-1
tools/testing/selftests/bpf/progs/verifier_gotol.c
···
5
5
#include "bpf_misc.h"
6
6
7
7
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
8
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || defined(__TARGET_ARCH_arm)) && \
8
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
9
+
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \
9
10
__clang_major__ >= 18
10
11
11
12
SEC("socket")
+88
-63
tools/testing/selftests/bpf/progs/verifier_ldsx.c
+88
-63
tools/testing/selftests/bpf/progs/verifier_ldsx.c
···
5
5
#include "bpf_misc.h"
6
6
7
7
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
8
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || defined(__TARGET_ARCH_arm)) && \
8
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
9
+
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \
9
10
__clang_major__ >= 18
10
11
11
12
SEC("socket")
···
14
13
__success __success_unpriv __retval(-2)
15
14
__naked void ldsx_s8(void)
16
15
{
17
-
asm volatile (" \
18
-
r1 = 0x3fe; \
19
-
*(u64 *)(r10 - 8) = r1; \
20
-
r0 = *(s8 *)(r10 - 8); \
21
-
exit; \
22
-
" ::: __clobber_all);
16
+
asm volatile (
17
+
"r1 = 0x3fe;"
18
+
"*(u64 *)(r10 - 8) = r1;"
19
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
20
+
"r0 = *(s8 *)(r10 - 8);"
21
+
#else
22
+
"r0 = *(s8 *)(r10 - 1);"
23
+
#endif
24
+
"exit;"
25
+
::: __clobber_all);
23
26
}
24
27
25
28
SEC("socket")
···
31
26
__success __success_unpriv __retval(-2)
32
27
__naked void ldsx_s16(void)
33
28
{
34
-
asm volatile (" \
35
-
r1 = 0x3fffe; \
36
-
*(u64 *)(r10 - 8) = r1; \
37
-
r0 = *(s16 *)(r10 - 8); \
38
-
exit; \
39
-
" ::: __clobber_all);
29
+
asm volatile (
30
+
"r1 = 0x3fffe;"
31
+
"*(u64 *)(r10 - 8) = r1;"
32
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
33
+
"r0 = *(s16 *)(r10 - 8);"
34
+
#else
35
+
"r0 = *(s16 *)(r10 - 2);"
36
+
#endif
37
+
"exit;"
38
+
::: __clobber_all);
40
39
}
41
40
42
41
SEC("socket")
···
48
39
__success __success_unpriv __retval(-1)
49
40
__naked void ldsx_s32(void)
50
41
{
51
-
asm volatile (" \
52
-
r1 = 0xfffffffe; \
53
-
*(u64 *)(r10 - 8) = r1; \
54
-
r0 = *(s32 *)(r10 - 8); \
55
-
r0 >>= 1; \
56
-
exit; \
57
-
" ::: __clobber_all);
42
+
asm volatile (
43
+
"r1 = 0xfffffffe;"
44
+
"*(u64 *)(r10 - 8) = r1;"
45
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
46
+
"r0 = *(s32 *)(r10 - 8);"
47
+
#else
48
+
"r0 = *(s32 *)(r10 - 4);"
49
+
#endif
50
+
"r0 >>= 1;"
51
+
"exit;"
52
+
::: __clobber_all);
58
53
}
59
54
60
55
SEC("socket")
61
56
__description("LDSX, S8 range checking, privileged")
62
57
__log_level(2) __success __retval(1)
63
-
__msg("R1_w=scalar(smin=-128,smax=127)")
58
+
__msg("R1_w=scalar(smin=smin32=-128,smax=smax32=127)")
64
59
__naked void ldsx_s8_range_priv(void)
65
60
{
66
-
asm volatile (" \
67
-
call %[bpf_get_prandom_u32]; \
68
-
*(u64 *)(r10 - 8) = r0; \
69
-
r1 = *(s8 *)(r10 - 8); \
70
-
/* r1 with s8 range */ \
71
-
if r1 s> 0x7f goto l0_%=; \
72
-
if r1 s< -0x80 goto l0_%=; \
73
-
r0 = 1; \
74
-
l1_%=: \
75
-
exit; \
76
-
l0_%=: \
77
-
r0 = 2; \
78
-
goto l1_%=; \
79
-
" :
61
+
asm volatile (
62
+
"call %[bpf_get_prandom_u32];"
63
+
"*(u64 *)(r10 - 8) = r0;"
64
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
65
+
"r1 = *(s8 *)(r10 - 8);"
66
+
#else
67
+
"r1 = *(s8 *)(r10 - 1);"
68
+
#endif
69
+
/* r1 with s8 range */
70
+
"if r1 s> 0x7f goto l0_%=;"
71
+
"if r1 s< -0x80 goto l0_%=;"
72
+
"r0 = 1;"
73
+
"l1_%=:"
74
+
"exit;"
75
+
"l0_%=:"
76
+
"r0 = 2;"
77
+
"goto l1_%=;"
78
+
:
80
79
: __imm(bpf_get_prandom_u32)
81
80
: __clobber_all);
82
81
}
···
94
77
__success __success_unpriv __retval(1)
95
78
__naked void ldsx_s16_range(void)
96
79
{
97
-
asm volatile (" \
98
-
call %[bpf_get_prandom_u32]; \
99
-
*(u64 *)(r10 - 8) = r0; \
100
-
r1 = *(s16 *)(r10 - 8); \
101
-
/* r1 with s16 range */ \
102
-
if r1 s> 0x7fff goto l0_%=; \
103
-
if r1 s< -0x8000 goto l0_%=; \
104
-
r0 = 1; \
105
-
l1_%=: \
106
-
exit; \
107
-
l0_%=: \
108
-
r0 = 2; \
109
-
goto l1_%=; \
110
-
" :
80
+
asm volatile (
81
+
"call %[bpf_get_prandom_u32];"
82
+
"*(u64 *)(r10 - 8) = r0;"
83
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
84
+
"r1 = *(s16 *)(r10 - 8);"
85
+
#else
86
+
"r1 = *(s16 *)(r10 - 2);"
87
+
#endif
88
+
/* r1 with s16 range */
89
+
"if r1 s> 0x7fff goto l0_%=;"
90
+
"if r1 s< -0x8000 goto l0_%=;"
91
+
"r0 = 1;"
92
+
"l1_%=:"
93
+
"exit;"
94
+
"l0_%=:"
95
+
"r0 = 2;"
96
+
"goto l1_%=;"
97
+
:
111
98
: __imm(bpf_get_prandom_u32)
112
99
: __clobber_all);
113
100
}
···
121
100
__success __success_unpriv __retval(1)
122
101
__naked void ldsx_s32_range(void)
123
102
{
124
-
asm volatile (" \
125
-
call %[bpf_get_prandom_u32]; \
126
-
*(u64 *)(r10 - 8) = r0; \
127
-
r1 = *(s32 *)(r10 - 8); \
128
-
/* r1 with s16 range */ \
129
-
if r1 s> 0x7fffFFFF goto l0_%=; \
130
-
if r1 s< -0x80000000 goto l0_%=; \
131
-
r0 = 1; \
132
-
l1_%=: \
133
-
exit; \
134
-
l0_%=: \
135
-
r0 = 2; \
136
-
goto l1_%=; \
137
-
" :
103
+
asm volatile (
104
+
"call %[bpf_get_prandom_u32];"
105
+
"*(u64 *)(r10 - 8) = r0;"
106
+
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
107
+
"r1 = *(s32 *)(r10 - 8);"
108
+
#else
109
+
"r1 = *(s32 *)(r10 - 4);"
110
+
#endif
111
+
/* r1 with s16 range */
112
+
"if r1 s> 0x7fffFFFF goto l0_%=;"
113
+
"if r1 s< -0x80000000 goto l0_%=;"
114
+
"r0 = 1;"
115
+
"l1_%=:"
116
+
"exit;"
117
+
"l0_%=:"
118
+
"r0 = 2;"
119
+
"goto l1_%=;"
120
+
:
138
121
: __imm(bpf_get_prandom_u32)
139
122
: __clobber_all);
140
123
}
+2
-1
tools/testing/selftests/bpf/progs/verifier_movsx.c
+2
-1
tools/testing/selftests/bpf/progs/verifier_movsx.c
···
5
5
#include "bpf_misc.h"
6
6
7
7
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
8
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || defined(__TARGET_ARCH_arm)) && \
8
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
9
+
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \
9
10
__clang_major__ >= 18
10
11
11
12
SEC("socket")
+2
-1
tools/testing/selftests/bpf/progs/verifier_sdiv.c
+2
-1
tools/testing/selftests/bpf/progs/verifier_sdiv.c
···
5
5
#include "bpf_misc.h"
6
6
7
7
#if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \
8
-
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || defined(__TARGET_ARCH_arm)) && \
8
+
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \
9
+
defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \
9
10
__clang_major__ >= 18
10
11
11
12
SEC("socket")
+20
-2
tools/testing/selftests/bpf/progs/xsk_xdp_progs.c
+20
-2
tools/testing/selftests/bpf/progs/xsk_xdp_progs.c
···
3
3
4
4
#include <linux/bpf.h>
5
5
#include <bpf/bpf_helpers.h>
6
-
#include "xsk_xdp_metadata.h"
6
+
#include <linux/if_ether.h>
7
+
#include "xsk_xdp_common.h"
7
8
8
9
struct {
9
10
__uint(type, BPF_MAP_TYPE_XSKMAP);
10
-
__uint(max_entries, 1);
11
+
__uint(max_entries, 2);
11
12
__uint(key_size, sizeof(int));
12
13
__uint(value_size, sizeof(int));
13
14
} xsk SEC(".maps");
···
51
50
meta->count = count++;
52
51
53
52
return bpf_redirect_map(&xsk, 0, XDP_DROP);
53
+
}
54
+
55
+
SEC("xdp") int xsk_xdp_shared_umem(struct xdp_md *xdp)
56
+
{
57
+
void *data = (void *)(long)xdp->data;
58
+
void *data_end = (void *)(long)xdp->data_end;
59
+
struct ethhdr *eth = data;
60
+
61
+
if (eth + 1 > data_end)
62
+
return XDP_DROP;
63
+
64
+
/* Redirecting packets based on the destination MAC address */
65
+
idx = ((unsigned int)(eth->h_dest[5])) / 2;
66
+
if (idx > MAX_SOCKETS)
67
+
return XDP_DROP;
68
+
69
+
return bpf_redirect_map(&xsk, idx, XDP_DROP);
54
70
}
55
71
56
72
char _license[] SEC("license") = "GPL";
+2
-2
tools/testing/selftests/bpf/test_loader.c
+2
-2
tools/testing/selftests/bpf/test_loader.c
···
69
69
{
70
70
if (!tester->log_buf) {
71
71
tester->log_buf_sz = TEST_LOADER_LOG_BUF_SZ;
72
-
tester->log_buf = malloc(tester->log_buf_sz);
72
+
tester->log_buf = calloc(tester->log_buf_sz, 1);
73
73
if (!ASSERT_OK_PTR(tester->log_buf, "tester_log_buf"))
74
74
return -ENOMEM;
75
75
}
···
538
538
bool unpriv)
539
539
{
540
540
struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv;
541
-
struct bpf_program *tprog, *tprog_iter;
541
+
struct bpf_program *tprog = NULL, *tprog_iter;
542
542
struct test_spec *spec_iter;
543
543
struct cap_state caps = {};
544
544
struct bpf_object *tobj;
+1
-1
tools/testing/selftests/bpf/test_progs.c
+1
-1
tools/testing/selftests/bpf/test_progs.c
+2
tools/testing/selftests/bpf/test_progs.h
+2
tools/testing/selftests/bpf/test_progs.h
···
417
417
#define SYS_NANOSLEEP_KPROBE_NAME "__s390x_sys_nanosleep"
418
418
#elif defined(__aarch64__)
419
419
#define SYS_NANOSLEEP_KPROBE_NAME "__arm64_sys_nanosleep"
420
+
#elif defined(__riscv)
421
+
#define SYS_NANOSLEEP_KPROBE_NAME "__riscv_sys_nanosleep"
420
422
#else
421
423
#define SYS_NANOSLEEP_KPROBE_NAME "sys_nanosleep"
422
424
#endif
+13
-2
tools/testing/selftests/bpf/urandom_read.c
+13
-2
tools/testing/selftests/bpf/urandom_read.c
···
11
11
#define _SDT_HAS_SEMAPHORES 1
12
12
#include "sdt.h"
13
13
14
+
#define SHARED 1
15
+
#include "bpf/libbpf_internal.h"
16
+
14
17
#define SEC(name) __attribute__((section(name), used))
15
18
16
19
#define BUF_SIZE 256
···
24
21
void urandlib_read_with_sema(int iter_num, int iter_cnt, int read_sz);
25
22
void urandlib_read_without_sema(int iter_num, int iter_cnt, int read_sz);
26
23
24
+
int urandlib_api(void);
25
+
COMPAT_VERSION(urandlib_api_old, urandlib_api, LIBURANDOM_READ_1.0.0)
26
+
int urandlib_api_old(void);
27
+
int urandlib_api_sameoffset(void);
28
+
27
29
unsigned short urand_read_with_sema_semaphore SEC(".probes");
28
30
29
-
static __attribute__((noinline))
30
-
void urandom_read(int fd, int count)
31
+
static noinline void urandom_read(int fd, int count)
31
32
{
32
33
char buf[BUF_SIZE];
33
34
int i;
···
89
82
}
90
83
91
84
urandom_read(fd, count);
85
+
86
+
urandlib_api();
87
+
urandlib_api_old();
88
+
urandlib_api_sameoffset();
92
89
93
90
close(fd);
94
91
return 0;
+22
tools/testing/selftests/bpf/urandom_read_lib1.c
+22
tools/testing/selftests/bpf/urandom_read_lib1.c
···
3
3
#define _SDT_HAS_SEMAPHORES 1
4
4
#include "sdt.h"
5
5
6
+
#define SHARED 1
7
+
#include "bpf/libbpf_internal.h"
8
+
6
9
#define SEC(name) __attribute__((section(name), used))
7
10
8
11
unsigned short urandlib_read_with_sema_semaphore SEC(".probes");
···
13
10
void urandlib_read_with_sema(int iter_num, int iter_cnt, int read_sz)
14
11
{
15
12
STAP_PROBE3(urandlib, read_with_sema, iter_num, iter_cnt, read_sz);
13
+
}
14
+
15
+
COMPAT_VERSION(urandlib_api_v1, urandlib_api, LIBURANDOM_READ_1.0.0)
16
+
int urandlib_api_v1(void)
17
+
{
18
+
return 1;
19
+
}
20
+
21
+
DEFAULT_VERSION(urandlib_api_v2, urandlib_api, LIBURANDOM_READ_2.0.0)
22
+
int urandlib_api_v2(void)
23
+
{
24
+
return 2;
25
+
}
26
+
27
+
COMPAT_VERSION(urandlib_api_sameoffset, urandlib_api_sameoffset, LIBURANDOM_READ_1.0.0)
28
+
DEFAULT_VERSION(urandlib_api_sameoffset, urandlib_api_sameoffset, LIBURANDOM_READ_2.0.0)
29
+
int urandlib_api_sameoffset(void)
30
+
{
31
+
return 3;
16
32
}
+2
-2
tools/testing/selftests/bpf/xdp_features.c
+2
-2
tools/testing/selftests/bpf/xdp_features.c
···
360
360
static int dut_run(struct xdp_features *skel)
361
361
{
362
362
int flags = XDP_FLAGS_UPDATE_IF_NOEXIST | XDP_FLAGS_DRV_MODE;
363
-
int state, err, *sockfd, ctrl_sockfd, echo_sockfd;
363
+
int state, err = 0, *sockfd, ctrl_sockfd, echo_sockfd;
364
364
struct sockaddr_storage ctrl_addr;
365
-
pthread_t dut_thread;
365
+
pthread_t dut_thread = 0;
366
366
socklen_t addrlen;
367
367
368
368
sockfd = start_reuseport_server(AF_INET6, SOCK_STREAM, NULL,
+1
-1
tools/testing/selftests/bpf/xdp_hw_metadata.c
+1
-1
tools/testing/selftests/bpf/xdp_hw_metadata.c
+1
-2
tools/testing/selftests/bpf/xsk.c
+1
-2
tools/testing/selftests/bpf/xsk.c
···
442
442
bpf_map_delete_elem(map_fd, &index);
443
443
}
444
444
445
-
int xsk_update_xskmap(struct bpf_map *map, struct xsk_socket *xsk)
445
+
int xsk_update_xskmap(struct bpf_map *map, struct xsk_socket *xsk, u32 index)
446
446
{
447
447
int map_fd, sock_fd;
448
-
u32 index = 0;
449
448
450
449
map_fd = bpf_map__fd(map);
451
450
sock_fd = xsk_socket__fd(xsk);
+1
-1
tools/testing/selftests/bpf/xsk.h
+1
-1
tools/testing/selftests/bpf/xsk.h
···
204
204
205
205
int xsk_attach_xdp_program(struct bpf_program *prog, int ifindex, u32 xdp_flags);
206
206
void xsk_detach_xdp_program(int ifindex, u32 xdp_flags);
207
-
int xsk_update_xskmap(struct bpf_map *map, struct xsk_socket *xsk);
207
+
int xsk_update_xskmap(struct bpf_map *map, struct xsk_socket *xsk, u32 index);
208
208
void xsk_clear_xskmap(struct bpf_map *map);
209
209
bool xsk_is_in_mode(u32 ifindex, int mode);
210
210
+12
tools/testing/selftests/bpf/xsk_xdp_common.h
+12
tools/testing/selftests/bpf/xsk_xdp_common.h
-5
tools/testing/selftests/bpf/xsk_xdp_metadata.h
-5
tools/testing/selftests/bpf/xsk_xdp_metadata.h
+322
-201
tools/testing/selftests/bpf/xskxceiver.c
+322
-201
tools/testing/selftests/bpf/xskxceiver.c
···
80
80
#include <linux/if_ether.h>
81
81
#include <linux/mman.h>
82
82
#include <linux/netdev.h>
83
+
#include <linux/bitmap.h>
83
84
#include <arpa/inet.h>
84
85
#include <net/if.h>
85
86
#include <locale.h>
···
103
102
#include <bpf/bpf.h>
104
103
#include <linux/filter.h>
105
104
#include "../kselftest.h"
106
-
#include "xsk_xdp_metadata.h"
107
-
108
-
static const char *MAC1 = "\x00\x0A\x56\x9E\xEE\x62";
109
-
static const char *MAC2 = "\x00\x0A\x56\x9E\xEE\x61";
105
+
#include "xsk_xdp_common.h"
110
106
111
107
static bool opt_verbose;
112
108
static bool opt_print_tests;
···
157
159
ptr[i] = htonl(pkt_nb << 16 | (i + start));
158
160
}
159
161
160
-
static void gen_eth_hdr(struct ifobject *ifobject, struct ethhdr *eth_hdr)
162
+
static void gen_eth_hdr(struct xsk_socket_info *xsk, struct ethhdr *eth_hdr)
161
163
{
162
-
memcpy(eth_hdr->h_dest, ifobject->dst_mac, ETH_ALEN);
163
-
memcpy(eth_hdr->h_source, ifobject->src_mac, ETH_ALEN);
164
+
memcpy(eth_hdr->h_dest, xsk->dst_mac, ETH_ALEN);
165
+
memcpy(eth_hdr->h_source, xsk->src_mac, ETH_ALEN);
164
166
eth_hdr->h_proto = htons(ETH_P_LOOPBACK);
165
167
}
166
168
···
258
260
cfg.bind_flags = ifobject->bind_flags;
259
261
if (shared)
260
262
cfg.bind_flags |= XDP_SHARED_UMEM;
261
-
if (ifobject->pkt_stream && ifobject->mtu > MAX_ETH_PKT_SIZE)
263
+
if (ifobject->mtu > MAX_ETH_PKT_SIZE)
262
264
cfg.bind_flags |= XDP_USE_SG;
263
265
264
266
txr = ifobject->tx_on ? &xsk->tx : NULL;
···
427
429
if (i == 0) {
428
430
ifobj->rx_on = false;
429
431
ifobj->tx_on = true;
430
-
ifobj->pkt_stream = test->tx_pkt_stream_default;
431
432
} else {
432
433
ifobj->rx_on = true;
433
434
ifobj->tx_on = false;
434
-
ifobj->pkt_stream = test->rx_pkt_stream_default;
435
435
}
436
436
437
437
memset(ifobj->umem, 0, sizeof(*ifobj->umem));
···
439
443
for (j = 0; j < MAX_SOCKETS; j++) {
440
444
memset(&ifobj->xsk_arr[j], 0, sizeof(ifobj->xsk_arr[j]));
441
445
ifobj->xsk_arr[j].rxqsize = XSK_RING_CONS__DEFAULT_NUM_DESCS;
446
+
if (i == 0)
447
+
ifobj->xsk_arr[j].pkt_stream = test->tx_pkt_stream_default;
448
+
else
449
+
ifobj->xsk_arr[j].pkt_stream = test->rx_pkt_stream_default;
450
+
451
+
memcpy(ifobj->xsk_arr[j].src_mac, g_mac, ETH_ALEN);
452
+
memcpy(ifobj->xsk_arr[j].dst_mac, g_mac, ETH_ALEN);
453
+
ifobj->xsk_arr[j].src_mac[5] += ((j * 2) + 0);
454
+
ifobj->xsk_arr[j].dst_mac[5] += ((j * 2) + 1);
442
455
}
443
456
}
444
457
···
531
526
532
527
static void pkt_stream_reset(struct pkt_stream *pkt_stream)
533
528
{
534
-
if (pkt_stream)
529
+
if (pkt_stream) {
535
530
pkt_stream->current_pkt_nb = 0;
531
+
pkt_stream->nb_rx_pkts = 0;
532
+
}
536
533
}
537
534
538
535
static struct pkt *pkt_stream_get_next_tx_pkt(struct pkt_stream *pkt_stream)
···
564
557
565
558
static void pkt_stream_restore_default(struct test_spec *test)
566
559
{
567
-
struct pkt_stream *tx_pkt_stream = test->ifobj_tx->pkt_stream;
568
-
struct pkt_stream *rx_pkt_stream = test->ifobj_rx->pkt_stream;
560
+
struct pkt_stream *tx_pkt_stream = test->ifobj_tx->xsk->pkt_stream;
561
+
struct pkt_stream *rx_pkt_stream = test->ifobj_rx->xsk->pkt_stream;
569
562
570
563
if (tx_pkt_stream != test->tx_pkt_stream_default) {
571
-
pkt_stream_delete(test->ifobj_tx->pkt_stream);
572
-
test->ifobj_tx->pkt_stream = test->tx_pkt_stream_default;
564
+
pkt_stream_delete(test->ifobj_tx->xsk->pkt_stream);
565
+
test->ifobj_tx->xsk->pkt_stream = test->tx_pkt_stream_default;
573
566
}
574
567
575
568
if (rx_pkt_stream != test->rx_pkt_stream_default) {
576
-
pkt_stream_delete(test->ifobj_rx->pkt_stream);
577
-
test->ifobj_rx->pkt_stream = test->rx_pkt_stream_default;
569
+
pkt_stream_delete(test->ifobj_rx->xsk->pkt_stream);
570
+
test->ifobj_rx->xsk->pkt_stream = test->rx_pkt_stream_default;
578
571
}
579
572
}
580
573
···
634
627
return nb_frags;
635
628
}
636
629
637
-
static void pkt_set(struct xsk_umem_info *umem, struct pkt *pkt, int offset, u32 len)
630
+
static void pkt_set(struct pkt_stream *pkt_stream, struct pkt *pkt, int offset, u32 len)
638
631
{
639
632
pkt->offset = offset;
640
633
pkt->len = len;
641
-
if (len > MAX_ETH_JUMBO_SIZE)
634
+
if (len > MAX_ETH_JUMBO_SIZE) {
642
635
pkt->valid = false;
643
-
else
636
+
} else {
644
637
pkt->valid = true;
638
+
pkt_stream->nb_valid_entries++;
639
+
}
645
640
}
646
641
647
642
static u32 pkt_get_buffer_len(struct xsk_umem_info *umem, u32 len)
···
651
642
return ceil_u32(len, umem->frame_size) * umem->frame_size;
652
643
}
653
644
654
-
static struct pkt_stream *pkt_stream_generate(struct xsk_umem_info *umem, u32 nb_pkts, u32 pkt_len)
645
+
static struct pkt_stream *__pkt_stream_generate(u32 nb_pkts, u32 pkt_len, u32 nb_start, u32 nb_off)
655
646
{
656
647
struct pkt_stream *pkt_stream;
657
648
u32 i;
···
665
656
for (i = 0; i < nb_pkts; i++) {
666
657
struct pkt *pkt = &pkt_stream->pkts[i];
667
658
668
-
pkt_set(umem, pkt, 0, pkt_len);
669
-
pkt->pkt_nb = i;
659
+
pkt_set(pkt_stream, pkt, 0, pkt_len);
660
+
pkt->pkt_nb = nb_start + i * nb_off;
670
661
}
671
662
672
663
return pkt_stream;
673
664
}
674
665
675
-
static struct pkt_stream *pkt_stream_clone(struct xsk_umem_info *umem,
676
-
struct pkt_stream *pkt_stream)
666
+
static struct pkt_stream *pkt_stream_generate(u32 nb_pkts, u32 pkt_len)
677
667
{
678
-
return pkt_stream_generate(umem, pkt_stream->nb_pkts, pkt_stream->pkts[0].len);
668
+
return __pkt_stream_generate(nb_pkts, pkt_len, 0, 1);
669
+
}
670
+
671
+
static struct pkt_stream *pkt_stream_clone(struct pkt_stream *pkt_stream)
672
+
{
673
+
return pkt_stream_generate(pkt_stream->nb_pkts, pkt_stream->pkts[0].len);
679
674
}
680
675
681
676
static void pkt_stream_replace(struct test_spec *test, u32 nb_pkts, u32 pkt_len)
682
677
{
683
678
struct pkt_stream *pkt_stream;
684
679
685
-
pkt_stream = pkt_stream_generate(test->ifobj_tx->umem, nb_pkts, pkt_len);
686
-
test->ifobj_tx->pkt_stream = pkt_stream;
687
-
pkt_stream = pkt_stream_generate(test->ifobj_rx->umem, nb_pkts, pkt_len);
688
-
test->ifobj_rx->pkt_stream = pkt_stream;
680
+
pkt_stream = pkt_stream_generate(nb_pkts, pkt_len);
681
+
test->ifobj_tx->xsk->pkt_stream = pkt_stream;
682
+
pkt_stream = pkt_stream_generate(nb_pkts, pkt_len);
683
+
test->ifobj_rx->xsk->pkt_stream = pkt_stream;
689
684
}
690
685
691
686
static void __pkt_stream_replace_half(struct ifobject *ifobj, u32 pkt_len,
692
687
int offset)
693
688
{
694
-
struct xsk_umem_info *umem = ifobj->umem;
695
689
struct pkt_stream *pkt_stream;
696
690
u32 i;
697
691
698
-
pkt_stream = pkt_stream_clone(umem, ifobj->pkt_stream);
699
-
for (i = 1; i < ifobj->pkt_stream->nb_pkts; i += 2)
700
-
pkt_set(umem, &pkt_stream->pkts[i], offset, pkt_len);
692
+
pkt_stream = pkt_stream_clone(ifobj->xsk->pkt_stream);
693
+
for (i = 1; i < ifobj->xsk->pkt_stream->nb_pkts; i += 2)
694
+
pkt_set(pkt_stream, &pkt_stream->pkts[i], offset, pkt_len);
701
695
702
-
ifobj->pkt_stream = pkt_stream;
696
+
ifobj->xsk->pkt_stream = pkt_stream;
697
+
pkt_stream->nb_valid_entries /= 2;
703
698
}
704
699
705
700
static void pkt_stream_replace_half(struct test_spec *test, u32 pkt_len, int offset)
···
714
701
715
702
static void pkt_stream_receive_half(struct test_spec *test)
716
703
{
717
-
struct xsk_umem_info *umem = test->ifobj_rx->umem;
718
-
struct pkt_stream *pkt_stream = test->ifobj_tx->pkt_stream;
704
+
struct pkt_stream *pkt_stream = test->ifobj_tx->xsk->pkt_stream;
719
705
u32 i;
720
706
721
-
test->ifobj_rx->pkt_stream = pkt_stream_generate(umem, pkt_stream->nb_pkts,
722
-
pkt_stream->pkts[0].len);
723
-
pkt_stream = test->ifobj_rx->pkt_stream;
707
+
test->ifobj_rx->xsk->pkt_stream = pkt_stream_generate(pkt_stream->nb_pkts,
708
+
pkt_stream->pkts[0].len);
709
+
pkt_stream = test->ifobj_rx->xsk->pkt_stream;
724
710
for (i = 1; i < pkt_stream->nb_pkts; i += 2)
725
711
pkt_stream->pkts[i].valid = false;
712
+
713
+
pkt_stream->nb_valid_entries /= 2;
714
+
}
715
+
716
+
static void pkt_stream_even_odd_sequence(struct test_spec *test)
717
+
{
718
+
struct pkt_stream *pkt_stream;
719
+
u32 i;
720
+
721
+
for (i = 0; i < test->nb_sockets; i++) {
722
+
pkt_stream = test->ifobj_tx->xsk_arr[i].pkt_stream;
723
+
pkt_stream = __pkt_stream_generate(pkt_stream->nb_pkts / 2,
724
+
pkt_stream->pkts[0].len, i, 2);
725
+
test->ifobj_tx->xsk_arr[i].pkt_stream = pkt_stream;
726
+
727
+
pkt_stream = test->ifobj_rx->xsk_arr[i].pkt_stream;
728
+
pkt_stream = __pkt_stream_generate(pkt_stream->nb_pkts / 2,
729
+
pkt_stream->pkts[0].len, i, 2);
730
+
test->ifobj_rx->xsk_arr[i].pkt_stream = pkt_stream;
731
+
}
726
732
}
727
733
728
734
static u64 pkt_get_addr(struct pkt *pkt, struct xsk_umem_info *umem)
···
756
724
pkt_stream->current_pkt_nb--;
757
725
}
758
726
759
-
static void pkt_generate(struct ifobject *ifobject, u64 addr, u32 len, u32 pkt_nb,
760
-
u32 bytes_written)
727
+
static void pkt_generate(struct xsk_socket_info *xsk, struct xsk_umem_info *umem, u64 addr, u32 len,
728
+
u32 pkt_nb, u32 bytes_written)
761
729
{
762
-
void *data = xsk_umem__get_data(ifobject->umem->buffer, addr);
730
+
void *data = xsk_umem__get_data(umem->buffer, addr);
763
731
764
732
if (len < MIN_PKT_SIZE)
765
733
return;
766
734
767
735
if (!bytes_written) {
768
-
gen_eth_hdr(ifobject, data);
736
+
gen_eth_hdr(xsk, data);
769
737
770
738
len -= PKT_HDR_SIZE;
771
739
data += PKT_HDR_SIZE;
···
815
783
816
784
if (pkt->valid && pkt->len > pkt_stream->max_pkt_len)
817
785
pkt_stream->max_pkt_len = pkt->len;
786
+
787
+
if (pkt->valid)
788
+
pkt_stream->nb_valid_entries++;
789
+
818
790
pkt_nb++;
819
791
}
820
792
···
832
796
struct pkt_stream *pkt_stream;
833
797
834
798
pkt_stream = __pkt_stream_generate_custom(test->ifobj_tx, pkts, nb_pkts, true);
835
-
test->ifobj_tx->pkt_stream = pkt_stream;
799
+
test->ifobj_tx->xsk->pkt_stream = pkt_stream;
836
800
837
801
pkt_stream = __pkt_stream_generate_custom(test->ifobj_rx, pkts, nb_pkts, false);
838
-
test->ifobj_rx->pkt_stream = pkt_stream;
802
+
test->ifobj_rx->xsk->pkt_stream = pkt_stream;
839
803
}
840
804
841
805
static void pkt_print_data(u32 *data, u32 cnt)
···
1040
1004
return TEST_PASS;
1041
1005
}
1042
1006
1043
-
static int receive_pkts(struct test_spec *test, struct pollfd *fds)
1007
+
static int __receive_pkts(struct test_spec *test, struct xsk_socket_info *xsk)
1044
1008
{
1045
-
struct timeval tv_end, tv_now, tv_timeout = {THREAD_TMOUT, 0};
1046
-
struct pkt_stream *pkt_stream = test->ifobj_rx->pkt_stream;
1047
-
struct xsk_socket_info *xsk = test->ifobj_rx->xsk;
1009
+
u32 frags_processed = 0, nb_frags = 0, pkt_len = 0;
1048
1010
u32 idx_rx = 0, idx_fq = 0, rcvd, pkts_sent = 0;
1011
+
struct pkt_stream *pkt_stream = xsk->pkt_stream;
1049
1012
struct ifobject *ifobj = test->ifobj_rx;
1050
1013
struct xsk_umem_info *umem = xsk->umem;
1014
+
struct pollfd fds = { };
1051
1015
struct pkt *pkt;
1016
+
u64 first_addr = 0;
1052
1017
int ret;
1053
1018
1054
-
ret = gettimeofday(&tv_now, NULL);
1019
+
fds.fd = xsk_socket__fd(xsk->xsk);
1020
+
fds.events = POLLIN;
1021
+
1022
+
ret = kick_rx(xsk);
1055
1023
if (ret)
1056
-
exit_with_error(errno);
1057
-
timeradd(&tv_now, &tv_timeout, &tv_end);
1024
+
return TEST_FAILURE;
1058
1025
1059
-
pkt = pkt_stream_get_next_rx_pkt(pkt_stream, &pkts_sent);
1060
-
while (pkt) {
1061
-
u32 frags_processed = 0, nb_frags = 0, pkt_len = 0;
1062
-
u64 first_addr;
1063
-
1064
-
ret = gettimeofday(&tv_now, NULL);
1065
-
if (ret)
1066
-
exit_with_error(errno);
1067
-
if (timercmp(&tv_now, &tv_end, >)) {
1068
-
ksft_print_msg("ERROR: [%s] Receive loop timed out\n", __func__);
1026
+
if (ifobj->use_poll) {
1027
+
ret = poll(&fds, 1, POLL_TMOUT);
1028
+
if (ret < 0)
1069
1029
return TEST_FAILURE;
1030
+
1031
+
if (!ret) {
1032
+
if (!is_umem_valid(test->ifobj_tx))
1033
+
return TEST_PASS;
1034
+
1035
+
ksft_print_msg("ERROR: [%s] Poll timed out\n", __func__);
1036
+
return TEST_CONTINUE;
1070
1037
}
1071
1038
1072
-
ret = kick_rx(xsk);
1073
-
if (ret)
1074
-
return TEST_FAILURE;
1039
+
if (!(fds.revents & POLLIN))
1040
+
return TEST_CONTINUE;
1041
+
}
1075
1042
1076
-
if (ifobj->use_poll) {
1077
-
ret = poll(fds, 1, POLL_TMOUT);
1078
-
if (ret < 0)
1079
-
return TEST_FAILURE;
1043
+
rcvd = xsk_ring_cons__peek(&xsk->rx, BATCH_SIZE, &idx_rx);
1044
+
if (!rcvd)
1045
+
return TEST_CONTINUE;
1080
1046
1081
-
if (!ret) {
1082
-
if (!is_umem_valid(test->ifobj_tx))
1083
-
return TEST_PASS;
1084
-
1085
-
ksft_print_msg("ERROR: [%s] Poll timed out\n", __func__);
1086
-
return TEST_FAILURE;
1047
+
if (ifobj->use_fill_ring) {
1048
+
ret = xsk_ring_prod__reserve(&umem->fq, rcvd, &idx_fq);
1049
+
while (ret != rcvd) {
1050
+
if (xsk_ring_prod__needs_wakeup(&umem->fq)) {
1051
+
ret = poll(&fds, 1, POLL_TMOUT);
1052
+
if (ret < 0)
1053
+
return TEST_FAILURE;
1087
1054
}
1088
-
1089
-
if (!(fds->revents & POLLIN))
1090
-
continue;
1091
-
}
1092
-
1093
-
rcvd = xsk_ring_cons__peek(&xsk->rx, BATCH_SIZE, &idx_rx);
1094
-
if (!rcvd)
1095
-
continue;
1096
-
1097
-
if (ifobj->use_fill_ring) {
1098
1055
ret = xsk_ring_prod__reserve(&umem->fq, rcvd, &idx_fq);
1099
-
while (ret != rcvd) {
1100
-
if (xsk_ring_prod__needs_wakeup(&umem->fq)) {
1101
-
ret = poll(fds, 1, POLL_TMOUT);
1102
-
if (ret < 0)
1103
-
return TEST_FAILURE;
1104
-
}
1105
-
ret = xsk_ring_prod__reserve(&umem->fq, rcvd, &idx_fq);
1106
-
}
1107
1056
}
1057
+
}
1108
1058
1109
-
while (frags_processed < rcvd) {
1110
-
const struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
1111
-
u64 addr = desc->addr, orig;
1059
+
while (frags_processed < rcvd) {
1060
+
const struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
1061
+
u64 addr = desc->addr, orig;
1112
1062
1113
-
orig = xsk_umem__extract_addr(addr);
1114
-
addr = xsk_umem__add_offset_to_addr(addr);
1063
+
orig = xsk_umem__extract_addr(addr);
1064
+
addr = xsk_umem__add_offset_to_addr(addr);
1115
1065
1066
+
if (!nb_frags) {
1067
+
pkt = pkt_stream_get_next_rx_pkt(pkt_stream, &pkts_sent);
1116
1068
if (!pkt) {
1117
1069
ksft_print_msg("[%s] received too many packets addr: %lx len %u\n",
1118
1070
__func__, addr, desc->len);
1119
1071
return TEST_FAILURE;
1120
1072
}
1121
-
1122
-
print_verbose("Rx: addr: %lx len: %u options: %u pkt_nb: %u valid: %u\n",
1123
-
addr, desc->len, desc->options, pkt->pkt_nb, pkt->valid);
1124
-
1125
-
if (!is_frag_valid(umem, addr, desc->len, pkt->pkt_nb, pkt_len) ||
1126
-
!is_offset_correct(umem, pkt, addr) ||
1127
-
(ifobj->use_metadata && !is_metadata_correct(pkt, umem->buffer, addr)))
1128
-
return TEST_FAILURE;
1129
-
1130
-
if (!nb_frags++)
1131
-
first_addr = addr;
1132
-
frags_processed++;
1133
-
pkt_len += desc->len;
1134
-
if (ifobj->use_fill_ring)
1135
-
*xsk_ring_prod__fill_addr(&umem->fq, idx_fq++) = orig;
1136
-
1137
-
if (pkt_continues(desc->options))
1138
-
continue;
1139
-
1140
-
/* The complete packet has been received */
1141
-
if (!is_pkt_valid(pkt, umem->buffer, first_addr, pkt_len) ||
1142
-
!is_offset_correct(umem, pkt, addr))
1143
-
return TEST_FAILURE;
1144
-
1145
-
pkt = pkt_stream_get_next_rx_pkt(pkt_stream, &pkts_sent);
1146
-
nb_frags = 0;
1147
-
pkt_len = 0;
1148
1073
}
1149
1074
1150
-
if (nb_frags) {
1151
-
/* In the middle of a packet. Start over from beginning of packet. */
1152
-
idx_rx -= nb_frags;
1153
-
xsk_ring_cons__cancel(&xsk->rx, nb_frags);
1154
-
if (ifobj->use_fill_ring) {
1155
-
idx_fq -= nb_frags;
1156
-
xsk_ring_prod__cancel(&umem->fq, nb_frags);
1157
-
}
1158
-
frags_processed -= nb_frags;
1159
-
}
1075
+
print_verbose("Rx: addr: %lx len: %u options: %u pkt_nb: %u valid: %u\n",
1076
+
addr, desc->len, desc->options, pkt->pkt_nb, pkt->valid);
1160
1077
1078
+
if (!is_frag_valid(umem, addr, desc->len, pkt->pkt_nb, pkt_len) ||
1079
+
!is_offset_correct(umem, pkt, addr) || (ifobj->use_metadata &&
1080
+
!is_metadata_correct(pkt, umem->buffer, addr)))
1081
+
return TEST_FAILURE;
1082
+
1083
+
if (!nb_frags++)
1084
+
first_addr = addr;
1085
+
frags_processed++;
1086
+
pkt_len += desc->len;
1161
1087
if (ifobj->use_fill_ring)
1162
-
xsk_ring_prod__submit(&umem->fq, frags_processed);
1163
-
if (ifobj->release_rx)
1164
-
xsk_ring_cons__release(&xsk->rx, frags_processed);
1088
+
*xsk_ring_prod__fill_addr(&umem->fq, idx_fq++) = orig;
1165
1089
1166
-
pthread_mutex_lock(&pacing_mutex);
1167
-
pkts_in_flight -= pkts_sent;
1168
-
pthread_mutex_unlock(&pacing_mutex);
1169
-
pkts_sent = 0;
1090
+
if (pkt_continues(desc->options))
1091
+
continue;
1092
+
1093
+
/* The complete packet has been received */
1094
+
if (!is_pkt_valid(pkt, umem->buffer, first_addr, pkt_len) ||
1095
+
!is_offset_correct(umem, pkt, addr))
1096
+
return TEST_FAILURE;
1097
+
1098
+
pkt_stream->nb_rx_pkts++;
1099
+
nb_frags = 0;
1100
+
pkt_len = 0;
1101
+
}
1102
+
1103
+
if (nb_frags) {
1104
+
/* In the middle of a packet. Start over from beginning of packet. */
1105
+
idx_rx -= nb_frags;
1106
+
xsk_ring_cons__cancel(&xsk->rx, nb_frags);
1107
+
if (ifobj->use_fill_ring) {
1108
+
idx_fq -= nb_frags;
1109
+
xsk_ring_prod__cancel(&umem->fq, nb_frags);
1110
+
}
1111
+
frags_processed -= nb_frags;
1112
+
}
1113
+
1114
+
if (ifobj->use_fill_ring)
1115
+
xsk_ring_prod__submit(&umem->fq, frags_processed);
1116
+
if (ifobj->release_rx)
1117
+
xsk_ring_cons__release(&xsk->rx, frags_processed);
1118
+
1119
+
pthread_mutex_lock(&pacing_mutex);
1120
+
pkts_in_flight -= pkts_sent;
1121
+
pthread_mutex_unlock(&pacing_mutex);
1122
+
pkts_sent = 0;
1123
+
1124
+
return TEST_CONTINUE;
1125
+
}
1126
+
1127
+
bool all_packets_received(struct test_spec *test, struct xsk_socket_info *xsk, u32 sock_num,
1128
+
unsigned long *bitmap)
1129
+
{
1130
+
struct pkt_stream *pkt_stream = xsk->pkt_stream;
1131
+
1132
+
if (!pkt_stream) {
1133
+
__set_bit(sock_num, bitmap);
1134
+
return false;
1135
+
}
1136
+
1137
+
if (pkt_stream->nb_rx_pkts == pkt_stream->nb_valid_entries) {
1138
+
__set_bit(sock_num, bitmap);
1139
+
if (bitmap_full(bitmap, test->nb_sockets))
1140
+
return true;
1141
+
}
1142
+
1143
+
return false;
1144
+
}
1145
+
1146
+
static int receive_pkts(struct test_spec *test)
1147
+
{
1148
+
struct timeval tv_end, tv_now, tv_timeout = {THREAD_TMOUT, 0};
1149
+
DECLARE_BITMAP(bitmap, test->nb_sockets);
1150
+
struct xsk_socket_info *xsk;
1151
+
u32 sock_num = 0;
1152
+
int res, ret;
1153
+
1154
+
ret = gettimeofday(&tv_now, NULL);
1155
+
if (ret)
1156
+
exit_with_error(errno);
1157
+
1158
+
timeradd(&tv_now, &tv_timeout, &tv_end);
1159
+
1160
+
while (1) {
1161
+
xsk = &test->ifobj_rx->xsk_arr[sock_num];
1162
+
1163
+
if ((all_packets_received(test, xsk, sock_num, bitmap)))
1164
+
break;
1165
+
1166
+
res = __receive_pkts(test, xsk);
1167
+
if (!(res == TEST_PASS || res == TEST_CONTINUE))
1168
+
return res;
1169
+
1170
+
ret = gettimeofday(&tv_now, NULL);
1171
+
if (ret)
1172
+
exit_with_error(errno);
1173
+
1174
+
if (timercmp(&tv_now, &tv_end, >)) {
1175
+
ksft_print_msg("ERROR: [%s] Receive loop timed out\n", __func__);
1176
+
return TEST_FAILURE;
1177
+
}
1178
+
sock_num = (sock_num + 1) % test->nb_sockets;
1170
1179
}
1171
1180
1172
1181
return TEST_PASS;
1173
1182
}
1174
1183
1175
-
static int __send_pkts(struct ifobject *ifobject, struct pollfd *fds, bool timeout)
1184
+
static int __send_pkts(struct ifobject *ifobject, struct xsk_socket_info *xsk, bool timeout)
1176
1185
{
1177
1186
u32 i, idx = 0, valid_pkts = 0, valid_frags = 0, buffer_len;
1178
-
struct pkt_stream *pkt_stream = ifobject->pkt_stream;
1179
-
struct xsk_socket_info *xsk = ifobject->xsk;
1187
+
struct pkt_stream *pkt_stream = xsk->pkt_stream;
1180
1188
struct xsk_umem_info *umem = ifobject->umem;
1181
1189
bool use_poll = ifobject->use_poll;
1190
+
struct pollfd fds = { };
1182
1191
int ret;
1183
1192
1184
1193
buffer_len = pkt_get_buffer_len(umem, pkt_stream->max_pkt_len);
···
1235
1154
return TEST_CONTINUE;
1236
1155
}
1237
1156
1157
+
fds.fd = xsk_socket__fd(xsk->xsk);
1158
+
fds.events = POLLOUT;
1159
+
1238
1160
while (xsk_ring_prod__reserve(&xsk->tx, BATCH_SIZE, &idx) < BATCH_SIZE) {
1239
1161
if (use_poll) {
1240
-
ret = poll(fds, 1, POLL_TMOUT);
1162
+
ret = poll(&fds, 1, POLL_TMOUT);
1241
1163
if (timeout) {
1242
1164
if (ret < 0) {
1243
1165
ksft_print_msg("ERROR: [%s] Poll error %d\n",
···
1291
1207
tx_desc->options = 0;
1292
1208
}
1293
1209
if (pkt->valid)
1294
-
pkt_generate(ifobject, tx_desc->addr, tx_desc->len, pkt->pkt_nb,
1210
+
pkt_generate(xsk, umem, tx_desc->addr, tx_desc->len, pkt->pkt_nb,
1295
1211
bytes_written);
1296
1212
bytes_written += tx_desc->len;
1297
1213
···
1319
1235
xsk->outstanding_tx += valid_frags;
1320
1236
1321
1237
if (use_poll) {
1322
-
ret = poll(fds, 1, POLL_TMOUT);
1238
+
ret = poll(&fds, 1, POLL_TMOUT);
1323
1239
if (ret <= 0) {
1324
1240
if (ret == 0 && timeout)
1325
1241
return TEST_PASS;
···
1365
1281
return TEST_PASS;
1366
1282
}
1367
1283
1284
+
bool all_packets_sent(struct test_spec *test, unsigned long *bitmap)
1285
+
{
1286
+
return bitmap_full(bitmap, test->nb_sockets);
1287
+
}
1288
+
1368
1289
static int send_pkts(struct test_spec *test, struct ifobject *ifobject)
1369
1290
{
1370
-
struct pkt_stream *pkt_stream = ifobject->pkt_stream;
1371
1291
bool timeout = !is_umem_valid(test->ifobj_rx);
1372
-
struct pollfd fds = { };
1373
-
u32 ret;
1292
+
DECLARE_BITMAP(bitmap, test->nb_sockets);
1293
+
u32 i, ret;
1374
1294
1375
-
fds.fd = xsk_socket__fd(ifobject->xsk->xsk);
1376
-
fds.events = POLLOUT;
1295
+
while (!(all_packets_sent(test, bitmap))) {
1296
+
for (i = 0; i < test->nb_sockets; i++) {
1297
+
struct pkt_stream *pkt_stream;
1377
1298
1378
-
while (pkt_stream->current_pkt_nb < pkt_stream->nb_pkts) {
1379
-
ret = __send_pkts(ifobject, &fds, timeout);
1380
-
if (ret == TEST_CONTINUE && !test->fail)
1381
-
continue;
1382
-
if ((ret || test->fail) && !timeout)
1383
-
return TEST_FAILURE;
1384
-
if (ret == TEST_PASS && timeout)
1385
-
return ret;
1299
+
pkt_stream = ifobject->xsk_arr[i].pkt_stream;
1300
+
if (!pkt_stream || pkt_stream->current_pkt_nb >= pkt_stream->nb_pkts) {
1301
+
__set_bit(i, bitmap);
1302
+
continue;
1303
+
}
1304
+
ret = __send_pkts(ifobject, &ifobject->xsk_arr[i], timeout);
1305
+
if (ret == TEST_CONTINUE && !test->fail)
1306
+
continue;
1307
+
1308
+
if ((ret || test->fail) && !timeout)
1309
+
return TEST_FAILURE;
1310
+
1311
+
if (ret == TEST_PASS && timeout)
1312
+
return ret;
1313
+
1314
+
ret = wait_for_tx_completion(&ifobject->xsk_arr[i]);
1315
+
if (ret)
1316
+
return TEST_FAILURE;
1317
+
}
1386
1318
}
1387
1319
1388
-
return wait_for_tx_completion(ifobject->xsk);
1320
+
return TEST_PASS;
1389
1321
}
1390
1322
1391
1323
static int get_xsk_stats(struct xsk_socket *xsk, struct xdp_statistics *stats)
···
1447
1347
* packet being invalid). Since the last packet may or may not have
1448
1348
* been dropped already, both outcomes must be allowed.
1449
1349
*/
1450
-
if (stats.rx_dropped == ifobject->pkt_stream->nb_pkts / 2 ||
1451
-
stats.rx_dropped == ifobject->pkt_stream->nb_pkts / 2 - 1)
1350
+
if (stats.rx_dropped == ifobject->xsk->pkt_stream->nb_pkts / 2 ||
1351
+
stats.rx_dropped == ifobject->xsk->pkt_stream->nb_pkts / 2 - 1)
1452
1352
return TEST_PASS;
1453
1353
1454
1354
return TEST_FAILURE;
···
1512
1412
return TEST_FAILURE;
1513
1413
}
1514
1414
1515
-
if (stats.tx_invalid_descs != ifobject->pkt_stream->nb_pkts / 2) {
1415
+
if (stats.tx_invalid_descs != ifobject->xsk->pkt_stream->nb_pkts / 2) {
1516
1416
ksft_print_msg("[%s] tx_invalid_descs incorrect. Got [%u] expected [%u]\n",
1517
-
__func__, stats.tx_invalid_descs, ifobject->pkt_stream->nb_pkts);
1417
+
__func__, stats.tx_invalid_descs,
1418
+
ifobject->xsk->pkt_stream->nb_pkts);
1518
1419
return TEST_FAILURE;
1519
1420
}
1520
1421
···
1607
1506
LIBBPF_OPTS(bpf_xdp_query_opts, opts);
1608
1507
void *bufs;
1609
1508
int ret;
1509
+
u32 i;
1610
1510
1611
1511
if (ifobject->umem->unaligned_mode)
1612
1512
mmap_flags |= MAP_HUGETLB | MAP_HUGE_2MB;
···
1630
1528
if (!ifobject->rx_on)
1631
1529
return;
1632
1530
1633
-
xsk_populate_fill_ring(ifobject->umem, ifobject->pkt_stream, ifobject->use_fill_ring);
1531
+
xsk_populate_fill_ring(ifobject->umem, ifobject->xsk->pkt_stream, ifobject->use_fill_ring);
1634
1532
1635
-
ret = xsk_update_xskmap(ifobject->xskmap, ifobject->xsk->xsk);
1636
-
if (ret)
1637
-
exit_with_error(errno);
1533
+
for (i = 0; i < test->nb_sockets; i++) {
1534
+
ifobject->xsk = &ifobject->xsk_arr[i];
1535
+
ret = xsk_update_xskmap(ifobject->xskmap, ifobject->xsk->xsk, i);
1536
+
if (ret)
1537
+
exit_with_error(errno);
1538
+
}
1638
1539
}
1639
1540
1640
1541
static void *worker_testapp_validate_tx(void *arg)
···
1667
1562
{
1668
1563
struct test_spec *test = (struct test_spec *)arg;
1669
1564
struct ifobject *ifobject = test->ifobj_rx;
1670
-
struct pollfd fds = { };
1671
1565
int err;
1672
1566
1673
1567
if (test->current_step == 1) {
1674
1568
thread_common_ops(test, ifobject);
1675
1569
} else {
1676
1570
xsk_clear_xskmap(ifobject->xskmap);
1677
-
err = xsk_update_xskmap(ifobject->xskmap, ifobject->xsk->xsk);
1571
+
err = xsk_update_xskmap(ifobject->xskmap, ifobject->xsk->xsk, 0);
1678
1572
if (err) {
1679
1573
ksft_print_msg("Error: Failed to update xskmap, error %s\n",
1680
1574
strerror(-err));
···
1681
1577
}
1682
1578
}
1683
1579
1684
-
fds.fd = xsk_socket__fd(ifobject->xsk->xsk);
1685
-
fds.events = POLLIN;
1686
-
1687
1580
pthread_barrier_wait(&barr);
1688
1581
1689
-
err = receive_pkts(test, &fds);
1582
+
err = receive_pkts(test);
1690
1583
1691
1584
if (!err && ifobject->validation_func)
1692
1585
err = ifobject->validation_func(ifobject);
···
1792
1691
if (ifobj2) {
1793
1692
if (pthread_barrier_init(&barr, NULL, 2))
1794
1693
exit_with_error(errno);
1795
-
pkt_stream_reset(ifobj2->pkt_stream);
1694
+
pkt_stream_reset(ifobj2->xsk->pkt_stream);
1796
1695
}
1797
1696
1798
1697
test->current_step++;
1799
-
pkt_stream_reset(ifobj1->pkt_stream);
1698
+
pkt_stream_reset(ifobj1->xsk->pkt_stream);
1800
1699
pkts_in_flight = 0;
1801
1700
1802
1701
signal(SIGUSR1, handler);
···
1820
1719
pthread_join(t0, NULL);
1821
1720
1822
1721
if (test->total_steps == test->current_step || test->fail) {
1722
+
u32 i;
1723
+
1823
1724
if (ifobj2)
1824
-
xsk_socket__delete(ifobj2->xsk->xsk);
1825
-
xsk_socket__delete(ifobj1->xsk->xsk);
1725
+
for (i = 0; i < test->nb_sockets; i++)
1726
+
xsk_socket__delete(ifobj2->xsk_arr[i].xsk);
1727
+
1728
+
for (i = 0; i < test->nb_sockets; i++)
1729
+
xsk_socket__delete(ifobj1->xsk_arr[i].xsk);
1730
+
1826
1731
testapp_clean_xsk_umem(ifobj1);
1827
1732
if (ifobj2 && !ifobj2->shared_umem)
1828
1733
testapp_clean_xsk_umem(ifobj2);
···
1900
1793
return res;
1901
1794
}
1902
1795
1903
-
static int swap_xsk_resources(struct ifobject *ifobj_tx, struct ifobject *ifobj_rx)
1796
+
static int swap_xsk_resources(struct test_spec *test)
1904
1797
{
1905
1798
int ret;
1906
1799
1907
-
xsk_socket__delete(ifobj_tx->xsk->xsk);
1908
-
xsk_socket__delete(ifobj_rx->xsk->xsk);
1909
-
ifobj_tx->xsk = &ifobj_tx->xsk_arr[1];
1910
-
ifobj_rx->xsk = &ifobj_rx->xsk_arr[1];
1800
+
test->ifobj_tx->xsk_arr[0].pkt_stream = NULL;
1801
+
test->ifobj_rx->xsk_arr[0].pkt_stream = NULL;
1802
+
test->ifobj_tx->xsk_arr[1].pkt_stream = test->tx_pkt_stream_default;
1803
+
test->ifobj_rx->xsk_arr[1].pkt_stream = test->rx_pkt_stream_default;
1804
+
test->ifobj_tx->xsk = &test->ifobj_tx->xsk_arr[1];
1805
+
test->ifobj_rx->xsk = &test->ifobj_rx->xsk_arr[1];
1911
1806
1912
-
ret = xsk_update_xskmap(ifobj_rx->xskmap, ifobj_rx->xsk->xsk);
1807
+
ret = xsk_update_xskmap(test->ifobj_rx->xskmap, test->ifobj_rx->xsk->xsk, 0);
1913
1808
if (ret)
1914
1809
return TEST_FAILURE;
1915
1810
···
1925
1816
if (testapp_validate_traffic(test))
1926
1817
return TEST_FAILURE;
1927
1818
1928
-
if (swap_xsk_resources(test->ifobj_tx, test->ifobj_rx))
1819
+
if (swap_xsk_resources(test))
1929
1820
return TEST_FAILURE;
1930
1821
return testapp_validate_traffic(test);
1931
1822
}
···
1961
1852
static int testapp_stats_rx_full(struct test_spec *test)
1962
1853
{
1963
1854
pkt_stream_replace(test, DEFAULT_UMEM_BUFFERS + DEFAULT_UMEM_BUFFERS / 2, MIN_PKT_SIZE);
1964
-
test->ifobj_rx->pkt_stream = pkt_stream_generate(test->ifobj_rx->umem,
1965
-
DEFAULT_UMEM_BUFFERS, MIN_PKT_SIZE);
1855
+
test->ifobj_rx->xsk->pkt_stream = pkt_stream_generate(DEFAULT_UMEM_BUFFERS, MIN_PKT_SIZE);
1966
1856
1967
1857
test->ifobj_rx->xsk->rxqsize = DEFAULT_UMEM_BUFFERS;
1968
1858
test->ifobj_rx->release_rx = false;
···
1972
1864
static int testapp_stats_fill_empty(struct test_spec *test)
1973
1865
{
1974
1866
pkt_stream_replace(test, DEFAULT_UMEM_BUFFERS + DEFAULT_UMEM_BUFFERS / 2, MIN_PKT_SIZE);
1975
-
test->ifobj_rx->pkt_stream = pkt_stream_generate(test->ifobj_rx->umem,
1976
-
DEFAULT_UMEM_BUFFERS, MIN_PKT_SIZE);
1867
+
test->ifobj_rx->xsk->pkt_stream = pkt_stream_generate(DEFAULT_UMEM_BUFFERS, MIN_PKT_SIZE);
1977
1868
1978
1869
test->ifobj_rx->use_fill_ring = false;
1979
1870
test->ifobj_rx->validation_func = validate_fill_empty;
···
2138
2031
return testapp_validate_traffic(test);
2139
2032
}
2140
2033
2034
+
static int testapp_xdp_shared_umem(struct test_spec *test)
2035
+
{
2036
+
struct xsk_xdp_progs *skel_rx = test->ifobj_rx->xdp_progs;
2037
+
struct xsk_xdp_progs *skel_tx = test->ifobj_tx->xdp_progs;
2038
+
2039
+
test->total_steps = 1;
2040
+
test->nb_sockets = 2;
2041
+
2042
+
test_spec_set_xdp_prog(test, skel_rx->progs.xsk_xdp_shared_umem,
2043
+
skel_tx->progs.xsk_xdp_shared_umem,
2044
+
skel_rx->maps.xsk, skel_tx->maps.xsk);
2045
+
2046
+
pkt_stream_even_odd_sequence(test);
2047
+
2048
+
return testapp_validate_traffic(test);
2049
+
}
2050
+
2141
2051
static int testapp_poll_txq_tmout(struct test_spec *test)
2142
2052
{
2143
2053
test->ifobj_tx->use_poll = true;
···
2241
2117
return true;
2242
2118
}
2243
2119
2244
-
static void init_iface(struct ifobject *ifobj, const char *dst_mac, const char *src_mac,
2245
-
thread_func_t func_ptr)
2120
+
static void init_iface(struct ifobject *ifobj, thread_func_t func_ptr)
2246
2121
{
2247
2122
LIBBPF_OPTS(bpf_xdp_query_opts, query_opts);
2248
2123
int err;
2249
-
2250
-
memcpy(ifobj->dst_mac, dst_mac, ETH_ALEN);
2251
-
memcpy(ifobj->src_mac, src_mac, ETH_ALEN);
2252
2124
2253
2125
ifobj->func_ptr = func_ptr;
2254
2126
···
2456
2336
{.name = "STAT_FILL_EMPTY", .test_func = testapp_stats_fill_empty},
2457
2337
{.name = "XDP_PROG_CLEANUP", .test_func = testapp_xdp_prog_cleanup},
2458
2338
{.name = "XDP_DROP_HALF", .test_func = testapp_xdp_drop},
2339
+
{.name = "XDP_SHARED_UMEM", .test_func = testapp_xdp_shared_umem},
2459
2340
{.name = "XDP_METADATA_COPY", .test_func = testapp_xdp_metadata},
2460
2341
{.name = "XDP_METADATA_COPY_MULTI_BUFF", .test_func = testapp_xdp_metadata_mb},
2461
2342
{.name = "SEND_RECEIVE_9K_PACKETS", .test_func = testapp_send_receive_mb},
···
2522
2401
modes++;
2523
2402
}
2524
2403
2525
-
init_iface(ifobj_rx, MAC1, MAC2, worker_testapp_validate_rx);
2526
-
init_iface(ifobj_tx, MAC2, MAC1, worker_testapp_validate_tx);
2404
+
init_iface(ifobj_rx, worker_testapp_validate_rx);
2405
+
init_iface(ifobj_tx, worker_testapp_validate_tx);
2527
2406
2528
2407
test_spec_init(&test, ifobj_tx, ifobj_rx, 0, &tests[0]);
2529
-
tx_pkt_stream_default = pkt_stream_generate(ifobj_tx->umem, DEFAULT_PKT_CNT, MIN_PKT_SIZE);
2530
-
rx_pkt_stream_default = pkt_stream_generate(ifobj_rx->umem, DEFAULT_PKT_CNT, MIN_PKT_SIZE);
2408
+
tx_pkt_stream_default = pkt_stream_generate(DEFAULT_PKT_CNT, MIN_PKT_SIZE);
2409
+
rx_pkt_stream_default = pkt_stream_generate(DEFAULT_PKT_CNT, MIN_PKT_SIZE);
2531
2410
if (!tx_pkt_stream_default || !rx_pkt_stream_default)
2532
2411
exit_with_error(ENOMEM);
2533
2412
test.tx_pkt_stream_default = tx_pkt_stream_default;
+9
-4
tools/testing/selftests/bpf/xskxceiver.h
+9
-4
tools/testing/selftests/bpf/xskxceiver.h
···
8
8
#include <limits.h>
9
9
10
10
#include "xsk_xdp_progs.skel.h"
11
+
#include "xsk_xdp_common.h"
11
12
12
13
#ifndef SOL_XDP
13
14
#define SOL_XDP 283
···
36
35
#define TEST_SKIP 2
37
36
#define MAX_INTERFACES 2
38
37
#define MAX_INTERFACE_NAME_CHARS 16
39
-
#define MAX_SOCKETS 2
40
38
#define MAX_TEST_NAME_SIZE 48
41
39
#define MAX_TEARDOWN_ITER 10
42
40
#define PKT_HDR_SIZE (sizeof(struct ethhdr) + 2) /* Just to align the data in the packet */
···
59
59
#define HUGEPAGE_SIZE (2 * 1024 * 1024)
60
60
#define PKT_DUMP_NB_TO_PRINT 16
61
61
#define RUN_ALL_TESTS UINT_MAX
62
+
#define NUM_MAC_ADDRESSES 4
62
63
63
64
#define print_verbose(x...) do { if (opt_verbose) ksft_print_msg(x); } while (0)
64
65
···
88
87
struct xsk_ring_prod tx;
89
88
struct xsk_umem_info *umem;
90
89
struct xsk_socket *xsk;
90
+
struct pkt_stream *pkt_stream;
91
91
u32 outstanding_tx;
92
92
u32 rxqsize;
93
+
u8 dst_mac[ETH_ALEN];
94
+
u8 src_mac[ETH_ALEN];
93
95
};
94
96
95
97
struct pkt {
···
108
104
u32 current_pkt_nb;
109
105
struct pkt *pkts;
110
106
u32 max_pkt_len;
107
+
u32 nb_rx_pkts;
108
+
u32 nb_valid_entries;
111
109
bool verbatim;
112
110
};
113
111
···
126
120
struct xsk_umem_info *umem;
127
121
thread_func_t func_ptr;
128
122
validation_func_t validation_func;
129
-
struct pkt_stream *pkt_stream;
130
123
struct xsk_xdp_progs *xdp_progs;
131
124
struct bpf_map *xskmap;
132
125
struct bpf_program *xdp_prog;
···
145
140
bool unaligned_supp;
146
141
bool multi_buff_supp;
147
142
bool multi_buff_zc_supp;
148
-
u8 dst_mac[ETH_ALEN];
149
-
u8 src_mac[ETH_ALEN];
150
143
};
151
144
152
145
struct test_spec {
···
170
167
pthread_mutex_t pacing_mutex = PTHREAD_MUTEX_INITIALIZER;
171
168
172
169
int pkts_in_flight;
170
+
171
+
static const u8 g_mac[ETH_ALEN] = {0x55, 0x44, 0x33, 0x22, 0x11, 0x00};
173
172
174
173
#endif /* XSKXCEIVER_H_ */