selftests/net: packetdrill: report benign debug flakes as xfail

A few recently added packetdrill tests that are known time sensitive
(e.g., because testing timestamping) occasionally fail in debug mode:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbg

These failures are well understood. Correctness of the tests is
verified in non-debug mode. Continue running in debug mode also, to
keep coverage with debug instrumentation.

But, only in debug mode, mark these tests with well understood
timing issues as XFAIL (known failing) rather than FAIL when failing.

Introduce an allow list xfail_list with known cases.

Expand the ktap infrastructure with XFAIL support.

Fixes: eab35989cc37 ("selftests/net: packetdrill: import tcp/fast_recovery, tcp/nagle, tcp/timestamping")
Reported-by: Jakub Kicinski <kuba@kernel.org>
Closes: https://lore.kernel.org/netdev/20241218100013.0c698629@kernel.org/
Signed-off-by: Willem de Bruijn <willemb@google.com>
Link: https://patch.msgid.link/20250103113142.129251-1-willemdebruijn.kernel@gmail.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>

authored by

Willem de Bruijn and committed by

Paolo Abeni 1 year ago 912d6f66 51cfbed1

+31 -7

2 changed files

expand all

tools

testing

selftests

kselftest

ktap_helpers.sh

net

packetdrill

ksft_runner.sh

+13 -2

tools/testing/selftests/kselftest/ktap_helpers.sh

··· 7 7 KTAP_TESTNO=1 8 8 KTAP_CNT_PASS=0 9 9 KTAP_CNT_FAIL=0 10 + KTAP_CNT_XFAIL=0 10 11 KTAP_CNT_SKIP=0 11 12 12 13 KSFT_PASS=0 ··· 70 69 KTAP_CNT_SKIP=$((KTAP_CNT_SKIP+1)) 71 70 } 72 71 72 + ktap_test_xfail() { 73 + description="$1" 74 + 75 + result="ok" 76 + directive="XFAIL" 77 + __ktap_test "$result" "$description" "$directive" 78 + 79 + KTAP_CNT_XFAIL=$((KTAP_CNT_XFAIL+1)) 80 + } 81 + 73 82 ktap_test_fail() { 74 83 description="$1" 75 84 ··· 110 99 ktap_finished() { 111 100 ktap_print_totals 112 101 113 - if [ $((KTAP_CNT_PASS + KTAP_CNT_SKIP)) -eq "$KSFT_NUM_TESTS" ]; then 102 + if [ $((KTAP_CNT_PASS + KTAP_CNT_SKIP + KTAP_CNT_XFAIL)) -eq "$KSFT_NUM_TESTS" ]; then 114 103 exit "$KSFT_PASS" 115 104 else 116 105 exit "$KSFT_FAIL" ··· 118 107 } 119 108 120 109 ktap_print_totals() { 121 - echo "# Totals: pass:$KTAP_CNT_PASS fail:$KTAP_CNT_FAIL xfail:0 xpass:0 skip:$KTAP_CNT_SKIP error:0" 110 + echo "# Totals: pass:$KTAP_CNT_PASS fail:$KTAP_CNT_FAIL xfail:$KTAP_CNT_XFAIL xpass:0 skip:$KTAP_CNT_SKIP error:0" 122 111 }

+18 -5

tools/testing/selftests/net/packetdrill/ksft_runner.sh

··· 23 23 ktap_exit_fail_msg "usage: $0 <script>" 24 24 exit "$KSFT_FAIL" 25 25 fi 26 - script="$1" 26 + script="$(basename $1)" 27 27 28 28 if [ -z "$(which packetdrill)" ]; then 29 29 ktap_skip_all "packetdrill not found in PATH" ··· 31 31 fi 32 32 33 33 declare -a optargs 34 + failfunc=ktap_test_fail 35 + 34 36 if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then 35 37 optargs+=('--tolerance_usecs=14000') 38 + 39 + # xfail tests that are known flaky with dbg config, not fixable. 40 + # still run them for coverage (and expect 100% pass without dbg). 41 + declare -ar xfail_list=( 42 + "tcp_fast_recovery_prr-ss.*.pkt" 43 + "tcp_timestamping.*.pkt" 44 + "tcp_user_timeout_user-timeout-probe.pkt" 45 + "tcp_zerocopy_epoll_.*.pkt" 46 + ) 47 + readonly xfail_regex="^($(printf '%s|' "${xfail_list[@]}"))$" 48 + [[ "$script" =~ ${xfail_regex} ]] && failfunc=ktap_test_xfail 36 49 fi 37 50 38 51 ktap_print_header 39 52 ktap_set_plan 2 40 53 41 - unshare -n packetdrill ${ipv4_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ 42 - && ktap_test_pass "ipv4" || ktap_test_fail "ipv4" 43 - unshare -n packetdrill ${ipv6_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ 44 - && ktap_test_pass "ipv6" || ktap_test_fail "ipv6" 54 + unshare -n packetdrill ${ipv4_args[@]} ${optargs[@]} $script > /dev/null \ 55 + && ktap_test_pass "ipv4" || $failfunc "ipv4" 56 + unshare -n packetdrill ${ipv6_args[@]} ${optargs[@]} $script > /dev/null \ 57 + && ktap_test_pass "ipv6" || $failfunc "ipv6" 45 58 46 59 ktap_finished

Configure Feed

Configure Feed