tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'selftests-mptcp-mark-unstable-subtests-as-flaky'

Matthieu Baerts says:

====================
selftests: mptcp: mark unstable subtests as flaky

Some subtests can be unstable, failing once every X runs. Fixing them
can take time: there could be an issue in the kernel or in the subtest,
and it is then important to do a proper analysis, not to hide real bugs.

To avoid creating noises on the different CIs where tests are more
unstable than on our side, some subtests have been marked as flaky. As a
result, errors with these subtests (if any) are ignored.

Note that the MPTCP CI will continue to track these flaky subtests. All
these unstable subtests are also tracked by our bug tracker.

These are fixes for the -net tree, because the instabilities are visible
there. The first patch introducing the flake support has no 'Fixes'
tags, mainly because it requires recent and important refactoring done
in all MPTCP selftests. Backporting that to old versions where the flaky
tests have been introduced would be too difficult, and probably not
worth it. The other patches, adding MPTCP_LIB_SUBTEST_FLAKY=1, have a
Fixes tag, simply to ease the backport of the future fixes removing them
along with the proper fix.
====================

Link: https://lore.kernel.org/r/20240524-upstream-net-20240524-selftests-mptcp-flaky-v1-0-a352362f3f8e@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

Jakub Kicinski 6e15774d 7a8cc96e

+40 -6
+9 -1
tools/testing/selftests/net/mptcp/mptcp_join.sh
··· 261 261 262 262 TEST_NAME="${1}" 263 263 264 + MPTCP_LIB_SUBTEST_FLAKY=0 # reset if modified 265 + 264 266 if skip_test; then 265 267 MPTCP_LIB_TEST_COUNTER=$((MPTCP_LIB_TEST_COUNTER+1)) 266 268 last_test_ignored=1 ··· 450 448 # $1: err msg 451 449 fail_test() 452 450 { 453 - ret=${KSFT_FAIL} 451 + if ! mptcp_lib_subtest_is_flaky; then 452 + ret=${KSFT_FAIL} 453 + fi 454 454 455 455 if [ ${#} -gt 0 ]; then 456 456 print_fail "${@}" ··· 3073 3069 fastclose_tests() 3074 3070 { 3075 3071 if reset_check_counter "fastclose test" "MPTcpExtMPFastcloseTx"; then 3072 + MPTCP_LIB_SUBTEST_FLAKY=1 3076 3073 test_linkfail=1024 fastclose=client \ 3077 3074 run_tests $ns1 $ns2 10.0.1.1 3078 3075 chk_join_nr 0 0 0 ··· 3082 3077 fi 3083 3078 3084 3079 if reset_check_counter "fastclose server test" "MPTcpExtMPFastcloseRx"; then 3080 + MPTCP_LIB_SUBTEST_FLAKY=1 3085 3081 test_linkfail=1024 fastclose=server \ 3086 3082 run_tests $ns1 $ns2 10.0.1.1 3087 3083 chk_join_nr 0 0 0 0 0 0 1 ··· 3101 3095 { 3102 3096 # single subflow 3103 3097 if reset_with_fail "Infinite map" 1; then 3098 + MPTCP_LIB_SUBTEST_FLAKY=1 3104 3099 test_linkfail=128 \ 3105 3100 run_tests $ns1 $ns2 10.0.1.1 3106 3101 chk_join_nr 0 0 0 +1 +0 1 0 1 "$(pedit_action_pkts)" ··· 3110 3103 3111 3104 # multiple subflows 3112 3105 if reset_with_fail "MP_FAIL MP_RST" 2; then 3106 + MPTCP_LIB_SUBTEST_FLAKY=1 3113 3107 tc -n $ns2 qdisc add dev ns2eth1 root netem rate 1mbit delay 5ms 3114 3108 pm_nl_set_limits $ns1 0 1 3115 3109 pm_nl_set_limits $ns2 0 1
+28 -2
tools/testing/selftests/net/mptcp/mptcp_lib.sh
··· 21 21 22 22 MPTCP_LIB_SUBTESTS=() 23 23 MPTCP_LIB_SUBTESTS_DUPLICATED=0 24 + MPTCP_LIB_SUBTEST_FLAKY=0 24 25 MPTCP_LIB_TEST_COUNTER=0 25 26 MPTCP_LIB_TEST_FORMAT="%02u %-50s" 26 27 MPTCP_LIB_IP_MPTCP=0 ··· 41 40 readonly MPTCP_LIB_COLOR_BLUE= 42 41 readonly MPTCP_LIB_COLOR_RESET= 43 42 fi 43 + 44 + # SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var can be set not to ignore errors 45 + # from subtests marked as flaky 46 + mptcp_lib_override_flaky() { 47 + [ "${SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY:-}" = 1 ] 48 + } 49 + 50 + mptcp_lib_subtest_is_flaky() { 51 + [ "${MPTCP_LIB_SUBTEST_FLAKY}" = 1 ] && ! mptcp_lib_override_flaky 52 + } 44 53 45 54 # $1: color, $2: text 46 55 mptcp_lib_print_color() { ··· 83 72 } 84 73 85 74 mptcp_lib_pr_fail() { 86 - mptcp_lib_print_err "[FAIL]${1:+ ${*}}" 75 + local title cmt 76 + 77 + if mptcp_lib_subtest_is_flaky; then 78 + title="IGNO" 79 + cmt=" (flaky)" 80 + else 81 + title="FAIL" 82 + fi 83 + 84 + mptcp_lib_print_err "[${title}]${cmt}${1:+ ${*}}" 87 85 } 88 86 89 87 mptcp_lib_pr_info() { ··· 228 208 229 209 # $1: test name 230 210 mptcp_lib_result_fail() { 231 - __mptcp_lib_result_add "not ok" "${1}" 211 + if mptcp_lib_subtest_is_flaky; then 212 + # It might sound better to use 'not ok # TODO' or 'ok # SKIP', 213 + # but some CIs don't understand 'TODO' and treat SKIP as errors. 214 + __mptcp_lib_result_add "ok" "${1} # IGNORE Flaky" 215 + else 216 + __mptcp_lib_result_add "not ok" "${1}" 217 + fi 232 218 } 233 219 234 220 # $1: test name
+3 -3
tools/testing/selftests/net/mptcp/simult_flows.sh
··· 244 244 do_transfer $small $large $time 245 245 lret=$? 246 246 mptcp_lib_result_code "${lret}" "${msg}" 247 - if [ $lret -ne 0 ]; then 247 + if [ $lret -ne 0 ] && ! mptcp_lib_subtest_is_flaky; then 248 248 ret=$lret 249 249 [ $bail -eq 0 ] || exit $ret 250 250 fi ··· 254 254 do_transfer $large $small $time 255 255 lret=$? 256 256 mptcp_lib_result_code "${lret}" "${msg}" 257 - if [ $lret -ne 0 ]; then 257 + if [ $lret -ne 0 ] && ! mptcp_lib_subtest_is_flaky; then 258 258 ret=$lret 259 259 [ $bail -eq 0 ] || exit $ret 260 260 fi ··· 290 290 run_test 10 10 1 25 "balanced bwidth with unbalanced delay" 291 291 292 292 # we still need some additional infrastructure to pass the following test-cases 293 - run_test 10 3 0 0 "unbalanced bwidth" 293 + MPTCP_LIB_SUBTEST_FLAKY=1 run_test 10 3 0 0 "unbalanced bwidth" 294 294 run_test 10 3 1 25 "unbalanced bwidth with unbalanced delay" 295 295 run_test 10 3 25 1 "unbalanced bwidth with opposed, unbalanced delay" 296 296