RDMA/rtrs-srv: Add check and closure for possible zombie paths

During several network incidents, a number of RTRS paths for a session
went through disconnect and reconnect phase. However, some of those did
not auto-reconnect successfully. Instead they failed with the following
logs,

On client,
kernel: rtrs_client L1991: <sess-name>: Connect rejected: status 28
(consumer defined), rtrs errno -104
kernel: rtrs_client L2698: <sess-name>: init_conns() failed: err=-104
path=gid:<gid1>@gid:<gid2> [mlx4_0:1]

On server, (log a)
kernel: ibtrs_server L1868: <>: Connection already exists: 0

When the misbehaving path was removed, and add_path was called to re-add
the path, the log on client side changed to, (log b)
kernel: rtrs_client L1991: <sess-name>: Connect rejected: status 28
(consumer defined), rtrs errno -17

There was no log on the server side for this, which is expected since
there is no logging in that path,
if (unlikely(__is_path_w_addr_exists(srv, &cm_id->route.addr))) {
err = -EEXIST;
goto err;

Because of the following check on server side,
if (unlikely(sess->state != IBTRS_SRV_CONNECTING)) {
ibtrs_err(s, "Session in wrong state: %s\n",

.. we know that the path in (log a) was in CONNECTING state.

The above state of the path persists for as long as we leave the session
be. This means that the path is in some zombie state, probably waiting
for the info_req packet to arrive, which never does.

The changes in this commits does 2 things.

1) Add logs at places where we see the errors happening. The logs would
shed more light at the state and lifetime of such zombie paths.

2) Close such zombie sessions, only if they are in CONNECTING state, and
after an inactivity period of 30 seconds.
i) The state check prevents closure of paths which are CONNECTED.
Also, from the above logs and code, we already know that the path could
only be on CONNECTING state, so we play safe and narrow our impact surface
area by closing only CONNECTING paths.
ii) The inactivity period is to allow requests for other cid to finish
processing, or for any stray packets to arrive/fail.

Signed-off-by: Md Haris Iqbal <haris.iqbal@ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Grzegorz Prajsner <grzegorz.prajsner@ionos.com>
Link: https://patch.msgid.link/20260107161517.56357-7-haris.iqbal@ionos.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>

authored by

Md Haris Iqbal and committed by

Leon Romanovsky 5 months ago c32eaba2 781c35b5

+41 -5

2 changed files

expand all

drivers

infiniband

ulp

rtrs

rtrs-srv.c

rtrs-srv.h

+40 -5

drivers/infiniband/ulp/rtrs/rtrs-srv.c

··· 905 905 tx_iu->dma_addr, 906 906 tx_iu->size, DMA_TO_DEVICE); 907 907 908 + /* 909 + * Now disable zombie connection closing. Since from the logs and code, 910 + * we know that it can never be in CONNECTED state. 911 + */ 912 + srv_path->connection_timeout = 0; 913 + 908 914 /* Send info response */ 909 915 err = rtrs_iu_post_send(&con->c, tx_iu, tx_sz, reg_wr); 910 916 if (err) { ··· 1537 1531 } 1538 1532 } 1539 1533 1534 + /* Let's close connections which have been waiting for more than 30 seconds */ 1535 + #define RTRS_MAX_CONN_TIMEOUT 30000 1536 + 1537 + static void rtrs_srv_check_close_path(struct rtrs_srv_path *srv_path) 1538 + { 1539 + struct rtrs_path *s = &srv_path->s; 1540 + 1541 + if (srv_path->state == RTRS_SRV_CONNECTING && srv_path->connection_timeout && 1542 + (jiffies_to_msecs(jiffies - srv_path->connection_timeout) > RTRS_MAX_CONN_TIMEOUT)) { 1543 + rtrs_err(s, "Closing zombie path\n"); 1544 + close_path(srv_path); 1545 + } 1546 + } 1547 + 1540 1548 static bool __is_path_w_addr_exists(struct rtrs_srv_sess *srv, 1541 1549 struct rdma_addr *addr) 1542 1550 { 1543 1551 struct rtrs_srv_path *srv_path; 1544 1552 1545 - list_for_each_entry(srv_path, &srv->paths_list, s.entry) 1553 + list_for_each_entry(srv_path, &srv->paths_list, s.entry) { 1546 1554 if (!sockaddr_cmp((struct sockaddr *)&srv_path->s.dst_addr, 1547 1555 (struct sockaddr *)&addr->dst_addr) && 1548 1556 !sockaddr_cmp((struct sockaddr *)&srv_path->s.src_addr, 1549 - (struct sockaddr *)&addr->src_addr)) 1557 + (struct sockaddr *)&addr->src_addr)) { 1558 + rtrs_err((&srv_path->s), 1559 + "Path (%s) with same addr exists (lifetime %u)\n", 1560 + rtrs_srv_state_str(srv_path->state), 1561 + (jiffies_to_msecs(jiffies - srv_path->connection_timeout))); 1562 + rtrs_srv_check_close_path(srv_path); 1550 1563 return true; 1564 + } 1565 + } 1551 1566 1552 1567 return false; 1553 1568 } ··· 1806 1779 } 1807 1780 if (__is_path_w_addr_exists(srv, &cm_id->route.addr)) { 1808 1781 err = -EEXIST; 1809 - pr_err("Path with same addr exists\n"); 1810 1782 goto err; 1811 1783 } 1812 1784 srv_path = kzalloc(sizeof(*srv_path), GFP_KERNEL); ··· 1852 1826 spin_lock_init(&srv_path->state_lock); 1853 1827 INIT_WORK(&srv_path->close_work, rtrs_srv_close_work); 1854 1828 rtrs_srv_init_hb(srv_path); 1829 + srv_path->connection_timeout = 0; 1855 1830 1856 1831 srv_path->s.dev = rtrs_ib_dev_find_or_add(cm_id->device, &dev_pd); 1857 1832 if (!srv_path->s.dev) { ··· 1958 1931 goto reject_w_err; 1959 1932 } 1960 1933 if (s->con[cid]) { 1961 - rtrs_err(s, "Connection already exists: %d\n", 1962 - cid); 1934 + rtrs_err(s, "Connection (%s) already exists: %d (lifetime %u)\n", 1935 + rtrs_srv_state_str(srv_path->state), cid, 1936 + (jiffies_to_msecs(jiffies - srv_path->connection_timeout))); 1937 + rtrs_srv_check_close_path(srv_path); 1963 1938 mutex_unlock(&srv->paths_mutex); 1964 1939 goto reject_w_err; 1965 1940 } ··· 1976 1947 goto reject_w_err; 1977 1948 } 1978 1949 } 1950 + 1951 + /* 1952 + * Start of any connection creation resets the timeout for the path. 1953 + */ 1954 + srv_path->connection_timeout = jiffies; 1955 + 1979 1956 err = create_con(srv_path, cm_id, cid); 1980 1957 if (err) { 1981 1958 rtrs_err((&srv_path->s), "create_con(), error %pe\n", ERR_PTR(err));

drivers/infiniband/ulp/rtrs/rtrs-srv.h

··· 89 89 unsigned int mem_bits; 90 90 struct kobject kobj; 91 91 struct rtrs_srv_stats *stats; 92 + unsigned long connection_timeout; 92 93 }; 93 94 94 95 static inline struct rtrs_srv_path *to_srv_path(struct rtrs_path *s)

Configure Feed

Configure Feed