tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

xprtrdma: Close lost-wakeup race in xprt_rdma_alloc_slot

xprt_rdma_alloc_slot() and xprt_rdma_free_slot() lack serialization
between the buffer pool and the backlog queue. A buffer freed
after rpcrdma_buffer_get() finds the pool empty but before
rpc_sleep_on() places the task on the backlog is returned to the
pool with no waiter to wake, leaving the task stuck on the backlog
indefinitely.

After joining the backlog, re-check the pool and route any
recovered buffer through xprt_wake_up_backlog(), whose queue lock
serializes with concurrent wakeups and avoids double-assignment
of slots.

Because xprt_rdma_free_slot() does not hold reserve_lock, the
XPRT_CONGESTED double-check in xprt_throttle_congested() is
ineffective: a task can join the backlog through that path after
free_slot has already found it empty and cleared the bit. Avoid
this by using xprt_add_backlog_noncongested(), which queues the
task without setting XPRT_CONGESTED, so every allocation reaches
xprt_rdma_alloc_slot() and its post-sleep re-check.

Fixes: edb41e61a54e ("xprtrdma: Make rpc_rqst part of rpcrdma_req")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>

authored by

Chuck Lever and committed by
Trond Myklebust 765bde47 10014209

+32 -1
+2
include/linux/sunrpc/xprt.h
··· 404 404 unsigned int max_req); 405 405 void xprt_free(struct rpc_xprt *); 406 406 void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task); 407 + void xprt_add_backlog_noncongested(struct rpc_xprt *xprt, 408 + struct rpc_task *task); 407 409 bool xprt_wake_up_backlog(struct rpc_xprt *xprt, struct rpc_rqst *req); 408 410 void xprt_cleanup_ids(void); 409 411
+16
net/sunrpc/xprt.c
··· 1663 1663 } 1664 1664 EXPORT_SYMBOL_GPL(xprt_add_backlog); 1665 1665 1666 + /** 1667 + * xprt_add_backlog_noncongested - queue task on backlog 1668 + * @xprt: transport whose backlog queue receives the task 1669 + * @task: task to queue 1670 + * 1671 + * Like xprt_add_backlog, but does not set XPRT_CONGESTED. 1672 + * For transports whose free_slot path does not synchronize 1673 + * with xprt_throttle_congested via reserve_lock. 1674 + */ 1675 + void xprt_add_backlog_noncongested(struct rpc_xprt *xprt, 1676 + struct rpc_task *task) 1677 + { 1678 + rpc_sleep_on(&xprt->backlog, task, xprt_complete_request_init); 1679 + } 1680 + EXPORT_SYMBOL_GPL(xprt_add_backlog_noncongested); 1681 + 1666 1682 static bool __xprt_set_rq(struct rpc_task *task, void *data) 1667 1683 { 1668 1684 struct rpc_rqst *req = data;
+14 -1
net/sunrpc/xprtrdma/transport.c
··· 511 511 512 512 out_sleep: 513 513 task->tk_status = -EAGAIN; 514 - xprt_add_backlog(xprt, task); 514 + xprt_add_backlog_noncongested(xprt, task); 515 + /* A buffer freed between buffer_get and rpc_sleep_on 516 + * goes back to the pool with no waiter to wake. 517 + * Re-check after joining the backlog to close that gap. 518 + */ 519 + req = rpcrdma_buffer_get(&r_xprt->rx_buf); 520 + if (req) { 521 + struct rpc_rqst *rqst = &req->rl_slot; 522 + 523 + if (!xprt_wake_up_backlog(xprt, rqst)) { 524 + memset(rqst, 0, sizeof(*rqst)); 525 + rpcrdma_buffer_put(&r_xprt->rx_buf, req); 526 + } 527 + } 515 528 } 516 529 517 530 /**