SUNRPC: Track consumed rq_pages entries

The rq_pages array holds pages allocated for incoming RPC requests.
Two transport receive paths NULL entries in rq_pages to prevent
svc_rqst_release_pages() from freeing pages that the transport has
taken ownership of:

- svc_tcp_save_pages() moves partial request data pages to
svsk->sk_pages during multi-fragment TCP reassembly.

- svc_rdma_clear_rqst_pages() moves request data pages to
head->rc_pages because they are targets of active RDMA Read WRs.

A new rq_pages_nfree field in struct svc_rqst records how many
entries were NULLed. svc_alloc_arg() uses it to refill only those
entries rather than scanning the full rq_pages array. In steady
state, the transport NULLs a handful of entries per RPC, so the
allocator visits only those entries instead of the full ~259 slots
(for 1MB messages).

Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>

Chuck Lever 2 months ago 7ed75042 26c8e6eb

+21 -3

5 changed files

expand all

include

linux

sunrpc

svc.h

net

sunrpc

svc.c

svc_xprt.c

svcsock.c

xprtrdma

svc_rdma_rw.c

+10

include/linux/sunrpc/svc.h

··· 143 143 * server thread needs to allocate more to replace those used in 144 144 * sending. 145 145 * 146 + * rq_pages request page contract: 147 + * 148 + * Transport receive paths that move request data pages out of 149 + * rq_pages -- TCP multi-fragment reassembly (svc_tcp_save_pages) 150 + * and RDMA Read I/O (svc_rdma_clear_rqst_pages) -- NULL those 151 + * entries to prevent svc_rqst_release_pages() from freeing pages 152 + * still in transport use, and set rq_pages_nfree to the count. 153 + * svc_alloc_arg() refills only that many rq_pages entries. 154 + * 146 155 * xdr_buf holds responses; the structure fits NFS read responses 147 156 * (header, data pages, optional tail) and enables sharing of 148 157 * client-side routines. ··· 213 204 struct folio *rq_scratch_folio; 214 205 struct xdr_buf rq_res; 215 206 unsigned long rq_maxpages; /* entries per page array */ 207 + unsigned long rq_pages_nfree; /* rq_pages entries NULLed by transport */ 216 208 struct page * *rq_pages; /* Call buffer pages */ 217 209 struct page * *rq_respages; /* Reply buffer pages */ 218 210 struct page * *rq_next_page; /* next reply page to use */

net/sunrpc/svc.c

··· 655 655 return false; 656 656 } 657 657 658 + rqstp->rq_pages_nfree = rqstp->rq_maxpages; 658 659 return true; 659 660 } 660 661

+8 -3

net/sunrpc/svc_xprt.c

··· 675 675 static bool svc_alloc_arg(struct svc_rqst *rqstp) 676 676 { 677 677 struct xdr_buf *arg = &rqstp->rq_arg; 678 - unsigned long pages; 678 + unsigned long pages, nfree; 679 679 680 680 pages = rqstp->rq_maxpages; 681 681 682 - if (!svc_fill_pages(rqstp, rqstp->rq_pages, pages)) 683 - return false; 682 + nfree = rqstp->rq_pages_nfree; 683 + if (nfree) { 684 + if (!svc_fill_pages(rqstp, rqstp->rq_pages, nfree)) 685 + return false; 686 + rqstp->rq_pages_nfree = 0; 687 + } 688 + 684 689 if (!svc_fill_pages(rqstp, rqstp->rq_respages, pages)) 685 690 return false; 686 691 rqstp->rq_next_page = rqstp->rq_respages;

net/sunrpc/svcsock.c

··· 1009 1009 svsk->sk_pages[i] = rqstp->rq_pages[i]; 1010 1010 rqstp->rq_pages[i] = NULL; 1011 1011 } 1012 + rqstp->rq_pages_nfree = npages; 1012 1013 } 1013 1014 1014 1015 static void svc_tcp_clear_pages(struct svc_sock *svsk)

net/sunrpc/xprtrdma/svc_rdma_rw.c

··· 1107 1107 head->rc_pages[i] = rqstp->rq_pages[i]; 1108 1108 rqstp->rq_pages[i] = NULL; 1109 1109 } 1110 + rqstp->rq_pages_nfree = head->rc_page_count; 1110 1111 } 1111 1112 1112 1113 /**

Configure Feed

Configure Feed