tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'net-rps-rfs-improvements'

Eric Dumazet says:

====================
net: rps/rfs improvements

Jason Xing attempted to optimize napi_schedule_rps() by avoiding
unneeded NET_RX_SOFTIRQ raises: [1], [2]

This is quite complex to implement properly. I chose to implement
the idea, and added a similar optimization in ____napi_schedule()

Overall, in an intensive RPC workload, with 32 TX/RX queues with RFS
I was able to observe a ~10% reduction of NET_RX_SOFTIRQ
invocations.

While this had no impact on throughput or cpu costs on this synthetic
benchmark, we know that firing NET_RX_SOFTIRQ from softirq handler
can force __do_softirq() to wakeup ksoftirqd when need_resched() is true.
This can have a latency impact on stressed hosts.

[1] https://lore.kernel.org/lkml/20230325152417.5403-1-kerneljasonxing@gmail.com/
[2] https://lore.kernel.org/netdev/20230328142112.12493-1-kerneljasonxing@gmail.com/
====================

Link: https://lore.kernel.org/r/20230328235021.1048163-1-edumazet@google.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>

Paolo Abeni 4ddd6375 7079d5e6

+37 -10
+1
include/linux/netdevice.h
··· 3188 3188 #ifdef CONFIG_RPS 3189 3189 struct softnet_data *rps_ipi_list; 3190 3190 #endif 3191 + bool in_net_rx_action; 3191 3192 #ifdef CONFIG_NET_FLOW_LIMIT 3192 3193 struct sd_flow_limit __rcu *flow_limit; 3193 3194 #endif
+36 -10
net/core/dev.c
··· 4360 4360 } 4361 4361 4362 4362 list_add_tail(&napi->poll_list, &sd->poll_list); 4363 - __raise_softirq_irqoff(NET_RX_SOFTIRQ); 4363 + /* If not called from net_rx_action() 4364 + * we have to raise NET_RX_SOFTIRQ. 4365 + */ 4366 + if (!sd->in_net_rx_action) 4367 + __raise_softirq_irqoff(NET_RX_SOFTIRQ); 4364 4368 } 4365 4369 4366 4370 #ifdef CONFIG_RPS ··· 4586 4582 } 4587 4583 4588 4584 /* 4589 - * Check if this softnet_data structure is another cpu one 4590 - * If yes, queue it to our IPI list and return 1 4591 - * If no, return 0 4585 + * After we queued a packet into sd->input_pkt_queue, 4586 + * we need to make sure this queue is serviced soon. 4587 + * 4588 + * - If this is another cpu queue, link it to our rps_ipi_list, 4589 + * and make sure we will process rps_ipi_list from net_rx_action(). 4590 + * 4591 + * - If this is our own queue, NAPI schedule our backlog. 4592 + * Note that this also raises NET_RX_SOFTIRQ. 4592 4593 */ 4593 - static int napi_schedule_rps(struct softnet_data *sd) 4594 + static void napi_schedule_rps(struct softnet_data *sd) 4594 4595 { 4595 4596 struct softnet_data *mysd = this_cpu_ptr(&softnet_data); 4596 4597 ··· 4604 4595 sd->rps_ipi_next = mysd->rps_ipi_list; 4605 4596 mysd->rps_ipi_list = sd; 4606 4597 4607 - __raise_softirq_irqoff(NET_RX_SOFTIRQ); 4608 - return 1; 4598 + /* If not called from net_rx_action() 4599 + * we have to raise NET_RX_SOFTIRQ. 4600 + */ 4601 + if (!mysd->in_net_rx_action) 4602 + __raise_softirq_irqoff(NET_RX_SOFTIRQ); 4603 + return; 4609 4604 } 4610 4605 #endif /* CONFIG_RPS */ 4611 4606 __napi_schedule_irqoff(&mysd->backlog); 4612 - return 0; 4613 4607 } 4614 4608 4615 4609 #ifdef CONFIG_NET_FLOW_LIMIT ··· 6652 6640 LIST_HEAD(list); 6653 6641 LIST_HEAD(repoll); 6654 6642 6643 + start: 6644 + sd->in_net_rx_action = true; 6655 6645 local_irq_disable(); 6656 6646 list_splice_init(&sd->poll_list, &list); 6657 6647 local_irq_enable(); ··· 6664 6650 skb_defer_free_flush(sd); 6665 6651 6666 6652 if (list_empty(&list)) { 6667 - if (!sd_has_rps_ipi_waiting(sd) && list_empty(&repoll)) 6668 - goto end; 6653 + if (list_empty(&repoll)) { 6654 + sd->in_net_rx_action = false; 6655 + barrier(); 6656 + /* We need to check if ____napi_schedule() 6657 + * had refilled poll_list while 6658 + * sd->in_net_rx_action was true. 6659 + */ 6660 + if (!list_empty(&sd->poll_list)) 6661 + goto start; 6662 + if (!sd_has_rps_ipi_waiting(sd)) 6663 + goto end; 6664 + } 6669 6665 break; 6670 6666 } 6671 6667 ··· 6700 6676 list_splice(&list, &sd->poll_list); 6701 6677 if (!list_empty(&sd->poll_list)) 6702 6678 __raise_softirq_irqoff(NET_RX_SOFTIRQ); 6679 + else 6680 + sd->in_net_rx_action = false; 6703 6681 6704 6682 net_rps_action_and_irq_enable(sd); 6705 6683 end:;