Date: Mon, 7 Sep 2020 23:09:36 +0530 From: Rajesh Kumar <rajfbsd@gmail.com> To: freebsd-net@freebsd.org Subject: 10G Network interface not reaching wire speed on Receive path with single thread iperf run Message-ID: <CAAO%2BANOENM1Bt7bL4wfEtdLbAU4Wo1RAW9DT6DeZA=YtEvNx3A@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi FreeBSD team, I am writing a network driver using the iflib framework and using the "iperf" tool for performance testing. 1) 10G link with 9000 MTU: Seeing line-rate numbers on both Tx and Rx path. 2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low numbers on Rx path. Base issue seems to be that Rx buffers are not getting re-filled fast enough. With low thread count, I see higher CPU utilization and lower throughput. But with a higher number of threads, I could see better throughput and lesser cpu utilization. iperf with 1 Thread: CPU Utilization is 80% - 95% - Throughput is ~4Gbps. iperf with 2 Thread: CPU Utilization is 80% - 90% - Throughput is ~7.5Gbps. iperf with 4 Thread: CPU Utilization is 40% - 75% - Throughput is ~9Gbps. iperf with 8 Thread: CPU Utilization is 30% - 70% - Throughput is ~9.2Gbps. So, something is causing CPU utilization higher with lesser thread, which kind of leads to Rx buffers not getting filled up faster and hence the performance numbers drop. From a code perspective, I couldn't see any evident CPU consuming operations in the Rx path. It's plain hardware descriptors reading/processing. Anyway we can debug to find what is causing more CPU utilization (with lesser threads) in my case? Any clues on debugging this issue? Thanks, Rajesh.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANOENM1Bt7bL4wfEtdLbAU4Wo1RAW9DT6DeZA=YtEvNx3A>