Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 7 Sep 2020 23:09:36 +0530
From:      Rajesh Kumar <rajfbsd@gmail.com>
To:        freebsd-net@freebsd.org
Subject:   10G Network interface not reaching wire speed on Receive path with single thread iperf run
Message-ID:  <CAAO%2BANOENM1Bt7bL4wfEtdLbAU4Wo1RAW9DT6DeZA=YtEvNx3A@mail.gmail.com>

next in thread | raw e-mail | index | archive | help
 Hi FreeBSD team,

I am writing a network driver using the iflib framework and using the
"iperf" tool for performance testing.

1) 10G link with 9000 MTU: Seeing line-rate numbers on both Tx and Rx path.
2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low
numbers on Rx path.

Base issue seems to be that Rx buffers are not getting re-filled fast
enough.  With low thread count, I see higher CPU utilization and lower
throughput.  But with a higher number of threads, I could see better
throughput and lesser cpu utilization.

iperf with 1 Thread: CPU Utilization is 80% - 95% - Throughput is ~4Gbps.
iperf with 2 Thread: CPU Utilization is 80% - 90% - Throughput is ~7.5Gbps.
iperf with 4 Thread: CPU Utilization is 40% - 75% - Throughput is ~9Gbps.
iperf with 8 Thread: CPU Utilization is 30% - 70% - Throughput is ~9.2Gbps.

So, something is causing CPU utilization higher with lesser thread, which
kind of leads to Rx buffers not getting filled up faster and hence the
performance numbers drop.  From a code perspective, I couldn't see any
evident CPU consuming operations in the Rx path.  It's plain hardware
descriptors reading/processing.

Anyway we can debug to find what is causing more CPU utilization (with
lesser threads) in my case? Any clues on debugging this issue?

Thanks,
Rajesh.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANOENM1Bt7bL4wfEtdLbAU4Wo1RAW9DT6DeZA=YtEvNx3A>