Date: Tue, 8 Sep 2020 23:10:14 +0530 From: Rajesh Kumar <rajfbsd@gmail.com> To: freebsd-net@freebsd.org Subject: Re: 10G Network interface not reaching wire speed on Receive path with single thread iperf run Message-ID: <CAAO%2BANPdpbkPB%2BwF%2BSdUWjSgMCHCwJ7d04RaXeu-G0PP2deXmg@mail.gmail.com> In-Reply-To: <CAAO%2BANPK6TX7HgR1sGHon87pyY8%2Bw82DO_6Zeb8mmYiPE4uAAg@mail.gmail.com> References: <CAAO%2BANOENM1Bt7bL4wfEtdLbAU4Wo1RAW9DT6DeZA=YtEvNx3A@mail.gmail.com> <CAHu1Y7021qxPkJ5q5mwKSZQB4Tn0_3JCta90RtcAeEMjbcLtHw@mail.gmail.com> <CAAO%2BANPK6TX7HgR1sGHon87pyY8%2Bw82DO_6Zeb8mmYiPE4uAAg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Guys, Has anyone got a chance to look into this? Any clues? Thanks, Rajesh. On Mon, Sep 7, 2020 at 11:33 PM Rajesh Kumar <rajfbsd@gmail.com> wrote: > Hi Michael, > > We use Interrupts for Rx packets. Whereas we poll for the link status. > > Thanks, > Rajesh. > > On Mon, Sep 7, 2020 at 11:17 PM Michael Sierchio <kudzu@tenebras.com> > wrote: > >> Are you using device polling or interrupts on the interface? >> >> On Mon, Sep 7, 2020 at 10:40 AM Rajesh Kumar <rajfbsd@gmail.com> wrote: >> >>> Hi FreeBSD team, >>> >>> I am writing a network driver using the iflib framework and using the >>> "iperf" tool for performance testing. >>> >>> 1) 10G link with 9000 MTU: Seeing line-rate numbers on both Tx and Rx >>> path. >>> 2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low >>> numbers on Rx path. >>> >>> Base issue seems to be that Rx buffers are not getting re-filled fast >>> enough. With low thread count, I see higher CPU utilization and lower >>> throughput. But with a higher number of threads, I could see better >>> throughput and lesser cpu utilization. >>> >>> iperf with 1 Thread: CPU Utilization is 80% - 95% - Throughput is ~4Gbp= s. >>> iperf with 2 Thread: CPU Utilization is 80% - 90% - Throughput is >>> ~7.5Gbps. >>> iperf with 4 Thread: CPU Utilization is 40% - 75% - Throughput is ~9Gbp= s. >>> iperf with 8 Thread: CPU Utilization is 30% - 70% - Throughput is >>> ~9.2Gbps. >>> >>> So, something is causing CPU utilization higher with lesser thread, whi= ch >>> kind of leads to Rx buffers not getting filled up faster and hence the >>> performance numbers drop. From a code perspective, I couldn't see any >>> evident CPU consuming operations in the Rx path. It's plain hardware >>> descriptors reading/processing. >>> >>> Anyway we can debug to find what is causing more CPU utilization (with >>> lesser threads) in my case? Any clues on debugging this issue? >>> >>> Thanks, >>> Rajesh. >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> >> >> -- >> >> "Well," Brahm=C4=81 said, "even after ten thousand explanations, a fool = is no >> wiser, but an intelligent person requires only two thousand five hundred= ." >> >> - The Mah=C4=81bh=C4=81rata >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANPdpbkPB%2BwF%2BSdUWjSgMCHCwJ7d04RaXeu-G0PP2deXmg>