Date: Wed, 2 Sep 2020 15:13:39 +0530 From: Rajesh Kumar <rajfbsd@gmail.com> To: Daniel Ebdrup Jensen <debdrup@freebsd.org> Cc: freebsd-drivers@freebsd.org, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Network throughput not reaching line rate. Need clarification on iflib. Message-ID: <CAAO%2BANN5_omTSxu1fwT%2BOuU8ZmMqdw%2BhBkG88fLnMAMhqqEHCA@mail.gmail.com> In-Reply-To: <CAAO%2BANO_uo4gAbThG7X_jVfbQUQf8_J4%2BV07YMAROFo4CG8=Hw@mail.gmail.com> References: <CAAO%2BANPvfRhLiCZX=0UWDToKWNACrZ66JgOd%2Bs_p9fAgc%2BrWTw@mail.gmail.com> <7cfc7c52-b548-19bd-343b-899aca45c654@selasky.org> <CAJEDNsi%2B-zC=LEv5y67X7pcKP83hLP=uSqcZRwdye_gWKRLjvQ@mail.gmail.com> <f64e8fb8-0a72-ffc0-2301-a081399483b6@freebsd.org> <CAJEDNsg-dQA=7dwXfeSVTuch9oX0HT23JkLDangQUQtg=2A6ww@mail.gmail.com> <CAAO%2BANO_uo4gAbThG7X_jVfbQUQf8_J4%2BV07YMAROFo4CG8=Hw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Guys, Apologies for posting a question on an old thread. I started this thread early this year and it happened to pause working on this topic due to COVID lockdown. I recently resumed working on this and hence posting my questions on the same thread. To restate the issue, I am observing low network throughput numbers in the Receive path (on a 10G link) and trying to improve the performance numbers. As suggested in this thread, I tried using "iperf" instead of "iperf3" and also tried multiple threads. The behavior remains the same between iperf and iperf3. With a single thread I don't see any improvement in the performance numbers. But with multiple threads, I see slight improvement on the numbers, but still not touching the line rate for a 10G link. The base issue here is that the Receive buffers are not getting refilled fast enough for the Received packets. Hence I see a lot of retries (iperf3 output showing retries) and the performance drops. I could see the CPU utilization more than 90% when running iperf/iperf3 traffic. This could be the reason why Receive buffers are not filled faster. Receive path in my driver is pretty straight forward using the iflib framework. "rxd_available" interface returns the number of newly received packets. "rxd_refill" and "rxd_flush" interfaces fills the receive descriptors with buffers from iflib and update the buffer pointer registers. "rxd_pkt_get" interface traverses through all the descriptors for a packet return the received pkt info to iflib. I don't see any unwanted lock, loops and other evident bottlenecks in this path. So, what could be the reason for higher CPU utilization? and How can I debug this? Please let me know if any details are needed. Thanks, Rajesh. On Mon, Mar 2, 2020 at 4:37 AM Rajesh Kumar <rajfbsd@gmail.com> wrote: > Hi Guys, > > Thanks for your time in responding. > > As you people stated, the problem here doesn't seem to be single threaded > vs multi threaded. I am experience similar behavior with "iperf". It's > mostly to do with iflib and my driver(mostly my driver). It looks like the > receiver is slow in reading the packets than the sender sending the > packets. In this case, iflib drives the packet read through the exposed > interfaces (rxd_available, rxd_pkt_get etc.,). So, how to make the > receiver side read the packet faster with iflib? Is there anything that I > should take care in my driver in this regard? > > Thanks, > Rajesh. > > > > > On Sat, Feb 29, 2020 at 3:47 AM Daniel Ebdrup Jensen <debdrup@freebsd.org> > wrote: > >> On Fri, Feb 28, 2020 at 7:39 PM Bruce A. Mah <bmah@kitchenlab.org> wrote: >> >> > [Resending with a From: address that hopefully works better.] >> > >> > If memory serves me right, Daniel Ebdrup Jensen wrote: >> > > Yes, iperf3 will default to single-threaded packet generation, et al. >> > which >> > > favours fast cores with frequency boosting facilities. >> > > You might want to use iperf2 as that's properly multi-threaded, or you >> > can >> > > use pkt-gen out of src/tools/tools/netmap/ or ports/net/pkt-gen. >> > >> > While it's true that iperf3 is single-threaded, it should be capable of >> > saturating a 10GE link with a single TCP connection, given proper >> > command-line arguments (in particular, specifying a sufficiently large >> > socket-buffer size with the -w option). >> > >> > But based on the symptom of packet loss, I'd say the single-threaded vs. >> > multi-threaded argument might not be relevant to the problem that the OP >> > has. >> > >> > Bruce. >> > >> > > On Fri, Feb 28, 2020 at 10:35 AM Hans Petter Selasky <hps@selasky.org >> > >> > > wrote: >> > > >> > >> On 2020-02-28 10:03, Rajesh Kumar wrote: >> > >>> Hi FreeBSD team, >> > >>> >> > >>> I am writing a network driver using iflib framework and using >> "iperf3" >> > >> tool >> > >>> for performance testing. >> > >>> >> > >> >> > >> Is there any difference with "iperf" tool and using multiple >> threads? I >> > >> think iperf3 is single threaded ??? >> > >> >> > >> --HPS >> > >> >> > >> _______________________________________________ >> > >> freebsd-hackers@freebsd.org mailing list >> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > >> To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > >> >> > > _______________________________________________ >> > > freebsd-hackers@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > > To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > > >> > >> >> Oh, I didn't mean to imply that that wasn't part of the issue - I'm sorry >> if I made it sound like that. >> I was just confirming what Hans was asking, and possibly using the excuse >> to mention some things in base/ports that I think are also pretty neat. :) >> >> Also no longer top-posting, which was rather ghastly of me. I apologise. >> _______________________________________________ >> freebsd-drivers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org >> " >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANN5_omTSxu1fwT%2BOuU8ZmMqdw%2BhBkG88fLnMAMhqqEHCA>