Date: Sat, 23 Jan 2016 10:00:48 -0800 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Marcus Cenzatti <cenzatti@hush.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Navdeep Parhar <nparhar@gmail.com> Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <CA%2BhQ2%2Bg4qzCZFsQ9meQh8uWacWSOz4RDDtw0AnPzJ4%2BE5-9Ymg@mail.gmail.com> In-Reply-To: <20160123174840.32B1DA0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <CA%2BhQ2%2Bg4kU4LA4PexRPBv7z49ZWh-mDqdpw18SeoYaBueHyjZg@mail.gmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com>
index | next in thread | previous in thread | raw e-mail
On Sat, Jan 23, 2016 at 9:48 AM, Marcus Cenzatti <cenzatti@hush.com> wrote: > > > On 1/23/2016 at 3:35 PM, "Luigi Rizzo" <rizzo@iet.unipi.it> wrote: >> >>On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti >><cenzatti@hush.com> wrote: >>> >>> >>> On 1/23/2016 at 1:40 PM, "Navdeep Parhar" <nparhar@gmail.com> >>wrote: >>>> >>>>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>>>> hello, >>>>> >>>>> I am testing a chelsio t520-so-cr connected to a Intel card >>with >>>>ix(4) >>>>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>>>another >>>>> chelsio or to a Intel card. However I can only get 800Kpps- >>1Mpps >>>>for >>>>> RX tests from both chelsio or Intel. >>>>> >>>>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>>>> >>>>> I tested it untuned first and later I have applied tuning >>>>> recommendations I found on BSDRP[1] website. Results still >>>>ranging >>>>> from 800Kpps to 1Mpps for RX. >>>>> >>>>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>>>with >>>>> both IP address and MAC address source/dest. >>>> >>>>The ncxl interfaces have their own MAC addresses. Make sure the >>>>sender >>>>uses the MAC of the receiver's ncxl interface as the destination >>>>MAC. >>>>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >>>> >>>>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>>>| grep >>>>tx_pause). If it's receiving frames on netmap interface the >>>>tx_pause >>>>counter should not move. >>>> >>>>Regards, >>>>Navdeep >>>> >>> >>> hello, >>> >>> yes, MAC addresses are correct, I did the tests again and >>tx_pause won't move, here is the full transcript for the tests: >>> >>> ===> BOX #1 CHELSIO >>> >>> chelsio# ifconfig -v ncxl0 >>> ncxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric >>0 mtu 1500 >>> ether 00:07:43:33:8d:c1 >>> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >>> media: Ethernet 10Gbase-SR <full-duplex> >>> status: active >>> >>> chelsio# ifconfig -v cxl0 >>> cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric >>0 mtu 1500 >>> >>options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN >>_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> >>> ether 00:07:43:33:8d:c0 >>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >>> media: Ethernet 10Gbase-SR <full-duplex> >>> status: active >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >>> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K >>DATE: 2011-06-28 >>> module temperature: 42.79 C Voltage: 3.23 Volts >>> RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) >>> >>> chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s >>00:07:e9:44:d2:ba >>> 311.132189 main [1715] interface is ncxl0 >>> 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to >>0.0.0.0:90 >>> 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 >> >> >>wait, the lower case -s and -d are for IP addresses, >>you need to use -S and -D for the MAC addresses. >>This way you are sending broadcasts, which likely >>means that the chelsio is replicating the packets >>to both the netmap and the regular port and the >>latter (which perhaps comes first) is likely >>dropping packets. >> >>cheers >>luigi > > woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > curiously, I have always used -s/-d with IP addresses on ix-ix testing this is why I never noticed the case, since ix always received 14Mpps, but you probably explained it since ix has one single deviceport per wire, hence the different behavior > > performance stills very low when compared to TX and to what is expected ok so next we can try and see what else is going on. please check the following: a) are you connected through a switch ? if so, try to send out some packets through the ncxl0 port (using pkt-gen and its native MAC address) so the switch can learn the address and does not need to replicate traffic on all ports (which generally is done at a limited rate). b) see if using different packet sizes (say 256, 512, 1024, 1500 passed as the -l option to pkt-gen) affects the rx rate. If the rate does not change (except for 1500 bytes) it may be a problem with interrupt moderation c) use progressively increasing packet rates on the sender, using -R xxxx (start at 500000 packets per second, and then go up until the receiver cannot sustain the tx rate. d) use a smaller batch size on the receiver (-b XXX, use values such as 2, 4, 8, 16...) and see if things improve. Smaller batch sizes make pkt-gen check the NIC more often thus overcoming possible problems with interrupt moderation. Let us know the outcome. Depending on what you see there are several possible explanations. cheers luigihome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BhQ2%2Bg4qzCZFsQ9meQh8uWacWSOz4RDDtw0AnPzJ4%2BE5-9Ymg>