Date: Sat, 23 Jan 2016 10:55:12 -0800 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Marcus Cenzatti <cenzatti@hush.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <CA%2BhQ2%2BjpemuCzRZy6QtQMyfpwOW17DtMr8SrUEXEm0w74nbmrg@mail.gmail.com> In-Reply-To: <20160123183447.4B150A0126@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <CA%2BhQ2%2Bg4kU4LA4PexRPBv7z49ZWh-mDqdpw18SeoYaBueHyjZg@mail.gmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <CA%2BhQ2%2Bg4qzCZFsQ9meQh8uWacWSOz4RDDtw0AnPzJ4%2BE5-9Ymg@mail.gmail.com> <20160123183447.4B150A0126@smtp.hushmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Jan 23, 2016 at 10:34 AM, Marcus Cenzatti <cenzatti@hush.com> wrote: >>> woops, my bad, yes probably we had some drop, with -S and -D now >>I get 1.2Mpps. >>> >>> curiously, I have always used -s/-d with IP addresses on ix-ix >>testing this is why I never noticed the case, since ix always >>received 14Mpps, but you probably explained it since ix has one >>single deviceport per wire, hence the different behavior >>> >>> performance stills very low when compared to TX and to what is >>expected >> >>ok so next we can try and see what else is going on. >>please check the following: >>a) are you connected through a switch ? if so, try to send >> out some packets through the ncxl0 port (using pkt-gen >> and its native MAC address) so the switch can learn the >> address and does not need to replicate traffic on all >> ports (which generally is done at a limited rate). >>b) see if using different packet sizes (say 256, 512, 1024, 1500 >> passed as the -l option to pkt-gen) affects the rx rate. >> If the rate does not change (except for 1500 bytes) >> it may be a problem with interrupt moderation >> >>c) use progressively increasing packet rates on the sender, >> using -R xxxx (start at 500000 packets per second, >> and then go up until the receiver cannot sustain the >> tx rate. >> >>d) use a smaller batch size on the receiver (-b XXX, use >> values such as 2, 4, 8, 16...) and see if things improve. >> Smaller batch sizes make pkt-gen check the NIC more often >> thus overcoming possible problems with interrupt moderation. >> >>Let us know the outcome. Depending on what you see there >>are several possible explanations. >> > > Ok, revisiting the summary > - TX host = Intel ix (host 1) > - RX host = Chelsio T520 (host 2) > - Simple topology host1==host2 directly connected intel port 0 (ix0) w/ chelsio port 0 (ncxl0). > > Tests results: > > => Batch 1 packet len > > TX at 256 bytes = 4.46Mpps/TX and 889Kpps/RX > TX at 256 bytes = 2.33Mpps/TX and 888Kpps/RX, 9.3Gbps on TX side according to pkt-gen > TX at 1024 bytes = 1.19Mpps/TX and 889Kpps/RX, 9.3Bps on TX > TX at 1500 bytes = 816Kpps/TX and 816Kpps/RX, 9.8Gbps on TX > > => Batch 2 rates > > -R 500000 / TX Speed: 499.99 Kpps Bandwidth: 240.00 Mbps (raw 336.00 Mbps) / RX 499Kpps > -R 700000 / TX Speed: 699.96 Kpps Bandwidth: 335.98 Mbps (raw 470.38 Mbps) / RX 699Kpps > -R 900000 / TX Speed: 899.98 Kpps Bandwidth: 431.99 Mbps (raw 604.78 Mbps) / RX 888Kpps > > reached the same limits on batch #1. > > => Batch 3 RX batch sizes, default pkt-gen packet len and fixed 900000 rate > > -r 2 / TX 899.98Kpps / RX 672Kpps > -r 4 / TX 899.98Kpps / RX 713Kpps > -r 8 / TX 899.98Kpps / RX 889Kpps > -r 16 / TX 899.98Kpps / RX 889Kpps > > Results make sense for rates bellow the max, but did not improve... only degraded. I think you are still sending to the broadcast addresss (otherwise i'd expect at least the 1.2 Mpps you were seeing before), and if this is the case the tests are not significant because you are hitting the in-nic duplication. Can you please re-run them sending to the MAC address of ncxl ? cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BhQ2%2BjpemuCzRZy6QtQMyfpwOW17DtMr8SrUEXEm0w74nbmrg>