Date: Thu, 10 Dec 2015 16:29:20 -0200 From: Denis Pearson <dennix.pearson@gmail.com> To: "Eggert, Lars" <lars@netapp.com> Cc: "Pieper, Jeffrey E" <jeffrey.e.pieper@intel.com>, Kevin Oberman <rkoberman@gmail.com>, Daniel Engberg <daniel.engberg.lists@pyret.net>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <CAFC3-mQ2d1GKCRCEF8%2BC-u1-EVy094w0zCDv7QHeYnaHK-NHrw@mail.gmail.com> In-Reply-To: <E21A5504-7780-4D84-AA5B-7A3F6F968FC7@netapp.com> References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <CAN6yY1t9Tw0j=uwaw1GK47r5=F-zeuz2hps_Ez3Y_QC-QSAGKA@mail.gmail.com> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com> <E21A5504-7780-4D84-AA5B-7A3F6F968FC7@netapp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars <lars@netapp.com> wrote: > On 2015-10-26, at 18:40, Eggert, Lars <lars@netapp.com> wrote: > > On 2015-10-26, at 17:08, Pieper, Jeffrey E <jeffrey.e.pieper@intel.com> > wrote: > >> As a caveat, this was using default netperf message sizes. > > > > I get the same ~3 Gb/s with the default netperf sizes and driver 1.4.5. > > Now there is version 1.4.8 on the Intel website, but it doesn't change > things for me. > I had the opportunity to see similar numbers and behavior while using XL710 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as of r292035 was providing expected numbers. While removing rxcsum/txcsum did not provide differences, fully removing RSS + disabling rx/cxsum support provided better numbers. However now with driver 1.4.8 and the same set of hardware setup, except for a different transceiver, I can get 36Gbps/24Mpps with no further tweaks, so if you can replace your transceiver, shall be a different test as a starting point. > > > When you tcpdump during the run, do you see TSO/LRO in effect, i.e., do > you see "segments" > 32K in the trace? > > I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note > how all the packets are 1448 bytes: > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 bytes > 17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq > 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 > ecr 0], length 0 > 17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq > 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale > 6,sackOK,TS val 3553932482 ecr 478099], length 0 > 17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, win > 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0 > 17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq 1:657, > ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], length 656 > 17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq 1:657, > ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr 478100], length > 656 > 17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq > 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478102 > ecr 0], length 0 > 17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq > 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale > 6,sackOK,TS val 2829000022 ecr 478102], length 0 > 17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, win > 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0 > 17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1:1449, > ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 > 17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], > length 1448 > 17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, > win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 > 17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr > 2829000022], length 1448 > 17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq > 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr > 2829000022], length 1448 > ... > > Doing the same trace over 10G ix interfaces shows most segments in the > 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G > throughput). > > Lars >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFC3-mQ2d1GKCRCEF8%2BC-u1-EVy094w0zCDv7QHeYnaHK-NHrw>