Date: Thu, 10 Dec 2015 16:18:30 +0000 From: "Eggert, Lars" <lars@netapp.com> To: "Pieper, Jeffrey E" <jeffrey.e.pieper@intel.com> Cc: Kevin Oberman <rkoberman@gmail.com>, Daniel Engberg <daniel.engberg.lists@pyret.net>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <E21A5504-7780-4D84-AA5B-7A3F6F968FC7@netapp.com> In-Reply-To: <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com> References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <CAN6yY1t9Tw0j=uwaw1GK47r5=F-zeuz2hps_Ez3Y_QC-QSAGKA@mail.gmail.com> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 2015-10-26, at 18:40, Eggert, Lars <lars@netapp.com> wrote: > On 2015-10-26, at 17:08, Pieper, Jeffrey E <jeffrey.e.pieper@intel.com> wrote: >> As a caveat, this was using default netperf message sizes. > > I get the same ~3 Gb/s with the default netperf sizes and driver 1.4.5. Now there is version 1.4.8 on the Intel website, but it doesn't change things for me. > When you tcpdump during the run, do you see TSO/LRO in effect, i.e., do you see "segments" > 32K in the trace? I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note how all the packets are 1448 bytes: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 ecr 0], length 0 17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3553932482 ecr 478099], length 0 17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, win 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0 17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq 1:657, ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], length 656 17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq 1:657, ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr 478100], length 656 17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478102 ecr 0], length 0 17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2829000022 ecr 478102], length 0 17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0 17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1:1449, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448 ... Doing the same trace over 10G ix interfaces shows most segments in the 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G throughput). Lars [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- iQIcBAEBCAAGBQJWaaXVAAoJEFS1wwm/cMFX40sP/jfIYTvgeHj9V3NvdkSE9dC/ dtcjwPMzi6zz9nfcjC2LWmqpeeRu0AbKG/luo3jF0n1lg6j+1G5PxnenvSAfZPsP 8aavCT1xzXf9gevBRD2XKrptbcR4weTmLMuvf/r0GGgeDlYLCpkWXVK1QOrbYl9M OUUsl+Z/2YiRD2+u2NxWk4+YZOqyiLyuCvDSbNtK0U0A4Cm8StAQMI8wbn18N8HD tEzA0wW1GOh8ByRKwCUuzcGkDVJtVAz4lt/0BZhNoWBKXOBQcJNbnCODSn3vjMrh kwe5+87X385jTD1LQsO+7YfUx5RfUW+XOmpVmTGNGRQO6KLH5zPCVU54s6hkGoGs Sz65aPLW5nJY7fP5o22LdxaLz41MrwAbEbQgdBgiQDzCy/M5unsdvDblwRpKo02Y V5NZNSNv6Q8mZnc+iEFT818cDWakQ3TrX8tRtvMYKmnGdgOqf4vrXMLlmBKEJbcO JIeJPFNLPsWL/Ly0ztcjKUPOLFusUsxqhGRclPRUkaYWyTTN1UPvqxmU9Z+l6nYu KfldgW3ZDgAmWQy4XJV3wzN7pQ7hLpyUgA7lnk/AUsZ+dfEKWEmJuJ67T/6GczsZ XwD2PVYxim/V9rkd5t3AHHiCt7JZMMeLGoMytRWKBMR/j7dnxRDXHELKavFnJ4LH vNtS+06NjxyaSrmuO4ue =YDe4 -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E21A5504-7780-4D84-AA5B-7A3F6F968FC7>