Date: Thu, 10 Dec 2015 16:18:30 +0000 From: "Eggert, Lars" <lars@netapp.com> To: "Pieper, Jeffrey E" <jeffrey.e.pieper@intel.com> Cc: Kevin Oberman <rkoberman@gmail.com>, Daniel Engberg <daniel.engberg.lists@pyret.net>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <E21A5504-7780-4D84-AA5B-7A3F6F968FC7@netapp.com> In-Reply-To: <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com> References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <CAN6yY1t9Tw0j=uwaw1GK47r5=F-zeuz2hps_Ez3Y_QC-QSAGKA@mail.gmail.com> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 2015-10-26, at 18:40, Eggert, Lars <lars@netapp.com> wrote: > On 2015-10-26, at 17:08, Pieper, Jeffrey E = <jeffrey.e.pieper@intel.com> wrote: >> As a caveat, this was using default netperf message sizes. >=20 > I get the same ~3 Gb/s with the default netperf sizes and driver = 1.4.5. Now there is version 1.4.8 on the Intel website, but it doesn't change = things for me. > When you tcpdump during the run, do you see TSO/LRO in effect, i.e., = do you see "segments" > 32K in the trace? I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note = how all the packets are 1448 bytes: tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 = bytes 17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq = 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 = ecr 0], length 0 17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq = 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale = 6,sackOK,TS val 3553932482 ecr 478099], length 0 17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, = win 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0 17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq = 1:657, ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], = length 656 17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq = 1:657, ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr = 478100], length 656 17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq = 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val = 478102 ecr 0], length 0 17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq = 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale = 6,sackOK,TS val 2829000022 ecr 478102], length 0 17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, = win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0 17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 1:1449, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], = length 1448 17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 ... Doing the same trace over 10G ix interfaces shows most segments in the = 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G = throughput). Lars --Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQIcBAEBCAAGBQJWaaXVAAoJEFS1wwm/cMFX40sP/jfIYTvgeHj9V3NvdkSE9dC/ dtcjwPMzi6zz9nfcjC2LWmqpeeRu0AbKG/luo3jF0n1lg6j+1G5PxnenvSAfZPsP 8aavCT1xzXf9gevBRD2XKrptbcR4weTmLMuvf/r0GGgeDlYLCpkWXVK1QOrbYl9M OUUsl+Z/2YiRD2+u2NxWk4+YZOqyiLyuCvDSbNtK0U0A4Cm8StAQMI8wbn18N8HD tEzA0wW1GOh8ByRKwCUuzcGkDVJtVAz4lt/0BZhNoWBKXOBQcJNbnCODSn3vjMrh kwe5+87X385jTD1LQsO+7YfUx5RfUW+XOmpVmTGNGRQO6KLH5zPCVU54s6hkGoGs Sz65aPLW5nJY7fP5o22LdxaLz41MrwAbEbQgdBgiQDzCy/M5unsdvDblwRpKo02Y V5NZNSNv6Q8mZnc+iEFT818cDWakQ3TrX8tRtvMYKmnGdgOqf4vrXMLlmBKEJbcO JIeJPFNLPsWL/Ly0ztcjKUPOLFusUsxqhGRclPRUkaYWyTTN1UPvqxmU9Z+l6nYu KfldgW3ZDgAmWQy4XJV3wzN7pQ7hLpyUgA7lnk/AUsZ+dfEKWEmJuJ67T/6GczsZ XwD2PVYxim/V9rkd5t3AHHiCt7JZMMeLGoMytRWKBMR/j7dnxRDXHELKavFnJ4LH vNtS+06NjxyaSrmuO4ue =YDe4 -----END PGP SIGNATURE----- --Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E21A5504-7780-4D84-AA5B-7A3F6F968FC7>