From owner-freebsd-net@freebsd.org Thu Dec 10 16:23:38 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75A189D74B2 for ; Thu, 10 Dec 2015 16:23:38 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx143.netapp.com (mx143.netapp.com [216.240.21.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mx143.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52EF81282 for ; Thu, 10 Dec 2015 16:23:37 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.20,408,1444719600"; d="asc'?scan'208";a="84121130" Received: from hioexcmbx08-prd.hq.netapp.com ([10.122.105.41]) by mx143-out.netapp.com with ESMTP; 10 Dec 2015 08:18:31 -0800 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx08-prd.hq.netapp.com (10.122.105.41) with Microsoft SMTP Server (TLS) id 15.0.1130.7; Thu, 10 Dec 2015 08:18:31 -0800 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::942f:e59f:f84a:7191%21]) with mapi id 15.00.1130.005; Thu, 10 Dec 2015 08:18:31 -0800 From: "Eggert, Lars" To: "Pieper, Jeffrey E" CC: Kevin Oberman , Daniel Engberg , "freebsd-net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRDva+ZO+9KY7om02T8EKrgQTcE559lvyAgABhmICAAFavAIAACEoAgAARDQCAABmYAIBGsssA Date: Thu, 10 Dec 2015 16:18:30 +0000 Message-ID: References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3112) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 16:23:38 -0000 --Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 2015-10-26, at 18:40, Eggert, Lars wrote: > On 2015-10-26, at 17:08, Pieper, Jeffrey E = wrote: >> As a caveat, this was using default netperf message sizes. >=20 > I get the same ~3 Gb/s with the default netperf sizes and driver = 1.4.5. Now there is version 1.4.8 on the Intel website, but it doesn't change = things for me. > When you tcpdump during the run, do you see TSO/LRO in effect, i.e., = do you see "segments" > 32K in the trace? I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note = how all the packets are 1448 bytes: tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 = bytes 17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq = 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 = ecr 0], length 0 17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq = 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale = 6,sackOK,TS val 3553932482 ecr 478099], length 0 17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, = win 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0 17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq = 1:657, ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], = length 656 17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq = 1:657, ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr = 478100], length 656 17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq = 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val = 478102 ecr 0], length 0 17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq = 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale = 6,sackOK,TS val 2829000022 ecr 478102], length 0 17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, = win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0 17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 1:1449, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], = length 1448 17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, = win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0 17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq = 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr = 2829000022], length 1448 ... Doing the same trace over 10G ix interfaces shows most segments in the = 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G = throughput). Lars --Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQIcBAEBCAAGBQJWaaXVAAoJEFS1wwm/cMFX40sP/jfIYTvgeHj9V3NvdkSE9dC/ dtcjwPMzi6zz9nfcjC2LWmqpeeRu0AbKG/luo3jF0n1lg6j+1G5PxnenvSAfZPsP 8aavCT1xzXf9gevBRD2XKrptbcR4weTmLMuvf/r0GGgeDlYLCpkWXVK1QOrbYl9M OUUsl+Z/2YiRD2+u2NxWk4+YZOqyiLyuCvDSbNtK0U0A4Cm8StAQMI8wbn18N8HD tEzA0wW1GOh8ByRKwCUuzcGkDVJtVAz4lt/0BZhNoWBKXOBQcJNbnCODSn3vjMrh kwe5+87X385jTD1LQsO+7YfUx5RfUW+XOmpVmTGNGRQO6KLH5zPCVU54s6hkGoGs Sz65aPLW5nJY7fP5o22LdxaLz41MrwAbEbQgdBgiQDzCy/M5unsdvDblwRpKo02Y V5NZNSNv6Q8mZnc+iEFT818cDWakQ3TrX8tRtvMYKmnGdgOqf4vrXMLlmBKEJbcO JIeJPFNLPsWL/Ly0ztcjKUPOLFusUsxqhGRclPRUkaYWyTTN1UPvqxmU9Z+l6nYu KfldgW3ZDgAmWQy4XJV3wzN7pQ7hLpyUgA7lnk/AUsZ+dfEKWEmJuJ67T/6GczsZ XwD2PVYxim/V9rkd5t3AHHiCt7JZMMeLGoMytRWKBMR/j7dnxRDXHELKavFnJ4LH vNtS+06NjxyaSrmuO4ue =YDe4 -----END PGP SIGNATURE----- --Apple-Mail=_F348DDF7-77DE-4388-8A43-01DE983F8350--