From owner-freebsd-net@freebsd.org Tue Oct 20 11:03:36 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52D48A19523 for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2ACD4121B for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 29DE5A19522; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29743A19521 for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx141.netapp.com (mx141.netapp.com [216.240.21.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D07F1121A; Tue, 20 Oct 2015 11:03:35 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,707,1437462000"; d="asc'?scan'208";a="75991804" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx141-out.netapp.com with ESMTP; 20 Oct 2015 04:03:15 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Tue, 20 Oct 2015 04:03:15 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Tue, 20 Oct 2015 04:03:15 -0700 From: "Eggert, Lars" To: Ian Smith CC: Kevin Oberman , Luigi Rizzo , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAA== Date: Tue, 20 Oct 2015 11:03:14 +0000 Message-ID: <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> In-Reply-To: <20151020190541.B15983@sola.nimnet.asn.au> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 11:03:36 -0000 --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, On 2015-10-20, at 10:24, Ian Smith wrote: > Actually, you want to set hw.acpi.cpu.cx_lowest=3DC1 instead. Done. On 2015-10-19, at 17:55, Luigi Rizzo wrote: > On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: >> The only other sysctls in ixl(4) that look relevant are: >>=20 >> hw.ixl.rx_itr >> The RX interrupt rate value, set to 8K by default. >>=20 >> hw.ixl.tx_itr >> The TX interrupt rate value, set to 4K by default. >>=20 >=20 > yes those. raise to 20-50k and see what you get in > terms of ping latency. While ixl(4) talks about 8K and 4K, the defaults actually seem to be: hw.ixl.tx_itr: 122 hw.ixl.rx_itr: 62 Doubling those values *increases* flood ping latency to ~200 usec (from = ~116 usec). Halving them to 62/31 decreases flood ping latency to ~50 usec, but = still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 = further drops latency to 24 usec, with no change in throughput. (Looking at the "interrupt Moderation parameters" #defines in = sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates = specified with some weird divider scheme.) With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. = Unfortunately, throughput is then also down to about 2 Gb/s. One thing I noticed in top is that one queue irq is using quite a bit of = CPU when I run iperf: 11 0 -92 - 0K 1152K CPU2 2 0:19 50.98% = intr{irq293: ixl1:q2} 11 0 -92 - 0K 1152K WAIT 3 0:02 5.18% = intr{irq294: ixl1:q3} 0 0 -92 0 0K 8944K - 25 0:01 1.07% = kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT 1 0:01 0.00% = intr{irq292: ixl1:q1} 11 0 -92 - 0K 1152K WAIT 0 0:00 0.00% = intr{irq291: ixl1:q0} 0 0 -92 0 0K 8944K - 22 0:00 0.00% = kernel{ixl1 adminq} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT -1 0:00 0.00% = intr{irq290: ixl1:aq} With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much = lower: 11 0 -92 - 0K 1152K WAIT 0 0:05 7.67% = intr{irq274: ix0:que } 0 0 -92 0 0K 8944K - 27 0:00 0.29% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 10 0:00 0.00% = kernel{ix0 linkq} 11 0 -92 - 0K 1152K WAIT 1 0:00 0.00% = intr{irq275: ix0:que } 11 0 -92 - 0K 1152K WAIT 3 0:00 0.00% = intr{irq277: ix0:que } 11 0 -92 - 0K 1152K WAIT 2 0:00 0.00% = intr{irq276: ix0:que } 11 0 -92 - 0K 1152K WAIT 18 0:00 0.00% = intr{irq278: ix0:link} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} Lars --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViYfcdZcnpRveo1xAQiS1gP/bGeKFsEN+BVIzUbtCDdGkoDsDL7eZIuZ Qfdw8J63Azn4DfnPvgKlH330ESm45UaSYvojSQev16rO6BfU53Sa0K+7nzaSyqzf kkyiOseAXHTTsAOnL/IA4djPcdYLSD2qGRyx1LpeNc3uMaVshcxlzeCgFmSLyJa4 17FAveCAMk4= =1HCx -----END PGP SIGNATURE----- --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B--