From owner-freebsd-net@freebsd.org Mon Oct 19 15:05:02 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED085A19E19 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CAB05182 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id C9665A19E18; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF183A19E17 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx144.netapp.com (mx144.netapp.com [216.240.21.25]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx144.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 649BA181; Mon, 19 Oct 2015 15:05:00 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,702,1437462000"; d="asc'?scan'208";a="74705270" Received: from hioexcmbx02-prd.hq.netapp.com ([10.122.105.35]) by mx144-out.netapp.com with ESMTP; 19 Oct 2015 08:03:54 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx02-prd.hq.netapp.com (10.122.105.35) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 19 Oct 2015 08:03:53 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Mon, 19 Oct 2015 08:03:54 -0700 From: "Eggert, Lars" To: Luigi Rizzo CC: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKAA= Date: Mon, 19 Oct 2015 15:03:53 +0000 Message-ID: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:05:02 -0000 --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, On 2015-10-19, at 16:20, Luigi Rizzo wrote: >=20 > i would look at the following: > - c states and clock speed - make sure you never go below C1, > and fix the clock speed to max. > Sure these parameters also affect the 10G card, but there > may be strange interaction that trigger the power saving > modes in different ways I already have powerd_flags=3D"-a max -b max -n max" in rc.conf, which I = hope should be enough. > - interrupt moderation (may affect ping latency, > do not remember how it is set in ixl but probably a sysctl ixl(4) describes two sysctls that sound like they control AIM, and they = default to off: hw.ixl.dynamic_tx_itr: 0 hw.ixl.dynamic_rx_itr: 0 > - number of queues (32 is a lot i wouldn't use more than 4-8), > may affect cpu-socket affinity With hw.ixl.max_queues=3D4 in loader.conf, performance is still = unchanged. > - tso and flow director - i have seen bad effects of > accelerations so i would run the iperf test with > of these features disabled on both sides, and then enable > them one at a time No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". How do I turn off flow director? > - queue sizes - the driver seems to use 1024 slots which is > about 1.5 MB queued, which in turn means you have 300us > (and possibly half of that) to drain the queue at 40Gbit/s. > 150-300us may seem an eternity, but if a couple of cores fall > into c7 your budget is gone and the loss will trigger a > retransmission and window halving etc. Also no change with "hw.ixl.ringsz=3D256" in loader.conf. This is really weird. Lars --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViUGW9ZcnpRveo1xAQjoBAP9F6YSpjWoZXl+C4yVN3eDWuzw9lCHfM7k r87TQ0avA1puKOTE84chmNCuhonnCvFpGWAscgmufZ/PeYkyR5TynFjwpMwjkVXB 4I18y/i7sOZqV98JOUWoVIBY+hhYb9BeitX7s+Ip6zoBgMILg82/mkn45tGb2gtV 6gC8c7VtXz4= =pAzg -----END PGP SIGNATURE----- --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2--