From owner-freebsd-net@freebsd.org Mon Oct 19 15:36:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E057A1965F for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 79FB9179F for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 7695AA1965E; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C5E9A1965D for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx141.netapp.com (mx141.netapp.com [216.240.21.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AEE5179E; Mon, 19 Oct 2015 15:36:05 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,702,1437462000"; d="asc'?scan'208";a="75767988" Received: from hioexcmbx03-prd.hq.netapp.com ([10.122.105.36]) by mx141-out.netapp.com with ESMTP; 19 Oct 2015 08:34:51 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx03-prd.hq.netapp.com (10.122.105.36) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 19 Oct 2015 08:34:48 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Mon, 19 Oct 2015 08:34:48 -0700 From: "Eggert, Lars" To: Luigi Rizzo CC: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAABpIA Date: Mon, 19 Oct 2015 15:34:47 +0000 Message-ID: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:36:06 -0000 --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, in order to eliminate network or hardware weirdness, I've rerun the test = with Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec = flood ping latency. Not great either, but in line with earlier = experiments with Mellanox NICs and an untuned Linux system. On 2015-10-19, at 17:11, Luigi Rizzo wrote: > I suspect it might not touch the c states, but better check. The = safest is > disable them in the bios. I'll try that. >> hw.ixl.dynamic_tx_itr: 0 >> hw.ixl.dynamic_rx_itr: 0 >>=20 >>=20 > There must be some other control for the actual (fixed, not dynamic) > moderation. The only other sysctls in ixl(4) that look relevant are: hw.ixl.rx_itr The RX interrupt rate value, set to 8K by default. hw.ixl.tx_itr The TX interrupt rate value, set to 4K by default. I'll play with those. >> Also no change with "hw.ixl.ringsz=3D256" in loader.conf. >=20 > Any better success with 2048 slots? > 3.5 gbit is what I used to see on the ixgbe with tso disabled, = probably > hitting a CPU bound. Will try. Thanks! Lars --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViUNm9ZcnpRveo1xAQihLQP/TL9IlhVgdX5D4soii576IC/ZSqq4xYw3 h2iT2QjQBNks6US3p2pv8ZT+5SkmtbN0c8GPFd3mmJ/PM2aov/JSb2wp+Xmz+zTD MOqjhGXKs84g5dc1Cb83dlkm2EmrrGxotpIvYp+ioui13BNqxDWFYVvIXyPN/wQP Dx+sdHklhHo= =74hg -----END PGP SIGNATURE----- --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02--