From owner-freebsd-net@freebsd.org Mon Aug 17 12:21:21 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CADDD9B982E; Mon, 17 Aug 2015 12:21:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 6F4F410BD; Mon, 17 Aug 2015 12:21:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:YM7nTxamSOtZlVGynGzpRfT/LSx+4OfEezUN459isYplN5qZpMi7bnLW6fgltlLVR4KTs6sC0LqN9fy6EjBbqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh7v0p8eYP14ArQH+SI0xBS3+lR/WuMgSjNkqAYcK4TyNnEF1ff9Lz3hjP1OZkkW0zM6x+Jl+73YY4Kp5pIZoGJ/3dKUgTLFeEC9ucyVsvJWq5lH+SxCS7C4cTnkOiUgPRAzE9w3hGJnrvybwreY73zOVesj/TLQxUDLl66ZwVB7uhiBAOSQ0/WvMholrkKtRpB/ymxsq74fSYYyRfNBkd6XcZshSEWZIWMBAfydaRIOhbYpJBuFHPOIO/KfnoF5blxq1BkGJDejszjJNzivs2KQx0OAsFCnb2wM9EtYWsDLfpYOmZ+8pTempwfyQnn34ZPRM1GK4sdCQfw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BFAgDX0NFV/61jaINdDoNhaQaDHrpFAQmBawqFL0oCgWYUAQEBAQEBAQGBCYIdggYBAQEDAQEBASArIAsQAgEIDgoCAg0WAwICIQYBCRURAgQIBwQBHASHeAMKCA26CI9pDYVXAQEBAQEBBAEBAQEBARgEgSKKMIJPgWgBAQcVATMHgmmBQwWHIo17hQSFBnWDN5Eng0+DZQImgz9aIjMHfwgXI4EEAQEB X-IronPort-AV: E=Sophos;i="5.15,694,1432612800"; d="scan'208";a="231258543" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 17 Aug 2015 08:21:14 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 3283815F577; Mon, 17 Aug 2015 08:21:14 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id PbWpkVNngp3n; Mon, 17 Aug 2015 08:21:13 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9000215F578; Mon, 17 Aug 2015 08:21:13 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id N6GgvCEtRE1Y; Mon, 17 Aug 2015 08:21:13 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6426915F577; Mon, 17 Aug 2015 08:21:13 -0400 (EDT) Date: Mon, 17 Aug 2015 08:21:12 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Cc: Christopher Forgeron , FreeBSD Net , FreeBSD stable , Slawa Olhovchenkov Message-ID: <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: tmoW4T+6Z7dNXU5bN4I24LLlQtGA7w== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Aug 2015 12:21:21 -0000 Daniel Braniss wrote: >=20 > > On Aug 17, 2015, at 1:41 PM, Christopher Forgeron > > wrote: > >=20 > > FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD > > 10.1. Before 10.1 it was less. > >=20 >=20 > this is NOT iperf/3 where i do get close to wire speed, > it=E2=80=99s NFS writes, i.e., almost real work :-) >=20 > > I used to tweak the card settings, but now it's just stock. You may wan= t to > > check your settings, the Mellanox may just have better defaults for you= r > > switch. > >=20 Have you tried disabling TSO for the Intel? With TSO enabled, it will be co= pying every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag(= ) when TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 825= 99 chip these days?) This has been fixed in the driver very recently, but those fixes won't be i= n 10.1. rick ps: If you could test with 10.2, it would be interesting to see how the ix = does with the current driver fixes in it? > > On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov > > wrote: > > On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >=20 > > > hi, > > > I have a host (Dell R730) with both cards, connected to an HP82= 00 > > > switch at 10Gb. > > > when writing to the same storage (netapp) this is what I get: > > > ix0: ~130MGB/s > > > mlxen0 ~330MGB/s > > > this is via nfs/tcpv3 > > > > > > I can get similar (bad) performance with the mellanox if I incr= ease > > > the file size > > > to 512MGB. > >=20 > > Look like mellanox have internal beffer for caching and do ACK acclerat= ing. > >=20 > > > so at face value, it seems the mlxen does a better use of resou= rces > > > than the intel. > > > Any ideas how to improve ix/intel's performance? > >=20 > > Are you sure about netapp performance? > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > > " > >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"