From owner-freebsd-stable@freebsd.org Sun Aug 23 12:17:07 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6559C9BA4B8; Sun, 23 Aug 2015 12:17:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 190851A56; Sun, 23 Aug 2015 12:17:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1ZTUCi-0005eT-Cl; Sun, 23 Aug 2015 15:16:52 +0300 Date: Sun, 23 Aug 2015 15:16:52 +0300 From: Slawa Olhovchenkov To: Daniel Braniss Cc: Rick Macklem , pyunyh@gmail.com, Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Gleb Smirnoff , Christopher Forgeron Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <20150823121652.GJ3158@zxy.spb.ru> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 12:17:07 -0000 On Sun, Aug 23, 2015 at 02:08:56PM +0300, Daniel Braniss wrote: > >> send me the patch and I'll test it ASAP. > >> danny > >> > > Patch is attached. The one for head will also include an update to the comment > > in sys/net/if_var.h, but that isn't needed for testing. > > > well, the plot thickens. > > Yesterday, before running the new kernel, I decided to re run my test, and to my surprise > i was getting good numbers, about 300MGB/s with and without TSO. > > this morning, the numbers were again bad, around 70MGB/s,what the ^%$#@! > > so, after some coffee, I run some more tests, and some conclusions: > using a netapp(*) as the nfs client: > - doing > ifconfig ix0 tso or -tso > does some magic and numbers are back to normal - for a while > > using another Fbsd/zfs as client all is nifty, actually a bit faster than the netapp (not a fair > comparison, since the zfs client is not heavily used) and I can't see any degradation. > > btw, this is with the patch applied, but was seeing similar numbers before the patch. > > running with tso, initially I get around 300MGB/s, but after a while(sorry can't be more scientific) > it drops down to about half, and finally to a pathetic 70MGB/s > > *: while running the tests I monitored the Netapp, and nothing out of the ordinary there. Can you do this https://lists.freebsd.org/pipermail/freebsd-stable/2015-August/083138.html ?