Date: Tue, 10 Sep 2013 15:42:16 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Mike Tancsa <mike@sentex.net>, Rick Macklem <rmacklem@uoguelph.ca> Cc: David Wolfskill <david@catwhisker.org>, FreeBSD Net <net@freebsd.org> Subject: Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) Message-ID: <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> In-Reply-To: <52275136.1010105@sentex.net> References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> <52275136.1010105@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
NFS has been broken since Day 1, so lets not come to conclusions about anyt= hing=0Aas it relates to NFS.=0A=0ABC=0A=0A=0A______________________________= __=0A From: Mike Tancsa <mike@sentex.net>=0ATo: Rick Macklem <rmacklem@uogu= elph.ca> =0ACc: FreeBSD Net <net@freebsd.org>; David Wolfskill <david@catwh= isker.org> =0ASent: Wednesday, September 4, 2013 11:26 AM=0ASubject: TSO he= lp or hindrance ? (was Re: TSO and FreeBSD vs Linux)=0A =0A=0AOn 9/4/2013 = 8:50 AM, Rick Macklem wrote:=0A> David Wolfskill wrote:=0A>>=0A>>=0A>> I no= ticed that when I tried to write files to NFS, I could write=0A>> small=0A>= > files OK, but larger ones seemed to ... hang.=0A>> * "ifconfig -v em0" sh= owed flags TSO4 & VLAN_HWTSO turned on.=0A>> * "sysctl net.inet.tcp.tso" sh= owed "1" -- enabled.=0A>>=0A>> As soon as I issued "sudo net.inet.tcp.tso= =3D0" ... the copy worked=0A>> without=0A>> a hitch or a whine.=A0 And I wa= s able to copy all 117709618 bytes, not=0A>> just=0A>> 2097152 (2^21).=0A>>= =0A>> Is the above expected?=A0 It came rather as a surprise to me.=0A>>=0A= > Not surprising to me, I'm afraid. When there are serious NFS problems=0A>= like this, it is often caused by a network fabric issue and broken=0A> TSO= is at the top of the list w.r.t. cause.=0A=0A=0AI was just experimenting a= bit with iSCSI via FreeNAS and was a little=0Adisappointed at the speeds I= was getting. So, I tried disabling tso on=0Aboth boxes and it did seem to = speed things up a bit.=A0 Data and testing=0Amethods attached in a txt file= .=0A=0AI did 3 cases.=0A=0AJust boot up FreeNAS and the initiator without t= weaks.=A0 That had the=0Aworst performance.=0Adisable tso on the nic as wel= l as via sysctl on both boxes. That had the=0Abest performance.=0Are-enable= tso on both boxes. That had better performance than the first=0Acase, but = still not as good as totally disabling it.=A0 I am guessing=0Asomething is = not quite being re-enabled properly ? But its different=0Athan the other tw= o cases ?!?=0A=0Atgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator i= s r254328 9.2=0AAMD64=0A=0AThe FreeNAS box has 16G of RAM, so the file is b= eing served out of cache=0Aas gstat shows no activity when sending out the = file=0A=0A=0A=0A=A0=A0=A0 ---Mike=0A=0A=0A-- =0A-------------------=0AMike = Tancsa, tel +1 519 651 3400=0ASentex Communications, mike@sentex.net=0AProv= iding Internet services since 1994 www.sentex.net=0ACambridge, Ontario Cana= da=A0 http://www.tancsa.com/=0A=0A_________________________________________= ______=0Afreebsd-net@freebsd.org mailing list=0Ahttp://lists.freebsd.org/ma= ilman/listinfo/freebsd-net=0ATo unsubscribe, send any mail to "freebsd-net-= unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Sep 10 22:54:34 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6C09DABB for <net@freebsd.org>; Tue, 10 Sep 2013 22:54:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2A593248F for <net@freebsd.org>; Tue, 10 Sep 2013 22:54:34 +0000 (UTC) Received: from [192.168.43.26] (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.7/8.14.7) with ESMTP id r8AMqvPS032514; Tue, 10 Sep 2013 18:52:57 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <522FA2C3.7010109@sentex.net> Date: Tue, 10 Sep 2013 18:52:51 -0400 From: Mike Tancsa <mike@sentex.net> Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Barney Cordoba <barney_cordoba@yahoo.com> Subject: Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> <52275136.1010105@sentex.net> <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> In-Reply-To: <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> X-Enigmail-Version: 1.4.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.74 on 64.7.153.18 Cc: David Wolfskill <david@catwhisker.org>, Rick Macklem <rmacklem@uoguelph.ca>, FreeBSD Net <net@freebsd.org> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 10 Sep 2013 22:54:34 -0000 On 9/10/2013 6:42 PM, Barney Cordoba wrote: > NFS has been broken since Day 1, so lets not come to conclusions about > anything > as it relates to NFS. iSCSI is NFS ? ---Mike > > BC > > ------------------------------------------------------------------------ > *From:* Mike Tancsa <mike@sentex.net> > *To:* Rick Macklem <rmacklem@uoguelph.ca> > *Cc:* FreeBSD Net <net@freebsd.org>; David Wolfskill <david@catwhisker.org> > *Sent:* Wednesday, September 4, 2013 11:26 AM > *Subject:* TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) > > On 9/4/2013 8:50 AM, Rick Macklem wrote: >> David Wolfskill wrote: >>> >>> >>> I noticed that when I tried to write files to NFS, I could write >>> small >>> files OK, but larger ones seemed to ... hang. >>> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. >>> * "sysctl net.inet.tcp.tso" showed "1" -- enabled. >>> >>> As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked >>> without >>> a hitch or a whine. And I was able to copy all 117709618 bytes, not >>> just >>> 2097152 (2^21). >>> >>> Is the above expected? It came rather as a surprise to me. >>> >> Not surprising to me, I'm afraid. When there are serious NFS problems >> like this, it is often caused by a network fabric issue and broken >> TSO is at the top of the list w.r.t. cause. > > > I was just experimenting a bit with iSCSI via FreeNAS and was a little > disappointed at the speeds I was getting. So, I tried disabling tso on > both boxes and it did seem to speed things up a bit. Data and testing > methods attached in a txt file. > > I did 3 cases. > > Just boot up FreeNAS and the initiator without tweaks. That had the > worst performance. > disable tso on the nic as well as via sysctl on both boxes. That had the > best performance. > re-enable tso on both boxes. That had better performance than the first > case, but still not as good as totally disabling it. I am guessing > something is not quite being re-enabled properly ? But its different > than the other two cases ?!? > > tgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator is r254328 9.2 > AMD64 > > The FreeNAS box has 16G of RAM, so the file is being served out of cache > as gstat shows no activity when sending out the file > > > > ---Mike > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net <mailto:mike@sentex.net> > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > > _______________________________________________ > freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > <mailto:freebsd-net-unsubscribe@freebsd.org>" > -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1378852936.95477.YahooMailNeo>