From owner-freebsd-net@FreeBSD.ORG Tue Sep 10 22:42:23 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 571647FD for ; Tue, 10 Sep 2013 22:42:23 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm18.bullet.mail.ne1.yahoo.com (nm18.bullet.mail.ne1.yahoo.com [98.138.90.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1881723E5 for ; Tue, 10 Sep 2013 22:42:22 +0000 (UTC) Received: from [98.138.90.50] by nm18.bullet.mail.ne1.yahoo.com with NNFMP; 10 Sep 2013 22:42:16 -0000 Received: from [98.138.88.234] by tm3.bullet.mail.ne1.yahoo.com with NNFMP; 10 Sep 2013 22:42:16 -0000 Received: from [127.0.0.1] by omp1034.mail.ne1.yahoo.com with NNFMP; 10 Sep 2013 22:42:16 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 367858.39232.bm@omp1034.mail.ne1.yahoo.com Received: (qmail 2624 invoked by uid 60001); 10 Sep 2013 22:42:16 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1378852936; bh=BImvDAgnFYFGDpdcT/DTTYL6vqhdaSF8OeFhyATH4kI=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=6mIWkb9VYRtRBBUSU+n8z056oQI3F+XJ3Vkb6H7X7TWYecd5iV1n+as/k5taDxIYhl3tYlRaY1g7FOdSf0mcMBLfC2/63DrTuzxKqd4bF9CntmxAKO3S/yMpu6JTQ5IJMvoyo8VP5TCh1d7/R0YZCR1i4meaTC4zBssX8QTTFJk= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=aDk9u6E4rbuC4whqpoVDY6chOgL6wEmlLrVsRjzxZ7LtW4RlZCGF4PWT4kbii01RwVj9Zpaotl1aCgmFOxM00Hatit32b0ntRRQ5VKLRAa+c7M5XZYvLFeLkUxgIG4xscVogm2ofKrSThlM/j320JQUMKgQBLP+M7NOH1VQGd6E=; X-YMail-OSG: 6SwEySMVM1nUPFeG.jFvPMIlbQRDl.w1iVVsoGa7HeY3FS8 xHIwEnQyjRqH0AtEIhTwHh00COPOE6Rpr9dJz7cMtVivPXXkA_2jfgRLdoAG NweJamhdRI_80OsiCmFs8UHIA2NDHWo9PAlNegh_QOEHXjCYSoDz_g03vKHl xWzPpJetcKhM_iyQCMmQJLYmw_CdWLLZoadqbazE7qhzmwEWmPradfbhJC85 VdahCNclRhzoltmtz_48aVxwmSRmP2U5A1FLzj5ccBVQaZEsunqCl1j508eq QjmqsEvk5f6yO7sQUm0s65dzDhw6vuZwGLgBHqpHsmAkY95uZjIknbc1ITyi 5iY3aTuaNySQdNCr5hdLokoLgu8.SwA.a0eBDdCtQId2lRwSAWONqQm8Sb2z O2qAd6lMTtTq6.k_xaeswegulCjjsKkhLJbNdhJ8sR9dbxm0gAMbFA9X17D2 snrHG476H8QsSr7vdjhWjfw9zJpwcbRo2TwCCBDWxv3ESysftbsd_fOSfz.W SUwv07QhzX2gjCBzI2m.kF5fFWM1RQhR.ZwDneHiHr.5lids3scyHSb7tkIT UtsLM3oft98jtisNnKX0n2PiwlLclmaRthV9RxR7eI8B7ozY_r6SKEI8oha0 Hmp3Cvv1Dke_IDjf.YEKENkKSgxyF8sf2rx0WDMJoqyxRhahIpG5d1rgCXiw _WY2wQRBE8wclBogNSyBCoom.0zD6obKHEF0JBKVwfSvAie8J Received: from [98.203.118.124] by web121601.mail.ne1.yahoo.com via HTTP; Tue, 10 Sep 2013 15:42:16 PDT X-Rocket-MIMEInfo: 002.001, TkZTIGhhcyBiZWVuIGJyb2tlbiBzaW5jZSBEYXkgMSwgc28gbGV0cyBub3QgY29tZSB0byBjb25jbHVzaW9ucyBhYm91dCBhbnl0aGluZwphcyBpdCByZWxhdGVzIHRvIE5GUy4KCkJDCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KIEZyb206IE1pa2UgVGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ.ClRvOiBSaWNrIE1hY2tsZW0gPHJtYWNrbGVtQHVvZ3VlbHBoLmNhPiAKQ2M6IEZyZWVCU0QgTmV0IDxuZXRAZnJlZWJzZC5vcmc.OyBEYXZpZCBXb2xmc2tpbGwgPGRhdmlkQGNhdHdoaXNrZXIub3IBMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> <52275136.1010105@sentex.net> Message-ID: <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Tue, 10 Sep 2013 15:42:16 -0700 (PDT) From: Barney Cordoba Subject: Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) To: Mike Tancsa , Rick Macklem In-Reply-To: <52275136.1010105@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: David Wolfskill , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Sep 2013 22:42:23 -0000 NFS has been broken since Day 1, so lets not come to conclusions about anyt= hing=0Aas it relates to NFS.=0A=0ABC=0A=0A=0A______________________________= __=0A From: Mike Tancsa =0ATo: Rick Macklem =0ACc: FreeBSD Net ; David Wolfskill =0ASent: Wednesday, September 4, 2013 11:26 AM=0ASubject: TSO he= lp or hindrance ? (was Re: TSO and FreeBSD vs Linux)=0A =0A=0AOn 9/4/2013 = 8:50 AM, Rick Macklem wrote:=0A> David Wolfskill wrote:=0A>>=0A>>=0A>> I no= ticed that when I tried to write files to NFS, I could write=0A>> small=0A>= > files OK, but larger ones seemed to ... hang.=0A>> * "ifconfig -v em0" sh= owed flags TSO4 & VLAN_HWTSO turned on.=0A>> * "sysctl net.inet.tcp.tso" sh= owed "1" -- enabled.=0A>>=0A>> As soon as I issued "sudo net.inet.tcp.tso= =3D0" ... the copy worked=0A>> without=0A>> a hitch or a whine.=A0 And I wa= s able to copy all 117709618 bytes, not=0A>> just=0A>> 2097152 (2^21).=0A>>= =0A>> Is the above expected?=A0 It came rather as a surprise to me.=0A>>=0A= > Not surprising to me, I'm afraid. When there are serious NFS problems=0A>= like this, it is often caused by a network fabric issue and broken=0A> TSO= is at the top of the list w.r.t. cause.=0A=0A=0AI was just experimenting a= bit with iSCSI via FreeNAS and was a little=0Adisappointed at the speeds I= was getting. So, I tried disabling tso on=0Aboth boxes and it did seem to = speed things up a bit.=A0 Data and testing=0Amethods attached in a txt file= .=0A=0AI did 3 cases.=0A=0AJust boot up FreeNAS and the initiator without t= weaks.=A0 That had the=0Aworst performance.=0Adisable tso on the nic as wel= l as via sysctl on both boxes. That had the=0Abest performance.=0Are-enable= tso on both boxes. That had better performance than the first=0Acase, but = still not as good as totally disabling it.=A0 I am guessing=0Asomething is = not quite being re-enabled properly ? But its different=0Athan the other tw= o cases ?!?=0A=0Atgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator i= s r254328 9.2=0AAMD64=0A=0AThe FreeNAS box has 16G of RAM, so the file is b= eing served out of cache=0Aas gstat shows no activity when sending out the = file=0A=0A=0A=0A=A0=A0=A0 ---Mike=0A=0A=0A-- =0A-------------------=0AMike = Tancsa, tel +1 519 651 3400=0ASentex Communications, mike@sentex.net=0AProv= iding Internet services since 1994 www.sentex.net=0ACambridge, Ontario Cana= da=A0 http://www.tancsa.com/=0A=0A_________________________________________= ______=0Afreebsd-net@freebsd.org mailing list=0Ahttp://lists.freebsd.org/ma= ilman/listinfo/freebsd-net=0ATo unsubscribe, send any mail to "freebsd-net-= unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Sep 10 22:54:34 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6C09DABB for ; Tue, 10 Sep 2013 22:54:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2A593248F for ; Tue, 10 Sep 2013 22:54:34 +0000 (UTC) Received: from [192.168.43.26] (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.7/8.14.7) with ESMTP id r8AMqvPS032514; Tue, 10 Sep 2013 18:52:57 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <522FA2C3.7010109@sentex.net> Date: Tue, 10 Sep 2013 18:52:51 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> <52275136.1010105@sentex.net> <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> In-Reply-To: <1378852936.95477.YahooMailNeo@web121601.mail.ne1.yahoo.com> X-Enigmail-Version: 1.4.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.74 on 64.7.153.18 Cc: David Wolfskill , Rick Macklem , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Sep 2013 22:54:34 -0000 On 9/10/2013 6:42 PM, Barney Cordoba wrote: > NFS has been broken since Day 1, so lets not come to conclusions about > anything > as it relates to NFS. iSCSI is NFS ? ---Mike > > BC > > ------------------------------------------------------------------------ > *From:* Mike Tancsa > *To:* Rick Macklem > *Cc:* FreeBSD Net ; David Wolfskill > *Sent:* Wednesday, September 4, 2013 11:26 AM > *Subject:* TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) > > On 9/4/2013 8:50 AM, Rick Macklem wrote: >> David Wolfskill wrote: >>> >>> >>> I noticed that when I tried to write files to NFS, I could write >>> small >>> files OK, but larger ones seemed to ... hang. >>> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. >>> * "sysctl net.inet.tcp.tso" showed "1" -- enabled. >>> >>> As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked >>> without >>> a hitch or a whine. And I was able to copy all 117709618 bytes, not >>> just >>> 2097152 (2^21). >>> >>> Is the above expected? It came rather as a surprise to me. >>> >> Not surprising to me, I'm afraid. When there are serious NFS problems >> like this, it is often caused by a network fabric issue and broken >> TSO is at the top of the list w.r.t. cause. > > > I was just experimenting a bit with iSCSI via FreeNAS and was a little > disappointed at the speeds I was getting. So, I tried disabling tso on > both boxes and it did seem to speed things up a bit. Data and testing > methods attached in a txt file. > > I did 3 cases. > > Just boot up FreeNAS and the initiator without tweaks. That had the > worst performance. > disable tso on the nic as well as via sysctl on both boxes. That had the > best performance. > re-enable tso on both boxes. That had better performance than the first > case, but still not as good as totally disabling it. I am guessing > something is not quite being re-enabled properly ? But its different > than the other two cases ?!? > > tgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator is r254328 9.2 > AMD64 > > The FreeNAS box has 16G of RAM, so the file is being served out of cache > as gstat shows no activity when sending out the file > > > > ---Mike > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/