From owner-freebsd-net@FreeBSD.ORG Sat May 18 12:11:25 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C6E69AC for ; Sat, 18 May 2013 12:11:25 +0000 (UTC) (envelope-from laurie_jennings_1977@yahoo.com) Received: from nm22-vm2.bullet.mail.ne1.yahoo.com (nm22-vm2.bullet.mail.ne1.yahoo.com [98.138.91.210]) by mx1.freebsd.org (Postfix) with ESMTP id DA8DEEA7 for ; Sat, 18 May 2013 12:11:24 +0000 (UTC) Received: from [98.138.226.180] by nm22.bullet.mail.ne1.yahoo.com with NNFMP; 18 May 2013 12:08:42 -0000 Received: from [98.138.89.169] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 18 May 2013 12:08:42 -0000 Received: from [127.0.0.1] by omp1025.mail.ne1.yahoo.com with NNFMP; 18 May 2013 12:08:42 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 813294.86267.bm@omp1025.mail.ne1.yahoo.com Received: (qmail 70265 invoked by uid 60001); 18 May 2013 12:08:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1368878922; bh=2OWRjfVN9nSoNJ2KUVouylMvxi1OWDQEaOkVtzO9NVs=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=MP7VsHhzPEAZnkC9Qrwab8BxjG3DqX7RkTU4dTORRpK2DUN5k3slKxmpjhJ7o99NpogUqE3nvPkOiQGrt+8sk4eMCkbaALbhe7MUQPjNjaSNIKl7YBfx5SMJ5vc8cRC2/kzCa30twl0SSTqw2TRtpcK4cqYmbNx1lNBU7TmP+04= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=s4bHXom68wNIVUSIO6llbaPl66/UOCR3SK0nUn5WUBDd4kFxmzC3p3x4/fZNP55krZri+ZoYxGB1v2Iejta4ZDEuShzBuF62CEUQ/wXsjwVvwBnIDdho28SD2Bkzw99rmPgg70G9ZLZwjVNv6lVK1CtvZ4Q3mWO/bJeQ1oGS5as=; X-YMail-OSG: yc2lvggVM1mff1KlpKDJIsdok02wl.wUQMByoAjtEWnorG3 b0HiSGWP6ZTKp49O3iHe.PGGGrG4rAz19fFXH6NnZcC6LJZYqjquyjHkAccD j4FnZxd9tPx2FcHK5kwJk_2UlKZuHlFZSDfLPEWH_gD3PLJhVmGxLNoDKSix HqxTh_KCpqmeDgYb80S0MRqucHGCi2LGWJxUr9.NEFnstsc.7cwkHHg_5o.n AGszEJLHPWNofEGgX77wTBtYi8GZq8f0kIOagpqFYR7ei_iceNgsKRdO1cEU NZCCX8cK0yZ0qHLr4k3J6PhiWqJ9WvRMvUXbA9de9vI15qftn63C9Ps6IpGy GzdT2HUWIzHo3zHZdJaOhC_89lv4eOGOpqm6T2Dj16zymo7sFqn0rYCR5Wac hJP8_Ee0TP4OJD4tTyjdH06dP.YNoMwroGGdtRVQULOjihxF8V2rUeUruyeS 9Vv0FV0I7Jh59k8kUxU8X.NPef4jfIdCg.TOH3Hfph80dFDDSj0ovGygjNQe Ai70KLLiabI9GVHwVsQus35Bs66RgrAhtA7h6zu9CuA-- Received: from [98.203.118.124] by web125801.mail.ne1.yahoo.com via HTTP; Sat, 18 May 2013 05:08:42 PDT X-Rocket-MIMEInfo: 002.001, Q2FuIHlvdSBvdXRsaW5lIHRoZSBjaGFuZ2VzPyBBbnl0aGluZyB3aXRoIHRoZSBnZW5lcmFsIHByb2Nlc3Npbmc_IEkgaGF2ZSB0byBtYWtlYSBjYXNlIHRvIGhvbGQgb2ZmIGEgZGVwbG95bWVudC4NCmFuZCB3aGF0IGhhcHBlbmVkIHRvIDksIDEwLCBhbmQgMTE_DQpMYXVyaWUNCg0KLS0tIE9uIEZyaSwgNS8xNy8xMywgSmFjayBWb2dlbCA8amZ2b2dlbEBnbWFpbC5jb20.IHdyb3RlOg0KDQpGcm9tOiBKYWNrIFZvZ2VsIDxqZnZvZ2VsQGdtYWlsLmNvbT4NClN1YmplY3Q6IFJlOiBJcyB0aGVyZSBhbnkgd2EBMAEBAQE- X-Mailer: YahooMailClassic/15.1.8 YahooMailWebService/0.8.142.542 Message-ID: <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> Date: Sat, 18 May 2013 05:08:42 -0700 (PDT) From: Laurie Jennings Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? To: Jack Vogel In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 May 2013 12:11:25 -0000 Can you outline the changes? Anything with the general processing? I have t= o makea case to hold off a deployment. and what happened to 9, 10, and 11? Laurie --- On Fri, 5/17/13, Jack Vogel wrote: From: Jack Vogel Subject: Re: Is there any way to limit the amount of data in an mbuf chain = submitted to a driver? To: "Laurie Jennings" Cc: "FreeBSD Net" , "Adrian Chadd" , "Richard Sharpe" Date: Friday, May 17, 2013, 3:32 PM No Laurie, I ran into an issue that I needed to resolve, and then my validation engineer went out of town a couple days. Should be early next week. Jack On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings < laurie_jennings_1977@yahoo.com> wrote: > Did you get to commit that new code yet? > > Laurie > > --- On *Sat, 5/4/13, Jack Vogel * wrote: > > > From: Jack Vogel > Subject: Re: Is there any way to limit the amount of data in an mbuf chai= n > submitted to a driver? > To: "Richard Sharpe" > Cc: "FreeBSD Net" , "Adrian Chadd" < > adrian@freebsd.org> > Date: Saturday, May 4, 2013, 2:18 PM > > > Ahh, Twinville, new hardware :)=C2=A0 The version at the tip is 2.5.8 and= I am > working on version 2.5.12 internally that I hope to commit next week... > so your version is "a bit old" :) I would do some testing on newer code. > > Jack > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > > >wrote: > > > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel > > wrote: > > > If you don't use TSO you will hurt your TX performance significantly > from > > > the tests that I've run. What exactly is the device you are using, I > > don't > > > have the source in front of me now, but I'm almost sure that the limi= t > is > > > not 64K but 256K, or are you using some ancient version of the driver= ? > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix0 pnpinfo vendor= =3D0x8086 device=3D0x1528 subvendor=3D0x8086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix1 pnpinfo vendor= =3D0x8086 device=3D0x1528 subvendor=3D0x8086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > > > The version calls itself ixgbe-2.4.4 ... > > > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > > > Jack > > > > > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > > realrichardsharpe@gmail.com > > > > > wrote: > > >> > > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd > > > > wrote: > > >> > On 4 May 2013 06:52, Richard Sharpe > > > > wrote: > > >> >> Hi folks, > > >> >> > > >> >> I understand better why I am seeing EINVAL intermittently when > > sending > > >> >> data from Samba via SMB2. > > >> >> > > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that > can > > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain > larger > > >> >> than that. > > >> >> > > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is > less > > >> >> than 64kiB of space available, so that is all TCP etc can put int= o > > the > > >> >> socket in one chain of mbufs. However, every now and then there i= s > > >> >> more than 65535 bytes available in the socket buffers, and we hav= e > an > > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > > >> >> > > >> >> To confirm this I am going to set SO_SNDBUF back to the default o= f > > >> >> 65536 and test again. My repros are very reliable. > > >> >> > > >> >> However, I wondered if my only way around this if I want to > continue > > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbu= f > > >> >> chains in the driver? > > >> > > > >> > Hm, is this is a problem without TSO? > > >> > > >> We are using the card without TSO, so I am thinking of changing that > > >> limit to 131072 and retesting. > > >> > > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > > problem. > > >> > > >> > Is the problem that the NIC can't handle a frame that big, or a > buffer > > >> > that big? > > >> > Ie - if you handed the hardware two descriptors of 64k each, for t= he > > >> > same IP datagram, will it complain? > > >> > > >> I can't find any documentation, but it seems that with TSO it cannot > > >> handle a frame that big. Actually, since we are not using TSO, there > > >> really should not be a problem with larger frames. > > >> > > >> > Or do you need to break it up into two separate IP datagrams, faci= ng > > >> > the driver, with a maximum size of 64k each? > > >> > > >> Not sure, but it looks like we need to do that. > > >> > > >> > > >> -- > > >> Regards, > > >> Richard Sharpe > > >> (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6= =9D=9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > > >> _______________________________________________ > > >> freebsd-net@freebsd.orgmailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g > " > > > > > > > > > > > > > > -- > > Regards, > > Richard Sharpe > > (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D= =9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > > > _______________________________________________ > freebsd-net@freebsd.org m= ailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat May 18 16:16:25 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 66EEF389 for ; Sat, 18 May 2013 16:16:25 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 27C8588F for ; Sat, 18 May 2013 16:16:25 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id cz11so4694764veb.6 for ; Sat, 18 May 2013 09:16:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=AjW0C2yl2nYSWjxeF6Bs2kgsSjeMu/Jq2K30UoLIW0Q=; b=AabCQ2d9ouHrRZuu+DRl7Oyymx7XNe+T2utvca3Fyvpw9lUxboZiEZudm/4xdOsX11 zrFVt+A71juGlydjVInbHisiUvSRH5WFyou2oUmmjaPvi/JToyXd+jFsF6ah136ud7cP aos4vo4KUWLXvYSSFhVAcA7JOfkgdMg+urm/TDOhI76aJwixmSj3a0XlJ9rPwrfB7Q1L XGU6qfGubo2+BaT2ONGsu0lmsUVU42dX2MqKUGzAUhzRgnGwC7jgEqa1Yl7h20AqBCF/ VTGAATSwFBol10q26CG1rdEFc05+qa42zp+bLSfLh2LLdCYvyP8XKXrkydAp2R7Xt1P2 QvIg== MIME-Version: 1.0 X-Received: by 10.52.171.135 with SMTP id au7mr307203vdc.126.1368893784098; Sat, 18 May 2013 09:16:24 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sat, 18 May 2013 09:16:23 -0700 (PDT) In-Reply-To: <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> References: <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> Date: Sat, 18 May 2013 09:16:23 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel To: Laurie Jennings Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 May 2013 16:16:25 -0000 Version numbers result from my work internally, and sometimes they increment due to a build issue, or a bug found that needs to be corrected, so unfortunately its not always a neat progression. I have thought at times about having a separate revision sequence but that would make for other issues, also not ever change made into the community source changes the version either. Its all because of the driver living in two different worlds you might say :) I don't have the list at hand right now, what were you planning to deploy instead? Jack On Sat, May 18, 2013 at 5:08 AM, Laurie Jennings < laurie_jennings_1977@yahoo.com> wrote: > Can you outline the changes? Anything with the general processing? I have > to make > a case to hold off a deployment. > > and what happened to 9, 10, and 11? > > Laurie > > > --- On *Fri, 5/17/13, Jack Vogel * wrote: > > > From: Jack Vogel > Subject: Re: Is there any way to limit the amount of data in an mbuf chai= n > submitted to a driver? > To: "Laurie Jennings" > Cc: "FreeBSD Net" , "Adrian Chadd" < > adrian@freebsd.org>, "Richard Sharpe" > Date: Friday, May 17, 2013, 3:32 PM > > No Laurie, I ran into an issue that I needed to resolve, and then my > validation engineer > went out of town a couple days. Should be early next week. > > Jack > > > > On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings < > laurie_jennings_1977@yahoo.com> > wrote: > > > Did you get to commit that new code yet? > > > > Laurie > > > > --- On *Sat, 5/4/13, Jack Vogel >* > wrote: > > > > > > From: Jack Vogel > > > > Subject: Re: Is there any way to limit the amount of data in an mbuf > chain > > submitted to a driver? > > To: "Richard Sharpe" > > > > Cc: "FreeBSD Net" >, > "Adrian Chadd" < > > adrian@freebsd.org > > > Date: Saturday, May 4, 2013, 2:18 PM > > > > > > Ahh, Twinville, new hardware :) The version at the tip is 2.5.8 and I = am > > working on version 2.5.12 internally that I hope to commit next week... > > so your version is "a bit old" :) I would do some testing on newer code= . > > > > Jack > > > > > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > > > > > >wrote: > > > > > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel > > > > > wrote: > > > > If you don't use TSO you will hurt your TX performance significantl= y > > from > > > > the tests that I've run. What exactly is the device you are using, = I > > > don't > > > > have the source in front of me now, but I'm almost sure that the > limit > > is > > > > not 64K but 256K, or are you using some ancient version of the > driver? > > > > > > ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0= x8086 > > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > > > ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0= x8086 > > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > > > > > The version calls itself ixgbe-2.4.4 ... > > > > > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > > > > > Jack > > > > > > > > > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > > > realrichardsharpe@gmail.com > > > > > > > > wrote: > > > >> > > > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd > > > > > > > wrote: > > > >> > On 4 May 2013 06:52, Richard Sharpe > > > > > > > > wrote: > > > >> >> Hi folks, > > > >> >> > > > >> >> I understand better why I am seeing EINVAL intermittently when > > > sending > > > >> >> data from Samba via SMB2. > > > >> >> > > > >> >> The ixgbe driver, for TSO reasons, limits the amount of data th= at > > can > > > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain > > larger > > > >> >> than that. > > > >> >> > > > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is > > less > > > >> >> than 64kiB of space available, so that is all TCP etc can put > into > > > the > > > >> >> socket in one chain of mbufs. However, every now and then there > is > > > >> >> more than 65535 bytes available in the socket buffers, and we > have > > an > > > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > > > >> >> > > > >> >> To confirm this I am going to set SO_SNDBUF back to the default > of > > > >> >> 65536 and test again. My repros are very reliable. > > > >> >> > > > >> >> However, I wondered if my only way around this if I want to > > continue > > > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large > mbuf > > > >> >> chains in the driver? > > > >> > > > > >> > Hm, is this is a problem without TSO? > > > >> > > > >> We are using the card without TSO, so I am thinking of changing th= at > > > >> limit to 131072 and retesting. > > > >> > > > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > > > problem. > > > >> > > > >> > Is the problem that the NIC can't handle a frame that big, or a > > buffer > > > >> > that big? > > > >> > Ie - if you handed the hardware two descriptors of 64k each, for > the > > > >> > same IP datagram, will it complain? > > > >> > > > >> I can't find any documentation, but it seems that with TSO it cann= ot > > > >> handle a frame that big. Actually, since we are not using TSO, the= re > > > >> really should not be a problem with larger frames. > > > >> > > > >> > Or do you need to break it up into two separate IP datagrams, > facing > > > >> > the driver, with a maximum size of 64k each? > > > >> > > > >> Not sure, but it looks like we need to do that. > > > >> > > > >> > > > >> -- > > > >> Regards, > > > >> Richard Sharpe > > > >> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > > >> _______________________________________________ > > > >> freebsd-net@freebsd.org > mailing list > > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > > > >> To unsubscribe, send any mail to " > freebsd-net-unsubscribe@freebsd.org > > > > " > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Richard Sharpe > > > (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > > > > _______________________________________________ > > freebsd-net@freebsd.org < > http://mc/compose?to=3Dfreebsd-net@freebsd.org>mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > > > > " > > > > > _______________________________________________ > freebsd-net@freebsd.org m= ailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > >