Date: Sat, 18 May 2013 05:08:42 -0700 (PDT) From: Laurie Jennings <laurie_jennings_1977@yahoo.com> To: Jack Vogel <jfvogel@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? Message-ID: <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> In-Reply-To: <CAFOYbcnu6jA5Zv1Qc=jeavGqWMqwkHzSo%2B8RVUGtZCPE3f=0Fg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Can you outline the changes? Anything with the general processing? I have t= o makea case to hold off a deployment. and what happened to 9, 10, and 11? Laurie --- On Fri, 5/17/13, Jack Vogel <jfvogel@gmail.com> wrote: From: Jack Vogel <jfvogel@gmail.com> Subject: Re: Is there any way to limit the amount of data in an mbuf chain = submitted to a driver? To: "Laurie Jennings" <laurie_jennings_1977@yahoo.com> Cc: "FreeBSD Net" <freebsd-net@freebsd.org>, "Adrian Chadd" <adrian@freebsd= .org>, "Richard Sharpe" <realrichardsharpe@gmail.com> Date: Friday, May 17, 2013, 3:32 PM No Laurie, I ran into an issue that I needed to resolve, and then my validation engineer went out of town a couple days. Should be early next week. Jack On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings < laurie_jennings_1977@yahoo.com> wrote: > Did you get to commit that new code yet? > > Laurie > > --- On *Sat, 5/4/13, Jack Vogel <jfvogel@gmail.com>* wrote: > > > From: Jack Vogel <jfvogel@gmail.com> > Subject: Re: Is there any way to limit the amount of data in an mbuf chai= n > submitted to a driver? > To: "Richard Sharpe" <realrichardsharpe@gmail.com> > Cc: "FreeBSD Net" <freebsd-net@freebsd.org>, "Adrian Chadd" < > adrian@freebsd.org> > Date: Saturday, May 4, 2013, 2:18 PM > > > Ahh, Twinville, new hardware :)=C2=A0 The version at the tip is 2.5.8 and= I am > working on version 2.5.12 internally that I hope to commit next week... > so your version is "a bit old" :) I would do some testing on newer code. > > Jack > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > <realrichardsharpe@gmail.com<http://mc/compose?to=3Drealrichardsharpe@gma= il.com> > >wrote: > > > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel <jfvogel@gmail.com<http://mc= /compose?to=3Djfvogel@gmail.com>> > wrote: > > > If you don't use TSO you will hurt your TX performance significantly > from > > > the tests that I've run. What exactly is the device you are using, I > > don't > > > have the source in front of me now, but I'm almost sure that the limi= t > is > > > not 64K but 256K, or are you using some ancient version of the driver= ? > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix0 pnpinfo vendor= =3D0x8086 device=3D0x1528 subvendor=3D0x8086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix1 pnpinfo vendor= =3D0x8086 device=3D0x1528 subvendor=3D0x8086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > > > The version calls itself ixgbe-2.4.4 ... > > > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > > > Jack > > > > > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > > realrichardsharpe@gmail.com<http://mc/compose?to=3Drealrichardsharpe@gm= ail.com> > > > > > wrote: > > >> > > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian@freebsd.org<ht= tp://mc/compose?to=3Dadrian@freebsd.org> > > > > wrote: > > >> > On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe@gmail.com<h= ttp://mc/compose?to=3Drealrichardsharpe@gmail.com> > > > > wrote: > > >> >> Hi folks, > > >> >> > > >> >> I understand better why I am seeing EINVAL intermittently when > > sending > > >> >> data from Samba via SMB2. > > >> >> > > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that > can > > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain > larger > > >> >> than that. > > >> >> > > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is > less > > >> >> than 64kiB of space available, so that is all TCP etc can put int= o > > the > > >> >> socket in one chain of mbufs. However, every now and then there i= s > > >> >> more than 65535 bytes available in the socket buffers, and we hav= e > an > > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > > >> >> > > >> >> To confirm this I am going to set SO_SNDBUF back to the default o= f > > >> >> 65536 and test again. My repros are very reliable. > > >> >> > > >> >> However, I wondered if my only way around this if I want to > continue > > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbu= f > > >> >> chains in the driver? > > >> > > > >> > Hm, is this is a problem without TSO? > > >> > > >> We are using the card without TSO, so I am thinking of changing that > > >> limit to 131072 and retesting. > > >> > > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > > problem. > > >> > > >> > Is the problem that the NIC can't handle a frame that big, or a > buffer > > >> > that big? > > >> > Ie - if you handed the hardware two descriptors of 64k each, for t= he > > >> > same IP datagram, will it complain? > > >> > > >> I can't find any documentation, but it seems that with TSO it cannot > > >> handle a frame that big. Actually, since we are not using TSO, there > > >> really should not be a problem with larger frames. > > >> > > >> > Or do you need to break it up into two separate IP datagrams, faci= ng > > >> > the driver, with a maximum size of 64k each? > > >> > > >> Not sure, but it looks like we need to do that. > > >> > > >> > > >> -- > > >> Regards, > > >> Richard Sharpe > > >> (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6= =9D=9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > > >> _______________________________________________ > > >> freebsd-net@freebsd.org<http://mc/compose?to=3Dfreebsd-net@freebsd.o= rg>mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g<http://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org>; > " > > > > > > > > > > > > > > -- > > Regards, > > Richard Sharpe > > (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D= =9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > > > _______________________________________________ > freebsd-net@freebsd.org <http://mc/compose?to=3Dfreebsd-net@freebsd.org>m= ailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<htt= p://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org> > " > > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat May 18 16:16:25 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 66EEF389 for <freebsd-net@freebsd.org>; Sat, 18 May 2013 16:16:25 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 27C8588F for <freebsd-net@freebsd.org>; Sat, 18 May 2013 16:16:25 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id cz11so4694764veb.6 for <freebsd-net@freebsd.org>; Sat, 18 May 2013 09:16:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=AjW0C2yl2nYSWjxeF6Bs2kgsSjeMu/Jq2K30UoLIW0Q=; b=AabCQ2d9ouHrRZuu+DRl7Oyymx7XNe+T2utvca3Fyvpw9lUxboZiEZudm/4xdOsX11 zrFVt+A71juGlydjVInbHisiUvSRH5WFyou2oUmmjaPvi/JToyXd+jFsF6ah136ud7cP aos4vo4KUWLXvYSSFhVAcA7JOfkgdMg+urm/TDOhI76aJwixmSj3a0XlJ9rPwrfB7Q1L XGU6qfGubo2+BaT2ONGsu0lmsUVU42dX2MqKUGzAUhzRgnGwC7jgEqa1Yl7h20AqBCF/ VTGAATSwFBol10q26CG1rdEFc05+qa42zp+bLSfLh2LLdCYvyP8XKXrkydAp2R7Xt1P2 QvIg== MIME-Version: 1.0 X-Received: by 10.52.171.135 with SMTP id au7mr307203vdc.126.1368893784098; Sat, 18 May 2013 09:16:24 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sat, 18 May 2013 09:16:23 -0700 (PDT) In-Reply-To: <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> References: <CAFOYbcnu6jA5Zv1Qc=jeavGqWMqwkHzSo+8RVUGtZCPE3f=0Fg@mail.gmail.com> <1368878922.70089.YahooMailClassic@web125801.mail.ne1.yahoo.com> Date: Sat, 18 May 2013 09:16:23 -0700 Message-ID: <CAFOYbckvv1-cNv_upiBWQQOdomq4g_X+jcBv-0SUU6JSG+PajQ@mail.gmail.com> Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel <jfvogel@gmail.com> To: Laurie Jennings <laurie_jennings_1977@yahoo.com> Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net <freebsd-net@freebsd.org> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 18 May 2013 16:16:25 -0000 Version numbers result from my work internally, and sometimes they increment due to a build issue, or a bug found that needs to be corrected, so unfortunately its not always a neat progression. I have thought at times about having a separate revision sequence but that would make for other issues, also not ever change made into the community source changes the version either. Its all because of the driver living in two different worlds you might say :) I don't have the list at hand right now, what were you planning to deploy instead? Jack On Sat, May 18, 2013 at 5:08 AM, Laurie Jennings < laurie_jennings_1977@yahoo.com> wrote: > Can you outline the changes? Anything with the general processing? I have > to make > a case to hold off a deployment. > > and what happened to 9, 10, and 11? > > Laurie > > > --- On *Fri, 5/17/13, Jack Vogel <jfvogel@gmail.com>* wrote: > > > From: Jack Vogel <jfvogel@gmail.com> > Subject: Re: Is there any way to limit the amount of data in an mbuf chai= n > submitted to a driver? > To: "Laurie Jennings" <laurie_jennings_1977@yahoo.com> > Cc: "FreeBSD Net" <freebsd-net@freebsd.org>, "Adrian Chadd" < > adrian@freebsd.org>, "Richard Sharpe" <realrichardsharpe@gmail.com> > Date: Friday, May 17, 2013, 3:32 PM > > No Laurie, I ran into an issue that I needed to resolve, and then my > validation engineer > went out of town a couple days. Should be early next week. > > Jack > > > > On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings < > laurie_jennings_1977@yahoo.com<http://mc/compose?to=3Dlaurie_jennings_197= 7@yahoo.com>> > wrote: > > > Did you get to commit that new code yet? > > > > Laurie > > > > --- On *Sat, 5/4/13, Jack Vogel <jfvogel@gmail.com<http://mc/compose?to= =3Djfvogel@gmail.com>>* > wrote: > > > > > > From: Jack Vogel <jfvogel@gmail.com<http://mc/compose?to=3Djfvogel@gmai= l.com> > > > > Subject: Re: Is there any way to limit the amount of data in an mbuf > chain > > submitted to a driver? > > To: "Richard Sharpe" <realrichardsharpe@gmail.com<http://mc/compose?to= =3Drealrichardsharpe@gmail.com> > > > > Cc: "FreeBSD Net" <freebsd-net@freebsd.org<http://mc/compose?to=3Dfreeb= sd-net@freebsd.org>>, > "Adrian Chadd" < > > adrian@freebsd.org <http://mc/compose?to=3Dadrian@freebsd.org>>; > > Date: Saturday, May 4, 2013, 2:18 PM > > > > > > Ahh, Twinville, new hardware :) The version at the tip is 2.5.8 and I = am > > working on version 2.5.12 internally that I hope to commit next week... > > so your version is "a bit old" :) I would do some testing on newer code= . > > > > Jack > > > > > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > > <realrichardsharpe@gmail.com<http://mc/compose?to=3Drealrichardsharpe@g= mail.com> > <http://mc/compose?to=3Drealrichardsharpe@gmail.com>; > > >wrote: > > > > > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel <jfvogel@gmail.com<http://= mc/compose?to=3Djfvogel@gmail.com> > <http://mc/compose?to=3Djfvogel@gmail.com>>; > > > wrote: > > > > If you don't use TSO you will hurt your TX performance significantl= y > > from > > > > the tests that I've run. What exactly is the device you are using, = I > > > don't > > > > have the source in front of me now, but I'm almost sure that the > limit > > is > > > > not 64K but 256K, or are you using some ancient version of the > driver? > > > > > > ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0= x8086 > > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > > > ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0= x8086 > > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > > > > > The version calls itself ixgbe-2.4.4 ... > > > > > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > > > > > Jack > > > > > > > > > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > > > realrichardsharpe@gmail.com<http://mc/compose?to=3Drealrichardsharpe@= gmail.com> > <http://mc/compose?to=3Drealrichardsharpe@gmail.com>; > > > > > > > wrote: > > > >> > > > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian@freebsd.org<= http://mc/compose?to=3Dadrian@freebsd.org>; > <http://mc/compose?to=3Dadrian@freebsd.org>; > > > > > > wrote: > > > >> > On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe@gmail.com= <http://mc/compose?to=3Drealrichardsharpe@gmail.com>; > <http://mc/compose?to=3Drealrichardsharpe@gmail.com>; > > > > > > > wrote: > > > >> >> Hi folks, > > > >> >> > > > >> >> I understand better why I am seeing EINVAL intermittently when > > > sending > > > >> >> data from Samba via SMB2. > > > >> >> > > > >> >> The ixgbe driver, for TSO reasons, limits the amount of data th= at > > can > > > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain > > larger > > > >> >> than that. > > > >> >> > > > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is > > less > > > >> >> than 64kiB of space available, so that is all TCP etc can put > into > > > the > > > >> >> socket in one chain of mbufs. However, every now and then there > is > > > >> >> more than 65535 bytes available in the socket buffers, and we > have > > an > > > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > > > >> >> > > > >> >> To confirm this I am going to set SO_SNDBUF back to the default > of > > > >> >> 65536 and test again. My repros are very reliable. > > > >> >> > > > >> >> However, I wondered if my only way around this if I want to > > continue > > > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large > mbuf > > > >> >> chains in the driver? > > > >> > > > > >> > Hm, is this is a problem without TSO? > > > >> > > > >> We are using the card without TSO, so I am thinking of changing th= at > > > >> limit to 131072 and retesting. > > > >> > > > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > > > problem. > > > >> > > > >> > Is the problem that the NIC can't handle a frame that big, or a > > buffer > > > >> > that big? > > > >> > Ie - if you handed the hardware two descriptors of 64k each, for > the > > > >> > same IP datagram, will it complain? > > > >> > > > >> I can't find any documentation, but it seems that with TSO it cann= ot > > > >> handle a frame that big. Actually, since we are not using TSO, the= re > > > >> really should not be a problem with larger frames. > > > >> > > > >> > Or do you need to break it up into two separate IP datagrams, > facing > > > >> > the driver, with a maximum size of 64k each? > > > >> > > > >> Not sure, but it looks like we need to do that. > > > >> > > > >> > > > >> -- > > > >> Regards, > > > >> Richard Sharpe > > > >> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > > >> _______________________________________________ > > > >> freebsd-net@freebsd.org<http://mc/compose?to=3Dfreebsd-net@freebsd= .org> > <http://mc/compose?to=3Dfreebsd-net@freebsd.org>mailing list > > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > > > >> To unsubscribe, send any mail to " > freebsd-net-unsubscribe@freebsd.org<http://mc/compose?to=3Dfreebsd-net-un= subscribe@freebsd.org> > <http://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org>; > > > " > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Richard Sharpe > > > (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > > > > _______________________________________________ > > freebsd-net@freebsd.org <http://mc/compose?to=3Dfreebsd-net@freebsd.org= > < > http://mc/compose?to=3Dfreebsd-net@freebsd.org>mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<h= ttp://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org> > <http://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org>; > > > " > > > > > _______________________________________________ > freebsd-net@freebsd.org <http://mc/compose?to=3Dfreebsd-net@freebsd.org>m= ailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<htt= p://mc/compose?to=3Dfreebsd-net-unsubscribe@freebsd.org> > " > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1368878922.70089.YahooMailClassic>