From owner-freebsd-net@FreeBSD.ORG Fri May 17 22:27:32 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 27A2CEA3 for ; Fri, 17 May 2013 22:27:32 +0000 (UTC) (envelope-from laurie_jennings_1977@yahoo.com) Received: from nm18-vm1.bullet.mail.ne1.yahoo.com (nm18-vm1.bullet.mail.ne1.yahoo.com [98.138.91.64]) by mx1.freebsd.org (Postfix) with ESMTP id CA69E186 for ; Fri, 17 May 2013 22:27:31 +0000 (UTC) Received: from [98.138.90.56] by nm18.bullet.mail.ne1.yahoo.com with NNFMP; 17 May 2013 22:25:42 -0000 Received: from [98.138.226.167] by tm9.bullet.mail.ne1.yahoo.com with NNFMP; 17 May 2013 22:25:42 -0000 Received: from [127.0.0.1] by omp1068.mail.ne1.yahoo.com with NNFMP; 17 May 2013 22:25:42 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 804200.47930.bm@omp1068.mail.ne1.yahoo.com Received: (qmail 54320 invoked by uid 60001); 17 May 2013 22:25:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1368829542; bh=QPvuYeUC6tSdfUtokpSPE2PyatbceDGSdl4MeQ/kn9k=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ByoH+D+HPiRdyjJ9tQquGiUJXIuqtiU671SK1KiLgNnA0xfZBlm5ssBb96j7VYZAo1MfHD+WuF/1RegEAS7rEaROVaNELfCVcPTJBBLzOO1qfR8Xv12nx33MUq+Zodl7EHPqrSHF70a6E38wBRHQ9chxflK9TVs1g5J8QyJKxa8= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ERhQEsgIs5LPzdfC1YPev9+gOBlb04o93kOnHX2RbKB6bIV4NHyiWIRnufdwOkKjoDROX6nzGMxXb9SYwV418QMcD/yb+DJx20unPCMWS4vOoyALEjXgpzTbN7Kd9YV73CzXcV+AI2lt/jKnpSdbfXfK2keSDvEeFHV7h9KBsdI=; X-YMail-OSG: 9S3FdNYVM1np7LOs9VJvWm7BPOdl7jG13Jtc037SDTrVB1p i3VBrRiiBuzrvc5qpbIcHt6xaWzG9tZB9Y13wTXwxD4uiV0eWkEqNrL8OHFE yvh1za0NlP6YR6cPQ2Xbi18uFF6QgfBCrkpejVsadVjOmaduDFD3wULOy0nc 9scJVrVyGBXmLgvu0tggVrF8TDa5J5Kr2IXHwK10G2EJ_Oy4r8Xnf3PMq1SV feHdQDzwQ.RpsfmkEFxbtUA.pNC0PFy_MRJbKcY1CDp5WLa1IR5bAYNmjcnH iE5tDuoS8DuMpRnMT6qwB2O_QqQq_.KTIbbbsRdm5LBJbLuXUoDqhaSuA4zH eu3E_8XM.G_9PLPl9VmaoZMQJSskCm6eynJdtbBqFA4_VHroHCxC_oZPC_wW tVFI143OlHYb8XuJH7aAhpj3yoZ_M3BuWti.5nIcgQmFVSBW4Jqk8NdDYgdc yuxBY3wh_N_8G0.IwsGdsUpFRhJYF78QY_oEi32VldXnToTNYP5TG9UGKjC6 7I6sg6Hup9YQ98.SU9hzNLXOn9gSha4Ee_Pd7961IPpU- Received: from [98.203.118.124] by web125804.mail.ne1.yahoo.com via HTTP; Fri, 17 May 2013 15:25:42 PDT X-Rocket-MIMEInfo: 002.001, RGlkIHlvdSBnZXQgdG8gY29tbWl0IHRoYXQgbmV3IGNvZGUgeWV0Pw0KTGF1cmllDQoNCi0tLSBPbiBTYXQsIDUvNC8xMywgSmFjayBWb2dlbCA8amZ2b2dlbEBnbWFpbC5jb20.IHdyb3RlOg0KDQpGcm9tOiBKYWNrIFZvZ2VsIDxqZnZvZ2VsQGdtYWlsLmNvbT4NClN1YmplY3Q6IFJlOiBJcyB0aGVyZSBhbnkgd2F5IHRvIGxpbWl0IHRoZSBhbW91bnQgb2YgZGF0YSBpbiBhbiBtYnVmIGNoYWluIHN1Ym1pdHRlZCB0byBhIGRyaXZlcj8NClRvOiAiUmljaGFyZCBTaGFycGUiIDxyZWFscmljaGFyZHNoYXJwZUABMAEBAQE- X-Mailer: YahooMailClassic/15.1.8 YahooMailWebService/0.8.142.542 Message-ID: <1368829542.53502.YahooMailClassic@web125804.mail.ne1.yahoo.com> Date: Fri, 17 May 2013 15:25:42 -0700 (PDT) From: Laurie Jennings Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? To: Richard Sharpe , Jack Vogel In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2013 22:27:32 -0000 Did you get to commit that new code yet? Laurie --- On Sat, 5/4/13, Jack Vogel wrote: From: Jack Vogel Subject: Re: Is there any way to limit the amount of data in an mbuf chain = submitted to a driver? To: "Richard Sharpe" Cc: "FreeBSD Net" , "Adrian Chadd" Date: Saturday, May 4, 2013, 2:18 PM Ahh, Twinville, new hardware :)=C2=A0 The version at the tip is 2.5.8 and I= am working on version 2.5.12 internally that I hope to commit next week... so your version is "a bit old" :) I would do some testing on newer code. Jack On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe wrote: > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel wrote: > > If you don't use TSO you will hurt your TX performance significantly fr= om > > the tests that I've run. What exactly is the device you are using, I > don't > > have the source in front of me now, but I'm almost sure that the limit = is > > not 64K but 256K, or are you using some ancient version of the driver? > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix0 pnpinfo vendor=3D= 0x8086 device=3D0x1528 subvendor=3D0x8086 > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0ix1 pnpinfo vendor=3D= 0x8086 device=3D0x1528 subvendor=3D0x8086 > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > The version calls itself ixgbe-2.4.4 ... > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > Jack > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > realrichardsharpe@gmail.com> > > wrote: > >> > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd > wrote: > >> > On 4 May 2013 06:52, Richard Sharpe > wrote: > >> >> Hi folks, > >> >> > >> >> I understand better why I am seeing EINVAL intermittently when > sending > >> >> data from Samba via SMB2. > >> >> > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that c= an > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain large= r > >> >> than that. > >> >> > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is les= s > >> >> than 64kiB of space available, so that is all TCP etc can put into > the > >> >> socket in one chain of mbufs. However, every now and then there is > >> >> more than 65535 bytes available in the socket buffers, and we have = an > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > >> >> > >> >> To confirm this I am going to set SO_SNDBUF back to the default of > >> >> 65536 and test again. My repros are very reliable. > >> >> > >> >> However, I wondered if my only way around this if I want to continu= e > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf > >> >> chains in the driver? > >> > > >> > Hm, is this is a problem without TSO? > >> > >> We are using the card without TSO, so I am thinking of changing that > >> limit to 131072 and retesting. > >> > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > problem. > >> > >> > Is the problem that the NIC can't handle a frame that big, or a buff= er > >> > that big? > >> > Ie - if you handed the hardware two descriptors of 64k each, for the > >> > same IP datagram, will it complain? > >> > >> I can't find any documentation, but it seems that with TSO it cannot > >> handle a frame that big. Actually, since we are not using TSO, there > >> really should not be a problem with larger frames. > >> > >> > Or do you need to break it up into two separate IP datagrams, facing > >> > the driver, with a maximum size of 64k each? > >> > >> Not sure, but it looks like we need to do that. > >> > >> > >> -- > >> Regards, > >> Richard Sharpe > >> (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D= =9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > > > > -- > Regards, > Richard Sharpe > (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D=9C= =E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri May 17 22:32:24 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B60A91DA; Fri, 17 May 2013 22:32:24 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f169.google.com (mail-vc0-f169.google.com [209.85.220.169]) by mx1.freebsd.org (Postfix) with ESMTP id 645881D5; Fri, 17 May 2013 22:32:24 +0000 (UTC) Received: by mail-vc0-f169.google.com with SMTP id kw10so4190192vcb.0 for ; Fri, 17 May 2013 15:32:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=wGWLZW7klgyeOeeYtiW3eCdFm+wF+xiUw6DiLmL6LEw=; b=Nl0eSo+R4cMrqx4I/RP7qCJArll8p97SCe67UNNhi6BBCX7Pl0tl+OV4LCZ7Uh1m9o RtIb7DOjHW2qYwA74TUI821ZSz3puLLjW2OdfybkUD8uOfGhqwT1SPE6PnA6poSo9TDz +W0hDx91KGGpDZcFMjLGCTLI/dmlws08Eba45XTbHfAvQVs6v+CqAcALyxpmZbZH1fiI xrct/qB8cS2zwfx+RuEIJlEJQgyUkQORCIL+oE5hOXtJRE34XCuWxhGWltXG7E3bfH4j xt+jsGjXSyI2Huq1/x/oR/ldhCjZ3W2f7Cv9ghf0/7kLPJbT9Pi3zkpqKL34ZtBcPE8E AyTA== MIME-Version: 1.0 X-Received: by 10.58.247.130 with SMTP id ye2mr30703522vec.35.1368829938120; Fri, 17 May 2013 15:32:18 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Fri, 17 May 2013 15:32:17 -0700 (PDT) In-Reply-To: <1368829542.53502.YahooMailClassic@web125804.mail.ne1.yahoo.com> References: <1368829542.53502.YahooMailClassic@web125804.mail.ne1.yahoo.com> Date: Fri, 17 May 2013 15:32:17 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel To: Laurie Jennings Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd , Richard Sharpe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2013 22:32:24 -0000 No Laurie, I ran into an issue that I needed to resolve, and then my validation engineer went out of town a couple days. Should be early next week. Jack On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings < laurie_jennings_1977@yahoo.com> wrote: > Did you get to commit that new code yet? > > Laurie > > --- On *Sat, 5/4/13, Jack Vogel * wrote: > > > From: Jack Vogel > Subject: Re: Is there any way to limit the amount of data in an mbuf chai= n > submitted to a driver? > To: "Richard Sharpe" > Cc: "FreeBSD Net" , "Adrian Chadd" < > adrian@freebsd.org> > Date: Saturday, May 4, 2013, 2:18 PM > > > Ahh, Twinville, new hardware :) The version at the tip is 2.5.8 and I am > working on version 2.5.12 internally that I hope to commit next week... > so your version is "a bit old" :) I would do some testing on newer code. > > Jack > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > > >wrote: > > > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel > > wrote: > > > If you don't use TSO you will hurt your TX performance significantly > from > > > the tests that I've run. What exactly is the device you are using, I > > don't > > > have the source in front of me now, but I'm almost sure that the limi= t > is > > > not 64K but 256K, or are you using some ancient version of the driver= ? > > > > ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x8= 086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > > ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x8= 086 > > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > > > The version calls itself ixgbe-2.4.4 ... > > > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > > > Jack > > > > > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > > realrichardsharpe@gmail.com > > > > > wrote: > > >> > > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd > > > > wrote: > > >> > On 4 May 2013 06:52, Richard Sharpe > > > > wrote: > > >> >> Hi folks, > > >> >> > > >> >> I understand better why I am seeing EINVAL intermittently when > > sending > > >> >> data from Samba via SMB2. > > >> >> > > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that > can > > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain > larger > > >> >> than that. > > >> >> > > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is > less > > >> >> than 64kiB of space available, so that is all TCP etc can put int= o > > the > > >> >> socket in one chain of mbufs. However, every now and then there i= s > > >> >> more than 65535 bytes available in the socket buffers, and we hav= e > an > > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > > >> >> > > >> >> To confirm this I am going to set SO_SNDBUF back to the default o= f > > >> >> 65536 and test again. My repros are very reliable. > > >> >> > > >> >> However, I wondered if my only way around this if I want to > continue > > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbu= f > > >> >> chains in the driver? > > >> > > > >> > Hm, is this is a problem without TSO? > > >> > > >> We are using the card without TSO, so I am thinking of changing that > > >> limit to 131072 and retesting. > > >> > > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > > problem. > > >> > > >> > Is the problem that the NIC can't handle a frame that big, or a > buffer > > >> > that big? > > >> > Ie - if you handed the hardware two descriptors of 64k each, for t= he > > >> > same IP datagram, will it complain? > > >> > > >> I can't find any documentation, but it seems that with TSO it cannot > > >> handle a frame that big. Actually, since we are not using TSO, there > > >> really should not be a problem with larger frames. > > >> > > >> > Or do you need to break it up into two separate IP datagrams, faci= ng > > >> > the driver, with a maximum size of 64k each? > > >> > > >> Not sure, but it looks like we need to do that. > > >> > > >> > > >> -- > > >> Regards, > > >> Richard Sharpe > > >> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > >> _______________________________________________ > > >> freebsd-net@freebsd.orgmailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g > " > > > > > > > > > > > > > > -- > > Regards, > > Richard Sharpe > > (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > > _______________________________________________ > freebsd-net@freebsd.org m= ailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > >