From owner-freebsd-net@FreeBSD.ORG Sat May 4 21:18:21 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7FD57CB5; Sat, 4 May 2013 21:18:21 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f178.google.com (mail-vc0-f178.google.com [209.85.220.178]) by mx1.freebsd.org (Postfix) with ESMTP id 2EF8FFA7; Sat, 4 May 2013 21:18:21 +0000 (UTC) Received: by mail-vc0-f178.google.com with SMTP id kw10so2265360vcb.37 for ; Sat, 04 May 2013 14:18:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=NjiBIsQ9rl6aLUu+yDp/VXeuMPye0sJDdJzKhu+XKOI=; b=cNKOqph3/isd9AAgVv/gAyRwsKyb8nIeUE4Dkl0VG2/hQWoLGTDvZwSP6Q0ldN4S4N SYAz/eUI5v7/Pssta4cbcnOgLAcG7UvWUgkfuLooB2yH98gq/SlsZLCxawkHNOvd7wco zkr2pS3lOLbuK3ap9s2QI9kWTSqPqD1BPtGH/L1Gww6SaKQUEXtqj5KG/gR8f8mLMASv w9KjMCImB3tVpCbJvVbmNtcdIXX8r/bOGIeIoHIYBEujzjUelTR+VgnH4soq9BOdOlGU ghG5sLV44hvEdDXk4qkxFvSCx5JgPAi+DhQgl3rD8N8Fjhl3Q3bWSGHv83MR1o+mIFhV csrg== MIME-Version: 1.0 X-Received: by 10.58.230.130 with SMTP id sy2mr5155893vec.22.1367702300388; Sat, 04 May 2013 14:18:20 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sat, 4 May 2013 14:18:20 -0700 (PDT) In-Reply-To: References: Date: Sat, 4 May 2013 14:18:20 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel To: Richard Sharpe Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 May 2013 21:18:21 -0000 Ahh, Twinville, new hardware :) The version at the tip is 2.5.8 and I am working on version 2.5.12 internally that I hope to commit next week... so your version is "a bit old" :) I would do some testing on newer code. Jack On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe wrote: > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel wrote: > > If you don't use TSO you will hurt your TX performance significantly fr= om > > the tests that I've run. What exactly is the device you are using, I > don't > > have the source in front of me now, but I'm almost sure that the limit = is > > not 64K but 256K, or are you using some ancient version of the driver? > > ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x808= 6 > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 > ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x808= 6 > subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 > > The version calls itself ixgbe-2.4.4 ... > > Hmmm, copyright is 2001-2010 ... so perhaps a bit old. > > > Jack > > > > > > > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < > realrichardsharpe@gmail.com> > > wrote: > >> > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd > wrote: > >> > On 4 May 2013 06:52, Richard Sharpe > wrote: > >> >> Hi folks, > >> >> > >> >> I understand better why I am seeing EINVAL intermittently when > sending > >> >> data from Samba via SMB2. > >> >> > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that c= an > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain large= r > >> >> than that. > >> >> > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is les= s > >> >> than 64kiB of space available, so that is all TCP etc can put into > the > >> >> socket in one chain of mbufs. However, every now and then there is > >> >> more than 65535 bytes available in the socket buffers, and we have = an > >> >> SMB packet that is larger than 65535 bytes, and we get hit. > >> >> > >> >> To confirm this I am going to set SO_SNDBUF back to the default of > >> >> 65536 and test again. My repros are very reliable. > >> >> > >> >> However, I wondered if my only way around this if I want to continu= e > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf > >> >> chains in the driver? > >> > > >> > Hm, is this is a problem without TSO? > >> > >> We are using the card without TSO, so I am thinking of changing that > >> limit to 131072 and retesting. > >> > >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the > problem. > >> > >> > Is the problem that the NIC can't handle a frame that big, or a buff= er > >> > that big? > >> > Ie - if you handed the hardware two descriptors of 64k each, for the > >> > same IP datagram, will it complain? > >> > >> I can't find any documentation, but it seems that with TSO it cannot > >> handle a frame that big. Actually, since we are not using TSO, there > >> really should not be a problem with larger frames. > >> > >> > Or do you need to break it up into two separate IP datagrams, facing > >> > the driver, with a maximum size of 64k each? > >> > >> Not sure, but it looks like we need to do that. > >> > >> > >> -- > >> Regards, > >> Richard Sharpe > >> (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D= =9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > > > > -- > Regards, > Richard Sharpe > (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D=9C= =E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D) >