From owner-freebsd-net@FreeBSD.ORG Sat May 4 21:19:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 334C2D50; Sat, 4 May 2013 21:19:53 +0000 (UTC) (envelope-from realrichardsharpe@gmail.com) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) by mx1.freebsd.org (Postfix) with ESMTP id 9890BFB7; Sat, 4 May 2013 21:19:52 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id m6so1559503wiv.9 for ; Sat, 04 May 2013 14:19:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=b2raqqmmhLHG13tyzwpDFJwq0wDi9liKipIBJRUTCVo=; b=D1lzqK1r67ugKCc6UW+l6N3UiIWmGlELjW7J5DgItr3jskY0fFTL0mOr3K249S1pep l1aVTfdjEVa9U1GYVKBdc2UcQFy0XYzRtVem9m4ZBejjDs7XEbnZjCKUcx0phtM4dCNk OCXnwB9nsE7FfyN85fvrTKWY6IUu42EU3Sz1ESRN1YzD2kMThhC8vdiz3Xlz/a/iheE3 6VoeCSg7FI110rPzUzEVYFcha0VYe8w5j3TGtVCijLm/LloBN9kOxNkG7LtVV6CfTk64 x8SLepYRJAQou7EjZf/o0KN3oiMJc7FrKdSWgQIbBoZyOtnvMdZU4Rnhap/v/iEsI1PC pZ3w== MIME-Version: 1.0 X-Received: by 10.180.90.203 with SMTP id by11mr3015897wib.10.1367702391647; Sat, 04 May 2013 14:19:51 -0700 (PDT) Received: by 10.194.179.194 with HTTP; Sat, 4 May 2013 14:19:51 -0700 (PDT) In-Reply-To: References: Date: Sat, 4 May 2013 14:19:51 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Richard Sharpe To: Jack Vogel Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Net , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 May 2013 21:19:53 -0000 On Sat, May 4, 2013 at 2:18 PM, Jack Vogel wrote: > Ahh, Twinville, new hardware :) The version at the tip is 2.5.8 and I am > working on version 2.5.12 internally that I hope to commit next week... > so your version is "a bit old" :) I would do some testing on newer code. I would love to. Where is the repo. > Jack > > > > On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe > wrote: >> >> On Sat, May 4, 2013 at 1:41 PM, Jack Vogel wrote: >> > If you don't use TSO you will hurt your TX performance significantly >> > from >> > the tests that I've run. What exactly is the device you are using, I >> > don't >> > have the source in front of me now, but I'm almost sure that the limit >> > is >> > not 64K but 256K, or are you using some ancient version of the driver? >> >> ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x80= 86 >> subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0 >> ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x80= 86 >> subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1 >> >> The version calls itself ixgbe-2.4.4 ... >> >> Hmmm, copyright is 2001-2010 ... so perhaps a bit old. >> >> > Jack >> > >> > >> > >> > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe >> > >> > wrote: >> >> >> >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd >> >> wrote: >> >> > On 4 May 2013 06:52, Richard Sharpe >> >> > wrote: >> >> >> Hi folks, >> >> >> >> >> >> I understand better why I am seeing EINVAL intermittently when >> >> >> sending >> >> >> data from Samba via SMB2. >> >> >> >> >> >> The ixgbe driver, for TSO reasons, limits the amount of data that >> >> >> can >> >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larg= er >> >> >> than that. >> >> >> >> >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is le= ss >> >> >> than 64kiB of space available, so that is all TCP etc can put into >> >> >> the >> >> >> socket in one chain of mbufs. However, every now and then there is >> >> >> more than 65535 bytes available in the socket buffers, and we have >> >> >> an >> >> >> SMB packet that is larger than 65535 bytes, and we get hit. >> >> >> >> >> >> To confirm this I am going to set SO_SNDBUF back to the default of >> >> >> 65536 and test again. My repros are very reliable. >> >> >> >> >> >> However, I wondered if my only way around this if I want to contin= ue >> >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf >> >> >> chains in the driver? >> >> > >> >> > Hm, is this is a problem without TSO? >> >> >> >> We are using the card without TSO, so I am thinking of changing that >> >> limit to 131072 and retesting. >> >> >> >> I am currently testing with SO_SNDBUF=3D32768 and have not hit the >> >> problem. >> >> >> >> > Is the problem that the NIC can't handle a frame that big, or a >> >> > buffer >> >> > that big? >> >> > Ie - if you handed the hardware two descriptors of 64k each, for th= e >> >> > same IP datagram, will it complain? >> >> >> >> I can't find any documentation, but it seems that with TSO it cannot >> >> handle a frame that big. Actually, since we are not using TSO, there >> >> really should not be a problem with larger frames. >> >> >> >> > Or do you need to break it up into two separate IP datagrams, facin= g >> >> > the driver, with a maximum size of 64k each? >> >> >> >> Not sure, but it looks like we need to do that. >> >> >> >> >> >> -- >> >> Regards, >> >> Richard Sharpe >> >> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) >> >> _______________________________________________ >> >> freebsd-net@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >> > >> > >> >> >> >> -- >> Regards, >> Richard Sharpe >> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > > --=20 Regards, Richard Sharpe (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE)