From owner-freebsd-net@FreeBSD.ORG Sat May 4 21:12:56 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9EC9DB12; Sat, 4 May 2013 21:12:56 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f169.google.com (mail-vc0-f169.google.com [209.85.220.169]) by mx1.freebsd.org (Postfix) with ESMTP id 39232F81; Sat, 4 May 2013 21:12:55 +0000 (UTC) Received: by mail-vc0-f169.google.com with SMTP id gd11so2302142vcb.28 for ; Sat, 04 May 2013 14:12:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ztkBxKtzdogpxb9VKwuQHREujAbStymC6HDtGM4lS4A=; b=cy3FgHIEQR5qqSScqzhOzzM17apFLX60ryqKmAvf36ZWSdtx9tAA6WP4549vid3K3n RK2/fpZLYrp5Mpt8pcr5EsRJKhbQ64qU5SXTKruZNqafhwusb15m3zCTbVFjpu5hp+E5 qir1J4HyNZB4WH7APfJgg+gEnzGdF4u+IC1OfkLxDkQRWw4xb+2MqdB19bvRcbH+6TvQ 7txEjk1aXOJi1r7q9Bdbw40dmnudwu+wkB+TOKJq3iKgsnQ9+6sm1on8aa0bHrb3bayg aHUW1FHJ8bYc/tXud+fV0kmugqKmkg2av1Qj1FufuPRe50PBrri/ANN3JLLSv50a2gEU B6jw== MIME-Version: 1.0 X-Received: by 10.220.137.145 with SMTP id w17mr5165082vct.73.1367701975262; Sat, 04 May 2013 14:12:55 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sat, 4 May 2013 14:12:55 -0700 (PDT) In-Reply-To: <518576EF.90706@freebsd.org> References: <518576EF.90706@freebsd.org> Date: Sat, 4 May 2013 14:12:55 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel To: Andre Oppermann Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd , Richard Sharpe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 May 2013 21:12:56 -0000 Hmmm, so its the stack, can that be easily increased Andre? Regards, Jack On Sat, May 4, 2013 at 2:00 PM, Andre Oppermann wrote: > On 04.05.2013 22:47, Jack Vogel wrote: > >> Yes, I checked: #define IXGBE_TSO_SIZE 262140 >> >> So, the driver is not limiting you to 64K assuming you are using a >> version of recent vintage. >> > > The stack won't generate TCP and IP packets larger than 64K. However > the ethernet header gets prepended to it and bumps the overall packet > size as seen by the driver to 64K + sizeof(ether_hdr) and possibly > additional VLAN headers. If the bus_dma_tag size is set to 64K then > bus_dma mapping will fail for such maximum size packets. > > -- > Andre > > > Jack >> >> >> >> On Sat, May 4, 2013 at 1:41 PM, Jack Vogel wrote: >> >> If you don't use TSO you will hurt your TX performance significantly fr= om >>> the tests that I've run. What exactly is the device you are using, I >>> don't >>> have the source in front of me now, but I'm almost sure that the limit = is >>> not 64K but 256K, or are you using some ancient version of the driver? >>> >>> Jack >>> >>> >>> >>> On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe < >>> realrichardsharpe@gmail.com> wrote: >>> >>> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd >>>> wrote: >>>> >>>>> On 4 May 2013 06:52, Richard Sharpe >>>>> >>>> wrote: >>>> >>>>> Hi folks, >>>>>> >>>>>> I understand better why I am seeing EINVAL intermittently when sendi= ng >>>>>> data from Samba via SMB2. >>>>>> >>>>>> The ixgbe driver, for TSO reasons, limits the amount of data that ca= n >>>>>> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larger >>>>>> than that. >>>>>> >>>>>> The SO_SNDBUF for that socket is set to 131972. Mostly there is less >>>>>> than 64kiB of space available, so that is all TCP etc can put into t= he >>>>>> socket in one chain of mbufs. However, every now and then there is >>>>>> more than 65535 bytes available in the socket buffers, and we have a= n >>>>>> SMB packet that is larger than 65535 bytes, and we get hit. >>>>>> >>>>>> To confirm this I am going to set SO_SNDBUF back to the default of >>>>>> 65536 and test again. My repros are very reliable. >>>>>> >>>>>> However, I wondered if my only way around this if I want to continue >>>>>> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf >>>>>> chains in the driver? >>>>>> >>>>> >>>>> Hm, is this is a problem without TSO? >>>>> >>>> >>>> We are using the card without TSO, so I am thinking of changing that >>>> limit to 131072 and retesting. >>>> >>>> I am currently testing with SO_SNDBUF=3D32768 and have not hit the >>>> problem. >>>> >>>> Is the problem that the NIC can't handle a frame that big, or a buffe= r >>>>> >>>> that big? >>>> >>>>> Ie - if you handed the hardware two descriptors of 64k each, for the >>>>> same IP datagram, will it complain? >>>>> >>>> >>>> I can't find any documentation, but it seems that with TSO it cannot >>>> handle a frame that big. Actually, since we are not using TSO, there >>>> really should not be a problem with larger frames. >>>> >>>> Or do you need to break it up into two separate IP datagrams, facing >>>>> the driver, with a maximum size of 64k each? >>>>> >>>> >>>> Not sure, but it looks like we need to do that. >>>> >>>> >>>> -- >>>> Regards, >>>> Richard Sharpe >>>> (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) >>>> ______________________________**_________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.or= g >>>> " >>>> >>> >>> >>> >>> ______________________________**_________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org<= freebsd-net-unsubscribe@freebsd.org> >> " >> >> >