From owner-freebsd-net@FreeBSD.ORG Sat May 4 20:41:18 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1F61E326; Sat, 4 May 2013 20:41:18 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f170.google.com (mail-vc0-f170.google.com [209.85.220.170]) by mx1.freebsd.org (Postfix) with ESMTP id C2E48EC3; Sat, 4 May 2013 20:41:17 +0000 (UTC) Received: by mail-vc0-f170.google.com with SMTP id gf12so2276344vcb.15 for ; Sat, 04 May 2013 13:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=4162ot7BsDr4AF7L8qoxyOlRf0uUNusCzBe9iGv8PR4=; b=vN73CQ4war7mGaZ2hQBJlVsQo04GjK4W6d9NncujWRW+ya5BYlFgmiX9PdHK6B7oEG XCPfWla831uqzbuMpoMVDFjtB0FVHtfyO24daS+Hb5EqErdXB1lY9BnMOmtlZ8oDZOcO c3Njrxr5dG9Y7Hw7DMLjghD/d3u9K+5pDxLCp1NbIOoMcHUqJgYc/vd2HvNvXqSoeHHW knTcqxaz9IQImEx1LT/vdEjeiIUO1CagR9sl+Vqw/a5tG/P42dzhUh92QKtXiMbGPjLL i34x2x3uYqkPC2JZhU+opTHtn9rXB2Lp9eLctZe6gqbXiEgU7DhVXG0s6Nzxmko+bU5G Tbgg== MIME-Version: 1.0 X-Received: by 10.59.0.226 with SMTP id bb2mr5154405ved.1.1367700070905; Sat, 04 May 2013 13:41:10 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sat, 4 May 2013 13:41:10 -0700 (PDT) In-Reply-To: References: Date: Sat, 4 May 2013 13:41:10 -0700 Message-ID: Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? From: Jack Vogel To: Richard Sharpe Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 May 2013 20:41:18 -0000 If you don't use TSO you will hurt your TX performance significantly from the tests that I've run. What exactly is the device you are using, I don't have the source in front of me now, but I'm almost sure that the limit is not 64K but 256K, or are you using some ancient version of the driver? Jack On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe wrote: > On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd wrote: > > On 4 May 2013 06:52, Richard Sharpe wrote= : > >> Hi folks, > >> > >> I understand better why I am seeing EINVAL intermittently when sending > >> data from Samba via SMB2. > >> > >> The ixgbe driver, for TSO reasons, limits the amount of data that can > >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larger > >> than that. > >> > >> The SO_SNDBUF for that socket is set to 131972. Mostly there is less > >> than 64kiB of space available, so that is all TCP etc can put into the > >> socket in one chain of mbufs. However, every now and then there is > >> more than 65535 bytes available in the socket buffers, and we have an > >> SMB packet that is larger than 65535 bytes, and we get hit. > >> > >> To confirm this I am going to set SO_SNDBUF back to the default of > >> 65536 and test again. My repros are very reliable. > >> > >> However, I wondered if my only way around this if I want to continue > >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf > >> chains in the driver? > > > > Hm, is this is a problem without TSO? > > We are using the card without TSO, so I am thinking of changing that > limit to 131072 and retesting. > > I am currently testing with SO_SNDBUF=3D32768 and have not hit the proble= m. > > > Is the problem that the NIC can't handle a frame that big, or a buffer > that big? > > Ie - if you handed the hardware two descriptors of 64k each, for the > > same IP datagram, will it complain? > > I can't find any documentation, but it seems that with TSO it cannot > handle a frame that big. Actually, since we are not using TSO, there > really should not be a problem with larger frames. > > > Or do you need to break it up into two separate IP datagrams, facing > > the driver, with a maximum size of 64k each? > > Not sure, but it looks like we need to do that. > > > -- > Regards, > Richard Sharpe > (=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE) > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"