From owner-freebsd-net@FreeBSD.ORG Wed May 30 22:36:58 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3BB961065670 for ; Wed, 30 May 2012 22:36:58 +0000 (UTC) (envelope-from bounces+73574-866e-freebsd-net=freebsd.org@sendgrid.me) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) by mx1.freebsd.org (Postfix) with SMTP id DBE738FC14 for ; Wed, 30 May 2012 22:36:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h= message-id:date:from:mime-version:to:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; s=smtpapi; bh=Ki1s/HupIZswUwhPcC/s395l3OE=; b=VY59otgXrc5adc5VbjjxT+RQRX6B iVysOPtqPxrTqvbKOdlFKMStRv05tXLV53DnXAoYP3NPgNvNl6/EVj07TJghBgkc 871hGXs7870PkFcMyJdZ37jmHw0yXwC+j30cLAAR9xRG8NC58M3gkqpvKeCZ9EVx olTo83D+S4uaOeM= Received: by 10.4.35.224 with SMTP id mf3.10363.4FC6A107A Wed, 30 May 2012 17:36:55 -0500 (CDT) Received: from mail.tarsnap.com (unknown [10.8.49.124]) by mi2 (SG) with ESMTP id 4fc6a107.5003.61c724 for ; Wed, 30 May 2012 17:36:55 -0500 (CST) Received: (qmail 77098 invoked from network); 30 May 2012 22:29:23 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by mail.tarsnap.com with ESMTP; 30 May 2012 22:29:23 -0000 Received: (qmail 34495 invoked from network); 30 May 2012 22:35:46 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 30 May 2012 22:35:46 -0000 Message-ID: <4FC6A0C1.3010307@freebsd.org> Date: Wed, 30 May 2012 15:35:45 -0700 From: Colin Percival User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:12.0) Gecko/20120509 Thunderbird/12.0.1 MIME-Version: 1.0 To: Andrew Gallatin References: <4FC635CC.5030608@freebsd.org> <4FC63D27.70807@cs.duke.edu> In-Reply-To: <4FC63D27.70807@cs.duke.edu> X-Enigmail-Version: 1.5pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Sendgrid-EID: ISGKkmHlRE12gnWy0TjFyGQSFR1WnpjQdICm3qu6YxGdYKAvy/gc4T2jUQa000uADlkkFAtN0740Q3+Xcn+vpnC6RFIdeP7ZMIhNA/0EifUH+J2P8T/3Ra8e1qTlj4/E8LGiirHKpD8Hr4xJamr0JA== X-SendGrid-Contentd-ID: {"test_id":1338417415} Cc: freebsd-net@freebsd.org Subject: Re: [please review] TSO mbuf chain length limiting patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2012 22:36:58 -0000 On 05/30/12 08:30, Andrew Gallatin wrote: > On 05/30/12 10:59, Colin Percival wrote: >> The Xen virtual network interface has an issue (ok, really the issue is with >> the linux back-end, but that's what most people are using) where it can't >> handle scatter-gather writes with lots of pieces, aka. long mbuf chains. >> This currently bites us hard with TSO enabled, since it produces said long >> mbuf chains. > > I've never been clear about what the max TSO size supported by FreeBSD > is. The NIC I maintain (mxge) is limited to 64K - epsilon for both > IPv4 *AND* IPv6. Up until now, this has been enforced by the 16-bit > ip length limit of IPv4 and we have not had IPv6 TSO until this week. > With IPv6, I'm worried that FreeBSD may now send packets down larger > than I could handle. In my case, however, the problem is not s/g list > length, but rather it is internal limits in the NIC which limit us to > 64K - epsilon for IPv6 as well. I think there may be other NICs in > the same boat for IPv6 (and maybe even some which cannot handle the > full 64K for IPv4). > > Your approach would not work well for my size limit. For > example, I'd have to set the limit to 4 mbufs to stay under 64KB. > This would be assuming the worst case of 16KB jumbo mbufs, so > that would limit me to ~8KB per TSO if 2KB mbufs were used. Right, the problem you describe isn't the one I was trying to solve. :-) > I think a better approach would be to have a limit on the size of the > pre-segmented TCP payload size sent to the driver. I tend to think > that this would be more generically useful, and it is a better match > for the NDIS APIs, where a driver must specify the max TSO size. I > think the changes to the TCP stack might be simpler (eg, they > would seem to jive better with the existing "maxmtu" approach). > > I think this could work for you as well. You could set the Xen max > tso size to be 32K (derived from 18 pages/skb, multiplied by a typical > 2KB mbuf size, with some slack built in). If the chain was too large, > you could m_defrag it down to size. Sounds good -- I don't want to m_defrag too often, but I imagine in most cases when TSO is being invoked most of the mbufs will have 2 kB each. This should also make the patch simpler by avoiding the need to modify uipc_mbuf.c; if we just limit the TSO payload size then the TCP stack can figure things out by itself. Are you working on a patch, or should I put one together? -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid