From owner-freebsd-current@FreeBSD.ORG Fri Aug 17 08:14:32 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02BD116A418 for ; Fri, 17 Aug 2007 08:14:32 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id C72F713C48A for ; Fri, 17 Aug 2007 08:14:31 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6C9D44C601; Fri, 17 Aug 2007 04:14:31 -0400 (EDT) Date: Fri, 17 Aug 2007 09:14:31 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Andrew Gallatin In-Reply-To: <18116.44624.144286.621286@grasshopper.cs.duke.edu> Message-ID: <20070817084418.N43264@fledge.watson.org> References: <18116.43755.107638.103132@grasshopper.cs.duke.edu> <18116.44624.144286.621286@grasshopper.cs.duke.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Kip Macy , freebsd-current@freebsd.org Subject: Re: IPSEC disables TSO X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2007 08:14:32 -0000 On Thu, 16 Aug 2007, Andrew Gallatin wrote: > > I thought about this after sending and realized that the appropriate > > response is that IPSEC needs to be careful to disable TSO when its in use > > for a connection. What you're seeing was most likely done as an expedient. > > Yes, exactly, there needs to be a smarter test that can distingiush if IPSEC > is actually in use on a connection or not; I should have been more clear > about this. The problem is that I have zero knowledge about IPSEC, so I > have no idea how to do this. > > I'm worried that people will compile IPSEC into the kernel to run an > encrypted tunnel (or the TCP MD5 signature stuff for BGP), and then be > rather surprised that their their "normal" TCP performance stinks. There are a few things going on here, and a few different approaches that could be taken. I'm pretty sure we have problems with the handling of various TCP TSO edge cases. Right now we handle one "disabling" edge case, and the way it's handled may be instructive in thinking about the problem: if a route changes, or TSO is disabled on the interface, EMSGSIZE will be returned from the IP output function, causing TCP to reevaluate the segmentation strategy it's using, and turn off TSO as required. This case may not properly handle IPSEC as it runs today, because it's reevaluation of whether to use TSO and what MTU size to use likely doesn't take IPSEC into account. However, you could imagine that it might in the future, in which case the right way to handle this case might be for IPSEC to simply encrypt the datagram and let the EMSGSIZE from its send propagate back up to IP and then TCP. Another way to deal with this whole scenario is to treat TSO the same way we treat offloaded checksums: provide a just-in-time segmentation routine to run at the last moment when we discover that TSO is not the right strategy. This isn't, probably, generally the right solution (i.e., don't always use TSO at the TCP layer), as we'd like TCP to make the segmentation decisions with more information in the general case (so you don't end up with awkwardly sized segments, etc). However, there may be other places where it does make sense. We also currently have no way to enable/re-enable TSO in the event that we start routing to an interface that has TSO enabled, or where TSO is turned on administratively, so once a connection has started, it can only ever downgrade from TSO operation to normal operation, never reverse the change. For some protocols, this is undesirable -- for example, for long-running NFS TCP connections, it would be nice if TSO would turn itself back on again. However, we should get basic TSO working properly before turning to issues like this :-). Robert N M Watson Computer Laboratory University of Cambridge