From owner-freebsd-fs@FreeBSD.ORG Fri Jan 18 19:59:12 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0E8716A418; Fri, 18 Jan 2008 19:59:12 +0000 (UTC) (envelope-from jaharkes@cs.cmu.edu) Received: from delft.aura.cs.cmu.edu (unknown [IPv6:2002:8002:ce58::1]) by mx1.freebsd.org (Postfix) with ESMTP id 9697413C455; Fri, 18 Jan 2008 19:59:12 +0000 (UTC) (envelope-from jaharkes@cs.cmu.edu) Received: from jaharkes by delft.aura.cs.cmu.edu with local (Exim 4.68) (envelope-from ) id 1JFxMr-0002tf-Nl; Fri, 18 Jan 2008 14:59:09 -0500 Date: Fri, 18 Jan 2008 14:59:09 -0500 From: Jan Harkes To: Robert Watson Message-ID: <20080118195909.GE7898@cs.cmu.edu> Mail-Followup-To: Robert Watson , Rune , "Marc G. Fournier" , freebsd-fs@FreeBSD.org, jcw@highperformance.net References: <18CC5A4A2AC36D7FF57615EE@ganymede.hub.org> <478AF6BC.8050604@highperformance.net> <20080114142124.Y55696@fledge.watson.org> <20080116085630.GA32361@pappardelle.tekno.chalmers.se> <20080117080359.U51764@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080117080359.U51764@fledge.watson.org> User-Agent: Mutt/1.5.17 (2007-12-11) Cc: freebsd-fs@FreeBSD.org, jcw@highperformance.net, Rune Subject: Re: [OpenAFS-devel] Re: AFS ... or equivalent ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2008 19:59:13 -0000 On Thu, Jan 17, 2008 at 08:11:35AM +0000, Robert Watson wrote: > that more fixes are required to make Coda stable and supportable. I've > added Jan to the CC, perhaps he can opine on what further work is > required, and if I missed any further patches he sent. As with Arla, You got everything that was in my queue. That doesn't mean that all problems have been resolved, but I could walk through the file system and read and write files without problems. I notice all the other mails, I'll have to look at those. > for some period (perhaps still), Coda worked poorly from behind a NAT, a > common property of many development environments. It isn't so much that Coda works poorly behind a NAT, but that a lot of masquerading firewalls assume that the only UDP traffic is DNS lookups and maybe NFS. Which means that they do unexpected things when one side starts to send several packets in rapid succession to the same destination. I've seen firewalls that drop everything but the first packet until a reply comes back. Or they only allow a single "reply" packet direction after a request. Both of these flaws break our windowed data tranfers because a single ack will move the window which can result in several outgoing UDP packets. Another interesting case is firewalls that don't seem to hash on the source addr,port,destination address, port tuple, but only remember the last destination we sent to. So if we send a message to several (replicated) servers, we only receive a response from a single one (the last one we talked to). Of course we retry the request to the others, so eventually we get all answers. But it isn't really very efficient we end up waiting for the estimated RTT and send 50% more data if we have 2 replicas. Which affects the estimated RTT, making the next round even slower. A more common scenario is a too short period during which the connection/redirection is considered active. Some masquerading firewalls have a default value that may be as low as 60 seconds. This means we have to send at least one packet every 60 seconds, there could be packet loss, so possibly even more often than that, just to keep the connection alive. If our packet reaches the firewall outside of this window, it will either drop the packet, or allocate a new port so that the receiver thinks this is a connection from a different client. The client ends up having to revalidate everything in it's cache to make sure it didn't miss a callback message. Originally a connection keepalive probe only happened once every 15 minutes if we hadn't heard anything else, the default is now 150 seconds because a lot of off-the-shelf wireless routers/nat firewalls seem to have a roughly 3 minute timeout. And these issues will affect any protocol over UDP that isn't a plain stateless request-response variant, and in some cases (f.i. dropping everything but the first packet in a sequence) are broken in all cases because a packet can be dropped/lost or corrupted after passing the firewall and the firewall isn't letting the retransmissions through. But that only happens once in a blue moon, and will in most cases just show as a stalled/failed DNS lookup and people just hit 'try again'. Jan