From owner-freebsd-current@FreeBSD.ORG Fri Nov 6 00:40:13 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A1F6106566B for ; Fri, 6 Nov 2009 00:40:13 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id CAD358FC0C for ; Fri, 6 Nov 2009 00:40:12 +0000 (UTC) Received: by ewy18 with SMTP id 18so386034ewy.43 for ; Thu, 05 Nov 2009 16:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:references:in-reply-to :mime-version:content-type:message-id:content-transfer-encoding:cc :from:subject:date:to:x-mailer; bh=bTJj3cFAyKCsnc7xIy7xG3uyPi65cCBRESYvykeaFX0=; b=rnMkSkoUiL3+quYbhqDpsTT/q+PmzDTgj+MHtZhAFEDnt8DC1aZStrNueucm1sEKO5 yoe7NXga4s+QzsEhArVO6cjf1F6LXDlx/iWBKlvkjNmU2eRorx+pKX+SeiqTmxTbdjGq nFMh1tVSqPrmOr+v8w/ZvYElPdwJ/RkCZOIxs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:references:in-reply-to:mime-version:content-type:message-id :content-transfer-encoding:cc:from:subject:date:to:x-mailer; b=o9SUnUVJTe71n3M7SQiQxqrK2lIqVG4OTRdin16bWj1JeayUp3OEU6dlyCetyav4kf ef/M8NB45Gmo42vcm7JVDG4fpWE4zUMSuguEoP8GCXQ0I5s3xNk92TjS+gZm167dmASF e8IWKTOB4Qbe2QDPDAF+FDMHL6VSJcFIoUYqc= Received: by 10.213.0.152 with SMTP id 24mr4458374ebb.52.1257468011786; Thu, 05 Nov 2009 16:40:11 -0800 (PST) Received: from rui-macbook.lan (bl7-119-248.dsl.telepac.pt [85.240.119.248]) by mx.google.com with ESMTPS id 5sm4784574eyf.23.2009.11.05.16.39.49 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 05 Nov 2009 16:39:57 -0800 (PST) Sender: Rui Paulo References: <4AF0B7DF.9030405@freebsd.org> In-Reply-To: Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Message-Id: <912FE42D-F2A2-422B-8681-AE27CA428DD5@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Rui Paulo Date: Fri, 6 Nov 2009 00:39:47 +0000 To: Rick Macklem X-Mailer: Apple Mail (2.1076) Cc: freebsd-current@freebsd.org Subject: Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 00:40:13 -0000 On 5 Nov 2009, at 16:36, Rick Macklem wrote: > Rick Macklem wrote: >> I can now reproduce what I think others were seeing as slow >> reconnects >> when using NFSv3 over TCP against a server that disconnects inactive >> TCP connections. I have had some luck figuring out what is going on >> and can reproduce it fairly easily, but I really need help from >> someone >> who understands the FreeBSD TCP stack. >> > Ok, I haven't made much progress on this, but here's what little I > currently know about it. > > The problem occurs after a server has dropped an inactive TCP > connection > for an NFS over TCP mount (in my case a Solaris10 server). When the > client > does a new connection it, for some reason, sends a RST at almost > exactly > the same time as the first RPC request on the new TCP connection, > causing > the server to shut it down. > > Ok, things I now know don't affect this are: > - doing the soshutdown(), soclose() on the old connection. I commented > them out and it still happened. > - Avoiding the sobind() on the new connection, done before the > soconnect(). > - Using a non-reserved port#. > (The above tests shot down pretty well all the "theories" I could > come up > with.) > > The only thing I've found that avoids the problem: > - putting a 2sec delay right before the soconnect() call. (A 1sec > delay > made it hard to reproduce and I've never reproduced it yet with a 2sec > delay.) > Not much of a fix, though. > > Now, here's where someone may be able to help? > > Grep'ng around, I found 4 places where the TCP stack called ip_output > () > (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and > tcp_syncache.c) and I put a printf like this just before them: > if (flags & TH_RST) > printf("sent a reset\n"); > > (The exact format varies for each, because of where the TCP > header flags are and have different printf messages.) > > Now, the weird part is, that when the extraneous RST is sent to the > server, I don't get any printf. (I do get a few of these, but at other > times for what appear to be legitimate RSTs.) > > I can't see anywhere else that the TCP stack would send an RST and, > so, > I'm stuck w.r.t. figuring out what is sending them? > > Anyone know of another place the TCP stack would make the send happen? > (Or is it queued earlier when I see the printf message, and then the > send is "triggered" inside the ip layer when the first data is sent on > the new connection?) > > rick, who is getting sick of looking at this:-) One option would be trying to calling ddb on top of ip_output() and checking the backtrace. -- Rui Paulo