From owner-freebsd-stable@FreeBSD.ORG Thu May 29 19:30:41 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55B4D1065676 for ; Thu, 29 May 2008 19:30:41 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 992D88FC19 for ; Thu, 29 May 2008 19:30:40 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m4TJUeqT025816; Thu, 29 May 2008 12:30:40 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m4TJUeGX025815; Thu, 29 May 2008 12:30:40 -0700 (PDT) Date: Thu, 29 May 2008 12:30:40 -0700 (PDT) From: Matthew Dillon Message-Id: <200805291930.m4TJUeGX025815@apollo.backplane.com> To: Robert Blayzor References: <1A19ABA2-61CD-4D92-A08D-5D9650D69768@mac.com> <23C02C8B-281A-4ABD-8144-3E25E36EDAB4@inoc.net> <483DE2E0.90003@FreeBSD.org> <483E36CE.3060400@FreeBSD.org> <483E3C26.3060103@paradise.net.nz> <483E4657.9060906@FreeBSD.org> <483EA513.4070409@earthlink.net> <96AFE8D3-7EAC-4A4A-8EFF-35A5DCEC6426@inoc.net> <483EAED1.2050404@FreeBSD.org> <200805291912.m4TJCG56025525@apollo.backplane.com> <14DA211A-A9C5-483A-8CB9-886E5B19A840@inoc.net> Cc: freebsd-stable@freebsd.org Subject: Re: Sockets stuck in FIN_WAIT_1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 19:30:41 -0000 :On May 29, 2008, at 3:12 PM, Matthew Dillon wrote: :> I guess nobody mentioned the obvious thing to check: Make sure :> TCP keepalive is turned on. :> :> sysctl net.inet.tcp.always_keepalive=1 : : :Thanks Matt. : :I also thought that a keepalives were not running and sessions just :stuck around forever, however I do have: : : :net.inet.tcp.keepidle=900000 :net.inet.tcp.keepintvl=30000 :net.inet.tcp.msl=5000 :net.inet.tcp.always_keepalive=1 (default) : : :I believe keep idle was defaulted to 2hrs, I changed it to 15 minutes :with a 30 second tick... I still found FIN_WAIT_1 sessions stuck for :several hours, if not infinite. : :Nonet he less, I have a new server up running 7.0-p1, I'll be pumping :a lot of traffic to that box soon and I'll see how that makes out. : :-- :Robert Blayzor, BOFH :INOC, LLC :rblayzor@inoc.net :http://www.inoc.net/~rblayzor/ If it is still giving you trouble I recommend using tcpdump to observe the IP/port pair of one of the stuck connections over the keepalive period and see if the keepalives are still being sent and, if they are, what kind of response you get from the other end. It is quite possible that the other ends of the connection are still live and that the issue could very well be a timeout setting in the server config file instead of something in the TCP stack. This is what you should see when a keepalive occurs over an idle connection: * A TCP packet w/ 0 data sent to the remote * A response from the remote: Either a pure ACK, or a TCP RESET If no response occurs from the remote the keepalive code will then retry a couple of times over keepintvl (every 30 seconds in your case), and if it still gets no response after I think 3 retries (30+30+30 = 90 seconds later) it should terminate the connection state. -Matt Matthew Dillon