From owner-freebsd-stable@FreeBSD.ORG Thu May 29 20:24:39 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AAFFA1065672 for ; Thu, 29 May 2008 20:24:39 +0000 (UTC) (envelope-from rblayzor.bulk@inoc.net) Received: from mx0-b.inoc.net (mx0-b.inoc.net [64.246.130.28]) by mx1.freebsd.org (Postfix) with ESMTP id 1EC308FC1C for ; Thu, 29 May 2008 20:24:38 +0000 (UTC) (envelope-from rblayzor.bulk@inoc.net) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=inoc.net; h=Received:From:To:Subject:Date; b=2WqBCDBJjtoznXQKBq9E1RmTNPyXGBu+QtMG9lpBR8UWyu8xxmIteTA5wkDTSyqMiYUkEksd6X280MZSEo0nBk0CTBX8gXhVNPbAcxOjnizuRSX8l4jPhxqf67AXdJHNqo3mMCd7VXU9W/uNdq14NeD0LH7rWSLf/nDj38E2Y0o=; Received: from void.ops.inoc.net (vanguard.noc.albyny.inoc.net [64.246.135.8]) by mx0-b.inoc.net (build v8.3.29) with ESMTP id 148977759-1941382 for multiple; Thu, 29 May 2008 20:24:34 +0000 (UTC) Message-Id: <0C827F66-09CE-476D-86E9-146AB255926B@inoc.net> From: Robert Blayzor To: Matthew Dillon In-Reply-To: <200805291930.m4TJUeGX025815@apollo.backplane.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v924) Date: Thu, 29 May 2008 16:24:33 -0400 References: <1A19ABA2-61CD-4D92-A08D-5D9650D69768@mac.com> <23C02C8B-281A-4ABD-8144-3E25E36EDAB4@inoc.net> <483DE2E0.90003@FreeBSD.org> <483E36CE.3060400@FreeBSD.org> <483E3C26.3060103@paradise.net.nz> <483E4657.9060906@FreeBSD.org> <483EA513.4070409@earthlink.net> <96AFE8D3-7EAC-4A4A-8EFF-35A5DCEC6426@inoc.net> <483EAED1.2050404@FreeBSD.org> <200805291912.m4TJCG56025525@apollo.backplane.com> <14DA211A-A9C5-483A-8CB9-886E5B19A840@inoc.net> <200805291930.m4TJUeGX025815@apollo.backplane.com> X-Mailer: Apple Mail (2.924) Cc: freebsd-stable@freebsd.org Subject: Re: Sockets stuck in FIN_WAIT_1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 May 2008 20:24:39 -0000 On May 29, 2008, at 3:30 PM, Matthew Dillon wrote: > It is quite possible that the other ends of the connection are > still > live and that the issue could very well be a timeout setting in the > server config file instead of something in the TCP stack. I think we're onto something here, but for some reason it doesn't make any sense. I have keepalives turned OFF in Apache: In the OS I have: net.inet.tcp.keepidle: 900000 net.inet.tcp.keepintvl: 30000 net.inet.tcp.keepinit: 75000 net.inet.tcp.always_keepalive: 1 I see FIN_WAIT_1 connections collecting like crazy again. I managed to investigate one that's been hung around for several minutes. 1.1.1.1 = our server ip 2.2.2.2 = remote client ip [mirror0:/usr/local/etc/apache22] netstat -an | grep 2.2.2.2 tcp4 0 25861 1.1.1.1.80 2.2.2.2.33379 FIN_WAIT_1 When I tcpdump this, I see something sending ack's back and forth every 60 seconds, but what? Apache? I'm not sure why. I don't see any timeouts in Apache for ~60 seconds. As you can see, sometimes we send an ack, but never see a reply. I'm gathering the OS level keepalives don't come into play because this session is not considered idle? 0:13:07.640426 IP 1.1.1.1.80 > 2.2.2.2.33379: . 4208136508:4208136509(1) ack 1471446041 win 520 20:13:07.736505 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:14:07.702647 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:15:07.764920 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:15:07.860988 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:16:07.827262 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:16:07.923341 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:17:07.889690 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:17:07.984770 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:18:07.952167 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:18:08.048249 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:19:08.014715 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:19:08.110803 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:20:08.077321 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:21:08.139995 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:21:08.236063 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:22:08.202435 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:22:08.297499 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 20:23:08.264631 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 20:23:08.360700 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 I'm finding several of these sessions doing the same exact thing.... -- Robert Blayzor, BOFH INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/