From owner-freebsd-net@FreeBSD.ORG Fri May 13 19:39:25 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62F9316A4CE for ; Fri, 13 May 2005 19:39:25 +0000 (GMT) Received: from mail.frii.com (phobos02.frii.net [216.17.128.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A52E43D77 for ; Fri, 13 May 2005 19:39:25 +0000 (GMT) (envelope-from matt@frii.com) Received: from elara.frii.com (elara.frii.com [216.17.128.39]) (using TLSv1 with cipher EDH-RSA-DES-CBC3-SHA (168/168 bits)) (No client certificate requested) by mail.frii.com (FRII) with ESMTP id C30331E1A50; Fri, 13 May 2005 13:39:22 -0600 (MDT) Date: Fri, 13 May 2005 13:39:21 -0600 (MDT) From: Matt Ruzicka X-X-Sender: mattr@elara.frii.com To: Maxim Konovalov In-Reply-To: <20050513230848.K89035@mp2.macomnet.net> Message-ID: References: <20050513134227.P616@odysseus.silby.com> <20050513230848.K89035@mp2.macomnet.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-net@freebsd.org Subject: Re: **net** Re: Outbound TCP issue, potentially related to'FreeBSD-SA-05:08.kmem [REVISED]' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 May 2005 19:39:25 -0000 hmm.. I'm starting to feel a bit silly maybe. Running the netstat and grep below showed that we had between 800-1700 lines in TIME_WAIT. I then ran the netcat test script while checking for TIME_WAITs. They climbed to the 4800 range then I started getting port failures. I then ran sysctl net.inet.ip.portrange.last=65535 at Mike's recommendation. I then re-ran the netcat script while checking for TIME_WAITs. This time they climbed to 15000 range before I starting getting errors, but this time I started getting errors from the netcat script. Can't get socket : No buffer space available This leads me to believe we were in fact running out of ports and while running this script also overran our socket buffers. I just got the second email from Maxim with the vmstat request as well.. Currently both boxes are sitting higher in the TIME_WAITs box 1: -->netstat -an | grep -c TIME_WAIT 2005/05/13 13:27:41 2455 -->vmstat -z | grep -i sock 2005/05/13 13:29:41 socket: 224, 16424, 3874, 12564, 1107241 box 2: -->netstat -an | grep -c TIME_WAIT 2005/05/13 13:27:10 3541 -->vmstat -z | grep -i sock 2005/05/13 13:29:50 socket: 224, 16424, 3987, 2760, 726779 When I check the vmstat while getting errors from the netcat script I get this. -->vmstat -z | grep -i sock 2005/05/13 13:33:20 socket: 224, 16424, 16438, 0, 1150867 A minute or so later we are back to this: -->netstat -an | grep -c TIME_WAIT 2005/05/13 13:33:55 2302 -->vmstat -z | grep -i sock 2005/05/13 13:34:27 socket: 224, 16424, 3282, 13156, 1155482 Here is my vmstat -z in a "normal" state. ITEM SIZE LIMIT USED FREE REQUESTS PIPE: 160, 0, 80, 124, 282288 SWAPMETA: 160, 233016, 0, 0, 0 unpcb: 160, 0, 46, 104, 74835 ripcb: 192, 16424, 0, 21, 1 divcb: 192, 16424, 0, 0, 0 syncache: 160, 15359, 5, 71, 316134 tcpcb: 576, 16424, 3562, 12836, 1040575 udpcb: 192, 16424, 13, 93, 51641 socket: 224, 16424, 3621, 12817, 1167053 KNOTE: 64, 0, 0, 128, 50789 NFSNODE: 352, 0, 77943, 14, 415454 NFSMOUNT: 544, 0, 4, 10, 4 VNODE: 192, 0, 79602, 110, 79602 NAMEI: 1024, 0, 0, 32, 28750812 VMSPACE: 192, 0, 165, 155, 187746 PROC: 416, 0, 175, 168, 187759 DP fakepg: 64, 0, 0, 0, 0 PV ENTRY: 28, 2690958, 809707, 711801, 259399026 MAP ENTRY: 48, 0, 15192, 16216, 10277033 KMAP ENTRY: 48, 65615, 1037, 200, 305137 MAP: 108, 0, 7, 3, 7 VM OBJECT: 92, 0, 77799, 117, 4201147 And during the failures.. ITEM SIZE LIMIT USED FREE REQUESTS PIPE: 160, 0, 76, 128, 283432 SWAPMETA: 160, 233016, 0, 0, 0 unpcb: 160, 0, 42, 108, 75353 ripcb: 192, 16424, 0, 21, 1 divcb: 192, 16424, 0, 0, 0 syncache: 160, 15359, 2, 74, 331720 tcpcb: 576, 16424, 16375, 23, 1074316 udpcb: 192, 16424, 13, 93, 51949 socket: 224, 16424, 16430, 8, 1201620 KNOTE: 64, 0, 0, 128, 51096 NFSNODE: 352, 0, 78365, 21, 417728 NFSMOUNT: 544, 0, 4, 10, 4 VNODE: 192, 0, 80024, 112, 80024 NAMEI: 1024, 0, 0, 32, 28983336 VMSPACE: 192, 0, 150, 170, 202142 PROC: 416, 0, 160, 183, 202155 DP fakepg: 64, 0, 0, 0, 0 PV ENTRY: 28, 2690958, 661633, 859875, 263111591 MAP ENTRY: 48, 0, 13004, 18404, 10546263 KMAP ENTRY: 48, 65615, 1034, 203, 306496 MAP: 108, 0, 7, 3, 7 VM OBJECT: 92, 0, 78224, 484, 4438390 Am I pretty much just looking at a tuning issue at this point I assume? Matthew Ruzicka - Systems Administrator Front Range Internet, Inc. matt@frii.net - (970) 212-0728 Got SPAM? Take back your email with MailArmory. http://www.MailArmory.com On Fri, 13 May 2005, Maxim Konovalov wrote: > On Fri, 13 May 2005, 12:58-0600, Matt Ruzicka wrote: > > > Yes, it still does. And actually the script Maxim attached to his last > > email (using our IP's) has an interesting side effect of causing the > > connections to fail. > > > > It doesn't fail right away, but within a few moments. > > > > -->./netcat-test 2005/05/13 12:46:51 > > fail > > fail > > fail > > fail > > ... > > Please run > > netstat -an | grep -c TIME_WAIT > > when fails occur. > > -- > Maxim Konovalov >