From owner-freebsd-net@FreeBSD.ORG Wed Apr 23 05:31:04 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA0241065670; Wed, 23 Apr 2008 05:31:04 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id A0F528FC17; Wed, 23 Apr 2008 05:31:04 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m3N5V0hu073780; Tue, 22 Apr 2008 22:31:04 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m3N5UtaP051259; Tue, 22 Apr 2008 22:30:55 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from xxxu000142.ocv.ne.jp.neville-neil.com (xxxu000142.ocv.ne.jp [203.205.0.142]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m3N5UsBa076781; Tue, 22 Apr 2008 22:30:55 -0700 (PDT) (envelope-from gnn@neville-neil.com) Date: Wed, 23 Apr 2008 14:30:50 +0900 Message-ID: From: gnn@freebsd.org To: Chris Pratt In-Reply-To: References: <48087C98.8060600@delphij.net> <382258DB-13B8-4108-B8F4-157F247A7E4B@hughes.net> <20080420103258.D67663@fledge.watson.org> <33AC96BF-B9AC-4303-9597-80BC341B7309@hughes.net> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.11.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: Robert Watson , net@freebsd.org Subject: Re: zonelimit issues... X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Apr 2008 05:31:04 -0000 At Tue, 22 Apr 2008 06:35:38 -0700, Chris Pratt wrote: > > > On Apr 21, 2008, at 12:43 AM, gnn@freebsd.org wrote: > > > ...snip > > > > Well there are plenty of us motivated to get at these issues. Can you > > do me a favor and characterize your traffic a bit? Is it mostly TCP, > > The traffic that seems to take us out is TCP port 80. I'll make a > generalized guess but it does seem to follow. We freeze on one of > two dramatically heavy use days for our industry (Sunday and Monday > evening). The hang will actually occur on Monday or Tuesday > following these days if sufficient traffic hits us. It has not > always followed this pattern but most frequently. There is always a > high presence of high frequency attacks of various sorts. For > example referer spam posts which hit us hard on our busy > evenings. So it is TCP and I would presume we usually have the > establishment of many useless sessions that could cause us to bump > up against limits and cause exhaustion coupled with our real traffic > peaks. > Interesting, but with TCP it should be easier to tune this, in particular because TCP has backoff once a packet drops. I gather you are using facilities, like accept filters, that make it easy to drop less useful traffic? > This thread has given me several things to try and I'm adjusting (e.g., > nmbclusters) upward to see what happens. Sounds good. Using netstat -m and netstat -an are a good way to watch this issue. -m is the number of mbufs/clusters in use and -an will show you all sockets, but what you want to check on s the number of bytes in the recv and send socket buffers, which are the 2nd and 3rd columns. > I should also mention that this system has the natural limitations > on it's traffic ceiling of two T1s on two NICs and a 3rd LAN NIC > fielding continuous round-robin mysql replication and rsync style > mirroring. It uses two bge interfaces and one server type em > interface. It's always troubled me that the zonelimit issues have > always been associated with higher volume circuits (in what I've > read). But since our issue is very directly related to traffic > levels and seem to occur at times where my monitors show us way over > committed on the two outward facing T1s, I'm still going to proceed > with the adjustments and see if it increases our survivability. Since zonelimit is a state reached when your system is out of resources it makes sense that the higher the traffic the sooner you'll reach it. > Thanks for your time on this. > No problem, it's what I like to do :-) Best, George