From owner-freebsd-questions@FreeBSD.ORG Fri Feb 13 16:06:08 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB8DB106566B for ; Fri, 13 Feb 2009 16:06:08 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [220.233.188.227]) by mx1.freebsd.org (Postfix) with ESMTP id 267708FC17 for ; Fri, 13 Feb 2009 16:06:07 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id n1DG5rMr099058; Sat, 14 Feb 2009 03:05:54 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sat, 14 Feb 2009 03:05:52 +1100 (EST) From: Ian Smith To: scuba@centroin.com.br In-Reply-To: <20090212195437.90C441065726@hub.freebsd.org> Message-ID: <20090214014803.R38905@sola.nimnet.asn.au> References: <20090212195437.90C441065726@hub.freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-questions@freebsd.org Subject: Re: Help with high LA X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2009 16:06:09 -0000 On Thu, 12 Feb 2009 10:04:41 -0200 (BRST) scuba@centroin.com.br wrote: > I need help for some strange problem with one of my servers, that can cost > my job. > > It's a FreeBSD 7.0-RELEASE-p5/amd64 running on a Dell PowerEdge III as a > Virtual machine of VMware ESXi. There are only two VM in this box, and one > of them (basicly a mail server) is running fine. > > The problem is with high loads on the other one, that runs (besides other > services) http and pop3. > > TOP show LA from 40 to 90 most of the time. > > I thought, at first, that was a disk botleneck due to some big mailboxes, > or something related to some Apache (2.2.9) fine tuning, but it's > something else. > > > If I stop pop3 and apache services (the most active of the box), the LA > drops to 1~2. > Starting only one of them (any one) the LA rise to 20~40. Sugesting that > it's not tied to a specific service. > > I did a test running just pop3 (Qpopper), pointing the mail spool to a > empty directory, to make shure that it's not a disk problem. And the LA > also goes to sky (~30). The same happens with only apache running pointing > to a simple http page. > > The console shows messages like: > > ipfw: install_state: Too many dynamic rules net.inet.ip.fw.dyn_max: 4096 # (here) Maximum number of dynamic rules. When you hit this limit, no more dynamic rules can be installed until old ones expire. To see which traffic is creating 'too many' dynamic rules, check: # ipfw -ted show | less -S ++G (-td for just active rules, but the expired ones tell useful stories) > I know I must review my rules and limit the number of keep-state entries, > but a tryed to rise the number of dynamic buckets via sysctl: > > sysctl -w net.inet.ip.fw.dyn_buckets=2048 > > But it seems it's not working, since the number of current buckets doesn't > pass 256: > > net.inet.ip.fw.curr_dyn_buckets: 256 But did you remember to flush? :) See ipfw(8) under 'SYSCTL VARIABLES'. You might also want to monitor and/or play with some of the other net.inet.ip.fw.dyn_* sysctls to see what's happening and how many dynamic rules you need with comfortable headroom for your workload/s. For TCP, keepalive and *lifetime timeouts may be relevant. I tend to use stateful rules for outbound UDP, and stateless setup and established rules for TCP services here, but your needs may differ. > I tryed to make some OS tuning, from the handbook, like increase the > maxcon: > > kern.ipc.somaxconn: 2048 > > but nothing seems to work. > > Other entries in the logs: > Feb 12 09:06:20 host1 inetd[1248]: accept (for ftp): Software caused > connection abort > Feb 12 09:06:20 host1 inetd[1248]: accept (for pop3): Software caused > connection abort > > I need some clues to undestand what is happening. > > Thank you, > > - Marcelo Yes, 'Too many dynamic rules'; further connections will surely fail. cheers, Ian