From owner-freebsd-questions@FreeBSD.ORG Wed Aug 13 18:59:36 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA2581065690 for ; Wed, 13 Aug 2008 18:59:36 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id A9F098FC1B for ; Wed, 13 Aug 2008 18:59:36 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 82AC0EBC09; Wed, 13 Aug 2008 14:59:35 -0400 (EDT) Date: Wed, 13 Aug 2008 14:59:45 -0400 From: Bill Moran To: Christopher Cowart Message-Id: <20080813145945.80d08986.wmoran@potentialtech.com> In-Reply-To: <20080813184603.GA25990@hal.rescomp.berkeley.edu> References: <20080813184603.GA25990@hal.rescomp.berkeley.edu> X-Mailer: Sylpheed 2.5.0 (GTK+ 2.12.11; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: Lots of accounting data X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2008 18:59:36 -0000 In response to Christopher Cowart : > Hello, > > I'm playing a game of cat and mouse with process accounting and disk > space. I built some boxes with 9GB /var partitions, rolled them into > production, and after about 4 days of full load, /var filled up. > > Looking at the size of /var/account/acct{,.0}, and figuring I'd be > seeing a 200% load increase in about a month, I created a new label from > the large chunk of free space I saved for situations like this. 40GB > mounted to /var/account: usage was down to 20%, and I thought the crisis > was averted. > > About a week and a half later, I get a disk full e-mail from nagios and > > +pid 94696 (gzip), uid 0 inumber 6 on /var/account: filesystem full > in my dailies again. My /var/account/acct file was 17GB in size. Add one > rotation before compression and I completely lose that feeling of > cleverness I had when I gave accounting a dedicated 40GB partition. > > If you're wondering how I can possibly have this much accounting data, > two `vmstat -f' invocations 100 seconds apart show 32282 forks (an > average of 323 per second). These boxes are running squid with a > redirect script to implement a captive portal. There are generally > several hundred unauthenticated users; all of their http traffic, from > firefox to the little weather widgets and spyware phoning home, gets > proxied through squid and subsquently a redirect script that, among > other things, does some text munging on the URL, and queries various > ipfw tables to determine what "context" the user is in. Some of this > could be optimized to launch fewer processes, but the code would be less > maintainable. > > I only really see two options, neither of which I particularly like: > * Throw more disk at the problem (but given what I've seen, I don't > like the odds that within a month or two, I'll realize I didn't give > it enough). > * Turn off accounting on these boxes. * Rotate and compress more frequently; and store less history? -- Bill Moran http://www.potentialtech.com