Date: Thu, 9 Sep 2004 13:03:33 -0400 From: Bill Moran <wmoran@potentialtech.com> To: Paul Schmehl <pauls@utdallas.edu> Cc: questions@freebsd.org Subject: Re: Phantom /var full messages Message-ID: <20040909130333.67242dc4.wmoran@potentialtech.com> In-Reply-To: <44A044721750C2FA9877513F@utd49554.utdallas.edu> References: <44A044721750C2FA9877513F@utd49554.utdallas.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Paul Schmehl <pauls@utdallas.edu> wrote: > I'm running snort 2.1.3 and mysql 3.23.58 on FreeBSD 4.9 RELEASE. All > applications are built from ports. > > Periodically I get /var full messages and everything comes to a grinding > halt. The problem is, /var isn't full. > > df -h will show /var at 104%, but du -h /var shows /var at 40% (for > example). This typically happens because a file has been deleted, but some program still has it open. The filesystem can't actually free up the data blocks until no other programs are using them. But du doesn't see the usage because there's no longer a file there to attribute it to. > If I shut down snort and mysql, wait for a minute and then start > them back up, df agrees with du again. You've already done the first diagnostic step. You know that either snort or MySQL is keeping a file handle after a file is deleted. > The system works fine because only /var is full (although things can get > squirrelly if I let it go long enough because the system can't write to the > logs or the mail spool), so I can still ssh in and run utilities. Yup. > I suspect this is some sort of filehandle not being released issue, but I'm > not sure how to track it down. I've got lsof installed, but I'm not an > expert on it yet. > > Any hints would be welcomed. What's the best way to troubleshoot this > problem? First, if you could isolate it to just snort or just MySQL. Typically, folks have this problem because they try to rotate log files without restarting the program that's logging to them. The rotate program compresses the current log file into a new file, then deletes the original file ... but the program is still logging to it. Thus the space fills up, but there is no file to see the space in. Restarting the program doing the logging causes the old file to disappear, and a new log file to be created. On a guess, Snort would be the first thing I'd look at. However, MySQL can create a TON of data if logging is enabled, so you may want to look closely at it as well. HTH -- Bill Moran Potential Technologies http://www.potentialtech.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040909130333.67242dc4.wmoran>