Date: Tue, 03 Apr 2007 18:00:53 +0100 From: Matthew Seaman <m.seaman@infracaninophile.co.uk> To: Sean Murphy <smurphy@calarts.edu> Cc: freebsd-questions@freebsd.org Subject: Re: var Filesystem Full Help Message-ID: <46128845.3030308@infracaninophile.co.uk> In-Reply-To: <46127E55.2040005@calarts.edu> References: <46127E55.2040005@calarts.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig197E378CA444C047429D5ED7 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable Sean Murphy wrote: > I am getting these errors on my var filesystem but df -h shows there is= > plenty of space available. >=20 > I am running FreeBSD 5.4 >=20 > muse2# df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/amrd0s1a 989M 56M 854M 6% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/amrd0s1e 989M 32M 878M 4% /tmp > /dev/amrd0s1f 9.5G 4.2G 4.6G 48% /usr > /dev/amrd0s1g 245G 9.4G 216G 4% /usr/home > /dev/amrd0s1d 1.9G 526M 1.3G 29% /var >=20 > muse2# tail /var/log/messages > Apr 3 09:00:44 muse2 kernel: pid 537 (mimedefang), uid 26 inumber > 126291 on /var: filesystem full > Apr 3 09:09:55 muse2 kernel: pid 52000 (httpd), uid 80 inumber 170037 > on /var: filesystem full > Apr 3 09:12:59 muse2 kernel: pid 34758 (mimedefang), uid 26 inumber > 127701 on /var: filesystem full >=20 > I have restarted the mimdefang process but I get the same messages. >=20 > What can I do? There are two reasons why a filesystem may give 'out of space' errors when df(1) still shows plenty of space available. i) Out of inodes. You can tell this by running 'df -i'. You're unlikely to run into this unless either you used non-standard settings when you newfs'd the partition or else the partition is full of a very large number of very files. If this is the case, then apart from rampantly deleting lots of stuff the only solution is to backup the filesystem somewhere, recreate the filesystem by running newfs with a more realistic set of parameters (bytes-per-inode should be smaller) and then recover the data from backup. ii) Open file descriptor on an unlinked file. This is much more likely to be the problem with the /var partition, seeing as it's a favourite place for log files. What can happen is this: a process has a file open (ie. it has an open file descriptor on the data) but a second process comes along and unlinks the original file.= That means that the file name and other meta data are removed from the=20 directory contents, but since another process has the file open, the spac= e taken up by the files' data is not returned to the generally available pool. Sounds daft at first, but that's the way Unix has worked since the= epoch and if you think about it, it makes sense really. Doing that deliberately can be exceedingly useful -- a program can reserve itself some scratch space that can't be accessed or altered by anything else[*] What tends to happen in /var is a side effect of not rotating log files correctly. newsyslog(8) and pals will move aside and compress an existin= g log file very happily, then they will send a signal to the program genera= ting the log file (by default assumed to be syslogd) to tell it to close and reopen any files it is logging to -- it's a common behaviour for Unix daemons to understand a SIGHUP to mean 'reinitialise yourself and reopen any files you're using' If newsyslog(8) signals the wrong process, or doesn't signal any process at all, or the process doesn't grok the SIGHUP, then you'll find you get exactly the sort of orphaned file with an open descriptor on it as descri= bed above. The way to debug this is to list all of the processes that have open desc= riptors on the partition: # fstat -f /var then it's a case of doing some detective work to try and identify which o= ut of the many processes listed is the culprit. Unfortunately fstat(1) does= n't tell you file names -- instead you get the files inode number as column 6= of the output. There is no generic method of mapping from inode number to filename (indeed, orphaned files like we've been discussing have an inode= number, but *no* filename); other than by doing exhaustive searches using= eg. find(1): # find /var -inum nnnnn -print In this case you're looking for the ones that don't return an answer. Cheers, Matthew [*] Well, not without rootly powers, ample clue and a reasonable expendit= ure of effort. --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW --------------enig197E378CA444C047429D5ED7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGEohL8Mjk52CukIwRCCIbAJ0XyyTZPPpdwlxpDMJLD1BFxxNZIACeO9Ta s9cVsZRh+HHV80Tvwxe5wPQ= =OCkw -----END PGP SIGNATURE----- --------------enig197E378CA444C047429D5ED7--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46128845.3030308>