Date: Sun, 30 Mar 2003 18:34:06 +0200 From: Paul Schenkeveld <fb-isp@psconsult.nl> To: "Arie J. Gerszt" <arie@gerszt.ch> Cc: freebsd-isp@FreeBSD.ORG Subject: Re: file system help needed Message-ID: <20030330183406.A35239@psconsult.nl> In-Reply-To: <FEEHKMHBPPGLAPHJCDIIEENLCFAA.arie@gerszt.ch>; from arie@gerszt.ch on Tue, Mar 25, 2003 at 06:14:59PM %2B0100 References: <DBEIKNMKGOBGNDHAAKGNEEDEKOAB.dave@hawk-systems.com> <FEEHKMHBPPGLAPHJCDIIEENLCFAA.arie@gerszt.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi all, I just saw this thread, hence this late reaction. Hopefully I'm late and you got your system in shape again but in case you still didn't find the culprit, here are my $0.02. First, did you ever "ktrace" a process and remove the ktrace.out file? In that case, "ktrace -C" will instruct the kernel to stop tracing and release all ktrace.out files, including those that you already removed. Then, I'd check for files that were removed but are still open by some process and continue to occupy diskspace. To check this: compare the output of "du -skx /" to the value reported under Used in the output of "df -l /". If the two are (almost) the same, all space is occupied by files that are still around in some directory, in that case you really have to look though all directories of the root filesystem to find what's there but should not be there (a common culprit is a regular file in /dev, possibly with a name looking like the name of a tape drive or even /dev/null). If however these two numbers are very different you've got removed files still occupying disk space because they are open. In that case you can either reboot the machine compelete if possible (make sure /sbin and /bin are back first otherwise you even don't have a mount command anymore and got yourself a chicken and egg situation), or stop processes one by one (and restart them afterwards) until df suddenly shows free space again (hopefully you've not enabled soft updates on / in which case recovery of free space will not show up in df until up to half a minute or so after stopping the process which kept the file open). Or you can take a more deterministic approach by looking au the output of fstat. The MOUNT column will show / for all inodes (files) open that are on the root filesystem. Concentrate on regular files (lines with a '-' as the first character of the MODE column) and look at the SZ|DV column. Huge numbers here indicate huge files. Skip lines with "text" in the FD column and try to find suspicious files using "find / -xdev -inum <inum>" using the inode number from the INUM column. This should reveal processes keeping deleted files open, kill those culprits and if they served a purpose, restart them again. Regards, Paul Schenkeveld, Consultant PSconsult ICT Services BV On Tue, Mar 25, 2003 at 06:14:59PM +0100, Arie J. Gerszt wrote: > Hi Everybody > > I have a huge problem as you see ... > > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/ad0s1a 99183 96309 -5060 106% / > /dev/ad0s1e 5458605 2866595 2155322 57% /usr > /dev/ad0s1g 17876344 1085421 15360816 7% /usr/www > /dev/ad0s1f 4465853 212592 3895993 5% /var > procfs 4 4 0 100% /proc > > This server is a production server and I can't add any dns zonefiles or > passwords, > because they sit in /etc which is full, evidently. > > What can I do to solve that rather quickly? The disk ad0 has free space: > > caramba# fdisk /dev/ad0 > ******* Working on device /dev/ad0 ******* > parameters extracted from in-core disklabel are: > cylinders=3649 heads=255 sectors/track=63 (16065 blks/cyl) > > Figures below won't work with BIOS for partitions not in cyl 1 > parameters to be used for BIOS calculations are: > cylinders=3649 heads=255 sectors/track=63 (16065 blks/cyl) > > Media sector size is 512 > Warning: BIOS sector numbering starts with sector 1 > Information from DOS bootblock is: > The data for partition 1 is: > sysid 165,(FreeBSD/NetBSD/386BSD) > start 63, size 58621122 (28623 Meg), flag 80 (active) > beg: cyl 0/ head 1/ sector 1; > end: cyl 1023/ head 254/ sector 63 > The data for partition 2 is: > <UNUSED> > The data for partition 3 is: > <UNUSED> > The data for partition 4 is: > <UNUSED> > caramba# > > > --> but i am not sure what to do, because I can't have any long downtime, > understandibily. > Thanks for help, > > Arie > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030330183406.A35239>