Date: Thu, 30 Sep 2010 00:11:09 +0200 From: Torbjorn Kristoffersen <torbjoern@gmail.com> To: freebsd-fs@freebsd.org Subject: Re: Strange ZFS problem, filesystem claims to be full when clearly not full Message-ID: <AANLkTikqX1Y3qbcdr-2six%2BPwv61k-Exwh142w5FFqbS@mail.gmail.com> In-Reply-To: <AANLkTi=q6adZv57mwNZVivOwLsfXBjVHki7tzP6-jD0G@mail.gmail.com> References: <AANLkTimRmBi=th1oia5ZuKcEtLR%2BYjK04KNYeZhu931A@mail.gmail.com> <20100929192534.GA97031@icarus.home.lan> <AANLkTi=q6adZv57mwNZVivOwLsfXBjVHki7tzP6-jD0G@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'm at a complete loss here. I shut down the jail completely, and I am watching the jail's ZFS filesystem grow as we speak. No process is using it. It only grows in "df" and "zfs list", I can't find any files that are growing. I have to re-set the quota to be higher and higher to accommodate the space. On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen < torbjoern@gmail.com> wrote: > Hi Jeremy. > > 1) I checked now, and found nothing extraordinary. Just processes that have > been running for a long while, such as screen, cron, sshd, bash, irssi, > syslogd, etc. > > 2) No compression used on this zfs filesystem (or any of the others). > > I completedly stopped the jail now, and removed some of the directories > with the most data in them, but to no avail. > > > On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick <freebsd@jdc.parodius.com > > wrote: > >> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote: >> > I have a ZFS "tank" called tpool, the server runs a couple of jails >> (each >> > with a zfs filesystem). There is a problem with one of these >> filesystems. >> > First, its disk usage as shown in ``df -h'': >> > ... >> > tpool/rb.org 100G 95G 4.6G 95% /jails/rb.org >> > ... >> > >> > The command ``zfs list'' shows the same: >> > .. >> > tpool/rb.org 95.4G 4.56G 95.4G /jails/rb.org >> > .. >> > >> > However, there is a very mysterious problem somewhere. >> > Something inside this jail is eating diskspace, but we can't find any >> > directories that is actually taking the diskspace. We first suspected >> either >> > fetchmail or spamassassin of causing a lot of space to be used, since >> some >> > of their directories were huge. (These were later deleted, and which is >> why >> > you see that 4.6GB is now available, before that 0GB was available). >> > >> > However, we can't find *any trace* of an actual directory or file that >> is >> > taking all the spac.e >> > >> > Take this for instance: >> > >> > outsidejail# du -sh rb.org >> > 43G rb.org >> > >> > How can this be? df and zfs are showing that the entire drive is nearly >> > full, yet I can't find any directory that is actually taking all this >> space. >> > I've carefully looked through every single directory within the jail >> trying >> > to find something that's taking all that space, but to no avail. >> > >> > ---- >> > My system stats: >> > # uname -a >> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC >> > 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> > # zpool get version tpool >> > NAME PROPERTY VALUE SOURCE >> > tpool version 14 default >> > # zpool status >> > pool: tpool >> > state: ONLINE >> > scrub: none requested >> > config: >> > >> > NAME STATE READ WRITE CKSUM >> > tpool ONLINE 0 0 0 >> > mirror ONLINE 0 0 0 >> > ad4s1d ONLINE 0 0 0 >> > ad6s1d ONLINE 0 0 0 >> > >> > errors: No known data errors >> > >> > [ Note that I've also done a scrub recently ] >> >> 1) Have you checked using fstat to ensure that no file descriptors >> remain open on any of your ZFS filesystems (not pools)? >> >> 2) Are you using compression on any of your ZFS filesystems? >> >> -- >> | Jeremy Chadwick jdc@parodius.com | >> | Parodius Networking http://www.parodius.com/ | >> | UNIX Systems Administrator Mountain View, CA, USA | >> | Making life hard for others since 1977. PGP: 4BD6C0CB | >> >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTikqX1Y3qbcdr-2six%2BPwv61k-Exwh142w5FFqbS>