From owner-freebsd-fs@FreeBSD.ORG Wed Sep 29 22:15:52 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 910CE1065673 for ; Wed, 29 Sep 2010 22:15:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 3A0288FC12 for ; Wed, 29 Sep 2010 22:15:51 +0000 (UTC) Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88]) by qmta03.westchester.pa.mail.comcast.net with comcast id CbLE1f0061uE5Es53mFsit; Wed, 29 Sep 2010 22:15:52 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta16.westchester.pa.mail.comcast.net with comcast id CmFq1f00E3LrwQ23cmFrtZ; Wed, 29 Sep 2010 22:15:52 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6B6099B418; Wed, 29 Sep 2010 15:15:49 -0700 (PDT) Date: Wed, 29 Sep 2010 15:15:49 -0700 From: Jeremy Chadwick To: Torbjorn Kristoffersen Message-ID: <20100929221549.GA343@icarus.home.lan> References: <20100929192534.GA97031@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek , Andriy Gapon Subject: Re: Strange ZFS problem, filesystem claims to be full when clearly not full X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Sep 2010 22:15:52 -0000 On Thu, Sep 30, 2010 at 12:11:09AM +0200, Torbjorn Kristoffersen wrote: > I'm at a complete loss here. I shut down the jail completely, and I am > watching the jail's ZFS filesystem grow as we speak. No process is using > it. It only grows in "df" and "zfs list", I can't find any files that are > growing. I have to re-set the quota to be higher and higher to accommodate > the space. > > On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen < > torbjoern@gmail.com> wrote: > > > Hi Jeremy. > > > > 1) I checked now, and found nothing extraordinary. Just processes that have > > been running for a long while, such as screen, cron, sshd, bash, irssi, > > syslogd, etc. > > > > 2) No compression used on this zfs filesystem (or any of the others). > > > > I completedly stopped the jail now, and removed some of the directories > > with the most data in them, but to no avail. > > > > > > On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick > > wrote: > > > >> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote: > >> > I have a ZFS "tank" called tpool, the server runs a couple of jails > >> (each > >> > with a zfs filesystem). There is a problem with one of these > >> filesystems. > >> > First, its disk usage as shown in ``df -h'': > >> > ... > >> > tpool/rb.org 100G 95G 4.6G 95% /jails/rb.org > >> > ... > >> > > >> > The command ``zfs list'' shows the same: > >> > .. > >> > tpool/rb.org 95.4G 4.56G 95.4G /jails/rb.org > >> > .. > >> > > >> > However, there is a very mysterious problem somewhere. > >> > Something inside this jail is eating diskspace, but we can't find any > >> > directories that is actually taking the diskspace. We first suspected > >> either > >> > fetchmail or spamassassin of causing a lot of space to be used, since > >> some > >> > of their directories were huge. (These were later deleted, and which is > >> why > >> > you see that 4.6GB is now available, before that 0GB was available). > >> > > >> > However, we can't find *any trace* of an actual directory or file that > >> is > >> > taking all the spac.e > >> > > >> > Take this for instance: > >> > > >> > outsidejail# du -sh rb.org > >> > 43G rb.org > >> > > >> > How can this be? df and zfs are showing that the entire drive is nearly > >> > full, yet I can't find any directory that is actually taking all this > >> space. > >> > I've carefully looked through every single directory within the jail > >> trying > >> > to find something that's taking all that space, but to no avail. > >> > > >> > ---- > >> > My system stats: > >> > # uname -a > >> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC > >> > 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > >> > # zpool get version tpool > >> > NAME PROPERTY VALUE SOURCE > >> > tpool version 14 default > >> > # zpool status > >> > pool: tpool > >> > state: ONLINE > >> > scrub: none requested > >> > config: > >> > > >> > NAME STATE READ WRITE CKSUM > >> > tpool ONLINE 0 0 0 > >> > mirror ONLINE 0 0 0 > >> > ad4s1d ONLINE 0 0 0 > >> > ad6s1d ONLINE 0 0 0 > >> > > >> > errors: No known data errors > >> > > >> > [ Note that I've also done a scrub recently ] > >> > >> 1) Have you checked using fstat to ensure that no file descriptors > >> remain open on any of your ZFS filesystems (not pools)? > >> > >> 2) Are you using compression on any of your ZFS filesystems? Andriy and Pawel, Do either of you have ideas as to what could cause the issue Torbjorn's experiencing? I swear I remember some bug or quirk that got fixed with regards to free space on ZFS, but as has been proven time and time again my memory is horrible. His kernel's 8.1-RELEASE dated July 19th. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |