From owner-freebsd-fs@FreeBSD.ORG Wed Sep 29 19:25:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFE89106566B for ; Wed, 29 Sep 2010 19:25:36 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 959D78FC1A for ; Wed, 29 Sep 2010 19:25:36 +0000 (UTC) Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36]) by qmta06.emeryville.ca.mail.comcast.net with comcast id CcTG1f0070mlR8UA6jRcVP; Wed, 29 Sep 2010 19:25:36 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta11.emeryville.ca.mail.comcast.net with comcast id CjRa1f00T3LrwQ28XjRbow; Wed, 29 Sep 2010 19:25:35 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id C6B8F9B418; Wed, 29 Sep 2010 12:25:34 -0700 (PDT) Date: Wed, 29 Sep 2010 12:25:34 -0700 From: Jeremy Chadwick To: Torbjorn Kristoffersen Message-ID: <20100929192534.GA97031@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Strange ZFS problem, filesystem claims to be full when clearly not full X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Sep 2010 19:25:36 -0000 On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote: > I have a ZFS "tank" called tpool, the server runs a couple of jails (each > with a zfs filesystem). There is a problem with one of these filesystems. > First, its disk usage as shown in ``df -h'': > ... > tpool/rb.org 100G 95G 4.6G 95% /jails/rb.org > ... > > The command ``zfs list'' shows the same: > .. > tpool/rb.org 95.4G 4.56G 95.4G /jails/rb.org > .. > > However, there is a very mysterious problem somewhere. > Something inside this jail is eating diskspace, but we can't find any > directories that is actually taking the diskspace. We first suspected either > fetchmail or spamassassin of causing a lot of space to be used, since some > of their directories were huge. (These were later deleted, and which is why > you see that 4.6GB is now available, before that 0GB was available). > > However, we can't find *any trace* of an actual directory or file that is > taking all the spac.e > > Take this for instance: > > outsidejail# du -sh rb.org > 43G rb.org > > How can this be? df and zfs are showing that the entire drive is nearly > full, yet I can't find any directory that is actually taking all this space. > I've carefully looked through every single directory within the jail trying > to find something that's taking all that space, but to no avail. > > ---- > My system stats: > # uname -a > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC > 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > # zpool get version tpool > NAME PROPERTY VALUE SOURCE > tpool version 14 default > # zpool status > pool: tpool > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tpool ONLINE 0 0 0 > mirror ONLINE 0 0 0 > ad4s1d ONLINE 0 0 0 > ad6s1d ONLINE 0 0 0 > > errors: No known data errors > > [ Note that I've also done a scrub recently ] 1) Have you checked using fstat to ensure that no file descriptors remain open on any of your ZFS filesystems (not pools)? 2) Are you using compression on any of your ZFS filesystems? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |