From owner-freebsd-fs@FreeBSD.ORG Thu Sep 30 08:37:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23B07106566B; Thu, 30 Sep 2010 08:37:02 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 979938FC14; Thu, 30 Sep 2010 08:37:01 +0000 (UTC) Received: from outgoing.leidinger.net (p57B3ABE8.dip.t-dialin.net [87.179.171.232]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id D986684400A; Thu, 30 Sep 2010 10:36:56 +0200 (CEST) Received: from webmail.leidinger.net (unknown [IPv6:fd73:10c7:2053:1::2:102]) by outgoing.leidinger.net (Postfix) with ESMTP id D41A91198; Thu, 30 Sep 2010 10:36:53 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id o8U8alK8006368; Thu, 30 Sep 2010 10:36:47 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Thu, 30 Sep 2010 10:36:47 +0200 Message-ID: <20100930103647.62193lbkp9yqx5k4@webmail.leidinger.net> Date: Thu, 30 Sep 2010 10:36:47 +0200 From: Alexander Leidinger To: Jeremy Chadwick References: <20100929192534.GA97031@icarus.home.lan> <20100929221549.GA343@icarus.home.lan> In-Reply-To: <20100929221549.GA343@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: D986684400A.A90B1 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=2.028, required 6, autolearn=disabled, J_CHICKENPOX_41 0.60, RDNS_NONE 1.27, TW_JL 0.08, TW_ZF 0.08) X-EBL-MailScanner-SpamScore: ss X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1286440618.16161@dE0iwi07uJgVRYkV8syYQQ X-EBL-Spam-Status: No Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek , Andriy Gapon Subject: Re: Strange ZFS problem, filesystem claims to be full when clearly not full X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Sep 2010 08:37:02 -0000 Quoting Jeremy Chadwick (from Wed, 29 Sep 2010 15:15:49 -0700): > On Thu, Sep 30, 2010 at 12:11:09AM +0200, Torbjorn Kristoffersen wrote: >> I'm at a complete loss here. I shut down the jail completely, and I am >> watching the jail's ZFS filesystem grow as we speak. No process is using >> it. It only grows in "df" and "zfs list", I can't find any files that are >> growing. I have to re-set the quota to be higher and higher to accommodate >> the space. >> >> On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen < >> torbjoern@gmail.com> wrote: >> >> > Hi Jeremy. >> > >> > 1) I checked now, and found nothing extraordinary. Just processes >> that have >> > been running for a long while, such as screen, cron, sshd, bash, irssi, >> > syslogd, etc. >> > >> > 2) No compression used on this zfs filesystem (or any of the others). >> > >> > I completedly stopped the jail now, and removed some of the directories >> > with the most data in them, but to no avail. >> > >> > >> > On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick > > > wrote: >> > >> >> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote: >> >> > I have a ZFS "tank" called tpool, the server runs a couple of jails >> >> (each >> >> > with a zfs filesystem). There is a problem with one of these >> >> filesystems. >> >> > First, its disk usage as shown in ``df -h'': >> >> > ... >> >> > tpool/rb.org 100G 95G 4.6G 95% /jails/rb.org >> >> > ... >> >> > >> >> > The command ``zfs list'' shows the same: >> >> > .. >> >> > tpool/rb.org 95.4G 4.56G 95.4G /jails/rb.org >> >> > .. >> >> > >> >> > However, there is a very mysterious problem somewhere. >> >> > Something inside this jail is eating diskspace, but we can't find any >> >> > directories that is actually taking the diskspace. We first suspected >> >> either >> >> > fetchmail or spamassassin of causing a lot of space to be used, since >> >> some >> >> > of their directories were huge. (These were later deleted, and which is >> >> why >> >> > you see that 4.6GB is now available, before that 0GB was available). >> >> > >> >> > However, we can't find *any trace* of an actual directory or file that >> >> is >> >> > taking all the spac.e >> >> > >> >> > Take this for instance: >> >> > >> >> > outsidejail# du -sh rb.org >> >> > 43G rb.org >> >> > >> >> > How can this be? df and zfs are showing that the entire drive >> is nearly >> >> > full, yet I can't find any directory that is actually taking all this >> >> space. >> >> > I've carefully looked through every single directory within the jail >> >> trying >> >> > to find something that's taking all that space, but to no avail. >> >> > >> >> > ---- >> >> > My system stats: >> >> > # uname -a >> >> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 >> 02:36:49 UTC >> >> > 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> >> > # zpool get version tpool >> >> > NAME PROPERTY VALUE SOURCE >> >> > tpool version 14 default >> >> > # zpool status >> >> > pool: tpool >> >> > state: ONLINE >> >> > scrub: none requested >> >> > config: >> >> > >> >> > NAME STATE READ WRITE CKSUM >> >> > tpool ONLINE 0 0 0 >> >> > mirror ONLINE 0 0 0 >> >> > ad4s1d ONLINE 0 0 0 >> >> > ad6s1d ONLINE 0 0 0 >> >> > >> >> > errors: No known data errors >> >> > >> >> > [ Note that I've also done a scrub recently ] >> >> >> >> 1) Have you checked using fstat to ensure that no file descriptors >> >> remain open on any of your ZFS filesystems (not pools)? >> >> >> >> 2) Are you using compression on any of your ZFS filesystems? > > Andriy and Pawel, > > Do either of you have ideas as to what could cause the issue Torbjorn's > experiencing? I swear I remember some bug or quirk that got fixed with > regards to free space on ZFS, but as has been proven time and time again > my memory is horrible. His kernel's 8.1-RELEASE dated July 19th. IIRC the commit you talk about was by Martin (CCed). I do not know if it is (already) MFCed. I'm not sure the bug you talk about is related to what Torbjorn is talking about. The fact that the free space is going down while the jail is shutdown (and I assume jls does not show his JID anymore, so all of its processes are really gone) points more to some other process (outside of the jail) which is filling some (maybe already deleted, so not visible anymore with du) file. Bye, Alexander. -- A wide-eyed, innocent UNICORN, poised delicately in a MEADOW filled with LILACS, LOLLIPOPS & small CHILDREN at the HUSH of twilight?? http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137