From owner-freebsd-fs@FreeBSD.ORG Thu Sep 30 17:08:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5A0F1065675 for ; Thu, 30 Sep 2010 17:08:55 +0000 (UTC) (envelope-from torbjoern@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 630E48FC1A for ; Thu, 30 Sep 2010 17:08:54 +0000 (UTC) Received: by bwz15 with SMTP id 15so2045521bwz.13 for ; Thu, 30 Sep 2010 10:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=Q9C6ztQhJlATbYUMqtzfJH68X3l3NO9+/ezEjA2o9UE=; b=T3mN+4uZQfCf4C9C5FwTYprdKCmQCjNBq0pEFnWjT+kRwCWmEwdCi2nwpUIFanq1c6 tCcSj4f0UApseUDMnPa4L8mSrhmt1v1ypWFu0lpbjHLZ+kVHRF5pX+ygNExRehCXgvjk 4j0nb7T/5+AKWZkOs8PwUurc3DKeVWjINPw7Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=clGNU903fGvzl0vxsIWw5Ifcb2BKQSYRl7S5HfWXYNmyHScY1d91n+5jegVr2G7Lpk syQmzNKKBRiWFHsyQZ+9HfVRUZFL98GvWAocZjSz8aC8xv1RXCvNEHJEIOjsAbAXnQcc kQ/RwBepbeDi5vcqk6n6szCKLcyr5hyucdpBU= MIME-Version: 1.0 Received: by 10.204.82.167 with SMTP id b39mr3002759bkl.164.1285866530661; Thu, 30 Sep 2010 10:08:50 -0700 (PDT) Received: by 10.204.71.138 with HTTP; Thu, 30 Sep 2010 10:08:50 -0700 (PDT) In-Reply-To: References: <4CA45444.6070002@dannysplace.net> <201009301438.o8UEckoY019473@lurza.secnetix.de> <20100930144845.GA19926@icarus.home.lan> <4CA4B12B.7050307@icyb.net.ua> Date: Thu, 30 Sep 2010 19:08:50 +0200 Message-ID: From: Torbjorn Kristoffersen To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Fwd: Strange ZFS problem, filesystem claims to be full when clearly not full X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Sep 2010 17:08:56 -0000 On Thu, Sep 30, 2010 at 5:47 PM, Andriy Gapon wrote: > on 30/09/2010 17:48 Jeremy Chadwick said the following: >> On Thu, Sep 30, 2010 at 04:38:46PM +0200, Oliver Fromme wrote: >>> Danny Carroll wrote: >>> =A0> [...] >>> =A0> It certainly smells like a process still writing to a file that is= unlinked. >>> =A0> I wonder if it would show up with lsof. >>> >>> If it's a file that was unlinked that is still held open by >>> a process, then lsof will definitely list it. =A0The command >>> >>> # lsof +L1 >>> >>> lists all open files with a link count of zero. =A0You can >>> restrict it to a certain file system like this: >>> >>> # lsof +aL1 /var >>> >>> Of course, lsof won't list the file name because the file >>> doesn't have a name anymore. =A0But it lists the process by >>> name, PID and user, the file system and the file size. >> >> Can someone explain how use of lsof in this regard is different than use >> of fstat(1) like I originally mentioned? =A0Does lsof do something more >> thorough or differently that what fstat does? > > I believe that there is no reason to prefer lsof except for those who spe= nt more > time with Linux than with FreeBSD. > I tried fstat earlier and now I tried lsof as suggested. =A0Doing lsof +L1 only gave me: COMMAND =A0PID =A0USER =A0 FD =A0 TYPE DEVICE SIZE/OFF NLINK =A0 NODE NAME mysqld =A01030 mysql =A0 =A04u =A0VREG =A0 0,99 =A0 =A0 =A0 =A00 =A0 =A0 0 = 800965 / (/dev/mirror/root) mysqld =A01030 mysql =A0 =A05u =A0VREG =A0 0,99 =A0 =A0 =A0 =A00 =A0 =A0 0 = 800969 / (/dev/mirror/root) mysqld =A01030 mysql =A0 =A06u =A0VREG =A0 0,99 =A0 =A0 =A0 =A00 =A0 =A0 0 = 800970 / (/dev/mirror/root) .... Basically, it only gives me mysqld which runs outside the jails. Nothing else was listed. I noticed that the filesystem has stopped growing now though, so that may also be the reason why lsof does not show anything anymore. The "du -sh /jails/rb.org" still gives a low usage value. Also, this is the output from df -h (I've since resized the ZFS quota to make the filesystem bigger for this jail): tpool/rb.org =A0 =A0 =A0200G =A0 =A0111G =A0 =A0 89G =A0 =A056% =A0 =A0/jai= ls/rb.org If the process causing this is gone, or is working correctly (seeing that the fs is no longer growing, I hope), can dead unlinked files still remain, is there a way to purge them?