From owner-freebsd-fs@FreeBSD.ORG Sun Oct 17 03:39:00 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22566106564A for ; Sun, 17 Oct 2010 03:39:00 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id C88088FC08 for ; Sun, 17 Oct 2010 03:38:59 +0000 (UTC) Received: by gxk4 with SMTP id 4so1065520gxk.13 for ; Sat, 16 Oct 2010 20:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :organization:user-agent:mime-version:to:cc:subject:references :in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; bh=ZI3iWXj00uFBuN9a7zEFYhkMq4RQSxZ/GL0mkT/5P70=; b=VpngejTfhgz3XsrYKlbFhm+gPq7ql6RE0XYJn7dvg/Ih6KcoTKz8ua9DnsaEyKRb8y zLRh9Vmzi8KZtu0a96QxnoRLBCR8BzSlMJq+fXrwmghG3WpcMhTg1dm4jGMyJ4lpiuNs OtOM4G3u0+rsnLVcOLFMKV0ApBOHVi3GpY/jk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:organization:user-agent:mime-version:to :cc:subject:references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=sw/lqXZ7iWin9+obB+hf58zSpPdcEHL/INF/RCM4nq422mYJE7vrsydlLPy6SXEL5P eCu6dkRFAD5mGsHQri0uqtP0LK+Qx5q71n1KluSFjQjTCsOuekwQ9LEszqX8RfTRtTiL B237j8TxkCcFjQXzcYb5BXVxMlKLVJ3CT5GOg= Received: by 10.236.102.145 with SMTP id d17mr3931643yhg.29.1287286738887; Sat, 16 Oct 2010 20:38:58 -0700 (PDT) Received: from centel.dataix.local ([99.181.158.202]) by mx.google.com with ESMTPS id j64sm7054954yha.24.2010.10.16.20.38.56 (version=SSLv3 cipher=RC4-MD5); Sat, 16 Oct 2010 20:38:57 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4CBA6FCF.3080301@DataIX.net> Date: Sat, 16 Oct 2010 23:38:55 -0400 From: jhell Organization: http://www.DataIX.net User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100917 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: David Samms References: <201010151909.o9FJ9cZf065459@lurza.secnetix.de> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS trouble: unbelievably large files created X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Oct 2010 03:39:00 -0000 On 10/15/2010 18:44, David Samms wrote: > On 10/15/10 15:09, Oliver Fromme wrote: >> > ls -kls shawxp/var/amavis/.spamBAD >> > total 13352131 >> > 771 -rw------- 1 110 110 684032 Oct 15 12:02 >> > auto-whitelist >> > 1 -rw------- 1 110 110 40 Oct 15 12:37 >> bayes.lock >> > 1284 -rw------- 1 110 110 1294336 Oct 15 12:38 >> bayes_seen >> > 4229 -rw------- 1 110 110 4227072 Oct 15 12:38 >> bayes_toks >> > 5025103 -rw------- 1 110 110 553184002048 Oct 15 12:38 >> > bayes_toks.expire3515 >> > 8320745 -rw------- 1 110 110 140743122878464 Oct 15 12:14 >> > bayes_toks.expire97254 >> >> Ok, so those files are so-called "sparse" files, i.e. they >> contain holes that don't actually occupy disk space. >> >> The numbers in the first column indicate the amount of >> physical disk space allocated (in KB). That's about 5 GB >> for the first file and 8 GB for the second (this is also >> consistent with the "total" value in the first line of the >> ls output, i.e. about 13 GB). >> >> That's still quite big, but certainly not in the TB range. >> I do not know why amavis creates such large sparse files, >> though. > > Under UFS I don't observe amavis creating the sparse files. The problem > is fairly repeatable, not on demand, but occurs several times a day. > High disk activity is a sure sign amavis is creating large files. Task > shuts down normaly and cpu load is low. > > This is a production server but I have moved all critical customer jails > back to UFS so can do testing if anyone has any ideas of what to look for. > I do not believe this to be a problem with amavis/amavisd, lets get that out of the way. Why I say this is because I have experienced the same problem on a server with UFS and p5-Mail-Spamassassin. I was logged into that server doing some work and there was plenty of disk space to do the work as it consisted of copying a ton of space eating files to the file-system where this happened. After operations were done, cleaned up the disk properly restoring space back for normal usage and then logged out. I came back the next day only to find my INBOX full of error messages and other junk from that server saying things like can't write to disk and such error messages. What it turned out to be was the same files that you had mentioned above but in my ~/.spamassassin/.bayes*. One of the files just like yours had grown to eat up the entire amount of disk space that was left besides the 8% that was reserved. I tried to find the offending email messages that might have caused this to happen but in some odd ~7 hours total time did not come up with a conclusion and could not afford to waste further time on that issue. IMHO this has to do with bayes and the usage of dbm, or db4* and something in a messages header or body that triggers it to happen but do not have enough time or conclusive evidence yet to suggest anything further. Ultimately I removed the offending files and went about business as usual until I see it happen again. Also tuned the quota for myself to be a much lower amount disk space so I should get an email as soon as it happens again without effecting the rest of operations on that disk. Regards, -- jhell,v