From owner-freebsd-questions@FreeBSD.ORG Thu Jun 19 03:46:02 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1CBC1065903 for ; Thu, 19 Jun 2008 03:46:00 +0000 (UTC) (envelope-from pschmehl_lists@tx.rr.com) Received: from mail.stovebolt.com (mail.stovebolt.com [66.221.101.249]) by mx1.freebsd.org (Postfix) with ESMTP id 8EFA18FC1F for ; Thu, 19 Jun 2008 03:45:59 +0000 (UTC) (envelope-from pschmehl_lists@tx.rr.com) Received: from [192.168.2.102] (cpe-24-175-90-48.tx.res.rr.com [24.175.90.48]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.stovebolt.com (Postfix) with ESMTPSA id 892E811438F for ; Wed, 18 Jun 2008 22:45:51 -0500 (CDT) Date: Wed, 18 Jun 2008 22:45:57 -0500 From: Paul Schmehl To: freebsd-questions@freebsd.org Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) X-Munged-Reply-To: To reply - figure it out MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: /var full X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Paul Schmehl List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2008 03:46:07 -0000 At 10PM (local time) this evening, a server started reporting that /var was full. When I ssh'd in to the server to investigate, df said /var was at 2% full (5.1G) and dh reported the same (5.1G). /var/log/dmesg.today is full of messages listing multiple entries with the same inode number followed by one entry listing dd as the culprit. +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132107 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 73338 (dd), uid 2 inumber 27131975 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full +pid 73365 (dd), uid 2 inumber 27131920 on /var: filesystem full And so on. What is going on? The system seems to be functioning normally. df and dh report normal disk usage. Top looks fine. /var/log/dmesg.today shows 309 total lines and 98 unique lines. Was this some sort of temporary glitch? Or something more ominous? Why would toor be running dd? Is it some sort of file recovery routine triggered by filesystem full messages? Paul Schmehl If it isn't already obvious, my opinions are my own and not those of my employer.