From owner-freebsd-bugs@FreeBSD.ORG Mon Apr 2 22:30:03 2007 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6D26316A40F for ; Mon, 2 Apr 2007 22:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 4B1FD13C4C4 for ; Mon, 2 Apr 2007 22:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l32MU3YA001950 for ; Mon, 2 Apr 2007 22:30:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l32MU39N001949; Mon, 2 Apr 2007 22:30:03 GMT (envelope-from gnats) Resent-Date: Mon, 2 Apr 2007 22:30:03 GMT Resent-Message-Id: <200704022230.l32MU39N001949@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, George Breahna Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B49AD16A405 for ; Mon, 2 Apr 2007 22:22:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [69.147.83.33]) by mx1.freebsd.org (Postfix) with ESMTP id 8D27213C480 for ; Mon, 2 Apr 2007 22:22:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l32MMLXI057397 for ; Mon, 2 Apr 2007 22:22:21 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id l32MHKIk057061; Mon, 2 Apr 2007 22:17:20 GMT (envelope-from nobody) Message-Id: <200704022217.l32MHKIk057061@www.freebsd.org> Date: Mon, 2 Apr 2007 22:17:20 GMT From: George Breahna To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.0 Cc: Subject: misc/111159: Kernel panic caused by UFS X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2007 22:30:03 -0000 >Number: 111159 >Category: misc >Synopsis: Kernel panic caused by UFS >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Apr 02 22:30:02 GMT 2007 >Closed-Date: >Last-Modified: >Originator: George Breahna >Release: 6.1-RELEASE-p12 >Organization: G-Tech IT Consulting >Environment: FreeBSD store016.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #0: Mon Jan 22 11:20:03 UTC 2007 pulsar@:/usr/src/sys/i386/compile/GTECH i386 >Description: The machine contains a 3ware 9550SX-4LP card. It has attached to it 4 x 500GB disks in RAID 10 config. Output from 3ware CLI is all normal, RAID ok, drives OK. The drives and the 3ware card have been updated to the latest firmware available from manufacturer. This machine is mainly used for backups of an e-mail server. Within 1 hour of starting the backups, the following error appears: Mar 31 00:29:05 store01 syslogd: kernel boot file is /boot/kernel/kernel Mar 31 00:29:05 store01 kernel: start = 0, len = 11431, fs = /usr Mar 31 00:29:05 store01 kernel: panic: ffs_alloccg: map corrupted Mar 31 00:29:05 store01 kernel: Uptime: 2d22h57m30s Mar 31 00:29:05 store01 kernel: Cannot dump. No dump device defined. Mar 31 00:29:05 store01 kernel: Automatic reboot in 15 seconds - press a key on the console to abort The first time it happened I rebooted in single user mode and rand a fsck -y on all partitions. It didn't encounter any errors except for the following type: fsck: /dev/da0s1f: INCORRECT BLOCK COUNT I=28342929 (8 should be 0) (CORRECTED) After that the system reboots fine, works excellent until the backups are started again. I should mention that the server ran fine for up to 22 days, sitting idle. The day I put it into production these issues appeared. I did a bit of research and the only other case where I found a very similar problem is with NetBSD, documented here: http://www.nabble.com/kern-35580%3A-creating-directories-on-large-UFS2--%3E-boom-tf3203420.html#a8895438 Some output from df: Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1a 988398 122102 787226 13% 4425 136885 3% / devfs 1 1 0 100% 0 0 100% /dev /dev/da0s1d 988398 60 909268 0% 25 141285 0% /tmp /dev/da0s1f 940800816 508178166 357358586 59% 15881895 105717079 13% /usr /dev/da0s1e 988398 452108 457220 50% 2506 138804 2% /var >How-To-Repeat: Create many directories/files on the file system. >Fix: >Release-Note: >Audit-Trail: >Unformatted: