From owner-freebsd-stable@FreeBSD.ORG Wed Nov 26 19:32:04 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 173141065672 for ; Wed, 26 Nov 2008 19:32:04 +0000 (UTC) (envelope-from lambert@lambertfam.org) Received: from sysmon.tcworks.net (sysmon.tcworks.net [65.66.76.4]) by mx1.freebsd.org (Postfix) with ESMTP id D24778FC1C for ; Wed, 26 Nov 2008 19:32:03 +0000 (UTC) (envelope-from lambert@lambertfam.org) Received: from sysmon.tcworks.net (localhost [127.0.0.1]) by sysmon.tcworks.net (8.13.1/8.13.1) with ESMTP id mAQJW2oO095824 for ; Wed, 26 Nov 2008 13:32:02 -0600 (CST) (envelope-from lambert@lambertfam.org) Received: (from lambert@localhost) by sysmon.tcworks.net (8.13.1/8.13.1/Submit) id mAQJW2S6095823 for freebsd-stable@freebsd.org; Wed, 26 Nov 2008 13:32:02 -0600 (CST) (envelope-from lambert@lambertfam.org) X-Authentication-Warning: sysmon.tcworks.net: lambert set sender to lambert@lambertfam.org using -f Date: Wed, 26 Nov 2008 13:32:02 -0600 From: Scott Lambert To: FreeBSD-STABLE Message-ID: <20081126193202.GB22462@sysmon.tcworks.net> Mail-Followup-To: FreeBSD-STABLE References: <20081125043124.GA18518@sysmon.tcworks.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081125043124.GA18518@sysmon.tcworks.net> User-Agent: Mutt/1.4.2.2i Subject: Re: FreeBSD 7.0-STABLE Jul 23: panic: ffs_blkfree: freeing free frag X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 19:32:04 -0000 On Mon, Nov 24, 2008 at 10:31:24PM -0600, Scott Lambert wrote: > I have a box I am using for hosting jailed web servers. I did a > test move of a jail from a FreeBSD 6 box to the FreeBSD 7 server, > web1.hosting. It took forever, 30 minutes to be exact, to create the > jail with the 3GB image file and restore the data from the FreeBSD 6 box > into it. > > I created a test archive of the running jail with ezjail-admin on the > FreeBSD 6 box and scp'd it to web1.hosting. That took 5 minutes. (I > was timing all of this to estimate how long it would really take later.) > > Once the archive was on web1.hosting, I created the new jail using the > archive to populate it. > > sudo ezjail-admin create -a test_host_tcworks_net-200811241856.40.tar.gz \ > -s 3G -i testhost.tcworks.net 192.168.1.238 > > That step took 40 minutes. According to 'systat -vm 1', da0 tended to > show around 90% utilization, da1 was about 23% and MB/s was about 1.6 > for both during the creation of the jail. > > After about 20 to 40 minutes of ensuring that the jail was working > properly with the compat6x libs, I decided to erase the test jail and > get ready for doing the transfer for real during the next maintenance > window. > > Just before the box stopped responding to me, I had run: > sudo ezjail-admin delete -w testhost.tcworks.net > > It might have been about 30 seconds after that I noticed it wasn't > responding. > > FreeBSD web1.hosting.tcworks.net 7.0-STABLE FreeBSD > 7.0-STABLE #1: Wed Jul 23 03:09:31 CDT 2008 > root@web1.hosting.tcworks.net:/usr/obj/usr/src/sys/GENERIC i386 > > I changed the fs = part of this line to avoid privacy issues for my > customer whose web host jail I've been working with. I just changed the > hostname: > > dev = md14, block = 1320903, fs = /home/ezjails/testhost.tcworks.net > > 21:23:17 Mon Nov 24 # kgdb /boot/kernel/kernel vmcore.0 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > dev = md14, block = 1320903, fs = /home/ezjails/testhost.tcworks.net > panic: ffs_blkfree: freeing free frag > cpuid = 0 > Uptime: 124d14h34m47s > Physical memory: 1011 MB > Dumping 199 MB: 184 168 152 136 120 104 88 72 56 40 24 8 > > #0 doadump () at pcpu.h:195 > 195 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:195 > #1 0xc077fd27 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 > #2 0xc077ffe9 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:572 > #3 0xc09667a3 in ffs_blkfree (ump=0xc865d400, fs=0xc4946800, > devvp=0xc7dc4880, bno=1320903, size=2048, inum=336842) > at /usr/src/sys/ufs/ffs/ffs_alloc.c:1918 > #4 0xc097a1af in handle_workitem_freeblocks (freeblks=0xc5aab600, flags=0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:2764 > #5 0xc097b1b8 in process_worklist_item (mp=0xc4859598, flags=Variable "flags" is not available. > ) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:962 > #6 0xc097cc62 in softdep_process_worklist (mp=0xc4859598, full=0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:845 > #7 0xc097f687 in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:756 > #8 0xc075d479 in fork_exit (callout=0xc097f210 , arg=0x0, > frame=0xe44edd38) at /usr/src/sys/kern/kern_fork.c:781 > #9 0xc0a6e170 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 > (kgdb) My non-developer eye thinks this looks like some of the other ffs_blkfree panics in the PR database which tend to involve deleteing large amounts of data in FreeBSD 5 and 6 and 7-CURRENT. http://www.freebsd.org/cgi/query-pr-summary.cgi?text=ffs_blkfree Some refer to "freeing free block", some to "freeing free frag". Would disabling soft-updates be a good temporary workaround? I can't afford to attempt to reproduce the problem, on this box, until I get some production stuff shifted. I am going to try on a different box, but everything about that box is different except that it uses a gmirror. Thanks for your help and Happy Thanksgiving, for those of you who celebrate it! -- Scott Lambert KC5MLE Unix SysAdmin lambert@lambertfam.org