From owner-freebsd-stable@FreeBSD.ORG Fri Jan 27 03:09:08 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A14541065673 for ; Fri, 27 Jan 2012 03:09:08 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 623EA8FC13 for ; Fri, 27 Jan 2012 03:09:08 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta07.westchester.pa.mail.comcast.net with comcast id ST831i0020bG4ec57T98Ww; Fri, 27 Jan 2012 03:09:08 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.westchester.pa.mail.comcast.net with comcast id ST971i00y1t3BNj3PT98Df; Fri, 27 Jan 2012 03:09:08 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5B37C102C19; Thu, 26 Jan 2012 19:09:06 -0800 (PST) Date: Thu, 26 Jan 2012 19:09:06 -0800 From: Jeremy Chadwick To: Gary Palmer Message-ID: <20120127030906.GA67449@icarus.home.lan> References: <20120127024815.GD17973@in-addr.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120127024815.GD17973@in-addr.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Panic on 7.4-RELEASE-p5 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jan 2012 03:09:08 -0000 On Thu, Jan 26, 2012 at 09:48:15PM -0500, Gary Palmer wrote: > Hi, > > My Soekris firewall just panic'd > > Fatal trap 12: page fault while in kernel mode > fault virtual address = 0x14001d > fault code = supervisor write, page not present > instruction pointer = 0x20:0xc06fd2d3 > stack pointer = 0x28:0xd63557bc > frame pointer = 0x28:0xd63558a4 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 1869 (tcpdump) > trap number = 12 > panic: page fault > KDB: stack backtrace: > db_trace_self_wrapper(c079bbfc,c07fb800,c078d680,d6355660,d6355660,...) at db_tr > ace_self_wrapper+0x26 > panic(c078d680,c07b829c,c36d6d24,1,1,...) at panic+0xed > trap_fatal(c2c352d0,140000,2,8,d63556c8,...) at trap_fatal+0x234 > trap_pfault(c2b13620,0,c368f460,4,c36d6b00,...) at trap_pfault+0x27a > trap(d635577c) at trap+0x34e > calltrap() at calltrap+0x6 > --- trap 0xc, eip = 0xc06fd2d3, esp = 0xd63557bc, ebp = 0xd63558a4 --- > softdep_disk_io_initiation(c2ade474,1000,c3003738,1,c2aab9b8,...) at softdep_dis > k_io_initiation+0xb3 > ffs_geom_strategy(c3003738,c2aab9b8,1000,edb000,0,...) at ffs_geom_strategy+0x10 > c > ufs_strategy(d6355910,d6355910,c07f3980,c378bbdc,c2aab9b8,...) at ufs_strategy+0 > x64 > bufstrategy(c378bc9c,c2aab9b8,c368f460,c2aab9b8,c2bd46b4,...) at bufstrategy+0x2 > e > bufwrite(c2aab9b8,c2aabb04,20,c368f460,0,...) at bufwrite+0xf4 > cluster_wbuild(c378bbdc,4000,3b7,0,8,...) at cluster_wbuild+0x6c9 > cluster_write(c378bbdc,c2bd46b4,edc000,0,7f,...) at cluster_write+0x715 > ffs_write(d6355bc0,c07091e4,c378bc34,0,c378bc64,...) at ffs_write+0x837 > VOP_WRITE_APV(c07e9500,d6355bc0,c368f460,c07a2aec,252,...) at VOP_WRITE_APV+0xa0 > vn_write(c2e2adf4,d6355c54,c36d7500,0,c368f460,...) at vn_write+0x26f > dofilewrite(d6355c54,ffffffff,ffffffff,0,c2e2adf4,...) at dofilewrite+0x84 > kern_writev(c368f460,4,d6355c54,d6355c74,1,...) at kern_writev+0x58 > write(c368f460,d6355cf8,c,c0798e1b,39bfbd8f,...) at write+0x50 > syscall(d6355d38) at syscall+0x1b9 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (4, FreeBSD ELF32, write), eip = 0x28354663, esp = 0xbfbfe85c, ebp = > 0xbfbfe878 --- > Uptime: 11d5h5m49s > Physical memory: 503 MB > Dumping 112 MB: 97 81 65 49 33 17 1 > Dump complete > Automatic reboot in 1 seconds - press a key on the console to abort > > fsck prevented the box from coming up automatically > > /dev/ufs/varlog: PARTIALLY TRUNCATED INODE I=1496123 > /dev/ufs/varlog: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. > THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY: > ufs: /dev/ufs/varlog (/var/log) > > Anyone seen this before? Any clues? Please shed some light as to your filesystem and disk setup. Partitioning details, any GEOM layers you use, you get the idea. The more verbose the better. Please be sure to state which filesystems use softupdates and which do not (this matters a lot in this situation). As for the actual crash -- I assume the filesystem in question which induced this has softupdates. Have you booted this box into single-user and issued a manual fsck on all the filesystems? If not, please do so ASAP and provide details of the results. These are confirmed cases of background_fsck not properly catching/dealing with all filesystem errors, thus letting some transparently pass through. You are running 7.4-p2, which makes this likelihood even more probable. This is why I advocate (and have for years) background_fsck="no" in rc.conf. I can dig up the (very long and semi-heated) thread discussing this if you want, but I probably won't get to it until next week. I have too much going on with job-related things outside of FreeBSD to spend a lot of time with it now. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |