From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 19:36:28 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 148E416A4B3 for ; Tue, 30 Sep 2003 19:36:28 -0700 (PDT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4710543FF2 for ; Tue, 30 Sep 2003 19:36:23 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.9p2/8.12.9) with ESMTP id h912Zl7R010485; Tue, 30 Sep 2003 22:35:47 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)h912Zldq010482; Tue, 30 Sep 2003 22:35:47 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 30 Sep 2003 22:35:46 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: "Michael W. Oliver" In-Reply-To: <200309292126.36578.michael@gargantuan.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-stable@freebsd.org Subject: Re: panics on 24 hour boundaries X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Robert Watson List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2003 02:36:28 -0000 On Mon, 29 Sep 2003, Michael W. Oliver wrote: > Hi stable, nice you see you again. I was one of those guys who was > seeing constand panics on 24 hour boundaries but couldn't provide a > backtrace due to the ar device not taking a dump. I installed a > dedicated drive just to take the dump, and then didn't have a panic for > a couple weeks. Now, I am back with, and I have traces to share. > > The first two, from 2003-09-27 and 2003-09-28 are from an older > kernel... the same one that was working great for a couple weeks. Due > to some severe storms around here, I shut the maching down voluntarily, > and once rebooted, it started the 24 hour panic crap again. > > So, since I didn't have anything much to lose, I cvsuped my src and > rebuilt and installed world and kernel on the evening of 2003-09-28. Lo > and behold, I had another 24 hour panic. The trace for this one is > phoenix_2003-09-29.txt. The full uname from this latest panic is: > > FreeBSD phoenix.gargantuan.com 4.9-PRERELEASE FreeBSD 4.9-PRERELEASE #1: > Sun Sep 28 18:16:30 EDT 2003 > root@phoenix.gargantuan.com:/usr/obj/usr/src/sys/PHOENIX i386 > > If this information isn't sufficient, please let me know what else I can > do to help troubleshoot this problem. Thanks in advance for your help, > it is much appreciated! Initial reactions: panics on 24 hour boundaries are, in my experience, often associated with the daily event. Once a day, the daily scripts run find several times on your file systems, causing every file and directory to be inspected for changes in setuid scripts, etc. This can trigger certain classes of race conditions and resource limits that you might otherwise not hit in normal operation -- and conviently, they run 24 hours apart :-). To try and confirm this suspicion, it would be interesting to know what time of day exactly the panics take place, and whether you can reproduce the panic by manually running the daily or security script. Also, out of curiosity: since you're experiencing crashes, I assume fsck has run on all your file systems. If not, you might want to boot to single user and run fsck on each file system manually to make sure there's no on-disk corruption of UFS meta-data. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories [NON-Text Body part not included]