From owner-freebsd-stable@FreeBSD.ORG Thu Aug 18 00:16:41 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D2A9106564A; Thu, 18 Aug 2011 00:16:41 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper-int.allbsd.org [IPv6:2001:2f0:104:e002::2]) by mx1.freebsd.org (Postfix) with ESMTP id 716368FC08; Thu, 18 Aug 2011 00:16:40 +0000 (UTC) Received: from alph.allbsd.org (p3028-ipbf608funabasi.chiba.ocn.ne.jp [125.175.94.28]) (authenticated bits=128) by mail.allbsd.org (8.14.4/8.14.4) with ESMTP id p7I0GI0T059114; Thu, 18 Aug 2011 09:16:28 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [IPv6:::1]) (authenticated bits=0) by alph.allbsd.org (8.14.4/8.14.4) with ESMTP id p7I0GDqV044396; Thu, 18 Aug 2011 09:16:15 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Thu, 18 Aug 2011 09:16:00 +0900 (JST) Message-Id: <20110818.091600.831954331552558249.hrs@allbsd.org> To: attilio@FreeBSD.org From: Hiroki Sato In-Reply-To: <20110818.043332.27079545013461535.hrs@allbsd.org> References: <20110818.023832.373949045518579359.hrs@allbsd.org> <20110818.043332.27079545013461535.hrs@allbsd.org> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.3 on Emacs 23.1 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: clamav-milter 0.97 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.allbsd.org [133.31.130.32]); Thu, 18 Aug 2011 09:16:33 +0900 (JST) X-Spam-Status: No, score=-102.2 required=13.0 tests=BAYES_00, CONTENT_TYPE_PRESENT,DIRECTOCNDYN,MIMEQENC,QENCPTR2,RCVD_IN_RP_RNBL, SPF_SOFTFAIL,USER_IN_WHITELIST autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on gatekeeper.allbsd.org Cc: kostikbel@gmail.com, freebsd-stable@FreeBSD.org, avg@FreeBSD.org Subject: Re: panic: spin lock held too long (RELENG_8 from today) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 00:16:41 -0000 Hiroki Sato wrote in <20110818.043332.27079545013461535.hrs@allbsd.org>: hr> Attilio Rao wrote hr> in : hr> = hr> at> 2011/8/17 Hiroki Sato : hr> at> > Hi, hr> at> > hr> at> > Mike Tancsa wrote hr> at> > =A0in <4E15A08C.6090407@sentex.net>: hr> at> > hr> at> > mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: hr> at> > mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: hr> at> > mi> >> hr> at> > mi> >> BTW, we had a similar panic, "spinlock held too long",= the spinlock hr> at> > mi> >> is the sched lock N, on busy 8-core box recently upgra= ded to the hr> at> > mi> >> stable/8. Unfortunately, machine hung dumping core, so= the stack trace hr> at> > mi> >> for the owner thread was not available. hr> at> > mi> >> hr> at> > mi> >> I was unable to make any conclusion from the data that= was present. hr> at> > mi> >> If the situation is reproducable, you coulld try to re= vert r221937. This hr> at> > mi> >> is pure speculation, though. hr> at> > mi> > hr> at> > mi> > Another crash just now after 5hrs uptime. I will try an= d revert r221937 hr> at> > mi> > unless there is any extra debugging you want me to add = to the kernel hr> at> > mi> > instead =A0? hr> at> > hr> at> > =A0I am also suffering from a reproducible panic on an 8-STAB= LE box, an hr> at> > =A0NFS server with heavy I/O load. =A0I could not get a kerne= l dump hr> at> > =A0because this panic locked up the machine just after it occ= urred, but hr> at> > =A0according to the stack trace it was the same as posted one= .= hr> at> > =A0Switching to an 8.2R kernel can prevent this panic. hr> at> > hr> at> > =A0Any progress on the investigation? hr> at> = hr> at> Hiroki, hr> at> how easilly can you reproduce it? hr> = hr> It takes 5-10 hours. I installed another kernel for debugging jus= t hr> now, so I think I will be able to collect more detail information = in hr> a couple of days. hr> = hr> at> It would be important to have a DDB textdump with these informa= tions: hr> at> - bt hr> at> - ps hr> at> - show allpcpu hr> at> - alltrace hr> at> = hr> at> Alternatively, a coredump which has the stop cpu patch which An= dryi can provide. hr> = hr> Okay, I will post them once I can get another panic. Thanks! I got the panic with a crash dump this time. The result of bt, ps, allpcpu, and traces can be found at the following URL: http://people.allbsd.org/~hrs/FreeBSD/pool-panic_20110818-1.txt -- Hiroki