From owner-freebsd-stable@FreeBSD.ORG Mon Jun 8 22:50:30 2009 Return-Path: Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1FC85106566B; Mon, 8 Jun 2009 22:50:30 +0000 (UTC) (envelope-from nakaji@kankyo-u.ac.jp) Received: from www.heimat.gr.jp (unknown [IPv6:2001:3e0:a84::1]) by mx1.freebsd.org (Postfix) with ESMTP id D9A1B8FC18; Mon, 8 Jun 2009 22:50:29 +0000 (UTC) (envelope-from nakaji@kankyo-u.ac.jp) X-Virus-Scanned: amavisd-new at heimat.gr.jp Received: from ra333.heimat.gr.jp.kankyo-u.ac.jp (ra333.heimat.gr.jp [IPv6:2001:3e0:a84:0:200:4cff:fe17:573c]) by www.heimat.gr.jp (8.14.3/8.14.3) with ESMTP id n58MoG99004928; Tue, 9 Jun 2009 07:50:17 +0900 (JST) (envelope-from nakaji@kankyo-u.ac.jp) From: NAKAJI Hiroyuki To: Attilio Rao References: <86d49h300c.fsf@ra333.heimat.gr.jp> <3bbf2fe10906060749xbbc2f2fy4c09f67711a63b@mail.gmail.com> Date: Tue, 09 Jun 2009 07:50:15 +0900 In-Reply-To: <3bbf2fe10906060749xbbc2f2fy4c09f67711a63b@mail.gmail.com> (Attilio Rao's message of "Sat, 6 Jun 2009 16:49:54 +0200") Message-ID: <863aaagx20.fsf@ra333.heimat.gr.jp> User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.0.94 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Status: No, score=-5.9 required=13.0 tests=BAYES_00, CONTENT_TYPE_PRESENT,NO_RELAYS,QENCPTR1 autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on www.heimat.gr.jp Cc: freebsd-stable@FreeBSD.ORG Subject: Re: Big problem still remains with 7.2-STABLE locking up X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2009 22:50:30 -0000 I got a lockup at 3 a.m. JST, but because I'm not ready for dcons I cannot show you guys the whole ddb session. I put a 'bt' output of kgdb. http://www.heimat.gr.jp/localhost/kgdbbtvmcore.0 Kernel config: include GENERIC ident HEIMAT options MSGBUF_SIZE=81920 makeoptions DEBUG=-g options KDB options KDB_TRACE options KDB_UNATTENDED options DDB options BREAK_TO_DEBUGGER options QUOTA options DEVICE_POLLING options HZ=1000 options SW_WATCHDOG options DEBUG_VFS_LOCKS options INVARIANTS options INVARIANT_SUPPORT options WITNESS Thanks. P.S. "allthreads" was not a usable command in my RELENG_7's ddb. >>>>> In <3bbf2fe10906060749xbbc2f2fy4c09f67711a63b@mail.gmail.com> >>>>> Attilio Rao wrote: > Anyways, the only one way we have to debug this is getting some help > by the user. > 1) Drop the option WITNESS_SPIKSPIN (as we would like to debug > spinlocks too) and LOCK_PROFILING (in order to create higher > contention and kill some barriers) > 2) Once you get the deadlock break in the DDB debugger > 3) Once you are in DDB informations which could be very useful are: db> show allpcpu db> show alllocks db> show lockedvnods db> ps db> allthreads > Note that this is a lot of printout so you won't be able of collecting > all these informations if not with a serial connection. > 4) Dump the content so that we can further look at locks structure > states once we identify something useful (ideally, keeping the machine > up in DDB for that would be very useful, but often not viable) > Let me know. > Attilio -- NAKAJI Hiroyuki