From owner-freebsd-stable@FreeBSD.ORG Sun Jan 12 21:10:40 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B0BB3A0A for ; Sun, 12 Jan 2014 21:10:40 +0000 (UTC) Received: from nm25-vm2.bullet.mail.ne1.yahoo.com (nm25-vm2.bullet.mail.ne1.yahoo.com [98.138.91.213]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 71E571120 for ; Sun, 12 Jan 2014 21:10:40 +0000 (UTC) Received: from [98.138.100.115] by nm25.bullet.mail.ne1.yahoo.com with NNFMP; 12 Jan 2014 21:10:38 -0000 Received: from [98.138.226.124] by tm106.bullet.mail.ne1.yahoo.com with NNFMP; 12 Jan 2014 21:10:38 -0000 Received: from [127.0.0.1] by smtp203.mail.ne1.yahoo.com with NNFMP; 12 Jan 2014 21:10:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1389561038; bh=TNagU6iXr9rEQB3ZX9tL4Aj7RR/is/l0cV4gtKuye7k=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding; b=t5ilIl2AibcZ4PSABFanoQhIUKBRFXNInto3dp7LWGtPtMpAdpMcnfYq+33rxPs9/Fpv7+Fk0xCZ6mv1g34fR7/n0iocBXdmPHVrSi74WBILkA1qS9uyBCyktVbYVYc+bdbakV8xZ2y8Ly/EjwBvt0HaaT89EVWBQWg6wlpC20M= X-Yahoo-Newman-Id: 744642.10380.bm@smtp203.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 3LeY7kAVM1mC0J4lfbNQda1oIffmNhqWn.J7fOhiaxSYVJ5 OYaSJrUBO5tTwpIll9c1wj.vrkuwNl3SNwr.scAYMxc9KjZ6YRdnbEhyHJP2 yZ1IC3Wv8QkYIh4c8dja8pJ_7Vp6VG8x5BCfqYj9omGxV1kPpyQ48UWA5bDQ _cbzireLHa1xKOg7KWepp5yqQa3sv2t265d9zRB2qRSe_7XlwzL9E2EV_9tU dgfKx1lwsX89TC9QCF7NPD5K4IbRxLG1AnRBdF2YokKAESCaXLpFB226EJ9l x6m1PtTK24I89f59wpazEYV4NEKbFEwrPkAt4ZxFPSMVLIxZ00Zh5zLP20Id gzJHhPGGrBWZ15JzEbL9XtY9sb37IJHru69x3GYFwq7DmeoVYLqPkQ4OjwHw xY7nRMi3RhDTyXCCQ_BqWkPXkyJJCRJP0Z5WDnZd_QjGYU1uQhBIVkkCF7F0 P2xudYQEp4USnZPZtgn.RqDvTbwChwX5dv1_rH38TeofkAj2XsujSNPH8mAU OcPPpsNv_znVirgYUxRYPjBySzXFRVJso1l2PbVKh610K15tMHaKXHH64i1q .0vCnmNjCH0YhDJ6iyKRwjvQUbxrnRouFkww8.7T8t4z5B5X6YBRtmxJnqor wFlWW9XG6vfK_uOdBMtdpvQr09XnBl3ULrLVc_LCq7Pan82fXm_UokE1nSQe 9ibiZBkeXRTfNFGaQrJZnaXS72OEIRMVwbZM- X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.100.138] (sean_bruno@24.23.220.111 with plain [63.250.193.228]) by smtp203.mail.ne1.yahoo.com with SMTP; 12 Jan 2014 13:10:38 -0800 PST Subject: Re: panic: stable/10 with Debugging enabled in kern_cons.c:500 From: Sean Bruno To: Konstantin Belousov In-Reply-To: <20140112034203.GA59496@kib.kiev.ua> References: <1389478327.46758.7.camel@powernoodle.corp.yahoo.com> <1389480297.46758.9.camel@powernoodle.corp.yahoo.com> <20140112034203.GA59496@kib.kiev.ua> Content-Type: text/plain; charset="us-ascii" Date: Sun, 12 Jan 2014 13:10:37 -0800 Message-ID: <1389561037.1395.10.camel@powernoodle.corp.yahoo.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: sbruno@freebsd.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jan 2014 21:10:40 -0000 On Sun, 2014-01-12 at 05:42 +0200, Konstantin Belousov wrote: > On Sat, Jan 11, 2014 at 02:44:57PM -0800, Sean Bruno wrote: > > On Sat, 2014-01-11 at 14:12 -0800, Sean Bruno wrote: > > > I can't imagine that I'm the first person to run with the following > > > debug options enabled: > > > > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > > > > > > But, I can't get stable/10 to post on a plain old supermicro box with > > > those debugging symbols enabled. > > > > > > > > > random: unblocking device. > > > panic: mtx_lock_spin: recursed on non-recursive mutex cnputs_mtx > > > @ /usr/src/sys/kern/kern_cons.c:500 > > > > > > cpuid = 0 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > > 0xfffffe1764e681b0 > > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1764e68260 > > > vpanic() at vpanic+0x126/frame 0xfffffe1764e682a0 > > > kassert_panic() at kassert_panic+0x136/frame 0xfffffe1764e68310 > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x166/frame > > > 0xfffffe1764e68350 > > > cnputs() at cnputs+0x32/frame 0xfffffe1764e68370 > > > putchar() at putchar+0x13a/frame 0xfffffe1764e683f0 > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e684f0 > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e685c0 > > > printf() at printf+0x43/frame 0xfffffe1764e68620 > > > witness_checkorder() at witness_checkorder+0xa99/frame > > > 0xfffffe1764e686b0 > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame > > > 0xfffffe1764e686f0 > > > uart_cnputc() at uart_cnputc+0x3b/frame 0xfffffe1764e68710 > > > cnputc() at cnputc+0x7f/frame 0xfffffe1764e68740 > > > cnputs() at cnputs+0x58/frame 0xfffffe1764e68760 > > > putchar() at putchar+0x13a/frame 0xfffffe1764e687e0 > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e688e0 > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e689b0 > > > printf() at printf+0x43/frame 0xfffffe1764e68a10 > > > witness_checkorder() at witness_checkorder+0xa99/frame > > > 0xfffffe1764e68aa0 > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame > > > 0xfffffe1764e68ae0 > > > msleep_spin_sbt() at msleep_spin_sbt+0x90/frame 0xfffffe1764e68b70 > > > random_kthread() at random_kthread+0x1d0/frame 0xfffffe1764e68bb0 > > > fork_exit() at fork_exit+0x84/frame 0xfffffe1764e68bf0 > > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1764e68bf0 > > > --- trap 0, rip = 0, rsp = 0xfffffe1764e68cb0, rbp = 0 --- > > > KDB: enter: panic > > > [ thread pid 14 tid 100057 ] > > > Stopped at kdb_enter+0x3e: movq $0,kdb_why > > > > > > Witness induced. Removed the witness option and this doesn't happen > > (obviously). Can I get someone familiar with console code to look into > > this? Kind of crappy when I'm investigating a resource starvation > > issue. > > Add > options WITNESS_SKIPSPIN > to the kernel config. This is required now, unfortunately. Confirmed. Thank you. It looks like the machine becomes seriously unuseable with this much debugging on. After a few hours of testing. I cannot run "svn up" any more and the system seems sluggish. root@redbuild03:/usr/src # svn up -r 260075 Updating '.': load: 2.06 cmd: svn 12480 [*kmem arena] 624.87r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 625.71r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 625.91r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.09r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.27r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.45r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.61r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.79r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 626.95r 0.35u 434.25s 0% 11672k load: 2.06 cmd: svn 12480 [*kmem arena] 627.13r 0.35u 434.25s 0% 11672k Is this a symptom of the level of debugging or am I starting to come close to a problem that needs investigation? sean