From owner-freebsd-current@freebsd.org Tue Jul 21 12:57:37 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 280779A7AB0 for ; Tue, 21 Jul 2015 12:57:37 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0E2E81117 for ; Tue, 21 Jul 2015 12:57:37 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0A9669A7AAF; Tue, 21 Jul 2015 12:57:37 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5CD19A7AAE for ; Tue, 21 Jul 2015 12:57:36 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AFE911116 for ; Tue, 21 Jul 2015 12:57:35 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id t6LCvYdB025337 for ; Tue, 21 Jul 2015 05:57:34 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id t6LCvYJa025336 for current@freebsd.org; Tue, 21 Jul 2015 05:57:34 -0700 (PDT) (envelope-from david) Date: Tue, 21 Jul 2015 05:57:34 -0700 From: David Wolfskill To: current@freebsd.org Subject: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines Message-ID: <20150721125734.GF1217@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="lIdIj/ncdD+VcXt4" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2015 12:57:37 -0000 --lIdIj/ncdD+VcXt4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 "successes" out of 4 tries); here's a bit =66rom the core.txt file: freebeast.catwhisker.org dumped core - see /var/crash/vmcore.1 Tue Jul 21 05:36:11 PDT 2015 FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1787 r= 285741M/285741:1100077: Tue Jul 21 04:48:37 PDT 2015 root@freebeast.cat= whisker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 panic: witness_warn GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: witness_warn cpuid =3D 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe083b9c0= 860 vpanic() at vpanic+0x189/frame 0xfffffe083b9c08e0 kassert_panic() at kassert_panic+0x132/frame 0xfffffe083b9c0950 witness_warn() at witness_warn+0x498/frame 0xfffffe083b9c0a20 ithread_loop() at ithread_loop+0x165/frame 0xfffffe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083b9c0ab0 --- trap 0, rip =3D 0, rsp =3D 0xfffffe083b9c0b70, rbp =3D 0 --- =2E.. (kgdb) #0 doadump (textdump=3D0) at pcpu.h:221 #1 0xffffffff80377dfe in db_dump (dummy=3D, dummy2=3D= false,=20 dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:533 #2 0xffffffff80377971 in db_command (cmd_table=3D0x0) at /usr/src/sys/ddb/db_command.c:440 #3 0xffffffff80377604 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #4 0xffffffff8037a19b in db_trap (type=3D, code=3D0) at /usr/src/sys/ddb/db_main.c:251 #5 0xffffffff80a56624 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff80e61bd1 in trap (frame=3D0xfffffe083b9c0790) at /usr/src/sys/amd64/amd64/trap.c:540 #7 0xffffffff80e41e02 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:235 #8 0xffffffff80a55cfe in kdb_enter (why=3D0xffffffff8136f098 "panic",=20 msg=3D0xffffffff80a5bee0 "UH\211AWAVATSH\203PI\211A\211= H\213\004%\201H\211E\201<%x\201") at cpufunc.h:63 #9 0xffffffff80a19739 in vpanic (fmt=3D,=20 ap=3D) at /usr/src/sys/kern/kern_shutdown.c:737 #10 0xffffffff80a19582 in kassert_panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:634 #11 0xffffffff80a74908 in witness_warn (flags=3D2, lock=3D,=20 fmt=3D0xffffffff81367827 "suspending ithread") at /usr/src/sys/kern/subr_witness.c:1757 #12 0xffffffff809e2985 in ithread_loop (arg=3D0xfffff8000770c820) at /usr/src/sys/kern/kern_intr.c:1345 #13 0xffffffff809df874 in fork_exit ( callout=3D0xffffffff809e2820 , arg=3D0xfffff8000770c820,= =20 frame=3D0xfffffe083b9c0ac0) at /usr/src/sys/kern/kern_fork.c:1006 #14 0xffffffff80e4233e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:610 #15 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb)=20 On boot, it dropped into the debugger; it was on the most recent instantiation that I manually issued a "dump" command from that environment, then rebooted under the previous kernel: FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1786 r= 285715M/285715:1100077: Mon Jul 20 04:22:26 PDT 2015 root@freebeast.cat= whisker.org:/common/S3/obj/usr/src/sys/GENERIC amd64 (And yes, it runs an unmodified GENERIC kernel.) The machine has been deployed only for a couple of months or so, but has been building stable/10 and head daily during that time. Until a couple of weeks ago, it was doing this for both i386 and amd64; since then, I dropped i386 from my home infrastructure, so it's been only amd64. In the stable/10 environment, it also make use of a 3-spindle zraid for running poudriere (to build the ports for my "production" machines), and it's been doing that quite well, also. Only other thing that I think of that's noteworthy is that its boot drive is an SSD (where I have not yet enabled TRIM, as it's a Crucial M500, and I need to be sure we don't try to use the queued TRIM commands on it, as there are reports that queued TRIM commands on the M500 will corrupt data). OK; please see for the dump(-related) files. (It's on a residential ADSL, so it's going to be slow. Sorry; I have a limited amount of bandwidth.) Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --lIdIj/ncdD+VcXt4 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJVrkG+XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7T/MP/0rWaTMUsk4VtlFLNeZZFevz xYMcKZjQOdLAKffXaHx3m4r4gr4LMmVGbL1gSRnUs+WxF/8FQHddJYYvObSwCCOD C1BM5CpmFT5Mx1P/uC+mH2+q/XrNAwsUy85jmyeHgOtwIRtfr0RAQXcF2N8IBfOh x+skG0NZ6q6Gcvssih7RMReA/xKaznnPeFCL6h2mU0FghuXiSaiZpyv4dEWRrtDp RKSz4y0f4o4m9oBzdu0iYizhJnh1v1J60k/jmk56BJjXC+qHm3tNSAoh1KY/66AZ skad8vDBj1VoshiaDhcZnDv7b981fIOvPg67/Djd1cUQMDq+JiAR5+pD1arG3iih fWyfMOn4uQuY4bk9pfnXenaR/tAm/qP36qv3w/L/NC9+QLHzF3r+QG/lr2yxF0Ar /a9oH68aUrRkoU1CPRicCYQ8KLHAaFnFSR0Tpg9k8JCFAzUeNzZGKXmVkpD59jFR 6+GpBVKMEvu7sTa/Nt2+E/aXlreaTrErkM+Xb/IVuR2gyoeeRL91PPTQ8W+/qSPB fdNvOLMjmUDyNRPqFm6uvUFafM6Sfip39HXnJz7aaezXSHT6X4TY1sPU04GHuO0c RN6XmCDs/fwTsQxx3WsSMe9XBAAXZPpVUVoyLKUUHo1dyblWXlI9bI5YauO2om+H mwzGpajSsx+oIRHVNJ6W =ZY89 -----END PGP SIGNATURE----- --lIdIj/ncdD+VcXt4--