From owner-freebsd-current@freebsd.org Wed Nov 14 07:05:26 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D907B110C7A5 for ; Wed, 14 Nov 2018 07:05:25 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B1536A8A4 for ; Wed, 14 Nov 2018 07:05:25 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id BBA39288BE for ; Wed, 14 Nov 2018 07:05:24 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Tue, 13 Nov 2018 23:05:23 -0800 (PST) From: Don Lewis Subject: recent 13.0-CURRENT double panic near start of poudriere run To: freebsd-current@FreeBSD.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-Rspamd-Queue-Id: 5B1536A8A4 X-Spamd-Result: default: False [-106.83 / 200.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ALLOW_DOMAIN_WHITELIST(-100.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[FreeBSD.org]; MX_GOOD(-0.01)[cached: mx1.FreeBSD.org]; NEURAL_HAM_SHORT(-1.00)[-1.000,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-3.72)[ip: (-9.76), ipnet: 96.47.64.0/20(-4.83), asn: 11403(-3.91), country: US(-0.10)]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2018 07:05:26 -0000 I've been doing a bunch of packing building today on my Ryzen box and just got this kernel panic near the start of a poudriere run: 13.0-CURRENT FreeBSD 13.0-CURRENT r340358 GENERIC amd64 [00:00:08] Processing PRIORITY_BOOST [00:00:08] Balancing pool [00:00:08] Recording filesystem state for prepkg... done [00:00:09] Building 315 packages using 16 builders [00:00:09] Starting/Cloning builders [ system paniced and dropped into ddb ] # /usr/local/bin/kgdb /boot/kernel/kernel /var/crash/vmcore.3 GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. done. Unread portion of the kernel message buffer: timeout stopping cpus panic: mutex sched lock 2 recursed at /usr/src/sys/kern/kern_synch.c:390 cpuid = 2 time = 1542176879 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00779645f0 vpanic() at vpanic+0x1a3/frame 0xfffffe0077964650 panic() at panic+0x43/frame 0xfffffe00779646b0 __mtx_assert() at __mtx_assert+0x69/frame 0xfffffe00779646c0 mi_switch() at mi_switch+0x42/frame 0xfffffe00779646f0 critical_exit_preempt() at critical_exit_preempt+0x66/frame 0xfffffe0077964710 lapic_handle_timer() at lapic_handle_timer+0xe4/frame 0xfffffe0077964750 Xtimerint() at Xtimerint+0xae/frame 0xfffffe0077964750 --- interrupt, rip = 0xffffffff80bf7860, rsp = 0xfffffe0077964820, rbp = 0xfffffe0077964840 --- generic_stop_cpus() at generic_stop_cpus+0x170/frame 0xfffffe0077964840 stop_cpus_hard() at stop_cpus_hard+0x22/frame 0xfffffe0077964870 vpanic() at vpanic+0xb0/frame 0xfffffe00779648f0 panic() at panic+0x43/frame 0xfffffe0077964950 mi_switch() at mi_switch+0x1ff/frame 0xfffffe0077964980 critical_exit_preempt() at critical_exit_preempt+0x66/frame 0xfffffe00779649a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0077964a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0077964ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0077964ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic __curthread () at ./machine/pcpu.h:230 230 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD)); (kgdb) bt #0 __curthread () at ./machine/pcpu.h:230 #1 doadump (textdump=-2118704160) at /usr/src/sys/kern/kern_shutdown.c:371 #2 0xffffffff804659bc in db_fncall_generic (addr=, rv=, nargs=, args=) at /usr/src/sys/ddb/db_command.c:609 #3 db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at /usr/src/sys/ddb/db_command.c:657 #4 0xffffffff804654f9 in db_command (last_cmdp=, cmd_table=, dopager=) at /usr/src/sys/ddb/db_command.c:481 #5 0xffffffff80465274 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #6 0xffffffff8046848f in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:252 #7 0xffffffff80be7925 in kdb_trap (type=3, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:692 #8 0xffffffff81074946 in trap (frame=0xfffffe0077964520) at /usr/src/sys/amd64/amd64/trap.c:619 #9 #10 kdb_enter (why=0xffffffff813065f5 "panic", msg=) at /usr/src/sys/kern/subr_kdb.c:479 #11 0xffffffff80b9f1f0 in vpanic (fmt=, ap=0xfffffe0077964690) --Type for more, q to quit, c to continue without paging--c at /usr/src/sys/kern/kern_shutdown.c:866 #12 0xffffffff80b9ef93 in panic (fmt=0xffffffff81e951d8 "\304\271,\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:804 #13 0xffffffff80b7ea79 in __mtx_assert (c=, what=, file=0xfffffe00779644e0 "", line=128) at /usr/src/sys/kern/kern_mutex.c:1064 #14 0xffffffff80baa6d2 in mi_switch (flags=1544, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:390 #15 0xffffffff80ba7726 in critical_exit_preempt () at /usr/src/sys/kern/kern_switch.c:243 #16 0xffffffff811ea434 in critical_exit () at /usr/src/sys/sys/systm.h:266 #17 lapic_handle_timer (frame=0xfffffe0077964760) at /usr/src/sys/x86/x86/local_apic.c:1335 #18 #19 ia32_pause () at ./machine/cpufunc.h:348 #20 generic_stop_cpus (map=..., type=255) at /usr/src/sys/kern/subr_smp.c:280 #21 0xffffffff80bf78f2 in stop_cpus_hard (map=...) at /usr/src/sys/kern/subr_smp.c:308 #22 0xffffffff80b9f0e0 in vpanic (fmt=0xffffffff81210702 "mi_switch: switch in a critical section", ap=0xfffffe0077964930) at /usr/src/sys/kern/kern_shutdown.c:828 #23 0xffffffff80b9ef93 in panic (fmt=0x4400 ) at /usr/src/sys/kern/kern_shutdown.c:804 #24 0xffffffff80baa88f in mi_switch (flags=1544, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:396 #25 0xffffffff80ba7726 in critical_exit_preempt () at /usr/src/sys/kern/kern_switch.c:243 #26 0xffffffff80bcf2a7 in sched_idletd (dummy=) at /usr/src/sys/kern/sched_ule.c:2829 #27 0xffffffff80b5f064 in fork_exit (callout=0xffffffff80bced90 , arg=0x0, frame=0xfffffe0077964ac0) at /usr/src/sys/kern/kern_fork.c:1057 #28 It actually appears to be a double panic, with the original panic being: mi_switch: switch in a critical section