From owner-freebsd-current@freebsd.org Sat Mar 24 05:59:21 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80980F6F8CF for ; Sat, 24 Mar 2018 05:59:21 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0BA3E7AB58 for ; Sat, 24 Mar 2018 05:59:21 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: by mailman.ysv.freebsd.org (Postfix) id C3F39F6F8C4; Sat, 24 Mar 2018 05:59:20 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7710AF6F8C3 for ; Sat, 24 Mar 2018 05:59:20 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from nsstlmta03p.bpe.bigpond.com (nsstlmta03p.bpe.bigpond.com [203.38.21.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "", Issuer "Openwave Messaging Inc." (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 576407AB56 for ; Sat, 24 Mar 2018 05:59:18 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from smtp.telstra.com ([10.10.24.4]) by nsstlfep33p-svc.bpe.nexus.telstra.com.au with ESMTP id <20180324035659.LQXL1063.nsstlfep33p-svc.bpe.nexus.telstra.com.au@smtp.telstra.com> for ; Sat, 24 Mar 2018 14:56:59 +1100 X-RG-Spam: Unknown X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgedtgedrvddvgdejiecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfupfevtfgpvffgnffuvfftteenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtuggfsehmtderredtredvnecuhfhrohhmpeetnhgurhgvficutfgvihhllhihuceorghrvghilhhlhiessghighhpohhnugdrnhgvthdrrghuqeenucffohhmrghinheprggtqdhrrdhnuhenucfkphepuddvgedrudeltddrgedtrddukedvnecurfgrrhgrmhephhgvlhhopegkvghnrdgrtgdqrhdrnhhupdhinhgvthepuddvgedrudeltddrgedtrddukedvpdhmrghilhhfrhhomhepoegrrhgvihhllhihsegsihhgphhonhgurdhnvghtrdgruheqnecu X-RG-VS-CLASS: clean X-Authentication-Info: Submitted using ID areilly@bigpond.net.au Received: from Zen.ac-r.nu (124.190.40.182) by smtp.telstra.com (9.0.019.22-1) (authenticated as areilly@bigpond.net.au) id 5A614436175617AC for current@freebsd.org; Sat, 24 Mar 2018 14:56:59 +1100 Date: Sat, 24 Mar 2018 14:56:53 +1100 From: Andrew Reilly To: current@freebsd.org Subject: 12-Current panics on boot (didn't a week ago.) Message-ID: <20180324035653.GA3411@Zen.ac-r.nu> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="sm4nu43k4a2Rpi4c" Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Mar 2018 05:59:21 -0000 --sm4nu43k4a2Rpi4c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi all, For reasons that still escape me, I haven't been able to get a kernel dump to debug, sorry. Just thought that I'd generate a fairly low-quality report, to see if anyone has some ideas. The last kernel that I have that booted OK (and I'm now running) is: FreeBSD Zen.ac-r.nu 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r331064M: Sat Mar 17 07:54:51 AEDT 2018 root@Zen:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 The machine is a: CPU: AMD Ryzen 7 1700 Eight-Core Processor (2994.46-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 Features=0x178bfbff Kernels built from head as of a couple of hours ago get through launching the other CPUs and then stops somewhere in random, apparently: SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 1497223020 Hz quality 1000 random: entpanic: mtx_lock() of spin mutex (null) @ /usr/src/sys/kern/subr_bus.c:617 cpuid = 0 time = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00004507a0 vpanic() at vpanic+0x18d/frame 0xfffffe0000450800 doadump () at doadump/frame 0xfffffe0000450880 __mtx_lock_flags() at __mtx_lock_flags+0x163/frame 0xfffffe00004508d0 devctl_queue_data_f() at devctl_queue_data_f+0x6a/frame 0xfffffe0000450900 g_dev_taste() at g_dev_taste+0x370/frame 0xfffffe0000450a10 g_new_provider_event() at g_new_provider_event+0xfa/frame 0xfffffe0000450a30 g_run_events() at g_run_events+0x151/frame 0xfffffe0000450a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000450ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000450ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 14 tid 100052 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> dump Cannot dump: no dump device specified. db> Now dumping worked fine the last time the kernel panicked: I have dumpdev=AUTO in rc.conf and I have swap on nvd0p3 (first) and /dev/zvol/root/swap (second, larger than the first.) Root on the nvd0p2 is ZFS, and ther's a four-drive raidZ with user directories and what-not on them, and another ZFS on an external USB drive that I use for backups, unmounted. In the new kernels, we clearly aren't even getting as far as finding the hubs and controllers, let alone the drives. I've attached dmesg.boot from the last boot from last week's good kernel. (While briefly in yoyo mode I turned the SMT back on, so now there are 16 cores instead of the eight mentioned in the crash dump. Didn't help, but I haven't turned it back off yet.) Cheers, Andrew --sm4nu43k4a2Rpi4c Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg.boot" Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #1 r331064M: Sat Mar 17 07:54:51 AEDT 2018 root@Zen:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Ryzen 7 1700 Eight-Core Processor (2994.46-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 Features=0x178bfbff Features2=0x7ed8320b AMD Features=0x2e500800 AMD Features2=0x35c233ff Structured Extended Features=0x209c01a9 XSAVE Features=0xf AMD Extended Feature Extensions ID EBX=0x7 SVM: (disabled in BIOS) NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33272578048 (31731 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 1 package(s) x 2 cache groups x 4 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20180313/tbfadt-796) ioapic0: Changing APIC ID to 17 ioapic1: Changing APIC ID to 18 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-55 on motherboard SMP: AP CPU #12 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #11 Launched! Timecounter "TSC-low" frequency 1497228045 Hz quality 1000 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff8101f5c0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 netmap: loaded module nexus0 vtvga0: on motherboard cryptosoft0: on motherboard aesni0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 350 Event timer "HPET2" frequency 14318180 Hz quality 350 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 amdsmn0: on hostb0 amdtemp0: on hostb0 pci0: at device 0.2 (no driver attached) pcib1: at device 1.1 on pci0 pci1: on pcib1 nvme0: mem 0xfe900000-0xfe903fff irq 24 at device 0.0 on pci1 pcib2: at device 1.3 on pci0 pci2: on pcib2 xhci0: mem 0xfe6a0000-0xfe6a7fff irq 32 at device 0.0 on pci2 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ahci0: mem 0xfe680000-0xfe69ffff irq 33 at device 0.1 on pci2 ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 pcib3: irq 34 at device 0.2 on pci2 pci3: on pcib3 pcib4: irq 32 at device 0.0 on pci3 pci4: on pcib4 pcib5: irq 33 at device 1.0 on pci3 pci5: on pcib5 pcib6: irq 32 at device 4.0 on pci3 pci6: on pcib6 pcib7: irq 33 at device 5.0 on pci3 pci7: on pcib7 pcib8: irq 34 at device 6.0 on pci3 pci8: on pcib8 pci8: at device 0.0 (no driver attached) pcib9: irq 35 at device 7.0 on pci3 pci9: on pcib9 igb0: port 0xf000-0xf01f mem 0xfe400000-0xfe41ffff,0xfe420000-0xfe423fff irq 32 at device 0.0 on pci9 igb0: attach_pre capping queues at 2 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 2 igb0: pxm cpus: 8 queue msgs: 4 admincnt: 1 igb0: using 2 rx queues 2 tx queues igb0: Using MSIX interrupts with 3 vectors igb0: allocated for 2 tx_queues igb0: allocated for 2 rx_queues igb0: Ethernet address: 70:85:c2:59:38:4e igb0: netmap queues/slots: TX 2/1024, RX 2/1024 pcib10: at device 3.1 on pci0 pci10: on pcib10 vgapci0: port 0xe000-0xe0ff mem 0xe0000000-0xefffffff,0xfe800000-0xfe83ffff irq 54 at device 0.0 on pci10 vgapci0: Boot video device hdac0: mem 0xfe860000-0xfe863fff irq 55 at device 0.1 on pci10 hdac0: hdac_get_capabilities: Invalid corb size (0) device_attach: hdac0 attach returned 6 pcib11: at device 7.1 on pci0 pci11: on pcib11 pci11: at device 0.0 (no driver attached) ccp0: mem 0xfe200000-0xfe2fffff,0xfe300000-0xfe301fff irq 36 at device 0.2 on pci11 random: registering fast source AMD CCP TRNG xhci1: mem 0xfe100000-0xfe1fffff irq 37 at device 0.3 on pci11 xhci1: 64 bytes context size, 64-bit DMA usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 pcib12: at device 8.1 on pci0 pci12: on pcib12 pci12: at device 0.0 (no driver attached) ahci1: mem 0xfe708000-0xfe708fff irq 42 at device 0.2 on pci12 ahci1: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS ahcich8: at channel 0 on ahci1 hdac0: mem 0xfe700000-0xfe707fff irq 43 at device 0.3 on pci12 isab0: at device 20.3 on pci0 isa0: on isab0 acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] orm0: at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0 hwpstate0: on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ugen0.1: <0x1022 XHCI root HUB> at usbus0 ugen1.1: <0x1022 XHCI root HUB> at usbus1 uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 nvd0: NVMe namespace nvd0: 228936MB (468862128 512 byte sectors) hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21 and 24,26 on hdaa0 pcm1: at nid 27 and 25 on hdaa0 pcm2: at nid 30 on hdaa0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 3.x device ada0: Serial Number PK1334PEHWZT7S ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 3815447MB (7814037168 512 byte sectors) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA8-ACS SATA 3.x device ada1: Serial Number PK1334PEHZBWXS ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 3815447MB (7814037168 512 byte sectors) ada2 at ahcich4 bus 0 scbus2 target 0 lun 0 ada2: ATA8-ACS SATA 3.x device ada2: Serial Number PK1334PEHYSZ6S ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 3815447MB (7814037168 512 byte sectors) ada3 at ahcich5 bus 0 scbus3 target 0 lun 0 ada3: ATA8-ACS SATA 3.x device ada3: Serial Number PK1334PEHZDA1S ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 3815447MB (7814037168 512 byte sectors) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:root [rw]... Root mount waiting for: usbus1 usbus0 uhub1: 8 ports with 8 removable, self powered uhub0: 22 ports with 22 removable, self powered Root mount waiting for: usbus1 usbus0 ugen0.2: at usbus0 ubt0 on uhub0 ubt0: on usbus0 ugen1.2: at usbus1 umass0 on uhub1 umass0: on usbus1 umass0: SCSI over Bulk-Only; quirks = 0x0100 umass0:5:0: Attached to scbus5 da0 at umass-sim0 bus 0 scbus5 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number 1153000117AD da0: 400.000MB/s transfers da0: 5723166MB (1465130646 4096 byte sectors) da0: quirks=0x2 Root mount waiting for: usbus0 ugen0.3: at usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 Link state changed to up igb0: link state changed to UP --sm4nu43k4a2Rpi4c--