From owner-freebsd-fs@FreeBSD.ORG Sun Mar 30 18:43:51 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD3717D2 for ; Sun, 30 Mar 2014 18:43:51 +0000 (UTC) Received: from mail-yh0-x22f.google.com (mail-yh0-x22f.google.com [IPv6:2607:f8b0:4002:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 95081D2B for ; Sun, 30 Mar 2014 18:43:51 +0000 (UTC) Received: by mail-yh0-f47.google.com with SMTP id 29so6804276yhl.6 for ; Sun, 30 Mar 2014 11:43:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=zIhWYLa4xUJUS3RnWC6rF+CZJXL7gUqZTEVIVyA3yag=; b=i/M2uWC+D+hJsCoh06MipIaFV47OKzf8ReJ1vjd6fhCmqg4tPL5cCGN+1Wm1+wji7d hcTxv5HBCsAgQqjoS1FEFP6Wmo4kYViyI/1O6ML+GJKXx0e0bKS5x5FDf1rQ2wwSa/AA Izbp2cpmOhovsHddBQXZWcDvxYLXLGHeRiUDdoQNwxFDJ2XbEfqhuB6ZD+5GvCf/WDG6 vwZ755DzeaLNZrAbTnzVYP39Nc0OPjx0ENxC0a4rStC11IaftGcroaj+ciadsUPJC87j e1Uxlpn8c58ZnyHhKzGT6Xw1DplXGL1Xcm8bn38dqfZeysB3l9zVatazLWOaJQ/Uu+zj An+g== MIME-Version: 1.0 X-Received: by 10.236.90.225 with SMTP id e61mr29343782yhf.15.1396205030779; Sun, 30 Mar 2014 11:43:50 -0700 (PDT) Received: by 10.170.95.212 with HTTP; Sun, 30 Mar 2014 11:43:50 -0700 (PDT) Date: Sun, 30 Mar 2014 20:43:50 +0200 Message-ID: Subject: ZFS panic: spin lock held too long From: Idwer Vollering To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2014 18:43:52 -0000 ==== dmesg (from a cold boot) ==== $ dmesg Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: AMD A8-5500 APU with Radeon(tm) HD Graphics (3194.26-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x610f01 Family = 0x15 Model = 0x10 Stepping = 1 Features=0x178bfbff Features2=0x3e98320b AMD Features=0x2e500800 AMD Features2=0x1ebbfff Standard Extended Features=0x8 TSC: P-state invariant, performance statistics real memory = 9110028288 (8688 MB) avail memory = 7711780864 (7354 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 16 cpu1 (AP): APIC ID: 17 cpu2 (AP): APIC ID: 18 cpu3 (AP): APIC ID: 19 ioapic0 irqs 0-23 on motherboard random: initialized kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 pcib0: port 0xcf8-0xcff iomem 0xa0000-0xbffff,0xc0000-0xdffff,0xe0000000-0xffffffff on acpi0 pci0: on pcib0 vgapci0: port 0x2000-0x20ff mem 0xe0000000-0xefffffff,0xf0100000-0xf013ffff at device 1.0 on pci0 vgapci0: Boot video device hdac0: mem 0xf0140000-0xf0143fff at device 1.1 on pci0 hdac0: hdac_get_capabilities: Invalid corb size (0) device_attach: hdac0 attach returned 6 xhci0: mem 0xf0148000-0xf0149fff at device 16.0 on pci0 xhci0: 32 byte context size. usbus0 on xhci0 xhci1: mem 0xf014a000-0xf014bfff at device 16.1 on pci0 xhci1: 32 byte context size. usbus1 on xhci1 ahci0: port 0x2410-0x2417,0x2420-0x2423,0x2418-0x241f,0x2424-0x2427,0x2400-0x240f mem 0xf014f000-0xf014f7ff at device 17.0 on pci0 ahci0: AHCI v1.30 with 8 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ohci0: mem 0xf014c000-0xf014cfff at device 18.0 on pci0 usbus2 on ohci0 ehci0: mem 0xf014f800-0xf014f8ff at device 18.2 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 ohci1: mem 0xf014d000-0xf014dfff at device 19.0 on pci0 usbus4 on ohci1 ehci1: mem 0xf014f900-0xf014f9ff at device 19.2 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci1 pci0: at device 20.0 (no driver attached) hdac0: mem 0xf0144000-0xf0147fff at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib1: at device 20.4 on pci0 pci1: on pcib1 ohci2: mem 0xf014e000-0xf014efff at device 20.5 on pci0 usbus6 on ohci2 sdhci_pci0: mem 0xf014fa00-0xf014faff at device 20.7 on pci0 sdhci_pci0: 1 slot(s) allocated pcib2: at device 21.0 on pci0 pci2: on pcib2 pcib3: at device 21.1 on pci0 pcib3: failed to allocate initial I/O port window: 0x1000-0x1fff pci3: on pcib3 re0: mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff at device 0.0 on pci3 re0: Using 1 MSI-X message re0: turning off MSI enable bit. re0: Chip rev. 0x48000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: acpi_button0: on acpi0 orm0: at iomem 0xed800-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 acpi_throttle0: on cpu0 hwpstate0: on cpu0 acpi_throttle1: on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 acpi_throttle2: on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 acpi_throttle3: on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21,23 and 24,26 on hdaa0 pcm1: at nid 27 and 25 on hdaa0 pcm2: at nid 30 on hdaa0 pcm3: at nid 17 on hdaa0 random: unblocking device. usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 5.0Gbps Super Speed USB v3.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 usbus6: 12Mbps Full Speed USB v1.0 ugen0.1: <0x1022> at usbus0 uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen6.1: at usbus6 uhub1: on usbus6 ugen5.1: at usbus5 uhub2: on usbus5 ugen4.1: at usbus4 uhub3: on usbus4 ugen3.1: at usbus3 uhub4: on usbus3 ugen2.1: at usbus2 uhub5: on usbus2 ugen1.1: <0x1022> at usbus1 uhub6: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: Serial Number ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: Serial Number ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad6 Netvsc initializing... SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 1597128784 Hz quality 1000 uhub1: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 usbus4 usbus3 usbus2 usbus1 usbus0 uhub3: 5 ports with 5 removable, self powered uhub5: 5 ports with 5 removable, self powered uhub0: 4 ports with 4 removable, self powered uhub6: 4 ports with 4 removable, self powered Root mount waiting for: usbus5 usbus3 uhub4: 5 ports with 5 removable, self powered uhub2: 5 ports with 5 removable, self powered Trying to mount root from zfs:zroot_mirror_hd103sj []... ugen3.2: at usbus3 run0: <1.0> on usbus3 run0: MAC/BBP RT3070 (rev 0x0201), RF RT2020 (MIMO 1T1R), address wlan1: Ethernet address: run0: firmware RT2870 ver. 0.236 loaded wlan1: link state changed to UP ==== dump 1 ==== $ sudo cat info.0 Dump header from device /dev/gpt/swap0 Architecture: amd64 Architecture Version: 2 Dump Length: 2289082368B (2183 MB) Blocksize: 512 Dumptime: Thu Mar 20 13:59:54 2014 Hostname: machete Magic: FreeBSD Kernel Dump Version String: FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC Panic String: spin lock held too long Dump Parity: 422405026 Bounds: 3 Dump Status: good $ sudo kgdb /boot/kernel/kernel.symbols vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: spin lock 0xffffffff814fa030 (smp rendezvous) held by 0xfffff8000fb42920 (tid 100428) too long timeout stopping cpus panic: spin lock held too long cpuid = 3 KDB: stack backtrace: #0 0xffffffff808e7dd0 at kdb_backtrace+0x60 #1 0xffffffff808af8b5 at panic+0x155 #2 0xffffffff8089cb71 at _mtx_lock_spin_cookie+0x241 #3 0xffffffff80c7ef54 at smp_targeted_tlb_shootdown+0xf4 #4 0xffffffff80c80335 at pmap_invalidate_page+0x265 #5 0xffffffff80c88983 at pmap_ts_referenced+0x6c3 #6 0xffffffff80b2182a at vm_pageout+0x10fa #7 0xffffffff8088198a at fork_exit+0x9a #8 0xffffffff80c758ce at fork_trampoline+0xe Uptime: 1h29m59s Dumping 2183 out of 7624 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko...done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_run.ko...done. Loaded symbols for /boot/kernel/if_run.ko Reading symbols from /boot/kernel/pf.ko...done. Loaded symbols for /boot/kernel/pf.ko #0 doadump (textdump=) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff808af530 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff808af8f4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=, tid=, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:554 #4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits = {4}}, vector=245, pmap=0xfffff8000f0bc9f8, addr1=34388402176, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179 #5 0xffffffff80c80335 in pmap_invalidate_page (pmap=, va=) at /usr/src/sys/amd64/amd64/pmap.c:1376 #6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff8021168a470) at /usr/src/sys/amd64/amd64/pmap.c:5744 #7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360 #8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730 , arg=0x0, frame=0xfffffe02151b6a40) at /usr/src/sys/kern/kern_fork.c:995 #9 0xffffffff80c758ce in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #10 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) up #1 0xffffffff808af530 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 447 doadump(TRUE); (kgdb) list 442 * been completed. 443 */ 444 EVENTHANDLER_INVOKE(shutdown_post_sync, howto); 445 446 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping) 447 doadump(TRUE); 448 449 /* Now that we're going to really halt the system... */ 450 EVENTHANDLER_INVOKE(shutdown_final, howto); 451 (kgdb) up #2 0xffffffff808af8f4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 754 kern_reboot(bootopt); (kgdb) list 749 /*thread_lock(td); */ 750 td->td_flags |= TDF_INPANIC; 751 /* thread_unlock(td); */ 752 if (!sync_on_panic) 753 bootopt |= RB_NOSYNC; 754 kern_reboot(bootopt); 755 } 756 757 /* 758 * Support for poweroff delay. (kgdb) up #3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=, tid=, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:554 554 panic("spin lock held too long"); (kgdb) list 549 printf( "spin lock %p (%s) held by %p (tid %d) too long\n", 550 m, m->lock_object.lo_name, td, td->td_tid); 551 #ifdef WITNESS 552 witness_display_spinlock(&m->lock_object, td, printf); 553 #endif 554 panic("spin lock held too long"); 555 } 556 557 #ifdef SMP 558 /* (kgdb) up #4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits = {4}}, vector=245, pmap=0xfffff8000f0bc9f8, addr1=34388402176, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179 1179 mtx_lock_spin(&smp_ipi_mtx); (kgdb) list 1174 if (CPU_EMPTY(&mask)) 1175 return; 1176 } 1177 if (!(read_rflags() & PSL_I)) 1178 panic("%s: interrupts disabled", __func__); 1179 mtx_lock_spin(&smp_ipi_mtx); 1180 smp_tlb_invpcid.addr = addr1; 1181 if (pmap == NULL) { 1182 smp_tlb_invpcid.pcid = 0; 1183 } else { (kgdb) up #5 0xffffffff80c80335 in pmap_invalidate_page (pmap=, va=) at /usr/src/sys/amd64/amd64/pmap.c:1376 1376 smp_masked_invlpg(other_cpus, pmap, va); (kgdb) list 1371 if (pmap_pcid_enabled) 1372 CPU_AND(&other_cpus, &pmap->pm_save); 1373 else 1374 CPU_AND(&other_cpus, &pmap->pm_active); 1375 if (!CPU_EMPTY(&other_cpus)) 1376 smp_masked_invlpg(other_cpus, pmap, va); 1377 } 1378 sched_unpin(); 1379 } 1380 (kgdb) up #6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff8021168a470) at /usr/src/sys/amd64/amd64/pmap.c:5744 5744 pmap_invalidate_page(pmap, pv->pv_va); (kgdb) list 5739 m)); 5740 pte = pmap_pde_to_pte(pde, pv->pv_va); 5741 if ((*pte & PG_A) != 0) { 5742 if (safe_to_clear_referenced(pmap, *pte)) { 5743 atomic_clear_long(pte, PG_A); 5744 pmap_invalidate_page(pmap, pv->pv_va); 5745 cleared++; 5746 } else if ((*pte & PG_W) == 0) { 5747 /* 5748 * Wired pages cannot be paged out so (kgdb) up #7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360 1360 act_delta += pmap_ts_referenced(m); (kgdb) list 1355 * 2) The ref was transitioning to one and we saw zero. 1356 * The page lock prevents a new reference to this page so 1357 * we need not check the reference bits. 1358 */ 1359 if (m->object->ref_count != 0) 1360 act_delta += pmap_ts_referenced(m); 1361 1362 /* 1363 * Advance or decay the act_count based on recent usage. 1364 */ (kgdb) up #8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730 , arg=0x0, frame=0xfffffe02151b6a40) at /usr/src/sys/kern/kern_fork.c:995 995 callout(arg, frame); (kgdb) list 990 * cpu_set_fork_handler intercepts this function call to 991 * have this call a non-return function to stay in kernel mode. 992 * initproc has its own fork handler, but it does return. 993 */ 994 KASSERT(callout != NULL, ("NULL callout in fork_exit")); 995 callout(arg, frame); 996 997 /* 998 * Check if a kernel thread misbehaved and returned from its main 999 * function. (kgdb) up #9 0xffffffff80c758ce in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 606 call fork_exit (kgdb) up #10 0x0000000000000000 in ?? () (kgdb) list 601 602 ENTRY(fork_trampoline) 603 movq %r12,%rdi /* function */ 604 movq %rbx,%rsi /* arg1 */ 605 movq %rsp,%rdx /* trapframe pointer */ 606 call fork_exit 607 MEXITCOUNT 608 jmp doreti /* Handle any ASTs */ 609 610 /* (kgdb) up Initial frame selected; you cannot go up. ==== dump 2 ==== $ sudo cat info.1 Dump header from device /dev/gpt/swap0 Architecture: amd64 Architecture Version: 2 Dump Length: 479866880B (457 MB) Blocksize: 512 Dumptime: Thu Mar 27 18:30:42 2014 Hostname: machete Magic: FreeBSD Kernel Dump Version String: FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC Panic String: spin lock held too long Dump Parity: 287807286 Bounds: 0 Dump Status: good $ sudo kgdb /boot/kernel/kernel.symbols vmcore.1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: spin lock 0xffffffff814fa030 (smp rendezvous) held by 0xfffff8000e525920 (tid 100396) too long timeout stopping cpus panic: spin lock held too long cpuid = 2 KDB: stack backtrace: #0 0xffffffff808e7dd0 at kdb_backtrace+0x60 #1 0xffffffff808af8b5 at panic+0x155 #2 0xffffffff8089cb71 at _mtx_lock_spin_cookie+0x241 #3 0xffffffff80c7ef54 at smp_targeted_tlb_shootdown+0xf4 #4 0xffffffff80c80335 at pmap_invalidate_page+0x265 #5 0xffffffff80c88983 at pmap_ts_referenced+0x6c3 #6 0xffffffff80b2182a at vm_pageout+0x10fa #7 0xffffffff8088198a at fork_exit+0x9a #8 0xffffffff80c758ce at fork_trampoline+0xe Uptime: 20m22s Dumping 457 out of 7624 MB:..4%..11%..21%..32%..42%..53%..63%..74%..81%..91% Reading symbols from /boot/kernel/zfs.ko...done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_run.ko...done. Loaded symbols for /boot/kernel/if_run.ko Reading symbols from /boot/kernel/pf.ko...done. Loaded symbols for /boot/kernel/pf.ko #0 doadump (textdump=) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff808af530 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff808af8f4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=, tid=, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:554 #4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits = {8}}, vector=245, pmap=0xfffff8000e2542f8, addr1=34385813504, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179 #5 0xffffffff80c80335 in pmap_invalidate_page (pmap=, va=) at /usr/src/sys/amd64/amd64/pmap.c:1376 #6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff80216069c60) at /usr/src/sys/amd64/amd64/pmap.c:5744 #7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360 #8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730 , arg=0x0, frame=0xfffffe02151b6a40) at /usr/src/sys/kern/kern_fork.c:995 #9 0xffffffff80c758ce in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #10 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) up #1 0xffffffff808af530 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 447 doadump(TRUE); (kgdb) list 442 * been completed. 443 */ 444 EVENTHANDLER_INVOKE(shutdown_post_sync, howto); 445 446 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping) 447 doadump(TRUE); 448 449 /* Now that we're going to really halt the system... */ 450 EVENTHANDLER_INVOKE(shutdown_final, howto); 451 (kgdb) up #2 0xffffffff808af8f4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 754 kern_reboot(bootopt); (kgdb) list 749 /*thread_lock(td); */ 750 td->td_flags |= TDF_INPANIC; 751 /* thread_unlock(td); */ 752 if (!sync_on_panic) 753 bootopt |= RB_NOSYNC; 754 kern_reboot(bootopt); 755 } 756 757 /* 758 * Support for poweroff delay. (kgdb) up #3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=, tid=, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:554 554 panic("spin lock held too long"); (kgdb) list 549 printf( "spin lock %p (%s) held by %p (tid %d) too long\n", 550 m, m->lock_object.lo_name, td, td->td_tid); 551 #ifdef WITNESS 552 witness_display_spinlock(&m->lock_object, td, printf); 553 #endif 554 panic("spin lock held too long"); 555 } 556 557 #ifdef SMP 558 /* (kgdb) up #4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits = {8}}, vector=245, pmap=0xfffff8000e2542f8, addr1=34385813504, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179 1179 mtx_lock_spin(&smp_ipi_mtx); (kgdb) list 1174 if (CPU_EMPTY(&mask)) 1175 return; 1176 } 1177 if (!(read_rflags() & PSL_I)) 1178 panic("%s: interrupts disabled", __func__); 1179 mtx_lock_spin(&smp_ipi_mtx); 1180 smp_tlb_invpcid.addr = addr1; 1181 if (pmap == NULL) { 1182 smp_tlb_invpcid.pcid = 0; 1183 } else { (kgdb) up #5 0xffffffff80c80335 in pmap_invalidate_page (pmap=, va=) at /usr/src/sys/amd64/amd64/pmap.c:1376 1376 smp_masked_invlpg(other_cpus, pmap, va); (kgdb) list 1371 if (pmap_pcid_enabled) 1372 CPU_AND(&other_cpus, &pmap->pm_save); 1373 else 1374 CPU_AND(&other_cpus, &pmap->pm_active); 1375 if (!CPU_EMPTY(&other_cpus)) 1376 smp_masked_invlpg(other_cpus, pmap, va); 1377 } 1378 sched_unpin(); 1379 } 1380 (kgdb) up #6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff80216069c60) at /usr/src/sys/amd64/amd64/pmap.c:5744 5744 pmap_invalidate_page(pmap, pv->pv_va); (kgdb) list 5739 m)); 5740 pte = pmap_pde_to_pte(pde, pv->pv_va); 5741 if ((*pte & PG_A) != 0) { 5742 if (safe_to_clear_referenced(pmap, *pte)) { 5743 atomic_clear_long(pte, PG_A); 5744 pmap_invalidate_page(pmap, pv->pv_va); 5745 cleared++; 5746 } else if ((*pte & PG_W) == 0) { 5747 /* 5748 * Wired pages cannot be paged out so (kgdb) up #7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360 1360 act_delta += pmap_ts_referenced(m); (kgdb) list 1355 * 2) The ref was transitioning to one and we saw zero. 1356 * The page lock prevents a new reference to this page so 1357 * we need not check the reference bits. 1358 */ 1359 if (m->object->ref_count != 0) 1360 act_delta += pmap_ts_referenced(m); 1361 1362 /* 1363 * Advance or decay the act_count based on recent usage. 1364 */ (kgdb) up #8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730 , arg=0x0, frame=0xfffffe02151b6a40) at /usr/src/sys/kern/kern_fork.c:995 995 callout(arg, frame); (kgdb) list 990 * cpu_set_fork_handler intercepts this function call to 991 * have this call a non-return function to stay in kernel mode. 992 * initproc has its own fork handler, but it does return. 993 */ 994 KASSERT(callout != NULL, ("NULL callout in fork_exit")); 995 callout(arg, frame); 996 997 /* 998 * Check if a kernel thread misbehaved and returned from its main 999 * function. (kgdb) up #9 0xffffffff80c758ce in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 606 call fork_exit Current language: auto; currently asm (kgdb) list 601 602 ENTRY(fork_trampoline) 603 movq %r12,%rdi /* function */ 604 movq %rbx,%rsi /* arg1 */ 605 movq %rsp,%rdx /* trapframe pointer */ 606 call fork_exit 607 MEXITCOUNT 608 jmp doreti /* Handle any ASTs */ 609 610 /* (kgdb) up #10 0x0000000000000000 in ?? () (kgdb) list 611 * To efficiently implement classification of trap and interrupt handlers 612 * for profiling, there must be only trap handlers between the labels btrap 613 * and bintr, and only interrupt handlers between the labels bintr and 614 * eintr. This is implemented (partly) by including files that contain 615 * some of the handlers. Before including the files, set up a normal asm 616 * environment so that the included files doen't need to know that they are 617 * included. 618 */ 619 620 #ifdef COMPAT_FREEBSD32 (kgdb) up Initial frame selected; you cannot go up.