From owner-freebsd-current@FreeBSD.ORG Thu Sep 10 23:05:55 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF365106566B for ; Thu, 10 Sep 2009 23:05:55 +0000 (UTC) (envelope-from duncan.young@pobox.com) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id A4BC78FC15 for ; Thu, 10 Sep 2009 23:05:55 +0000 (UTC) Received: from compute1.internal (compute1.internal [10.202.2.41]) by gateway1.messagingengine.com (Postfix) with ESMTP id 0A4986909D for ; Thu, 10 Sep 2009 18:47:20 -0400 (EDT) Received: from heartbeat2.messagingengine.com ([10.202.2.161]) by compute1.internal (MEProxy); Thu, 10 Sep 2009 18:47:20 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:mime-version:to:subject:content-type:content-transfer-encoding; s=smtpout; bh=DnjaE39KKggHiyJCkQrUSMRSkN0=; b=Ahg9xOmAoV6Qr1gr/RXnj9ezfAUSzd9eO37RQl82/qShbhvZ0cTmd71SoERFDTCrw+8rfKUj1WIN3XbzqRybs4jKsgzCAH+93m6poXjZNP9fAUzOsgdDX5rFVEZQlh+3y1Z4sDNTPLeffcgY1yjfkPONviQa3jjDFNR3DoWhFys= X-Sasl-enc: BpCXCYYK4f/ROyVkA8bhHLXeXVEj4fYf0WK54bPhu8uO 1252622838 Received: from [0.0.0.0] (c122-108-168-198.rochd4.qld.optusnet.com.au [122.108.168.198]) by mail.messagingengine.com (Postfix) with ESMTPSA id 46F62225C6 for ; Thu, 10 Sep 2009 18:47:18 -0400 (EDT) Message-ID: <4AA981F4.3090607@pobox.com> Date: Fri, 11 Sep 2009 08:47:16 +1000 From: Duncan Young User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: freebsd-current@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: zfs woes X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2009 23:05:56 -0000 Hi all, Again its happened (my need to restore my main zpool from backups). I have been having regular crashes (only recently been getting crash dumps though), and in the the process of upgrading to 7.1-STABLE, upgrading zpools (now 13) and zfs (now 3), I now have a problem with my main data zpool. I was having these problems before the upgrade. After zpool status saying that I had corrupted files (actually a zfs filesystem), I tried destroying it (the zfs filesystem), in the hope that this may make my problems go away). The machine hung. zpool scrub works, reporting 16 errors. zpool status -v crashes. starting multiuser hangs (probably on mounting the dud fs). I can start single user, and I am currently additional backups of the filesytems I can access (most off the corrupted zpool). Any accessing the corrupted zfs filesystem crashes the machine. Now I think it's fair enough that the filesystem is corrupted (I still see a need for some type of fsck), but I don't see that the machine should crash because of it. Any sugestions? Script started on Fri Sep 11 17:22:23 2009 # uname -a FreeBSD 7.2-STABLE FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST 2009 root@:/usr/obj/usr/src/sys/TRIPLE0 amd64 # cd /usr/src/sys/TRIPLE0 # kgdb ./kernel.debug /var/crash/vmcore/.25 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST 2009 root@:/usr/obj/usr/src/sys/TRIPLE0 module_register: module g_label already exists! Module g_label failed to register: 17 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) II X4 810 Processor (2608.81-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f42 Stepping = 2 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x37ff TSC: P-state invariant Cores per package: 4 usable memory = 8441266176 (8050 MB) avail memory = 8131702784 (7754 MB) ACPI APIC Table: <072709 APIC1045> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 This module (opensolaris) contains code covered by the Common Development and Distribution License (CDDL) see http://opensolaris.org/os/licensing/opensolaris_license/ ACPI Warning (tbfadt-0505): Optional field "Pm2ControlBlock" has zero address or length: 0 0/1 [20070320] ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <072709 XSDT1045> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed ACPI HPET table warning: Sequence is non-zero (2) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfbce0000-0xfbceffff,0xfbb00000-0xfbbfffff irq 18 at device 5.0 on pci1 hdac0: mem 0xfbcfc000-0xfbcfffff irq 19 at device 5.1 on pci1 hdac0: HDA Driver Revision: 20090624_0136 hdac0: [ITHREAD] pcib2: irq 16 at device 4.0 on pci0 pci2: on pcib2 hptrr0: port 0xd800-0xd8ff mem 0xfbe00000-0xfbefffff irq 16 at device 0.0 on pci2 hptrr: adapter at PCI 2:0:0, IRQ 16 pcib3: irq 17 at device 5.0 on pci0 pci3: on pcib3 em0: port 0xec00-0xec1f mem 0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 17 at device 0.0 on pci3 em0: Using MSIX interrupts em0: [ITHREAD] em0: [ITHREAD] em0: [ITHREAD] em0: Ethernet address: 00:1b:21:3d:eb:e7 atapci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfbaffc00-0xfbafffff irq 22 at device 17.0 on pci0 atapci0: [ITHREAD] atapci0: AHCI Version 01.10 controller with 6 ports detected ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] ata6: on atapci0 ata6: [ITHREAD] ata7: on atapci0 ata7: [ITHREAD] ohci0: mem 0xfbafd000-0xfbafdfff irq 16 at device 18.0 on pci0 ohci0: [GIANT-LOCKED] ohci0: [ITHREAD] usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 3 ports with 3 removable, self powered ohci1: mem 0xfbafe000-0xfbafefff irq 16 at device 18.1 on pci0 ohci1: [GIANT-LOCKED] ohci1: [ITHREAD] usb1: OHCI version 1.0, legacy support usb1: on ohci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 3 ports with 3 removable, self powered ehci0: mem 0xfbaff800-0xfbaff8ff irq 17 at device 18.2 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb2: EHCI version 1.0 usb2: companion controllers, 3 ports each: usb0 usb1 usb2: on ehci0 usb2: USB revision 2.0 uhub2: on usb2 uhub2: 6 ports with 6 removable, self powered ohci2: mem 0xfbafb000-0xfbafbfff irq 18 at device 19.0 on pci0 ohci2: [GIANT-LOCKED] ohci2: [ITHREAD] usb3: OHCI version 1.0, legacy support usb3: on ohci2 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 3 ports with 3 removable, self powered ohci3: mem 0xfbafc000-0xfbafcfff irq 18 at device 19.1 on pci0 ohci3: [GIANT-LOCKED] ohci3: [ITHREAD] usb4: OHCI version 1.0, legacy support usb4: on ohci3 usb4: USB revision 1.0 uhub4: on usb4 uhub4: 3 ports with 3 removable, self powered ehci1: mem 0xfbaff400-0xfbaff4ff irq 19 at device 19.2 on pci0 ehci1: [GIANT-LOCKED] ehci1: [ITHREAD] usb5: EHCI version 1.0 usb5: companion controllers, 3 ports each: usb3 usb4 usb5: on ehci1 usb5: USB revision 2.0 uhub5: on usb5 uhub5: 6 ports with 6 removable, self powered pci0: at device 20.0 (no driver attached) atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] hdac1: mem 0xfbaf4000-0xfbaf7fff irq 16 at device 20.2 on pci0 hdac1: HDA Driver Revision: 20090624_0136 hdac1: [ITHREAD] isab0: at device 20.3 on pci0 isa0: on isab0 pcib4: at device 20.4 on pci0 pci4: on pcib4 ohci4: mem 0xfbafa000-0xfbafafff irq 18 at device 20.5 on pci0 ohci4: [GIANT-LOCKED] ohci4: [ITHREAD] usb6: OHCI version 1.0, legacy support usb6: on ohci4 usb6: USB revision 1.0 uhub6: on usb6 uhub6: 2 ports with 2 removable, self powered acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model MouseMan+, device ID 0 acpi_aiboost0: on acpi0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] cpu0: on acpi0 acpi_throttle0: on cpu0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cryptosoft0: on motherboard orm0: at iomem 0xd7800-0xd87ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled ulpt0: on uhub0 ulpt0: using bi-directional mode WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounters tick every 1.000 msec vboxdrv: fAsync=0 offMin=0x12c offMax=0x219 hptrr: start channel [0,0] hptrr: start channel [0,1] hptrr: start channel [0,2] hptrr: start channel [0,3] hptrr: [0 0] Start channel soft reset. hptrr: [0 2] Start channel soft reset. hptrr: [0 3] Start channel soft reset. hptrr: channel [0,0] started successfully hptrr: channel [0,2] started successfully hptrr: channel [0,3] started successfully hptrr: [0 1] Failed to perform channel hard reset. hptrr0: [GIANT-LOCKED] hptrr0: [ITHREAD] The GEOM class LABEL is already loaded. ZFS filesystem version 13 ZFS storage pool version 13 da0 at hptrr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da1 at hptrr0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-0 device da2 at hptrr0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-0 device acd0: DVDR at ata0-master PIO4 ad4: 476940MB at ata2-master SATA300 acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x40 0x00 0x01 (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe0:ata0:0:0:0): CAM Status: SCSI Status Error (probe0:ata0:0:0:0): SCSI Status: Check Condition (probe0:ata0:0:0:0): NOT READY csi:0,0,bb,0 asc:3a,0 (probe0:ata0:0:0:0): Medium not present (probe0:ata0:0:0:0): Unretryable error cd0 at ata0 bus 0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 16.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present ad6: 953869MB at ata3-master SATA300 ad10: 953869MB at ata5-master SATA300 ad12: 476940MB at ata6-master SATA300 ad14: 953869MB at ata7-master SATA300 hdac0: HDA Codec #0: ATI RS690/780 HDMI pcm0: at cad 0 nid 1 on hdac0 hdac1: HDA Codec #0: VIA VT1708S_0 pcm1: at cad 0 nid 1 on hdac1 pcm2: at cad 0 nid 1 on hdac1 pcm3: at cad 0 nid 1 on hdac1 SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! Trying to mount root from zfs:rootzfs <118>Loading configuration files. <118>kernel dumps on /dev/ad4s1b <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. GEOM_ELI: Device label/swapB.eli created. GEOM_ELI: Encryption: AES-CBC 256 GEOM_ELI: Crypto: software <118>swapon: adding /dev/label/swapB.eli as swap device <118>/etc/rc: WARNING: Use of the early.sh script is deprecated <118>/etc/rc: WARNING: Please use a new-style rc.d script instead <118>/etc/rc: WARNING: See rc(8) for more information <118>gmirror: <118>No such device: swap_AB. <118> <118>Starting file system checks: <118>/dev/label/bstrapA: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/label/bstrapA: clean, 303581 free (261 frags, 37915 blocks, 0.1% fragmentation) <118>/dev/label/bstrapB: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/label/bstrapB: clean, 303502 free (318 frags, 37898 blocks, 0.1% fragmentation) <118>Setting hostuuid: e039c626-5f54-da11-8e5a-e6b6a8a23cf0. <118>Setting hostid: 0xd449202f. <118>Mounting local file systems: <118>. Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x50 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff80ad7a5e stack pointer = 0x10:0xffffff80a5b2f620 frame pointer = 0x10:0xffffff80a5b2f6a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 314 (txg_thread_enter) trap number = 12 panic: page fault cpuid = 1 Uptime: 2m11s Physical memory: 8050 MB Dumping 1553 MB: 1538 1522 1506 1490 1474 1458 1442 1426 1410 1394 1378 1362 1346 1330 1314 1298 1282 1266 1250 1234 1218 1202 1186 1170 1154 1138 1122 1106 1090 1074 1058 1042 1026 1010 994 978 962 946 930 914 898 882 866 850 834 818 802 786 770 754 738 722 706 690 674 658 642 626 610 594 578 562 546 530 514 498 482 466 450 434 418 402 386 370 354 338 322 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /bootdirA/boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /bootdirA/boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from /bootdirA/boot/kernel/geom_eli.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_eli.ko Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /bootdirA/boot/kernel/crypto.ko.symbols...done. done. Loaded symbols for /boot/kernel/crypto.ko Reading symbols from /boot/kernel/zlib.ko...Reading symbols from /bootdirA/boot/kernel/zlib.ko.symbols...done. done. Loaded symbols for /boot/kernel/zlib.ko Reading symbols from /boot/kernel/geom_label.ko...Reading symbols from /bootdirA/boot/kernel/geom_label.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_label.ko Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /bootdirA/boot/kernel/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_mirror.ko Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /bootdirA/boot/kernel/snd_hda.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_hda.ko Reading symbols from /boot/kernel/sound.ko...Reading symbols from /bootdirA/boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/cd9660_iconv.ko...Reading symbols from /bootdirA/boot/kernel/cd9660_iconv.ko.symbols...done. done. Loaded symbols for /boot/kernel/cd9660_iconv.ko Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from /bootdirA/boot/kernel/libiconv.ko.symbols...done. done. Loaded symbols for /boot/kernel/libiconv.ko Reading symbols from /boot/kernel/aio.ko...Reading symbols from /bootdirA/boot/kernel/aio.ko.symbols...done. done. Loaded symbols for /boot/kernel/aio.ko Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/kernel/sem.ko...Reading symbols from /bootdirA/boot/kernel/sem.ko.symbols...done. done. Loaded symbols for /boot/kernel/sem.ko #0 doadump () at pcpu.h:195 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0x0000000000000004 in ?? () #2 0xffffffff803aab90 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #3 0xffffffff803aafb2 in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:574 #4 0xffffffff80651d03 in trap_fatal (frame=0xffffff0009572390, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:770 #5 0xffffffff806520d5 in trap_pfault (frame=0xffffff80a5b2f570, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:686 #6 0xffffffff80652a63 in trap (frame=0xffffff80a5b2f570) at /usr/src/sys/amd64/amd64/trap.c:457 #7 0xffffffff8063cb4e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:218 #8 0xffffffff80ad7a5e in find_block (th=0xffffff8006930000, zseg=0xffffff000dafc580, dnp=0xffffff8006d35600, depth=Variable "depth" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:400 #9 0xffffffff80ad81a3 in traverse_more (th=0xffffff8006930000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:672 #10 0xffffffff80ad8368 in traverse_dsl_dataset (ds=0x23, txg_start=1, advance=Variable "advance" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/d---Type to continue, or q to quit--- mu_traverse.c:731 #11 0xffffffff80ae41c9 in dsl_dataset_destroy_sync () from /boot/kernel/zfs.ko #12 0xffffffff80ae6d2a in dsl_sync_task_group_sync (dstg=0xffffff000d256600, tx=0xffffff000d1fca80) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:186 #13 0xffffffff80ae6873 in dsl_pool_sync (dp=0xffffff000d521000, txg=3430336) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:316 #14 0xffffffff80af59c3 in spa_sync (spa=0xffffff0006d08000, txg=3430336) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3988 #15 0xffffffff80afd772 in txg_sync_thread (arg=Variable "arg" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:352 #16 0xffffffff80385452 in fork_exit ( callout=0xffffffff80afd490 , arg=0xffffff000d521000, frame=0xffffff80a5b2fc80) at /usr/src/sys/kern/kern_fork.c:811 #17 0xffffffff8063cf2e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:554 #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000001 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000dde000 in ?? () #43 0xffffffff80950d40 in tdq_cpu () #44 0xffffffff8095c940 in tdq_groups () #45 0xffffffff8095c8c0 in tdq_cpu () #46 0xffffff0009572390 in ?? () #47 0x0000000000000000 in ?? () #48 0xffffff80a5b2f1e8 in ?? () #49 0xffffff0009572390 in ?? () #50 0xffffffff803ce154 in sched_switch (td=0xffffffff80afd490, newtd=0x0, flags=0) at /usr/src/sys/kern/sched_ule.c:1938 #51 0x0000000000000000 in ?? () #52 0x0000000000000000 in ?? () #53 0x0000000000000000 in ?? () #54 0x0000000000000000 in ?? () #55 0x0000000000000000 in ?? () #56 0x0000000000000000 in ?? () #57 0x0000000000000000 in ?? () #58 0x0000000000000000 in ?? () #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x0000000000000000 in ?? () #71 0x0000000000000000 in ?? () #72 0x0000000000000000 in ?? () #73 0x0000000000000000 in ?? () #74 0x0000000000000000 in ?? () #75 0x0000000000000000 in ?? () #76 0x0000000000000000 in ?? () #77 0x0000000000000000 in ?? () #78 0x0000000000000000 in ?? () #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () #117 0x0000000000000000 in ?? () #118 0x0000000000000000 in ?? () Cannot access memory at address 0xffffff80a5b30000 (kgdb) q # ^D Script done on Fri Sep 11 17:26:02 2009