From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 10:35:22 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E18A106566B; Mon, 2 Nov 2009 10:35:22 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 417A48FC0C; Mon, 2 Nov 2009 10:35:20 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop1.sarenet.es (Postfix) with ESMTP id EC2845EF3; Mon, 2 Nov 2009 11:35:19 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Borja Marcos In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> Date: Mon, 2 Nov 2009 11:35:18 +0100 Content-Transfer-Encoding: 7bit Message-Id: <435C3337-40EA-4A79-8774-D394D71342D3@sarenet.es> References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1076) Cc: freebsd-fs@freebsd.org, stef@memberwebs.com Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 10:35:22 -0000 On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote: > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. I've started the test. For now I see a bunch of LOR complaints. # dmesg Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #0: Mon Oct 26 14:40:09 CET 2009 root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2094.43-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features = 0xfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x80082201> AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 780140544 (744 MB) avail memory = 730873856 (697 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0 mpt0: port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:fd:ab:03 em1: port 0x2040-0x207f mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:fd:ab:f9 pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:fd:ab:0d em3: port 0x2400-0x243f mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:fd:ab:17 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem 0xc0000-0xc7fff,0xca000-0xcafff, 0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff, 0xe4000-0xe7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/ loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2094428941 Hz quality 800 Timecounters tick every 10.000 msec Waiting 5 seconds for SCSI devices to settle acd0: CDROM at ata1-master UDMA33 da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a Expensive timeout(9) function: 0xffffffff8046e5b0(0xffffffff80e4f5e0) 0.002432990 s Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000277000) 0.008108572 s Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000265000) 0.035305324 s lock order reversal: 1st 0xffffff800f0c3cc8 bufwait (bufwait) @ /pool/newsrc/src/sys/kern/ vfs_bio.c:2559 2nd 0xffffff000c721a00 dirhash (dirhash) @ /pool/newsrc/src/sys/ufs/ ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_remove() at ufsdirhash_remove+0x16 ufs_dirremove() at ufs_dirremove+0x181 ufs_remove() at ufs_remove+0x92 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7 kern_unlinkat() at kern_unlinkat+0x252 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffe8b8, rbp = 0x800902300 --- Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...2 2 1 0 0 done All buffers synced. lock order reversal: 1st 0xffffff0002558308 ufs (ufs) @ /pool/newsrc/src/sys/kern/ vfs_mount.c:1200 2nd 0xffffff0002558cc8 devfs (devfs) @ /pool/newsrc/src/sys/kern/ vfs_subr.c:2083 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd03 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b devfs_allocv() at devfs_allocv+0x100 devfs_root() at devfs_root+0x48 dounmount() at dounmount+0x474 vfs_unmountall() at vfs_unmountall+0x54 boot() at boot+0x7d3 reboot() at reboot+0x68 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (55, FreeBSD ELF64, reboot), rip = 0x40892c, rsp = 0x7fffffffe738, rbp = 0x4023d0 --- Uptime: 2h1m35s Rebooting... Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #2: Mon Nov 2 12:08:37 CET 2009 root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2094.42-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features = 0xfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x80082201> AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 780140544 (744 MB) avail memory = 730472448 (696 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0 mpt0: port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:fd:ab:03 em1: port 0x2040-0x207f mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:fd:ab:f9 pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:fd:ab:0d em3: port 0x2400-0x243f mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:fd:ab:17 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem 0xc0000-0xc7fff,0xca000-0xcafff, 0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff, 0xe4000-0xe7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/ loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2094424190 Hz quality 800 Timecounters tick every 10.000 msec Waiting 5 seconds for SCSI devices to settle acd0: CDROM at ata1-master UDMA33 da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) Expensive timeout(9) function: 0xffffffff806fd940(0) 0.002228495 s da2 at mpt0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a lock order reversal: 1st 0xffffff000284d010 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: 2506 2nd 0xffffff0002851058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:421 3rd 0xffffff000284cdd8 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: 3014 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 arc_released() at arc_released+0x34 dbuf_set_data() at dbuf_set_data+0x62 dbuf_read_done() at dbuf_read_done+0x10b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dbuf_findbp() at dbuf_findbp+0x19b dbuf_hold_impl() at dbuf_hold_impl+0x164 dbuf_hold() at dbuf_hold+0x1b dnode_hold_impl() at dnode_hold_impl+0xc5 dmu_buf_hold() at dmu_buf_hold+0x34 zap_lockdir() at zap_lockdir+0x6e zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e dsl_pool_open() at dsl_pool_open+0xe3 spa_load() at spa_load+0x3a9 spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:549 2nd 0xffffff00028580d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:1196 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Expensive timeout(9) function: 0xffffffff80875e50(0xffffffff80e3ff80) 0.002947581 s _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode_sync.c:381 2nd 0xffffff000279e140 osi->os_lock (osi->os_lock) @ /pool/newsrc/ src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:1116 2nd 0xffffff0002881738 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff00028f4438 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/dbuf.c:1905 2nd 0xffffff000255e2f0 spa->spa_sync_bplist.bpl_lock (spa- >spa_sync_bplist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff0002935098 zfs (zfs) @ /pool/newsrc/src/sys/modules/ zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:437 2nd 0xffffff000266d310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:863 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23a zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f4a4dc, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff000292c078 zp->z_name_lock (zp->z_name_lock) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/zfs_dir.c:212 2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:863 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23a zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x257 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x2eb namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff000266d210 zfsvfs->z_teardown_inactive_lock (zfsvfs- >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3724 2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:1023 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x8d zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vput() at vput+0x250 kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002914210 zfsvfs->z_teardown_inactive_lock (zfsvfs- >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:915 2nd 0xffffff000278f4f8 ds->ds_rwlock (ds->ds_rwlock) @ /pool/newsrc/ src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffbbf8, rbp = 0x7fffffffc930 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002857b88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002ee9888 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:624 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002851d28 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff0002914250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_vfsops.c:1314 2nd 0xffffff0002914310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:963 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffbbf8, rbp = 0x7fffffffc930 --- #