From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 08:28:50 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C1C81065670; Mon, 2 Nov 2009 08:28:50 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 1D2F58FC1C; Mon, 2 Nov 2009 08:28:49 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop1.sarenet.es (Postfix) with ESMTP id 08586601C; Mon, 2 Nov 2009 09:28:47 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Borja Marcos In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> Date: Mon, 2 Nov 2009 09:28:47 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1076) Cc: freebsd-fs@freebsd.org, stef@memberwebs.com Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 08:28:50 -0000 On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote: > On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote: >> Borja Marcos wrote: >>> I've been sending some alltraces to pjd about this easy to reproduce >>> problem. I am using zfs send/zfs receive to replicate a dataset >>> from one >>> server to another. At 1 minute intervals, an incremental snapshot is >>> sent to update the dataset copy. If there is reading activity on the >>> dataset copy, a deadlock can happen rendering ZFS and all the FS >>> subsystem unusable. I've tried with 8.0RC2 and it still happens. > > I was able to reproduce it, but I don't have fix yet. > >> FWIW, another (or the same) zfs recv deadlock I've been trying to >> get to >> the bottom of: >> >> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ >> 006999.html > > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. Aye aye, Sir! What do you want me to collect? Just an alltrace? Anything else? If you want the VMWare images to examine the panic by yourself just let me know. Borja. From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 10:35:22 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E18A106566B; Mon, 2 Nov 2009 10:35:22 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 417A48FC0C; Mon, 2 Nov 2009 10:35:20 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop1.sarenet.es (Postfix) with ESMTP id EC2845EF3; Mon, 2 Nov 2009 11:35:19 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Borja Marcos In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> Date: Mon, 2 Nov 2009 11:35:18 +0100 Content-Transfer-Encoding: 7bit Message-Id: <435C3337-40EA-4A79-8774-D394D71342D3@sarenet.es> References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1076) Cc: freebsd-fs@freebsd.org, stef@memberwebs.com Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 10:35:22 -0000 On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote: > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. I've started the test. For now I see a bunch of LOR complaints. # dmesg Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #0: Mon Oct 26 14:40:09 CET 2009 root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2094.43-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features = 0xfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x80082201> AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 780140544 (744 MB) avail memory = 730873856 (697 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0 mpt0: port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:fd:ab:03 em1: port 0x2040-0x207f mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:fd:ab:f9 pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:fd:ab:0d em3: port 0x2400-0x243f mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:fd:ab:17 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem 0xc0000-0xc7fff,0xca000-0xcafff, 0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff, 0xe4000-0xe7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/ loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2094428941 Hz quality 800 Timecounters tick every 10.000 msec Waiting 5 seconds for SCSI devices to settle acd0: CDROM at ata1-master UDMA33 da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a Expensive timeout(9) function: 0xffffffff8046e5b0(0xffffffff80e4f5e0) 0.002432990 s Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000277000) 0.008108572 s Expensive timeout(9) function: 0xffffffff8031ffc0(0xffffff8000265000) 0.035305324 s lock order reversal: 1st 0xffffff800f0c3cc8 bufwait (bufwait) @ /pool/newsrc/src/sys/kern/ vfs_bio.c:2559 2nd 0xffffff000c721a00 dirhash (dirhash) @ /pool/newsrc/src/sys/ufs/ ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_remove() at ufsdirhash_remove+0x16 ufs_dirremove() at ufs_dirremove+0x181 ufs_remove() at ufs_remove+0x92 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7 kern_unlinkat() at kern_unlinkat+0x252 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp = 0x7fffffffe8b8, rbp = 0x800902300 --- Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...2 2 1 0 0 done All buffers synced. lock order reversal: 1st 0xffffff0002558308 ufs (ufs) @ /pool/newsrc/src/sys/kern/ vfs_mount.c:1200 2nd 0xffffff0002558cc8 devfs (devfs) @ /pool/newsrc/src/sys/kern/ vfs_subr.c:2083 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd03 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b devfs_allocv() at devfs_allocv+0x100 devfs_root() at devfs_root+0x48 dounmount() at dounmount+0x474 vfs_unmountall() at vfs_unmountall+0x54 boot() at boot+0x7d3 reboot() at reboot+0x68 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (55, FreeBSD ELF64, reboot), rip = 0x40892c, rsp = 0x7fffffffe738, rbp = 0x4023d0 --- Uptime: 2h1m35s Rebooting... Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #2: Mon Nov 2 12:08:37 CET 2009 root@:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2094.42-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features = 0xfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x80082201> AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 780140544 (744 MB) avail memory = 730472448 (696 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0 mpt0: port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:fd:ab:03 em1: port 0x2040-0x207f mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:fd:ab:f9 pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:fd:ab:0d em3: port 0x2400-0x243f mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:fd:ab:17 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem 0xc0000-0xc7fff,0xca000-0xcafff, 0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff,0xdc000-0xdffff, 0xe4000-0xe7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/ loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2094424190 Hz quality 800 Timecounters tick every 10.000 msec Waiting 5 seconds for SCSI devices to settle acd0: CDROM at ata1-master UDMA33 da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) Expensive timeout(9) function: 0xffffffff806fd940(0) 0.002228495 s da2 at mpt0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a lock order reversal: 1st 0xffffff000284d010 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: 2506 2nd 0xffffff0002851058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:421 3rd 0xffffff000284cdd8 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: 3014 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 arc_released() at arc_released+0x34 dbuf_set_data() at dbuf_set_data+0x62 dbuf_read_done() at dbuf_read_done+0x10b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dbuf_findbp() at dbuf_findbp+0x19b dbuf_hold_impl() at dbuf_hold_impl+0x164 dbuf_hold() at dbuf_hold+0x1b dnode_hold_impl() at dnode_hold_impl+0xc5 dmu_buf_hold() at dmu_buf_hold+0x34 zap_lockdir() at zap_lockdir+0x6e zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e dsl_pool_open() at dsl_pool_open+0xe3 spa_load() at spa_load+0x3a9 spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:549 2nd 0xffffff00028580d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:1196 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Expensive timeout(9) function: 0xffffffff80875e50(0xffffffff80e3ff80) 0.002947581 s _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode_sync.c:381 2nd 0xffffff000279e140 osi->os_lock (osi->os_lock) @ /pool/newsrc/ src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffd808, rbp = 0x801222140 --- lock order reversal: 1st 0xffffff0002850e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:1116 2nd 0xffffff0002881738 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff00028f4438 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/dbuf.c:1905 2nd 0xffffff000255e2f0 spa->spa_sync_bplist.bpl_lock (spa- >spa_sync_bplist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff0002935098 zfs (zfs) @ /pool/newsrc/src/sys/modules/ zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:437 2nd 0xffffff000266d310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:863 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23a zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f4a4dc, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff000292c078 zp->z_name_lock (zp->z_name_lock) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/zfs_dir.c:212 2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:863 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23a zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x257 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x2eb namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff000266d210 zfsvfs->z_teardown_inactive_lock (zfsvfs- >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3724 2nd 0xffffff000266d330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:1023 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x8d zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vput() at vput+0x250 kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd94fc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002914210 zfsvfs->z_teardown_inactive_lock (zfsvfs- >z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:915 2nd 0xffffff000278f4f8 ds->ds_rwlock (ds->ds_rwlock) @ /pool/newsrc/ src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffbbf8, rbp = 0x7fffffffc930 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002857b88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002ee9888 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dnode.c:624 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff000278f438 ds->ds_deadlist.bpl_lock (ds- >ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/bplist.c:331 2nd 0xffffff0002851d28 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_space_birthrange() at bplist_space_birthrange+0xb1 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- lock order reversal: 1st 0xffffff0002914250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_vfsops.c:1314 2nd 0xffffff0002914310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_znode.c:963 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp = 0x7fffffffbbf8, rbp = 0x7fffffffc930 --- # From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 11:02:04 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 784ED1065676 for ; Mon, 2 Nov 2009 11:02:04 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id 4F1388FC24 for ; Mon, 2 Nov 2009 11:02:02 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop2.sarenet.es (Postfix) with ESMTP id E99A473124 for ; Mon, 2 Nov 2009 12:02:01 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Mime-Version: 1.0 (Apple Message framework v1076) From: Borja Marcos In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> Date: Mon, 2 Nov 2009 12:02:01 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1076) Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 11:02:04 -0000 (Resending to the list, I replied only to Pawel) On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote: > On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote: >> Borja Marcos wrote: >>> I've been sending some alltraces to pjd about this easy to reproduce >>> problem. I am using zfs send/zfs receive to replicate a dataset >>> from one >>> server to another. At 1 minute intervals, an incremental snapshot is >>> sent to update the dataset copy. If there is reading activity on the >>> dataset copy, a deadlock can happen rendering ZFS and all the FS >>> subsystem unusable. I've tried with 8.0RC2 and it still happens. > > I was able to reproduce it, but I don't have fix yet. > >> FWIW, another (or the same) zfs recv deadlock I've been trying to >> get to >> the bottom of: >> >> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ >> 006999.html > > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. No panic, it just deadlocked. Last trace of LORs Nov 2 13:45:44 kernel: lock order reversal: Nov 2 13:45:44 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:189 Nov 2 13:45:44 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode.c:705 Nov 2 13:45:44 kernel: KDB: stack backtrace: Nov 2 13:45:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:45:44 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:45:44 kernel: dnode_setdirty() at dnode_setdirty+0xbc Nov 2 13:45:44 kernel: dbuf_dirty() at dbuf_dirty+0x516 Nov 2 13:45:44 kernel: bplist_enqueue() at bplist_enqueue+0xbd Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x1fe Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:45:44 kernel: lock order reversal: Nov 2 13:45:44 kernel: 1st 0xffffff0002eb9d38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:45:44 kernel: 2nd 0xffffff0002ed3000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543 Nov 2 13:45:44 kernel: KDB: stack backtrace: Nov 2 13:45:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:45:44 kernel: _sx_slock() at _sx_slock+0x55 Nov 2 13:45:44 kernel: dbuf_read() at dbuf_read+0x2ad Nov 2 13:45:44 kernel: dbuf_will_dirty() at dbuf_will_dirty+0x53 Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 Nov 2 13:45:44 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x159 Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:45:44 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:03 kernel: lock order reversal: Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:152 Nov 2 13:46:03 kernel: 2nd 0xffffff0002f01ae0 dn->dn_dbufs_mtx (dn- >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1518 Nov 2 13:46:03 kernel: KDB: stack backtrace: Nov 2 13:46:03 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x58 Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3 Nov 2 13:46:03 kernel: bplist_space_birthrange() at bplist_space_birthrange+0x60 Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:03 kernel: lock order reversal: Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:152 Nov 2 13:46:03 kernel: 2nd 0xffffffff81121d50 h->hash_mutexes[i] (h- >hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191 Nov 2 13:46:03 kernel: KDB: stack backtrace: Nov 2 13:46:03 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x111 Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3 Nov 2 13:46:03 kernel: bplist_space_birthrange() at bplist_space_birthrange+0x60 Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:35 kernel: lock order reversal: Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock (dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171 Nov 2 13:46:35 kernel: 2nd 0xffffff0002784938 ds->ds_opening_lock (ds->ds_opening_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:483 Nov 2 13:46:35 kernel: KDB: stack backtrace: Nov 2 13:46:35 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:35 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:35 kernel: dmu_objset_create_impl() at dmu_objset_create_impl+0x50 Nov 2 13:46:35 kernel: dmu_objset_create_sync() at dmu_objset_create_sync+0xfc Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:35 kernel: lock order reversal: Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock (dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171 Nov 2 13:46:35 kernel: 2nd 0xffffff0002fb7578 zfs (zfs) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/zfs_znode.c:152 Nov 2 13:46:35 kernel: KDB: stack backtrace: Nov 2 13:46:35 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:35 kernel: __lockmgr_args() at __lockmgr_args+0xd03 Nov 2 13:46:35 kernel: vop_stdlock() at vop_stdlock+0x39 Nov 2 13:46:35 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b Nov 2 13:46:35 kernel: _vn_lock() at _vn_lock+0x57 Nov 2 13:46:35 kernel: zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x65 Nov 2 13:46:35 kernel: zfs_create_fs() at zfs_create_fs+0x2ab Nov 2 13:46:35 kernel: dmu_objset_create_sync() at dmu_objset_create_sync+0x116 Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:36 kernel: lock order reversal: Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1724 Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c0708 dn->dn_dbufs_mtx (dn- >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode_sync.c:373 Nov 2 13:46:36 kernel: KDB: stack backtrace: Nov 2 13:46:36 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x57 Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0xd4 Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54 Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55 Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173 Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618 Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:36 kernel: lock order reversal: Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1724 Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c03d8 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:409 Nov 2 13:46:36 kernel: KDB: stack backtrace: Nov 2 13:46:36 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x1b8 Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0xd4 Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54 Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55 Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173 Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618 Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:57 kernel: lock order reversal: Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:46:57 kernel: 2nd 0xffffff0002fdb4b0 dn->dn_mtx (dn- >dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1066 Nov 2 13:46:57 kernel: KDB: stack backtrace: Nov 2 13:46:57 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa03 Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x38b Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:46:57 last message repeated 5 times Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:57 kernel: lock order reversal: Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:46:57 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode.c:705 Nov 2 13:46:57 kernel: KDB: stack backtrace: Nov 2 13:46:57 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:57 kernel: dnode_setdirty() at dnode_setdirty+0xbc Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa53 Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x38b Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:46:57 last message repeated 5 times Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:47:28 kernel: lock order reversal: Nov 2 13:47:28 kernel: 1st 0xffffff0002fb6a58 syncer (syncer) @ / pool/newsrc/src/sys/kern/vfs_subr.c:1693 Nov 2 13:47:28 kernel: 2nd 0xffffff001a868350 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863 Nov 2 13:47:28 kernel: KDB: stack backtrace: Nov 2 13:47:28 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:47:28 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:47:28 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:47:28 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:47:28 kernel: zfs_zget() at zfs_zget+0x23a Nov 2 13:47:28 kernel: zfs_get_data() at zfs_get_data+0x5e Nov 2 13:47:28 kernel: zil_commit() at zil_commit+0x5aa Nov 2 13:47:28 kernel: zfs_sync() at zfs_sync+0xa6 Nov 2 13:47:28 kernel: sync_fsync() at sync_fsync+0x13a Nov 2 13:47:28 kernel: VOP_F Nov 2 13:47:28 kernel: SYNC_APV() at VOP_FSYNC_APV+0xb5 Nov 2 13:47:28 kernel: sync_vnode() at sync_vnode+0x157 Nov 2 13:47:28 kernel: sched_sync() at sched_sync+0x1d2 Nov 2 13:47:28 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:47:28 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:47:28 kernel: --- trap 0, rip = 0, rsp = 0xffffff8017aebd30, rbp = 0 --- Nov 2 13:47:44 kernel: lock order reversal: Nov 2 13:47:44 kernel: 1st 0xffffff002325d4c0 zp->z_parent_lock (zp- >z_parent_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/zfs_dir.c:379 Nov 2 13:47:44 kernel: 2nd 0xffffff0002914370 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863 Nov 2 13:47:44 kernel: KDB: stack backtrace: Nov 2 13:47:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:47:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:47:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:47:44 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:47:44 kernel: zfs_zget() at zfs_zget+0x23a Nov 2 13:47:44 kernel: zfs_dirlook() at zfs_dirlook+0x1fc Nov 2 13:47:44 kernel: zfs_lookup() at zfs_lookup+0x257 Nov 2 13:47:44 kernel: zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d Nov 2 13:47:44 kernel: VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV +0xaf Nov 2 13:47:44 kernel: vfs_cache_lookup() at vfs_cache_lookup+0xf0 Nov 2 13:47:44 kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 Nov 2 13:47:44 kernel: lookup() at lookup+0x2eb Nov 2 13:47:44 kernel: namei() at namei+0x4a9 Nov 2 13:47:44 kernel: kern_chdir() at kern_chdir+0x78 Nov 2 13:47:44 kernel: syscall() at syscall+0x1d0 Nov 2 13:47:44 kernel: Xfast_syscall() at Xfast_syscall+0xe1 Nov 2 13:47:44 kernel: --- syscall (12, FreeBSD ELF64, chdir), rip = 0x800da8a9c, rsp = 0x7fffffffe8b8, rbp = 0x8010135e0 --- From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 11:06:53 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF23C1065676 for ; Mon, 2 Nov 2009 11:06:53 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8B2818FC24 for ; Mon, 2 Nov 2009 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA2B6rA9033575 for ; Mon, 2 Nov 2009 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA2B6qsk033573 for freebsd-fs@FreeBSD.org; Mon, 2 Nov 2009 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 2 Nov 2009 11:06:52 GMT Message-Id: <200911021106.nA2B6qsk033573@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138367 fs [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/138109 fs [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 138 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 15:17:44 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E5DA1065676; Mon, 2 Nov 2009 15:17:44 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D9B048FC15; Mon, 2 Nov 2009 15:17:43 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA2FHhHr059160; Mon, 2 Nov 2009 15:17:43 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA2FHhQV059156; Mon, 2 Nov 2009 15:17:43 GMT (envelope-from jh) Date: Mon, 2 Nov 2009 15:17:43 GMT Message-Id: <200911021517.nA2FHhQV059156@freefall.freebsd.org> To: jh@FreeBSD.org, freebsd-fs@FreeBSD.org, jh@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/122047: [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / UF_APPEND flag on EXT2FS (maybe others) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 15:17:44 -0000 Synopsis: [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / UF_APPEND flag on EXT2FS (maybe others) Responsible-Changed-From-To: freebsd-fs->jh Responsible-Changed-By: jh Responsible-Changed-When: Mon Nov 2 15:17:43 UTC 2009 Responsible-Changed-Why: Take. http://www.freebsd.org/cgi/query-pr.cgi?pr=122047 From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 17:15:59 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EF831065676; Mon, 2 Nov 2009 17:15:59 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2534C8FC21; Mon, 2 Nov 2009 17:15:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA2HFxVv062261; Mon, 2 Nov 2009 17:15:59 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA2HFw5i062257; Mon, 2 Nov 2009 17:15:58 GMT (envelope-from jh) Date: Mon, 2 Nov 2009 17:15:58 GMT Message-Id: <200911021715.nA2HFw5i062257@freefall.freebsd.org> To: csaba.henk@creo.hu, jh@FreeBSD.org, freebsd-fs@FreeBSD.org, jh@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/105093: [ext2fs] [patch] ext2fs on read-only media cannot be mounted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 17:15:59 -0000 Synopsis: [ext2fs] [patch] ext2fs on read-only media cannot be mounted State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Mon Nov 2 17:13:28 UTC 2009 State-Changed-Why: Apparently this has been fixed. Can you confirm? Responsible-Changed-From-To: freebsd-fs->jh Responsible-Changed-By: jh Responsible-Changed-When: Mon Nov 2 17:13:28 UTC 2009 Responsible-Changed-Why: Track. http://www.freebsd.org/cgi/query-pr.cgi?pr=105093 From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 17:26:20 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C3EA1065670; Mon, 2 Nov 2009 17:26:20 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E6CE28FC1D; Mon, 2 Nov 2009 17:26:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA2HQJhi070412; Mon, 2 Nov 2009 17:26:19 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA2HQJIX070408; Mon, 2 Nov 2009 17:26:19 GMT (envelope-from jh) Date: Mon, 2 Nov 2009 17:26:19 GMT Message-Id: <200911021726.nA2HQJIX070408@freefall.freebsd.org> To: eoakes@comcast.net, jh@FreeBSD.org, freebsd-fs@FreeBSD.org, jh@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/77826: [ext2fs] ext2fs usb filesystem will not mount RW X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 17:26:20 -0000 Synopsis: [ext2fs] ext2fs usb filesystem will not mount RW State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Mon Nov 2 17:22:10 UTC 2009 State-Changed-Why: Note that submitter has been asked for feedback. Responsible-Changed-From-To: freebsd-fs->jh Responsible-Changed-By: jh Responsible-Changed-When: Mon Nov 2 17:22:10 UTC 2009 Responsible-Changed-Why: Track. http://www.freebsd.org/cgi/query-pr.cgi?pr=77826 From owner-freebsd-fs@FreeBSD.ORG Tue Nov 3 11:09:04 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28290106566B; Tue, 3 Nov 2009 11:09:04 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id DC5B28FC08; Tue, 3 Nov 2009 11:09:03 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop1.sarenet.es (Postfix) with ESMTP id 28436614F; Tue, 3 Nov 2009 12:09:02 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Borja Marcos In-Reply-To: <20091029205121.GB3418@garage.freebsd.pl> Date: Tue, 3 Nov 2009 12:08:54 +0100 Content-Transfer-Encoding: 7bit Message-Id: <9AA2C968-F09D-473D-BD13-F13B3F94ED60@sarenet.es> References: <20091029205121.GB3418@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1076) Cc: freebsd-fs@freebsd.org, Ronald Klop Subject: Re: zfs receive gives: internal error: Argument list too long X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2009 11:09:04 -0000 On Oct 29, 2009, at 9:51 PM, Pawel Jakub Dawidek wrote: > On Wed, Oct 28, 2009 at 09:51:46PM +0100, Ronald Klop wrote: >> Hi, >> >> I'm forwarding this, because there was no answer on freebsd-stable. >> >> Does anybody know about this and have some tips on how to solve it? > > Could you try this patch: > > http://people.freebsd.org/~pjd/patches/zfs_recv_E2BIG.patch It's caused a panic for me on 8.0-RC2/amd64. Seems a new problem, never saw a panic in this situation before. How to reproduce: With /usr/src and /usr/obj in a dataset, just cd /usr/src make clean Instant panic, in less than 20 seconds. Trying to get panic information, unfortunately I'm running on VMWare Fussion and the silly thing doesn't offer the equivalent of a serial console. Borja. From owner-freebsd-fs@FreeBSD.ORG Tue Nov 3 20:26:51 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B32A1065676 for ; Tue, 3 Nov 2009 20:26:51 +0000 (UTC) (envelope-from csaba.henk@gmail.com) Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id 40ECB8FC08 for ; Tue, 3 Nov 2009 20:26:51 +0000 (UTC) Received: by pwj8 with SMTP id 8so2932293pwj.3 for ; Tue, 03 Nov 2009 12:26:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type; bh=Y7c+tKpAa1pkYh5Nwb5u5SKPiXw413CHm3y2lM5lop8=; b=UGQup8y/xrU9A1SCwlJk5sgpz6Wanmcot2QSXUOS2ZAqSLIsjXPhnkZqyeaYuScp+i GtBRD46N1UCPqr+zHyiZW1ceMSWKOuKgTO3bTwoXK3uTXy6GXnTt+0LewU2F9xzSSnZ9 1BOnMRZsgZxfbTDbKyWswmvK6WHdw2U2OHX4I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=LJ4AqnPS4UiNzRNLsY2rlUDPFXyAZUYOYV+DCq1td/P2u0zEWEDxdOV46AfWtXbKsH IGm8HU1EqKTrVqJDT4Vl3xbJlVY2/V6QzEnrcQS6PDYE2xbA9chvf5rCGpFNSnsIkppE I1vBZXs15EIdVlLKwHPfwa6cvt1BjmbmYeqzc= MIME-Version: 1.0 Sender: csaba.henk@gmail.com Received: by 10.142.75.17 with SMTP id x17mr43764wfa.154.1257278461889; Tue, 03 Nov 2009 12:01:01 -0800 (PST) In-Reply-To: <200911021715.nA2HFw5i062257@freefall.freebsd.org> References: <200911021715.nA2HFw5i062257@freefall.freebsd.org> Date: Tue, 3 Nov 2009 21:01:01 +0100 X-Google-Sender-Auth: 47fa4ca6c4552f47 Message-ID: <100d90a90911031201y2c36ee70ida572ed2e63f3ba1@mail.gmail.com> From: Csaba Henk To: jh@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: [creo] Re: kern/105093: [ext2fs] [patch] ext2fs on read-only media cannot be mounted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2009 20:26:51 -0000 On Mon, Nov 2, 2009 at 6:15 PM, wrote: > Synopsis: [ext2fs] [patch] ext2fs on read-only media cannot be mounted > > Apparently this has been fixed. Can you confirm? > > http://www.freebsd.org/cgi/query-pr.cgi?pr=105093 I don't have a system at hand where I could test it but there were other reports that this problem has gone away, so I think it's indeed fixed. Csaba From owner-freebsd-fs@FreeBSD.ORG Thu Nov 5 06:25:38 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38744106566C for ; Thu, 5 Nov 2009 06:25:38 +0000 (UTC) (envelope-from ai@kliksys.ru) Received: from gate.kliksys.ru (gate.kliksys.ru [78.110.241.113]) by mx1.freebsd.org (Postfix) with ESMTP id E0DBA8FC17 for ; Thu, 5 Nov 2009 06:25:37 +0000 (UTC) Received: from [192.168.1.13] (helo=one.kliksys.ru) by gate.kliksys.ru with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1N5vc6-000BwW-Qg for freebsd-fs@freebsd.org; Thu, 05 Nov 2009 09:14:31 +0300 Date: Thu, 5 Nov 2009 09:14:30 +0300 From: Artemiev Igor To: freebsd-fs@freebsd.org Message-ID: <20091105061430.GA92808@one.kliksys.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam_score: 0.0 Subject: Low zfs prefetch hits - why? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2009 06:25:38 -0000 #sysctl vfs.zfs.zfetch vfs.zfs.zfetch.array_rd_sz: 524288 vfs.zfs.zfetch.block_cap: 8 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 Trying to sequential read 1G file by 128K chunks. #./arcstat.pl -f Time,pmis,pm% Time pmis pm% 05:48:40 749 100 05:48:41 1K 100 05:48:42 1K 100 05:48:43 1K 100 05:48:44 1K 100 05:48:45 1K 100 05:48:46 1K 100 05:48:47 257 99 I thought, blocks didn`t prefetching and wrote small dtrace script: #!/usr/sbin/dtrace -qs fbt:zfs:dmu_zfetch:entry { printf("dmu_zfetch: offset=%ld size=%ld prefetched=%ld\n", arg1, arg2, arg3); } fbt:zfs:dbuf_prefetch:entry { printf(" zfetching block %d\n", arg1); } Here output: dmu_zfetch: offset=325058560 size=131072 prefetched=32 dmu_zfetch: offset=325189632 size=131072 prefetched=32 dmu_zfetch: offset=325320704 size=131072 prefetched=32 dmu_zfetch: offset=325451776 size=131072 prefetched=32 dmu_zfetch: offset=325582848 size=131072 prefetched=32 dmu_zfetch: offset=326369280 size=131072 prefetched=32 dmu_zfetch: offset=326500352 size=131072 prefetched=32 dmu_zfetch: offset=326631424 size=131072 prefetched=32 dmu_zfetch: offset=326762496 size=131072 prefetched=32 dmu_zfetch: offset=326893568 size=131072 prefetched=32 dmu_zfetch: offset=327024640 size=131072 prefetched=32 zfetching block 2503 zfetching block 2504 zfetching block 2505 zfetching block 2506 zfetching block 2507 zfetching block 2508 zfetching block 2509 zfetching block 2510 dmu_zfetch: offset=327155712 size=131072 prefetched=32 dmu_zfetch: offset=327286784 size=131072 prefetched=32 dmu_zfetch: offset=327417856 size=131072 prefetched=32 dmu_zfetch: offset=327548928 size=131072 prefetched=32 dmu_zfetch: offset=327680000 size=131072 prefetched=32 Why this happening? Statistic is unrelevant? With sendfile(2) prefetching completely didn`t work - no forward read ahead. From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 08:47:38 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09065106568F for ; Fri, 6 Nov 2009 08:47:38 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 8CA988FC12 for ; Fri, 6 Nov 2009 08:47:37 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id nA68lYfP000741 for ; Fri, 6 Nov 2009 09:47:35 +0100 Received: from pmp.uni-hannover.de (arc.pmp.uni-hannover.de [130.75.117.1]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 54AAF24 for ; Fri, 6 Nov 2009 09:47:34 +0100 (CET) Date: Fri, 6 Nov 2009 09:47:34 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: freebsd-fs@freebsd.org Message-Id: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.7.1 (GTK+ 2.12.11; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-PMX-Version: 5.5.5.374460, Antispam-Engine: 2.7.1.369594, Antispam-Data: 2009.11.6.83630 Subject: zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 08:47:38 -0000 Hi, unfortunately I got no answer concerning this problem so far on -stable and -current (apart from the suggestion to try it again here :-). I can reproduce the panic, and if someone can guide me what to do with kdb, gdb, zdb or whatever tool might be needed to get the information needed to fix this, I'm all ears... cu Gerrit Begin forwarded message: Date: Wed, 4 Nov 2009 09:29:00 +0100 From: Gerrit K=FChn To: freebsd-stable@FreeBSD.ORG Cc:=20 Subject: zfs panic mounting fs after crash with RC2 Hi, Yesterday I had the opportunity to play around with my yet-to-become new fileserver a bit more. Originally I had installed 7.2-R, which I upgraded to 8-0-RC2 yesterday. After that I upgraded my zpool consisting of 4 disks in raidz1 constallation to v13. Some time later I tried to use powerd which was obviously a bad idea: it crashed the machine immediately. I will give a separate report on that later as it is probably related to the hardware, which is a bit exotic (VIA VB8001 board with 64bit Via Nano processor). However, the worst thing for me is, that after rebooting from that crash, one of my zfs fs cannot be mounted anymore. As soon as I try to mount it I get a kernel panic. I can still access the properties (I made use of "canmount=3Dnoauto" for the first time :-), but I cannot do a snapshot of the fs (funny enough, zfs complains that the fs is busy, while in reality it is not even mounted - so how could it be busy?). I took a picture of the kernel panic and put it here (don't know if there is any useful information in it): The pool as such seems to be fine, all other fs in it can be mounted and used, only trying to mount tank/sys/var triggers this panic. Are there any suggestions what I could do to get my fs back? Please let me know if (and how) I can provide more debugging information. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 12:10:34 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D200C106566C for ; Fri, 6 Nov 2009 12:10:34 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from tensor.andric.com (cl-327.ede-01.nl.sixxs.net [IPv6:2001:7b8:2ff:146::2]) by mx1.freebsd.org (Postfix) with ESMTP id 987188FC17 for ; Fri, 6 Nov 2009 12:10:34 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7:0:83a:c07c:8fa2:a7d8] (unknown [IPv6:2001:7b8:3a7:0:83a:c07c:8fa2:a7d8]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id C8DE65C59; Fri, 6 Nov 2009 13:10:33 +0100 (CET) Message-ID: <4AF4123A.4080301@andric.com> Date: Fri, 06 Nov 2009 13:10:34 +0100 From: Dimitry Andric User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.5pre) Gecko/20091031 Shredder/3.0pre MIME-Version: 1.0 To: =?ISO-8859-1?Q?Gerrit_K=FChn?= References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> In-Reply-To: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 12:10:34 -0000 On 2009-11-06 09:47, Gerrit K=FChn wrote: > unfortunately I got no answer concerning this problem so far on -stable= > and -current (apart from the suggestion to try it again here :-). > I can reproduce the panic, and if someone can guide me what to do with = kdb, > gdb, zdb or whatever tool might be needed to get the information needed= to > fix this, I'm all ears... At least a backtrace would be nice. :) From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 12:40:01 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B4D61065679 for ; Fri, 6 Nov 2009 12:40:01 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 727E38FC20 for ; Fri, 6 Nov 2009 12:39:59 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id nA6CducB014203; Fri, 6 Nov 2009 13:39:57 +0100 Received: from pmp.uni-hannover.de (arc.pmp.uni-hannover.de [130.75.117.1]) by www.pmp.uni-hannover.de (Postfix) with SMTP id CC3F824; Fri, 6 Nov 2009 13:39:56 +0100 (CET) Date: Fri, 6 Nov 2009 13:39:56 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Dimitry Andric Message-Id: <20091106133956.1091ed8c.gerrit@pmp.uni-hannover.de> In-Reply-To: <4AF4123A.4080301@andric.com> References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> <4AF4123A.4080301@andric.com> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.7.1 (GTK+ 2.12.11; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.5.374460, Antispam-Engine: 2.7.1.369594, Antispam-Data: 2009.11.6.122423 Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 12:40:01 -0000 On Fri, 06 Nov 2009 13:10:34 +0100 Dimitry Andric wrote about Re: zfs panic mounting fs after crash with RC2: DA> > unfortunately I got no answer concerning this problem so far on DA> > -stable and -current (apart from the suggestion to try it again DA> > here :-). I can reproduce the panic, and if someone can guide me DA> > what to do with kdb, gdb, zdb or whatever tool might be needed to DA> > get the information needed to fix this, I'm all ears... DA> At least a backtrace would be nice. :) I know. Unfortunately I know not much about debugging the kernel. I read , but I do not get a kernel core file, because I run the system from a CF card and use the hds completely for zfs. I have no swap partition I could dump to. Is it possible to dump onto a zfs fs? Or is there any other way for debugging? cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 12:50:24 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3F49106566B for ; Fri, 6 Nov 2009 12:50:24 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 48BCC8FC15 for ; Fri, 6 Nov 2009 12:50:23 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id nA6CoKgj014611; Fri, 6 Nov 2009 13:50:21 +0100 Received: from pmp.uni-hannover.de (arc.pmp.uni-hannover.de [130.75.117.1]) by www.pmp.uni-hannover.de (Postfix) with SMTP id D837324; Fri, 6 Nov 2009 13:50:20 +0100 (CET) Date: Fri, 6 Nov 2009 13:50:20 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Dimitry Andric Message-Id: <20091106135020.7c837bc7.gerrit@pmp.uni-hannover.de> In-Reply-To: <4AF4123A.4080301@andric.com> References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> <4AF4123A.4080301@andric.com> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.7.1 (GTK+ 2.12.11; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.5.374460, Antispam-Engine: 2.7.1.369594, Antispam-Data: 2009.11.6.123328 Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 12:50:24 -0000 On Fri, 06 Nov 2009 13:10:34 +0100 Dimitry Andric wrote about Re: zfs panic mounting fs after crash with RC2: DA> > unfortunately I got no answer concerning this problem so far on DA> > -stable and -current (apart from the suggestion to try it again DA> > here :-). I can reproduce the panic, and if someone can guide me DA> > what to do with kdb, gdb, zdb or whatever tool might be needed to DA> > get the information needed to fix this, I'm all ears... DA> At least a backtrace would be nice. :) Thinking about my situation and assuming that I cannot dump directly onto a zfs fs, I could probably either plug in an usb stick and try to dump onto that or recompile the kernel with ddb to try online debugging. Any suggestions? cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 15:08:57 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EFB0106568F; Fri, 6 Nov 2009 15:08:57 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id 75A778FC1A; Fri, 6 Nov 2009 15:08:56 +0000 (UTC) Received: by fxm27 with SMTP id 27so259620fxm.3 for ; Fri, 06 Nov 2009 07:08:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=RIQM3SXZgDP6srJWYEzoinN15v+wUFUZTrYO0/pI7SY=; b=iZrPQaLlZ3flohzj4TPpjvDrv0mdRe31nA5rVYIzRO31bM3z5DGXM9SpP0eiA8SneY RHGn1kYDWmqr7KGYMNG/57V/7MPvgSxpuzd1Tf0foe9KMu3FMul4aCAAy5PyUGszpONL wC30VfhpIMSjhC4PKyS0KucPHVvb2rKODro3Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=VAXHCSZ5OHH4mCRHB3QQbkeDHna9CYaVWUXFncWlofCNqlN44t+8VztHEQlByOVxoH IW51Mel9eXskNmVpA62gLLxAaQfz5uQefmH9ekS7+1zpX6oJ9wjwKKZtqt8JmDQDBKu1 RS3GJfdlfPMTYTePaAQ/QYKuQdZRWTgVzcvUU= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.4.27 with SMTP id 27mr159617fap.48.1257520134645; Fri, 06 Nov 2009 07:08:54 -0800 (PST) Date: Fri, 6 Nov 2009 16:08:54 +0100 X-Google-Sender-Auth: c973b7991f827212 Message-ID: <3bbf2fe10911060708t50f5ead8t76329b379c56a5eb@mail.gmail.com> From: Attilio Rao To: freebsd-fs@freebsd.org, Ed Maste , Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: Subject: [PATCH] Transform vfs.root.mountfrom into a list of fs:device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 15:08:57 -0000 This patch adds the possibility to specify multiple couplets of fs:device for the environ vfs.root.mountfrom (space separated) rather just one item: http://www.freebsd.org/~attilio/Sandvine/STABLE_8/vfsrootmountfrom/vfsrootmountfrom.diff The one going to boot is the first valid one. While there, this patch also fixes a nit into a comment. This patch has been contributed back by Sandvine Incorporated. Please review. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 19:34:06 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97B071065693 for ; Fri, 6 Nov 2009 19:34:06 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-ew0-f209.google.com (mail-ew0-f209.google.com [209.85.219.209]) by mx1.freebsd.org (Postfix) with ESMTP id 288488FC18 for ; Fri, 6 Nov 2009 19:34:05 +0000 (UTC) Received: by ewy5 with SMTP id 5so1338580ewy.36 for ; Fri, 06 Nov 2009 11:34:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:content-transfer-encoding; bh=KUHRqe16cNcLjgOovfIWGmCC0xJoqR0FvVx3O9eRqj0=; b=BUiI0AobLvAlrExOiPoVwmclgU4XWweo8uQnSPimpQahRpTAF0UBoRsMIjot5TLrSw h80hx5GwK72Vqu4cgOvBAx7keC88lMyF5vKAk48a3yI9BgT8DTI9eV0EYbQ+zdtvlJnl Y1Z1B+gDJoaal/M00jHNUH48iN+IesrfWNZnw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=Htw0NcgoiV/iFdouaVLVjdOHfxcaCuG/qswycWcIE6sheNAify1qVN3Flv+KJLROZg 6GGLalEMn9yzOlpPQYdd74AnVamkw8usNVbddj6nyUkzbUq5XXCQCBzElBcCx8rNIv2I 8ITwg0JjMdaz7Gp1FikNvYaCbJTb5eihR32Hc= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.89.193 with SMTP id c43mr1429838wef.221.1257534096161; Fri, 06 Nov 2009 11:01:36 -0800 (PST) In-Reply-To: <4AF46CA9.1040904@quip.cz> References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> <4AEEBD4B.1050407@quip.cz> <4AEEDB3B.5020600@quip.cz> <4AF46CA9.1040904@quip.cz> From: Ivan Voras Date: Fri, 6 Nov 2009 20:01:16 +0100 X-Google-Sender-Auth: 51b74079c92339e5 Message-ID: <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> To: Miroslav Lachman <000.fbsd@quip.cz> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 19:34:06 -0000 2009/11/6 Miroslav Lachman <000.fbsd@quip.cz>: > I do not understand why there are 10MB/s read from disks when network > traffic dropped to around 1MB/s (8Mbps) > > root@cage ~/# iostat -w 20 > =C2=A0 =C2=A0 =C2=A0tty =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ad4 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ad6 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 cpu > =C2=A0tin tout =C2=A0KB/t tps =C2=A0MB/s =C2=A0 KB/t tps =C2=A0MB/s =C2= =A0us ni sy in id > =C2=A0 0 =C2=A0 14 41.66 =C2=A053 =C2=A02.17 =C2=A041.82 =C2=A053 =C2=A02= .18 =C2=A0 0 =C2=A00 =C2=A02 =C2=A00 97 > =C2=A0 0 =C2=A0 18 50.92 =C2=A096 =C2=A04.77 =C2=A054.82 114 =C2=A06.12 = =C2=A0 0 =C2=A00 =C2=A03 =C2=A01 96 > =C2=A0 0 =C2=A0 =C2=A06 53.52 101 =C2=A05.29 =C2=A054.98 108 =C2=A05.81 = =C2=A0 1 =C2=A00 =C2=A04 =C2=A01 94 > =C2=A0 0 =C2=A0 =C2=A06 54.82 =C2=A098 =C2=A05.26 =C2=A055.89 108 =C2=A05= .89 =C2=A0 0 =C2=A00 =C2=A03 =C2=A01 96 Yes, this could limit your IO if the requests are random enough. Unfortunately I don't know how would you track down what is really going on. Maybe some tracing with DTrace? I'd tell you to use "top -m io" to see if there is a process responsible, but apparently these statistics are not updated for ZFS, which in itself may be a bug (which is why I'm crossposting to freebsd-fs). From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 22:14:45 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0F701065672 for ; Fri, 6 Nov 2009 22:14:45 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 5167F8FC12 for ; Fri, 6 Nov 2009 22:14:44 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id nA6MEfbp002808; Fri, 6 Nov 2009 23:14:43 +0100 Received: from pmp.uni-hannover.de (theq.pmp.uni-hannover.de [130.75.117.4]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 6E39524; Fri, 6 Nov 2009 23:14:41 +0100 (CET) Date: Fri, 6 Nov 2009 23:14:40 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Dimitry Andric Message-Id: <20091106231440.4f0f2cbb.gerrit@pmp.uni-hannover.de> In-Reply-To: <4AF4123A.4080301@andric.com> References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> <4AF4123A.4080301@andric.com> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.4.2 (GTK+ 2.10.12; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-PMX-Version: 5.5.5.374460, Antispam-Engine: 2.7.1.369594, Antispam-Data: 2009.11.6.220033 Cc: freebsd-fs@freebsd.org Subject: trace for zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 22:14:46 -0000 On Fri, 06 Nov 2009 13:10:34 +0100 Dimitry Andric wrote about Re: zfs panic mounting fs after crash with RC2: DA> On 2009-11-06 09:47, Gerrit K=FChn wrote: DA> > unfortunately I got no answer concerning this problem so far on DA> > -stable and -current (apart from the suggestion to try it again DA> > here :-). I can reproduce the panic, and if someone can guide me DA> > what to do with kdb, gdb, zdb or whatever tool might be needed to DA> > get the information needed to fix this, I'm all ears... DA> At least a backtrace would be nice. :) I recomplied the kernel with ddb support and got the following trace (using mount -t zfs instead of zfs mount this time, but getting the same panic): I have the system still sitting at this point and can also 100% reproduce the panic. Please let me know if (and how) any further information can get pulled out of the debugger. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 23:02:27 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68901106566C for ; Fri, 6 Nov 2009 23:02:27 +0000 (UTC) (envelope-from james-freebsd-fs2@jrv.org) Received: from mail.jrv.org (adsl-70-243-84-13.dsl.austtx.swbell.net [70.243.84.13]) by mx1.freebsd.org (Postfix) with ESMTP id E24BF8FC0A for ; Fri, 6 Nov 2009 23:02:26 +0000 (UTC) Received: from kremvax.housenet.jrv (kremvax.housenet.jrv [192.168.3.124]) by mail.jrv.org (8.14.3/8.14.3) with ESMTP id nA6N2Nhe037117; Fri, 6 Nov 2009 17:02:24 -0600 (CST) (envelope-from james-freebsd-fs2@jrv.org) Authentication-Results: mail.jrv.org; domainkeys=pass (testing) header.from=james-freebsd-fs2@jrv.org DomainKey-Signature: a=rsa-sha1; s=enigma; d=jrv.org; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:cc:subject: references:in-reply-to:content-type:content-transfer-encoding; b=JNskFQDLotR+FX/5Ffzjgd/GoyHiaBt+/BhIQNrJjYOoP4EWQAby2kW2XWQlBq7HE z0qEr1aZd8DA7Ds9gk9Y+CMImNHOgfIP2RPawrekVjfhjnXI0fL6QOB2PCV2xtvp+cz 94HgP3ifGs0plonm4PmLQF7nnrD4N30xkIKNyjY= Message-ID: <4AF4AAFF.2080104@jrv.org> Date: Fri, 06 Nov 2009 17:02:23 -0600 From: "James R. Van Artsdalen" User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Gerrit_K=FChn?= References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> <4AF4123A.4080301@andric.com> <20091106231440.4f0f2cbb.gerrit@pmp.uni-hannover.de> In-Reply-To: <20091106231440.4f0f2cbb.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: trace for zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 23:02:27 -0000 Gerrit K=FChn wrote: > I recomplied the kernel with ddb support and got the following trace > (using mount -t zfs instead of zfs mount this time, but getting the sam= e > panic): You may be able to recover your pool by changing the line below, but I have never tried it: it may clobber the pool. You definitely don't want this change normally! It may be necessary to avoid calling zil_destroy here too. How the ZIL got corrupted - if it did - is a harder question. What kind of hard disk is this, and how is it connected to the system? Was there any redundancy (mirror, raidz)? void zil_replay(objset_t *os, void *arg, uint64_t *txgp, zil_replay_func_t *replay_func[TX_MAX_TYPE], zil_replay_cleaner_t *replay_cleaner) { zilog_t *zilog =3D dmu_objset_zil(os); const zil_header_t *zh =3D zilog->zl_header; zil_replay_arg_t zr; =3D=3D> if (1 || zil_empty(zilog)) { zil_destroy(zilog, B_TRUE); return; } From owner-freebsd-fs@FreeBSD.ORG Fri Nov 6 23:53:24 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A80FD1065698 for ; Fri, 6 Nov 2009 23:53:24 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 33BDC8FC1A for ; Fri, 6 Nov 2009 23:53:23 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id nA6NrLaO005085; Sat, 7 Nov 2009 00:53:22 +0100 Received: from pmp.uni-hannover.de (theq.pmp.uni-hannover.de [130.75.117.4]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 7CF5E24; Sat, 7 Nov 2009 00:53:21 +0100 (CET) Date: Sat, 7 Nov 2009 00:53:20 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: "James R. Van Artsdalen" Message-Id: <20091107005320.cd6a9fad.gerrit@pmp.uni-hannover.de> In-Reply-To: <4AF4AAFF.2080104@jrv.org> References: <20091106094734.4b056899.gerrit@pmp.uni-hannover.de> <4AF4123A.4080301@andric.com> <20091106231440.4f0f2cbb.gerrit@pmp.uni-hannover.de> <4AF4AAFF.2080104@jrv.org> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.4.2 (GTK+ 2.10.12; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.5.374460, Antispam-Engine: 2.7.1.369594, Antispam-Data: 2009.11.6.234227 Cc: freebsd-fs@freebsd.org Subject: Re: trace for zfs panic mounting fs after crash with RC2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2009 23:53:24 -0000 On Fri, 06 Nov 2009 17:02:23 -0600 "James R. Van Artsdalen" wrote about Re: trace for zfs panic mounting fs after crash with RC2: JRVA> > I recomplied the kernel with ddb support and got the following JRVA> > trace (using mount -t zfs instead of zfs mount this time, but JRVA> > getting the same panic): JRVA> You may be able to recover your pool by changing the line below, but JRVA> I have never tried it: it may clobber the pool. You definitely JRVA> don't want this change normally! It may be necessary to avoid JRVA> calling zil_destroy here too. Well, as I said before, the pool itself and all other filesystems in it are fine. The pool can be imported and all other filesystems can be mounted and used. Just one of it panics the system when I try to mount it. JRVA> How the ZIL got corrupted - if it did - is a harder question. What JRVA> kind of hard disk is this, and how is it connected to the system? JRVA> Was there any redundancy (mirror, raidz)? These are 4x2.5" 400GB drives (WD4000BEVT) in a RAID-Z1 setup on a Supermicro AOC-USAS-L8i controller (LSI chip, mpt driver) in a VIA VB8001 board (powered by a Via Nano 1.6GHz) with 4GB of memory. The system paniced when I tried to run powerd, after reboot the pool came back fine, but the system paniced again when trying to mount this particular fs (tank/sys/var). Before this happened I had one similar issue when the system crashed (probably because I was mechanically pushing the controller card a bit too hard during operation when trying to fix some SATA cables). However, after this crash the whole pool did not come back and the system paniced when trying to import the pool - but also with ZIL-replay problems. As this happened right after installing the base system, I simply re-did the pool and re-installed the system. However, with this similar problem after a quite "normal" crash (hey, I only started powerd) my confidence is a bit low and I would like to have this fixed before I really put data on the machine. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 03:12:19 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E8D2106566C; Sat, 7 Nov 2009 03:12:19 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7480E8FC13; Sat, 7 Nov 2009 03:12:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA73CJEu075822; Sat, 7 Nov 2009 03:12:19 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA73CJ7m075818; Sat, 7 Nov 2009 03:12:19 GMT (envelope-from linimon) Date: Sat, 7 Nov 2009 03:12:19 GMT Message-Id: <200911070312.nA73CJ7m075818@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/140338: [vm] [zfs] [panic] FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 03:12:19 -0000 Old Synopsis: FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld New Synopsis: [vm] [zfs] [panic] FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Nov 7 03:10:45 UTC 2009 Responsible-Changed-Why: Seems to happen with a combination of vm and zfs settings. Since I have to pick an assignee, use the fs@ one. http://www.freebsd.org/cgi/query-pr.cgi?pr=140338 From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 08:04:49 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98D5A106566B; Sat, 7 Nov 2009 08:04:49 +0000 (UTC) (envelope-from xclin@cs.nctu.edu.tw) Received: from csmailer.cs.nctu.edu.tw (csmailer.cs.nctu.edu.tw [140.113.235.125]) by mx1.freebsd.org (Postfix) with ESMTP id 176E68FC08; Sat, 7 Nov 2009 08:04:49 +0000 (UTC) Received: from csmailer.cs.nctu.edu.tw (localhost [127.0.0.1]) by csmailer.cs.nctu.edu.tw (Postfix) with ESMTP id 9557522CBA; Sat, 7 Nov 2009 15:48:02 +0800 (CST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cs.nctu.edu.tw; h=date :from:to:cc:subject:message-id:mime-version:content-type; s= rsa1024; bh=pfgwpGTUq4jt8e2x3IaAj+P7kg8=; b=K4/dwrPFyEsfT30tjm3p wos3RwEwlGxSH3KSUEA/T/Ye/pWhA9rzOqY15mgyhGRim1p3OQyhH/c238t1qYXk HLm5Y6ISFVnZoJ5hL1Wnu2DVLwarfa/Zl6VAq3MMw5pdAYLOQBMRoRTjnGrgl5XJ CdAcrM4JvjNBMpszgxyW4+U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cs.nctu.edu.tw; h=date:from :to:cc:subject:message-id:mime-version:content-type; q=dns; s= rsa1024; b=lT4qb4lbV4GywympGh1FCaZMIo/i8+AHk68/5LTeT0HaTQVvfYaqx +0wdf4LXcIL+Ps/5cJPWTCFVBK7v2ciYZUY8bHLTuJTKVOONpd+sB+jA88AgUt58 yf/x2zzUToef7nlWE5rx2Ou2PeMqB+dildUmGmD6W4OLaFc+ybwaNU= Received: from bsd3.cs.nctu.edu.tw (bsd3.cs.nctu.edu.tw [140.113.235.133]) by csmailer.cs.nctu.edu.tw (Postfix) with ESMTP id 5F66022CB8; Sat, 7 Nov 2009 15:48:02 +0800 (CST) Received: (from xclin@localhost) by bsd3.cs.nctu.edu.tw (8.14.3/8.14.3/Submit) id nA77mN5A080344; Sat, 7 Nov 2009 15:48:23 +0800 (CST) (envelope-from xclin) Date: Sat, 7 Nov 2009 15:48:23 +0800 From: Chen-Chuan Lin To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org Message-ID: <20091107074823.GA78260@cs.nctu.edu.tw> MIME-Version: 1.0 Content-Type: text/plain; charset=big5 Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Cc: xclin@cs.nctu.edu.tw Subject: ZFSBoot with zpool inside bsdlabel X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 08:04:49 -0000 Hi all, I want to build a zfs only system and need multiboot with Microsoft Windows. So I can only choose MBR and zfsboot. And I found that zfsboot doesn't work in my environment. I create 2 slices, 1 for windows and 2 for FreeBSD. In slice 2, I create 2 partitions, A for zfs and B for swap. If partition A's offset is 0, zfsboot works fine, but if partition A is behind B, zfsboot1 can't locate zfsboot2. Also, zfsboot2 can't find my zpool. 0 1 n n+512 n+1024 (sector relative to +------+------+----------+--------+--------+-------+---- current slice) | boot | disk | ... swap | vdev | vdev | Boot |.... | code | label| | label0 | label1 | Block | +------+------+----------+--------+--------+-------+--- | Partition B | Partition A +------------------------+------------------------------ Original zfsboot only support 0 512 1024 +------+--------+-------+-------+---- | boot | vdev | vdev | Boot | +------+ label | label | Block | ..... | 0 | 1 | | +-------------+-+-------+-------+---- where boot code is zfsboot1 and boot block is zfsboot2 I found that zfsboot won't read any disklabel and assume zpool is placed in the whole slice instead of a single partition. So I modified zfsboot.c (part of zfsboot2) and zfsldr.S (zfsboot1). Make prove_drive() in zfsboot.c will scan zfs metadata no only in slice but also partition A if there is disklabel in the slice. In zfsldr.S. There is no enough space to put additional code to scan disk label, so i make a change. Every thing before main.5 is unchanged. And then scan disklable, check magic number, partition type, and calculate the offset of partition A. Finally, read BTX & BTX client from there, instead of sector 1024 of the slice. The rest of thing (BTX relocation & set A20) has been moved into zfsboot1.5, which store in sector 2 of the booting slice and will be loaded into 0xA00 (I don't know this memory location is ok or not). zfsboot1 will load boot1.5, load BTX and then jump to boot1.5. Boot 1.5 does the rest of things and jump to BTX to continue the booting procedure. The steps to install bootcode has become (assume zpool is in ad0s2a) dd if=zfsboot1 of=/dev/ad0s2 dd if=zfsboot15 of=/dev/ad0s2 seek=2 dd if=zfsboot2 of=/dev/ad0s2a seek=1024 There is a sample zfsboot at http://140.113.17.225/~xclin/zfsboot (with some debug message and only for i386/amd64) The first 512 bytes are zfsboot1, the second 512 bytes are zfsboot1.5, and the remaining 32K are zfsboot2 dd if=zfsboot of=/dev/ad0s2 count=1 dd if=zfsboot of=/dev/ad0s2 count=1 skip=1 seek=2 dd if=zfsboot of=/dev/ad0s2a skip=2 seek=1024 I don't know where this function is necessary or not... The following are patches & zfsboot1.5's code (or http://140.113.17.225/~xclin/zfsboot.patch) --- sys/boot/i386/zfsboot/Makefile.orig 2009-10-30 23:48:15.356651674 +0800 +++ sys/boot/i386/zfsboot/Makefile 2009-10-30 23:49:59.515556507 +0800 @@ -15,6 +15,7 @@ REL1= 0x700 ORG1= 0x7c00 +ORG15= 0xa00 ORG2= 0x2000 CFLAGS= -Os -g \ @@ -45,8 +46,8 @@ CLEANFILES= zfsboot -zfsboot: zfsboot1 zfsboot2 - cat zfsboot1 zfsboot2 > zfsboot +zfsboot: zfsboot1 zfsboot1.5 zfsboot2 + cat zfsboot1 zfsboot1.5 zfsboot2 > zfsboot CLEANFILES+= zfsboot1 zfsldr.out zfsldr.o @@ -56,6 +57,14 @@ zfsldr.out: zfsldr.o ${LD} ${LDFLAGS} -e start -Ttext ${ORG1} -o ${.TARGET} zfsldr.o +CLEANFILES+= zfsboot1.5 boot15.out boot15.o + +zfsboot1.5: boot15.out + objcopy -S -O binary boot15.out ${.TARGET} + +boot15.out: boot15.o + ${LD} ${LDFLAGS} -e start -Ttext ${ORG15} -o ${.TARGET} boot15.o + CLEANFILES+= zfsboot2 zfsboot.ld zfsboot.ldr zfsboot.bin zfsboot.out \ zfsboot.o zfsboot.s zfsboot.s.tmp zfsboot.h sio.o --- sys/boot/i386/zfsboot/boot15.S.orig 1970-01-01 08:00:00.000000000 +0800 +++ sys/boot/i386/zfsboot/boot15.S 2009-10-30 23:37:11.310256951 +0800 @@ -0,0 +1,93 @@ + +/* Memory Locations */ + .set MEM_ORG_15,0xa00 # Origin of boot15 + .set MEM_BUF,0x8000 # Load area + .set MEM_BTX,0x9000 # BTX start + .set MEM_JMP,0x9010 # BTX entry point + .set MEM_USR,0xa000 # Client start + .set BDA_BOOT,0x472 # Boot howto flag + +/* Misc. Constants */ + .set SIZ_PAG,0x1000 # Page size + .set SIZ_SEC,0x200 # Sector size + .set NSECT,0x40 + + .global start + .code16 + + .set NREADP, 0x7c27 + +start: jmp main + +main: + mov $msg_hello, %si + callw putstr + +/* + * We have already loaded BTX and client into $MEM_BUF in boot1. The + * only thing boot1.5 need to do is just relocate BTX and client and + * then jump to entry point + */ + +main.7: mov $MEM_BUF,%si # BTX (before reloc) + mov 0xa(%si),%bx # Get BTX length and set + mov $NSECT*SIZ_SEC-1,%di # Size of load area (less one) + mov %di,%si # End of load + add $MEM_BUF,%si # area + sub %bx,%di # End of client, 0xc000 rel + mov %di,%cx # Size of + inc %cx # client + mov $(MEM_USR+2*SIZ_PAG)>>4,%dx # Segment + mov %dx,%es # addressing 0xc000 + std # Move with decrement + rep # Relocate + movsb # client + mov %ds,%dx # Back to + mov %dx,%es # zero segment + mov $MEM_BUF,%si # BTX (before reloc) + mov $MEM_BTX,%di # BTX + mov %bx,%cx # Get BTX length + cld # Increment this time + rep # Relocate + movsb # BTX + +/* + * Enable A20 so we can access memory above 1 meg. + * Use the zero-valued %cx as a timeout for embedded hardware which do not + * have a keyboard controller. + */ +seta20: cli # Disable interrupts +seta20.1: dec %cx # Timeout? + jz seta20.3 # Yes + inb $0x64,%al # Get status + testb $0x2,%al # Busy? + jnz seta20.1 # Yes + movb $0xd1,%al # Command: Write + outb %al,$0x64 # output port +seta20.2: inb $0x64,%al # Get status + testb $0x2,%al # Busy? + jnz seta20.2 # Yes + movb $0xdf,%al # Enable + outb %al,$0x60 # A20 +seta20.3: sti # Enable interrupts + + + jmp start+MEM_JMP-MEM_ORG_15 # Start BTX + +putstr.0: mov $0x7,%bx + movb $0xe,%ah + int $0x10 +putstr: lodsb + testb %al,%al + jne putstr.0 + +ereturn: movb $0x1,%ah + stc +return: retw + + +msg_hello: .asciz "welcom to boot1.5\r\n" + + .org 0x1FE, 0x90 +/* useless but check if code is more then 512byte */ +part_magic: .word 0xaa55 --- sys/boot/i386/zfsboot/zfsboot.c.orig 2009-10-30 23:48:15.355651268 +0800 +++ sys/boot/i386/zfsboot/zfsboot.c 2009-10-30 23:50:44.761515700 +0800 @@ -22,6 +22,7 @@ #ifdef GPT #include #endif +#include #include #include @@ -154,6 +155,7 @@ struct dmadat { char rdbuf[READ_BUF_SIZE]; /* for reading large things */ char secbuf[READ_BUF_SIZE]; /* for MBR/disklabel */ + char labbuf[READ_BUF_SIZE]; /* for MBR/disklabel */ }; static struct dmadat *dmadat; @@ -524,6 +526,32 @@ */ dsk = copy_dsk(dsk); } + else if(dp[i].dp_typ == DOSPTYP_386BSD) + { + struct disklabel *label; + char *lhdr = dmadat->labbuf; + /* save slice offset if we want to scan all partitions */ + u_int32_t dp_start = dp[i].dp_start; + + if(drvread(dsk, lhdr, 1 ,1)) + continue; + + label = lhdr; + + if(label->d_magic !=DISKMAGIC || label->d_magic2 != DISKMAGIC) + continue; + + if(!label->d_partitions[0].p_size) + continue; + + dsk->start += label->d_partitions[0].p_offset; + dsk->start -= label->d_partitions[RAW_PART].p_offset; + + if(vdev_probe(vdev_read, dsk, spap) == 0) { + spap =0; + dsk = copy_dsk(dsk); + } + } } } --- sys/boot/i386/zfsboot/zfsldr.S.orig 2009-10-30 23:48:15.353651293 +0800 +++ sys/boot/i386/zfsboot/zfsldr.S 2009-10-30 23:51:13.431489398 +0800 @@ -18,6 +18,7 @@ /* Memory Locations */ .set MEM_REL,0x700 # Relocation address .set MEM_ARG,0x900 # Arguments + .set MEM_BOOT15,0xa00 # Boot1.5 Location .set MEM_ORG,0x7c00 # Origin .set MEM_BUF,0x8000 # Load area .set MEM_BTX,0x9000 # BTX start @@ -38,6 +39,16 @@ .set SIZ_SEC,0x200 # Sector size .set NSECT,0x40 + +/* Disklabel Constants */ + .set DL_DISKMAGIC,0x82564557 # Disklabel Magic + .set DL_MAGIC1,0x0 # Magic1 + .set DL_MAGIC2,0x84 # Magic2 + .set DL_SECT,0x94 # Number of sector + .set DL_PARTA_OFFSET,0x98 # PartA offset + .set DL_PARTA_TYPE,0xA0 # PartA Type + .set DL_RAW_OFFSET,0xb8 # Raw Part offset + .globl start .globl xread .code16 @@ -194,9 +205,64 @@ * area and target area do not overlap. */ main.5: mov %dx,MEM_ARG # Save args + movw 0x8(%si),%ax # Backup + movw %ax,MEM_BUF+SIZ_SEC+0x8 # original + movw 0xa(%si),%ax # slice + movw %ax,MEM_BUF+SIZ_SEC+0xa # start +/* + * Because we dont's have enough space, relocateing BTX will take + * place in boot 1.5. Load it from sector 2 and relocate into 0xa00 + */ + movb $1,%dh # Load + movw $2,%ax # Boot 1.5 from + callw nread.1 # Read disk + mov $MEM_BUF,%si # Boot 1.5 (before) + mov $MEM_BOOT15,%di # Boot 1.5 + mov $SIZ_SEC,%cx # Size + rep # Relocate + movsb # Boot 1.5 +/* + * Read sector 1 from slice to check out whethre there is disklabel + * or not. If so, check magic, partition type and size of partition A + * And then caculate partition A's offset and put it in + * MEM_BUF+SIZ_SEC+0x8, otherwise, it remain unchanged + */ + movw $MEM_BUF+SIZ_SEC,%si # offset + movb $1,%dh # Sector count + movw $1,%ax # Offset to disklabel + callw nread.1 # Read disk + mov $MEM_BUF,%si # Disklabel + cmpl $DL_DISKMAGIC,0x0(%si) # Check + jne main.6 # msgic1 + cmpl $DL_DISKMAGIC,0x84(%si) # Check + jne main.6 # magic2 + cmpl $0,DL_SECT(%si) # Check label a + je main.6 # size + cmpb $27,DL_PARTA_TYPE(%si) # Check label a + jne main.6 # type ZFS=27 + + movw MEM_BUF+SIZ_SEC+0x8,%ax # Partition A + movw MEM_BUF+SIZ_SEC+0xa,%cx # start at + addw DL_PARTA_OFFSET(%si),%ax # slice + adcw DL_PARTA_OFFSET+2(%si),%cx # offset + + subw DL_RAW_OFFSET(%si),%ax # part a + sbbw DL_RAW_OFFSET+2(%si),%cx # offset - + movw %ax,MEM_BUF+SIZ_SEC+0x8 # raw part + movw %cx,MEM_BUF+SIZ_SEC+0xa # offset +/* + * We have boot partitions's offset in $MEM_BUS+SIZ_SEC+8, put it in + * %si and load. And then read boot2 from disk. The rest of things + * will be done in boot 1.5 + */ +main.6: + movw $MEM_BUF+SIZ_SEC,%si # offset movb $NSECT,%dh # Sector count movw $1024,%ax # Offset to boot2 callw nread.1 # Read disk + + jmp start+MEM_BOOT15-MEM_ORG # Goto Boot1.5 + +#if 0 /* below is the original boot code */ main.6: mov $MEM_BUF,%si # BTX (before reloc) mov 0xa(%si),%bx # Get BTX length and set mov $NSECT*SIZ_SEC-1,%di # Size of load area (less one) @@ -241,14 +307,16 @@ jmp start+MEM_JMP-MEM_ORG # Start BTX +#endif /* * Trampoline used to call read from within boot1. */ nread: xor %ax,%ax # Sector offset in partition nread.1: mov $MEM_BUF,%bx # Transfer buffer + xor %cx,%cx # Clear add 0x8(%si),%ax # Get - mov 0xa(%si),%cx # LBA + adc 0xa(%si),%cx # LBA push %cs # Read from callw xread.1 # disk jnc return # If success, return From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 08:40:03 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62077106566C for ; Sat, 7 Nov 2009 08:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1AA338FC12 for ; Sat, 7 Nov 2009 08:40:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA78e2uf088427 for ; Sat, 7 Nov 2009 08:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA78e2qc088426; Sat, 7 Nov 2009 08:40:02 GMT (envelope-from gnats) Date: Sat, 7 Nov 2009 08:40:02 GMT Message-Id: <200911070840.nA78e2qc088426@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Kai Gallasch Cc: Subject: Re: kern/140338: FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Kai Gallasch List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 08:40:03 -0000 The following reply was made to PR kern/140338; it has been noted by GNATS. From: Kai Gallasch To: bug-followup@FreeBSD.org Cc: Patrick Lamaiziere , linimon@FreeBSD.org Subject: Re: kern/140338: FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld Date: Sat, 7 Nov 2009 09:09:22 +0100 Am Fri, 6 Nov 2009 19:36:54 +0100 schrieb Patrick Lamaiziere : > Le Fri, 6 Nov 2009 17:28:40 GMT, > Kai Gallasch : > > Hello, > > > ZFS filesystem version 13 > > ZFS storage pool version 13 > > It seems you are using ZFS on this box? No. The server is not in production and never was with FreeBSD 8. Only the kernel module is loaded, due to zfs_enable="YES" in rc.conf as a preparation for using ZFS with 8.0-RELEASE The problem described in the PR occured when zfs_enable="YES" was not set in rc.conf I see no direct connection to with this ZFS. > > On the same box, I've used super-pages for a longtime on FreeBSD 7.2 > and with 8.0/BETA without any problem (but without ZFS too). Since > I've turned off super-pages, ZFS is stable. I tested superpages on 7.2-STABLE with ZFS and had to deactivate them, after the server became instable. --Kai From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 14:28:13 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA46E106566C; Sat, 7 Nov 2009 14:28:13 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B0E078FC12; Sat, 7 Nov 2009 14:28:13 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nA7ESDNa097470; Sat, 7 Nov 2009 14:28:13 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nA7ESDhU097466; Sat, 7 Nov 2009 14:28:13 GMT (envelope-from gavin) Date: Sat, 7 Nov 2009 14:28:13 GMT Message-Id: <200911071428.nA7ESDhU097466@freefall.freebsd.org> To: gavin@FreeBSD.org, freebsd-fs@FreeBSD.org, freebsd-bugs@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: kern/140338: [vm][panic] FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 14:28:13 -0000 Old Synopsis: [vm] [zfs] [panic] FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld New Synopsis: [vm][panic] FreeBSD 8.0 RC2 with vm.pmap.pg_ps_enabled=1 kernel panic with makeworld Responsible-Changed-From-To: freebsd-fs->freebsd-bugs Responsible-Changed-By: gavin Responsible-Changed-When: Sat Nov 7 14:25:06 UTC 2009 Responsible-Changed-Why: Not ZFS related after all http://www.freebsd.org/cgi/query-pr.cgi?pr=140338 From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 20:18:31 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 076B5106566B; Sat, 7 Nov 2009 20:18:31 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id B01AA8FC08; Sat, 7 Nov 2009 20:18:30 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 7179C19E023; Sat, 7 Nov 2009 21:18:28 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 17B8D19E019; Sat, 7 Nov 2009 21:18:25 +0100 (CET) Message-ID: <4AF5D611.7060408@quip.cz> Date: Sat, 07 Nov 2009 21:18:25 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.4) Gecko/20091017 SeaMonkey/2.0 MIME-Version: 1.0 To: Ivan Voras References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> <4AEEBD4B.1050407@quip.cz> <4AEEDB3B.5020600@quip.cz> <4AF46CA9.1040904@quip.cz> <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> In-Reply-To: <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 20:18:31 -0000 Ivan Voras wrote: > 2009/11/6 Miroslav Lachman<000.fbsd@quip.cz>: > >> I do not understand why there are 10MB/s read from disks when network >> traffic dropped to around 1MB/s (8Mbps) >> >> root@cage ~/# iostat -w 20 >> tty ad4 ad6 cpu >> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id >> 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 >> 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 >> 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 >> 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 > > Yes, this could limit your IO if the requests are random enough. > Unfortunately I don't know how would you track down what is really > going on. Maybe some tracing with DTrace? > > I'd tell you to use "top -m io" to see if there is a process > responsible, but apparently these statistics are not updated for ZFS, > which in itself may be a bug (which is why I'm crossposting to > freebsd-fs). DTrace is totally out of my skills ;( There is otput of top -m io sorted by VCSW displaying JID. last pid: 17724; load averages: 0.01, 0.07, 0.08 up 74+20:49:49 21:03:40 195 processes: 1 running, 193 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 3.6% system, 0.4% interrupt, 96.1% idle Mem: 462M Active, 2385M Inact, 977M Wired, 21M Cache, 399M Buf, 100M Free Swap: 6144M Total, 2024K Used, 6142M Free PID JID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17681 8 www 657 64 0 0 0 0 0.00% lighttpd 17683 8 www 379 41 0 0 0 0 0.00% lighttpd 17680 8 www 136 5 0 0 0 0 0.00% lighttpd 17682 8 www 85 0 0 0 0 0 0.00% lighttpd 4689 1 90 10 0 0 0 0 0 0.00% fb_inet_server 3403 1 90 10 0 0 0 0 0 0.00% fb_inet_server 2632 1 90 10 0 0 0 0 0 0.00% fb_inet_server All four top consumers is Lighttpd workers. And as you noted, read, write, fault, total and percent are not updated on machine with ZFS, so I can't compare it with UFS2 based machine. Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS related things than me :]) Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 20:42:48 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CFB31065672; Sat, 7 Nov 2009 20:42:48 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.24]) by mx1.freebsd.org (Postfix) with ESMTP id 0686A8FC13; Sat, 7 Nov 2009 20:42:47 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 25so494505eya.9 for ; Sat, 07 Nov 2009 12:42:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type; bh=J+IeO8WFaYJU4R+atcehkRzY2PtOwLTC2bEdxOOsooU=; b=hzxJa8sPdIklQeigyZkpFfhF5btSI4mT/EPE0iFLfHgxlOkFT5XhSm1Eh/LVJGck7k C4OVZAzFDNrFFu0J2P+3Qk9T9y9AY1qBYEnmkAkzameH7cnieW8eDT3ZD211lwvSKXvl N9qRNB13rqN5b11KbsboSERpS8TgX1fQS/29Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; b=DKFbBzvf42L/ZwDipKg966SHkLh19iYTJAhobuS3mSPXg6kVLwlOBWKVaKl+R/EdzP HoKeiMoOJooIl1ZuN+9of1/p52FoQu8gtdkHkPCHKJ+vO0aHO1+bpcSL9Hq2gjU0Ptqh wfBFzV6SqwOBSmBAyUOPCuLucaZSZTdPJ0yQ8= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.91.82 with SMTP id g60mr1850027wef.98.1257626567097; Sat, 07 Nov 2009 12:42:47 -0800 (PST) In-Reply-To: <4AF5D611.7060408@quip.cz> References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> <4AEEBD4B.1050407@quip.cz> <4AEEDB3B.5020600@quip.cz> <4AF46CA9.1040904@quip.cz> <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> <4AF5D611.7060408@quip.cz> From: Ivan Voras Date: Sat, 7 Nov 2009 21:42:27 +0100 X-Google-Sender-Auth: 0ce8a3466e8bd976 Message-ID: <9bbcef730911071242m5ad91720xcccb7586c6848ffd@mail.gmail.com> To: Miroslav Lachman <000.fbsd@quip.cz> Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 20:42:48 -0000 2009/11/7 Miroslav Lachman <000.fbsd@quip.cz>: > > And as you noted, read, write, fault, total and percent are not updated on > machine with ZFS, so I can't compare it with UFS2 based machine. > Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS > related things than me :]) Not much... it depends on from where the stats are collected - there is a fair bit of file system infrastructure that ZFS bypasses and if these stats come from it, they cannot be collected. The stat is apparently updated around sys/kern/vfs_cluster.c: 233 . I'm not very familiar with this layer but since it uses struct buf and the ZFS doesn't use bufcache, this is probably one of the things that is bypassed, though it would be nice if it weren't since this code also defines and uses the vfs.write_behind and vfs.read_max sysctls. Also, since ZFS uses its own threads for IO and the stats are for curthread, it looks like it would maybe need careful work to actually assign the IO stats to the correct thread; otherwise it may be sufficient to add it to vdev_disk.c in vdev_disk_physio(). I don't really know this code and this is mostly mechanical analisys - it might be wrong. At least I'd like to read someone's comment about what is curthread in this code path.