From owner-freebsd-current@FreeBSD.ORG Thu Mar 29 17:11:30 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DF6C416A402 for ; Thu, 29 Mar 2007 17:11:30 +0000 (UTC) (envelope-from mike@nux.co.uk) Received: from smtp.nildram.co.uk (smtp.nildram.co.uk [195.112.4.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4F34913C458 for ; Thu, 29 Mar 2007 17:11:30 +0000 (UTC) (envelope-from mike@nux.co.uk) Received: from office.nux.co.uk (unknown [82.133.40.67]) by smtp.nildram.co.uk (Postfix) with ESMTP id EF63B2BAA0B for ; Thu, 29 Mar 2007 17:46:58 +0100 (BST) Received: (qmail 90341 invoked by uid 2223); 29 Mar 2007 16:47:01 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 29 Mar 2007 16:47:01 -0000 Date: Thu, 29 Mar 2007 17:47:01 +0100 (BST) From: Mike Wolman X-X-Sender: mike@nux.eros.office To: freebsd-current@freebsd.org Message-ID: <20070329174620.Q90308@nux.eros.office> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Thu, 29 Mar 2007 19:20:21 +0000 Subject: ggated, gmirror + zfs panic X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 17:11:31 -0000 Hi, I am experimenting with ggate, gmirror and zfs to create a redundant file system using 2 machines. I currently do not have 10 spare drives so i have been using md devices to test the setup with. If I run through the below commands i am able to crash the machine every time. I have a vmcore available but think this should be easy to reproduce. On Secondary Machine # dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k # mdconfig -a -t vnode -f /home/mike-play/file0 -u 0 # mdconfig -a -t vnode -f /home/mike-play/file1 -u 1 # mdconfig -a -t vnode -f /home/mike-play/file2 -u 2 # mdconfig -a -t vnode -f /home/mike-play/file3 -u 3 # mdconfig -a -t vnode -f /home/mike-play/file4 -u 4 edit /etc/gg.exports 192.168.5.0/24 RW /dev/md0 192.168.5.0/24 RW /dev/md1 192.168.5.0/24 RW /dev/md2 192.168.5.0/24 RW /dev/md3 192.168.5.0/24 RW /dev/md4 Start ggated # ggated -v On Primary Machine # dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k # mdconfig -a -t vnode -f /home/mike-play/file0 -u 0 # mdconfig -a -t vnode -f /home/mike-play/file1 -u 1 # mdconfig -a -t vnode -f /home/mike-play/file2 -u 2 # mdconfig -a -t vnode -f /home/mike-play/file3 -u 3 # mdconfig -a -t vnode -f /home/mike-play/file4 -u 4 connect up the ggated exports: # ggatec create -u 0 192.168.5.212 /dev/md0 # ggatec create -u 1 192.168.5.212 /dev/md1 # ggatec create -u 2 192.168.5.212 /dev/md2 # ggatec create -u 3 192.168.5.212 /dev/md3 # ggatec create -u 4 192.168.5.212 /dev/md4 Setup the individual mirrors for each device: # gmirror label -v -b prefer zmirror0 /dev/ggate0 # gmirror insert -p90 zmirror0 /dev/md0 # gmirror label -v -b prefer zmirror1 /dev/ggate1 # gmirror insert -p90 zmirror1 /dev/md1 # gmirror label -v -b prefer zmirror2 /dev/ggate2 # gmirror insert -p90 zmirror2 /dev/md2 # gmirror label -v -b prefer zmirror3 /dev/ggate3 # gmirror insert -p90 zmirror3 /dev/md3 # gmirror label -v -b prefer zmirror4 /dev/ggate4 # gmirror insert -p90 zmirror4 /dev/md4 Create zfs pool from these mirrors: # zpool create tank raidz2 \ /dev/mirror/zmirror0 \ /dev/mirror/zmirror1 \ /dev/mirror/zmirror2 \ /dev/mirror/zmirror3 \ /dev/mirror/zmirror4 At this point the primary machine panics and enters the kernel debugger. If i exclude gmirror from the setup and only use the secondary machines md devices the zpool create works as expected: On Secondary machine: # dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k # dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k # mdconfig -a -t vnode -f /home/mike-play/file0 -u 0 # mdconfig -a -t vnode -f /home/mike-play/file1 -u 1 # mdconfig -a -t vnode -f /home/mike-play/file2 -u 2 # mdconfig -a -t vnode -f /home/mike-play/file3 -u 3 # mdconfig -a -t vnode -f /home/mike-play/file4 -u 4 edit /etc/gg.exports 192.168.5.0/24 RW /dev/md0 192.168.5.0/24 RW /dev/md1 192.168.5.0/24 RW /dev/md2 192.168.5.0/24 RW /dev/md3 192.168.5.0/24 RW /dev/md4 Start ggated # ggated -v On Primary machine: # ggatec create -u 0 192.168.5.212 /dev/md0 # ggatec create -u 1 192.168.5.212 /dev/md1 # ggatec create -u 2 192.168.5.212 /dev/md2 # ggatec create -u 3 192.168.5.212 /dev/md3 # ggatec create -u 4 192.168.5.212 /dev/md4 # zpool create tank raidz2 \ /dev/ggate0 \ /dev/ggate1 \ /dev/ggate2 \ /dev/ggate3 \ /dev/ggate4 # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 4.84G 206K 4.84G 0% ONLINE - f7# dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT-200703 #0: Wed Mar 28 12:34:58 BST 2007 mike@f7.eros.office:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) XP 3200+ (2194.51-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x6a0 Stepping = 0 Features=0x383fbff AMD Features=0xc0400800 real memory = 1056899072 (1007 MB) avail memory = 1024897024 (977 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xd8000000-0xdbffffff,0xdc000000-0xdcffffff irq 16 at device 0.0 on pci1 re0: port 0x9000-0x90ff mem 0xde010000-0xde0100ff irq 16 at device 8.0 on pci0 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 00:0e:2e:92:7d:0a re0: [FILTER] fwohci0: port 0x9400-0x947f mem 0xde011000-0xde0117ff irq 17 at device 9.0 on pci0 fwohci0: [ITHREAD] fwohci0: OHCI version 1.0 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:30:67:00:00:04:c4:15 fwohci0: Phy 1394a available S400, 3 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:30:67:04:c4:15 fwe0: Ethernet address: 02:30:67:04:c4:15 fwe0: if_start running deferred for Giant sbp0: on firewire0 fwohci0: Initiate bus reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) atapci0: port 0x9800-0x9807,0x9c00-0x9c03,0xa000-0xa007,0xa400-0xa403,0xa800-0xa80f,0xac00-0xacff irq 20 at device 15.0 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb000-0xb00f at device 15.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] uhci0: port 0xb400-0xb41f irq 21 at device 16.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xb800-0xb81f irq 21 at device 16.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xbc00-0xbc1f irq 21 at device 16.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xc000-0xc01f irq 21 at device 16.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xde012000-0xde0120ff irq 21 at device 16.4 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered isab0: at device 17.0 on pci0 isa0: on isab0 pci0: at device 17.5 (no driver attached) vr0: port 0xc800-0xc8ff mem 0xde013000-0xde0130ff irq 23 at device 18.0 on pci0 miibus1: on vr0 ukphy0: PHY 1 on miibus1 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto vr0: using obsoleted if_watchdog interface vr0: Ethernet address: 00:e0:4c:d5:10:a5 vr0: [ITHREAD] acpi_tz0: on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] pmtimer0 on isa0 orm0: at iomem 0xc8000-0xcffff pnpid ORM0000 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2194511372 Hz quality 800 Timecounters tick every 1.000 msec ad0: 39205MB at ata0-master UDMA133 acd0: CDROM at ata1-slave UDMA33 Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted ZFS filesystem version 3 ZFS storage pool version 3