From owner-freebsd-questions@FreeBSD.ORG Wed Feb 6 16:24:32 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C64816A41A for ; Wed, 6 Feb 2008 16:24:32 +0000 (UTC) (envelope-from bc@default.co.yu) Received: from kaitan.default.co.yu (kaitan.default.co.yu [87.237.201.133]) by mx1.freebsd.org (Postfix) with SMTP id 5A30C13C457 for ; Wed, 6 Feb 2008 16:24:31 +0000 (UTC) (envelope-from bc@default.co.yu) Received: (qmail 23278 invoked by uid 89); 6 Feb 2008 16:57:13 +0100 Received: from unknown (HELO ?10.5.7.108?) (bc@default.co.yu@213.198.226.187) by 0 with SMTP; 6 Feb 2008 16:57:13 +0100 From: Bogdan Culibrk To: freebsd-questions@freebsd.org Content-Type: text/plain Date: Wed, 06 Feb 2008 16:57:13 +0100 Message-Id: <1202313434.5415.209.camel@serafim> Mime-Version: 1.0 X-Mailer: Evolution 2.12.1 Content-Transfer-Encoding: 7bit Subject: Some ZFS experience X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:24:32 -0000 Hi list monkeys! I've been using ZFS on freebsd since August 2007. It was running on 4x 9GB SCSI drives in raidz1 configuration powered by some old bx133 motherboard with 2x CPUs on 350MHz with 512MB of RAM. Applications running on it were couple of jails spinning low traffic mail servers and mysqls. All was running "smooth" considering the amount of fancy hardware :) However about week or so I upgraded that box by placing all 4 drives in new machine with P4 grade CPU and 1GB of ram, *but* I failed to connect all 4 drives in new enviroment with 1 scsi cable (was too short to reach 4th drive) and decided to give it a shot and see what happens with 1 hdd offline (omg i was expecting that will work). What happened: 1. Base system running on gmirror volume consisting of 4 mirrors booted up normally in degraded mode with 3 of 4 drives online. 2. /data powered by raidz1 zfs was showing 3 drives, 2 online and 1 faulted. zfs list was showing that volume is unavailable due lack of spares. I'm not sure which drive I didnt connect, was it da0, da1, da2 or da3 so i tried one more combination by cabling 3 drives in little different fashion. This time 2 of 3 drives showed as faulted. I've left it and got extra cabling the next day. Connected all 4 drives and ZFS pool continued working normally like it was in previous box. What was wrong when 3 drives were connected? Wasnt supposed to raidz1 survive lack of 1 drive? Or I did something wrong there? I didnt do 'zpool export' prior to physical migration of the drives. I'm using FreeBSD 7.0-PRERELEASE pulled from cvsup before 2 weeks. No custom patches. conf files and dmesg follows: cat /boot/loader.conf geom_mirror_load="YES" zfs_load="YES" vfs.zfs.zil_disable="1" vfs.zfs.prefetch_disable="1" #vm.kmem_size="335708160" vm.kmem_size="512M" vm.kmem_size_max="512M" hw.ata.wc=0 FreeBSD anarki.default.co.yu 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #3: Thu Jan 31 13:27:25 CET 2008 dmesg: Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-PRERELEASE #3: Thu Jan 31 13:27:25 CET 2008 root@anarki.default.co.yu:/usr/obj/usr/src/sys/ANARKI Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Celeron(R) D CPU 3.06GHz (3081.50-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf64 Stepping = 4 Features=0xbfebfbff Features2=0xe41d AMD Features=0x20100000 AMD Features2=0x1 real memory = 1065287680 (1015 MB) avail memory = 1028866048 (981 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 3f6f0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 cpu0: on acpi0 p4tcc0: on cpu0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd007 mem 0xe2000000-0xe207ffff,0xd0000000-0xdfffffff,0xe2080000-0xe20bffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: detected 7932k stolen memory agp0: aperture size is 256M pcib2: at device 30.0 on pci0 pci2: on pcib2 ahc0: port 0xb000-0xb0ff mem 0xe1000000-0xe1000fff irq 18 at device 2.0 on pci2 ahc0: [ITHREAD] aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs re0: port 0xb400-0xb4ff mem 0xe1001000-0xe10010ff irq 21 at device 5.0 on pci2 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 00:1d:7d:32:fc:ab re0: [FILTER] isab0: at device 31.0 on pci0 isa0: on isab0 pci0: at device 31.3 (no driver attached) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xca7ff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounter "TSC" frequency 3081499911 Hz quality 800 Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle ZFS filesystem version 6 ZFS storage pool version 6 da0 at ahc0 bus 0 target 1 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 40.000MB/s transfers (20.000MHz, offset 63, 16bit) da0: Command Queueing Enabled da0: 17357MB (35548320 512 byte sectors: 255H 63S/T 2212C) da1 at ahc0 bus 0 target 2 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) da1: Command Queueing Enabled da1: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C) da2 at ahc0 bus 0 target 4 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) da2: Command Queueing Enabled da2: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C) da3 at ahc0 bus 0 target 12 lun 0 da3: Fixed Direct Access SCSI-3 device da3: 40.000MB/s transfers (20.000MHz, offset 63, 16bit) da3: Command Queueing Enabled da3: 17357MB (35548320 512 byte sectors: 255H 63S/T 2212C) GEOM_MIRROR: Device mirror/sys1s1 launched (4/4). Trying to mount root from ufs:/dev/mirror/sys1s1a