From owner-freebsd-bugs@FreeBSD.ORG Wed Nov 3 09:40:11 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32DEC1065679 for ; Wed, 3 Nov 2010 09:40:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0F21E8FC17 for ; Wed, 3 Nov 2010 09:40:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oA39eAef080273 for ; Wed, 3 Nov 2010 09:40:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oA39eADg080272; Wed, 3 Nov 2010 09:40:10 GMT (envelope-from gnats) Resent-Date: Wed, 3 Nov 2010 09:40:10 GMT Resent-Message-Id: <201011030940.oA39eADg080272@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Emil Smolenski Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64226106566C for ; Wed, 3 Nov 2010 09:37:50 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 523B58FC14 for ; Wed, 3 Nov 2010 09:37:50 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id oA39boNQ045529 for ; Wed, 3 Nov 2010 09:37:50 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id oA39bo3F045528; Wed, 3 Nov 2010 09:37:50 GMT (envelope-from nobody) Message-Id: <201011030937.oA39bo3F045528@www.freebsd.org> Date: Wed, 3 Nov 2010 09:37:50 GMT From: Emil Smolenski To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't work X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Nov 2010 09:40:11 -0000 >Number: 151910 >Category: kern >Synopsis: [zfs] booting from raidz/raidz2 on ciss(4) doesn't work >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Nov 03 09:40:10 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Emil Smolenski >Release: FreeBSD 8.1-RELEASE >Organization: >Environment: FreeBSD 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: FreeBSD 8.1 installed using Fixit# environment on HP Proliant 185 G5 with HP SmartArray does not boot from ZFS pool in raidz or raidz2 setup. Mirror-based configurations work as expected. There are 6 disks, each configured on HP SmartArray as single disk RAID0 array. Thus there are 6 logical devices (da[0-5]). System information gathered from Fixit# environment: # dmesg (...) ciss0: port 0xe800-0xe8ff mem 0xdef00000-0xdeffffff,0xdeeff000-0xdeefffff irq 35 at device 0.0 on pci4 ciss0: PERFORMANT Transport ciss0: [ITHREAD] (...) da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 1430767MB (2930211632 512 byte sectors: 255H 32S/T 65535C) (...) da1 at ciss0 bus 0 scbus0 target 1 lun 0 (...) da2 at ciss0 bus 0 scbus0 target 2 lun 0 (...) da3 at ciss0 bus 0 scbus0 target 3 lun 0 (...) da4 at ciss0 bus 0 scbus0 target 4 lun 0 (...) da5 at ciss0 bus 0 scbus0 target 5 lun 0 (...) # diskinfo -v da0 da0 512 # sectorsize 1500268355584 # mediasize in bytes (1.4T) 2930211632 # mediasize in sectors 0 # stripesize 0 # stripeoffset 359094 # Cylinders according to firmware. 255 # Heads according to firmware. 32 # Sectors according to firmware. PAFGL0T9SXH13E # Disk ident. (...) # camcontrol devlist -v scbus0 on ciss0 bus 0: at scbus0 target 0 lun 0 (da0,pass0) at scbus0 target 1 lun 0 (da1,pass1) at scbus0 target 2 lun 0 (da2,pass2) at scbus0 target 3 lun 0 (da3,pass3) at scbus0 target 4 lun 0 (da4,pass4) at scbus0 target 5 lun 0 (da5,pass5) scbus1 on ciss0 bus 32: scbus-1 on xpt0 bus 0: <> at scbus-1 target -1 lun -1 (xpt0) # pciconf -lv (...) ciss0@pci0:4:0:0: class=0x010400 card=0x3234103c chip=0x3230103c rev=0x04 hdr=0x00 class = mass storage subclass = RAID (...) I've done several tests with different configurations. Common setup: Each logical device (array) has GPT scheme with following partitions: => 34 2930211565 da0 GPT (1.4T) 34 128 1 freebsd-boot (64K) 162 2097152 2 freebsd-swap (1.0G) 2097314 4194304 3 freebsd-zfs (2.0G) 6291618 2923919981 - free - (1.4T) (...) Each freebsd-zfs partition has GPT label: gpt/test0, gpt/test1, etc. ZFS pool is created with following default options: canmount=off, checksum=fletcher4, atime=off, setuid=off. There are some datasets with compress=lzjb or compress=gzip option set. Boot code is initialized on each disk this way: # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 Results from tests: 1) + One logical device -> works 2) + Mirror built on 2 logical devices -> works 3) + Mirror built on 6 LDs -> works 4) - RAIDZ2 built on 3 LDs -> doesn't work 5) - RAIDZ2 built on 6 LDs -> doesn't work 6) - RAIDZ built on 3 LDs -> doesn't work 7) - RAIDZ built on 6 LDs -> doesn't work 8) + RAIDZ built on 3 USB sticks on the same machine -> works 9) + RAIDZ built on 3 LDs on another machine (aacdX devices) -> works There are three different error messages I encountered. It depends on where bootcodes were built. a) Error message encountered when using bootcodes from Fixit# media: error 1 lba 32 error 1 lba 1 error 1 lba 32 error 1 lba 1 No ZFS pools located, can't boot b) Error message encountered when using bootcodes that I built myself: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS object directory Cant' find root filesystem - giving up ZFS: unexpected object set type 0 ZFS: unexpected object set type 0 FreeBSD/x86 boot Default: test:/boot/kernel/kernel boot: ZFS: unexpected object set type 0 c) I also saw message similar to shown above, but with "ZFS: can't read MOS" message. Example output from "status" command at "boot:" prompt (when error b appears): boot: status pool: test config: NAME STATE test ONLINE raidz1 ONLINE /dev/gpt/test0 ONLINE /dev/gpt/test1 ONLINE /dev/gpt/test2 ONLINE So, the bootcode sees healthy ZFS pool with all devs available, but it can't boot from it. More details on _working_ configuration (2) (mirror on 2 LDs): works# zpool list test NAME SIZE USED AVAIL CAP HEALTH ALTROOT test 1.98G 434M 1.56G 21% ONLINE - works# zpool status test pool: test state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 gpt/test0 ONLINE 0 0 0 gpt/test1 ONLINE 0 0 0 errors: No known data errors works# zdb -uuu test Uberblock magic = 0000000000bab10c version = 14 txg = 77 guid_sum = 2401146990467298568 timestamp = 1287650011 UTC = Thu Oct 21 08:33:31 2010 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:e90ca00:200> DVA[1]=<0:26037600:200> DVA[2]=<0:3e00ce00:200> fletcher4 lzjb LE contiguous birth=77 fill=169 cksum=b02658433:48b259d7157:f39e911b0c82:2286d07e38a6a3 More details on _NOT_working_ configuration (6) (raidz on 3 LDs): doesntwork# zpool list test NAME SIZE USED AVAIL CAP HEALTH ALTROOT test 5.97G 655M 5.33G 10% ONLINE - doesntwork# zpool status test pool: test state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1 ONLINE 0 0 0 gpt/test0 ONLINE 0 0 0 gpt/test1 ONLINE 0 0 0 gpt/test2 ONLINE 0 0 0 errors: No known data errors doesntwork# zdb -uuu test Uberblock magic = 0000000000bab10c version = 14 txg = 78 guid_sum = 8302404134133891378 timestamp = 1287656704 UTC = Thu Oct 21 10:25:04 2010 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:5b9a7000:400> DVA[1]=<0:a205d800:400> DVA[2]=<0:ea016000:400> fletcher4 lzjb LE contiguous birth=78 fill=159 cksum=a13038304:4232b1013de:dc9ba0c2751d:1f1144c425d7a4 Please see also possibly the same issue reported on mailing list: http://lists.freebsd.org/pipermail/freebsd-stable/2010-October/059610.html I also tried mm's mfsBSD with ZFSv15. System installed from it also doesn't boot. Same with FreeBSD 8.0-RELEASE. I can provide more details on this issue and test patches. >How-To-Repeat: 1. Install ZFS-only FreeBSD 8.1-RELEASE on HP SmartArray. Use raidz or raidz2. 2. Reboot. >Fix: >Release-Note: >Audit-Trail: >Unformatted: