Date: Wed, 20 Nov 2013 11:18:27 +0000 From: Richard Kearsley <richard@kearsley.me> To: freebsd-fs@freebsd.org Subject: boot zfs array from hba Message-ID: <528C9A83.1060606@kearsley.me>
next in thread | raw e-mail | index | archive | help
Hi
I am having some problems booting a raidz1 array from a lsi 2308 HBA on
a supermicro X9DRD-7LN4F-JBOD board
I believe the problem is that the bios only exposes one of the disks on
the HBA to the bootloader, resulting in the loader thinking the array is
degraded
FreeBSD must detect the disks itself at some point - through mps(4), but
it does not happen early enough in the boot procedure...
My question is there a trick, or something I'm missing, to be able to
boot an array from a HBA?
I'm using FreeBSD 9.1
The error I get at boot is as follows:
ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset u
ZFS: can't open root filesystem
gptzfsboot: failed to mount default pool ssd
FreeBSD/x86 boot
Default: ssd::
boot:
I can then run:
boot: status pool: ssd
which gives the output:
pool: ssd
config:
NAME STATE
ssd CLOSED
raidz1 OFFLINE
/dev/gpt/ssd0.nop ONLINE
/dev/gpt/ssd1.nop OFFLINE
/dev/gpt/ssd2.nop OFFLINE
/dev/gpt/ssd3.nop OFFLINE
/dev/gpt/ssd4.nop OFFLINE
I'm able to install in the same way if I move the 5 disks to the
on-board ports of the motherboard - and everything works as expected
Below is how i set up the array during install:
Clear old partition table if disks are not new:
`dd if=/dev/zero of=/dev/da0 bs=1M count=10`
`dd if=/dev/zero of=/dev/da1 bs=1M count=10`
`dd if=/dev/zero of=/dev/da2 bs=1M count=10`
`dd if=/dev/zero of=/dev/da3 bs=1M count=10`
`dd if=/dev/zero of=/dev/da4 bs=1M count=10`
Create gpt tables
`gpart create -s gpt da0`
`gpart create -s gpt da1`
`gpart create -s gpt da2`
`gpart create -s gpt da3`
`gpart create -s gpt da4`
Create boot partitions
`gpart add -s 222 -a 4k -t freebsd-boot -l boot0 da0`
`gpart add -s 222 -a 4k -t freebsd-boot -l boot1 da1`
`gpart add -s 222 -a 4k -t freebsd-boot -l boot2 da2`
`gpart add -s 222 -a 4k -t freebsd-boot -l boot3 da3`
`gpart add -s 222 -a 4k -t freebsd-boot -l boot4 da4`
Create swap partitions
`gpart add -s 1g -a 4k -t freebsd-swap -l swap0 da0`
`gpart add -s 1g -a 4k -t freebsd-swap -l swap1 da1`
`gpart add -s 1g -a 4k -t freebsd-swap -l swap2 da2`
`gpart add -s 1g -a 4k -t freebsd-swap -l swap3 da3`
`gpart add -s 1g -a 4k -t freebsd-swap -l swap4 da4`
Create zfs partitions:
`gpart add -a 4k -t freebsd-zfs -l ssd0 da0`
`gpart add -a 4k -t freebsd-zfs -l ssd1 da1`
`gpart add -a 4k -t freebsd-zfs -l ssd2 da2`
`gpart add -a 4k -t freebsd-zfs -l ssd3 da3`
`gpart add -a 4k -t freebsd-zfs -l ssd4 da4`
Clear any old zfs data
`dd if=/dev/zero of=/dev/gpt/ssd0 count=560 bs=512`
`dd if=/dev/zero of=/dev/gpt/ssd1 count=560 bs=512`
`dd if=/dev/zero of=/dev/gpt/ssd2 count=560 bs=512`
`dd if=/dev/zero of=/dev/gpt/ssd3 count=560 bs=512`
`dd if=/dev/zero of=/dev/gpt/ssd4 count=560 bs=512`
Install boot loader
`gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0`
`gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1`
`gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2`
`gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3`
`gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da4`
Align 4k sectors for ZFS
`gnop create -S 4096 /dev/gpt/ssd0`
`gnop create -S 4096 /dev/gpt/ssd1`
`gnop create -S 4096 /dev/gpt/ssd2`
`gnop create -S 4096 /dev/gpt/ssd3`
`gnop create -S 4096 /dev/gpt/ssd4`
Create zpool
`zpool create -f -m none -o altroot=/mnt -o cachefile=/tmp/zpool.cache
ssd raidz1 /dev/gpt/ssd0.nop /dev/gpt/ssd1.nop /dev/gpt/ssd2.nop
/dev/gpt/ssd3.nop /dev/gpt/ssd4.nop`
`zfs set atime=off ssd`
Create mounts
`zfs create -o mountpoint=/ ssd/root`
`zfs create -o mountpoint=/data/small-0 ssd/small-0`
`zfs create -o mountpoint=/data/large-2 ssd/large-2`
Set bootfs
`zpool set bootfs=ssd/root ssd`
exit, installer runs...
Open a shell? `NO`
Complete `Live CD`
Edit fstab at `/mnt/etc/fstab`
`/dev/gpt/swap0 none swap sw 0 0`
`/dev/gpt/swap1 none swap sw 0 0`
`/dev/gpt/swap2 none swap sw 0 0`
`/dev/gpt/swap3 none swap sw 0 0`
`/dev/gpt/swap4 none swap sw 0 0`
Edit `/mnt/boot/loader.conf`
`zfs_load="YES" `
`vfs.root.mountfrom="zfs:ssd/root"`
Edit `/mnt/etc/rc.conf`
`zfs_enable="YES"`
Zpool cache trick
`zpool export ssd`
`zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache ssd`
`cp /tmp/zpool.cache /mnt/boot/zfs/`
reboot...
Many thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?528C9A83.1060606>