From owner-freebsd-doc@FreeBSD.ORG Thu Mar 7 22:08:23 2013 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 98C8FFAE for ; Thu, 7 Mar 2013 22:08:23 +0000 (UTC) (envelope-from cr@caltel.com) Received: from mail1.caltel.com (mail1.caltel.com [66.102.144.6]) by mx1.freebsd.org (Postfix) with ESMTP id 694DB757 for ; Thu, 7 Mar 2013 22:08:22 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAIUOOVFCZpCq/2dsb2JhbABDxlF0gmtAPRABAQQYAwIBAgFYCAEBiA+aVKB2jVuBWwSDGQOIbI1fgR6ER4sMgyqBUQ X-IPAS-Result: Ap4EAIUOOVFCZpCq/2dsb2JhbABDxlF0gmtAPRABAQQYAwIBAgFYCAEBiA+aVKB2jVuBWwSDGQOIbI1fgR6ER4sMgyqBUQ X-IronPort-AV: E=Sophos;i="4.84,804,1355126400"; d="scan'208";a="16819753" Received: from host-170.a66-102-144.caltel.com (HELO codys-mac.local) ([66.102.144.170]) by smtp.caltel.com with ESMTP/TLS/DHE-RSA-CAMELLIA256-SHA; 07 Mar 2013 14:07:04 -0800 Message-ID: <51390F88.3070101@caltel.com> Date: Thu, 07 Mar 2013 14:07:04 -0800 From: Cody Ritts Organization: CalTel User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20130216 Thunderbird/17.0.3 MIME-Version: 1.0 To: freebsd-doc@freebsd.org Subject: Wiki / ZFS Booting Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 22:08:23 -0000 Hello all, I have some suggestions for the wiki regarding root on zfs. In short, I think it would be EXTREMELY helpful to consolidate all of the pages into two. 1. Booting to ZFS 2. Advanced ZFS partitioning The long story is that I have just spent several days documenting my efforts to get multiple systems to boot from ZFS. First, thank you to all who have contributed to the wiki, I would not have been able to get it done without you. That being said these are 11 pages I have found on the wiki: /RootOnZFS, /RootOnZFS/UFSBoot, /RootOnZFS/ZFSBootPartition, /RootOnZFS/ZFSBootSlice, /RootOnZFS/GPTZFSBoot, /RootOnZFS/GPTZFSBoot/Mirror, /RootOnZFS/GPTZFSBoot/RAIDZ1, /RootOnZFS/GPTZFSBoot/RAIDZ2, /RootOnZFS/GPTZFSBoot/9.0-RELEASE, /ZFSOnRoot, /ZFSOnRootWithZFSboot I have read all of those, plus another half dozen on the internet. While the steps are largely the same, they all seem to have their own little differences. I have copied and pasted the commands from more than one of them into virtual machines, and they would not boot. My overall experience was extremely draining. I expected that when I am copying and pasting commands from a wiki, and it does not boot, the problem is not with the commands. When there are different commands being run on one example from the other, I have to go research which one is right. I spent a lot of time chasing my tail. I don't want to just complain that it should be better, I would like to share the internal documentation that I put together from all of the web pages, man pages, and server testing. I am not 100% certain I am doing everything the "right" way, but I am 100% CERTAIN that I have COPY/PASTED each and every one of these commands into a FreeBSD-9.1-RELEASE LiveCD shell and then successfully booted. These commands cover all of the information that I needed to know that was spread out across the internet. Hopefully someone can find this information useful who has the time, inclination, and write access to the wiki. Cheers, Cody ###### Boot to LiveCD on 9.1-RELEASE media ###### ## -- I like copy/paste accuracy, so I enable sshd on the liveCD dhclient em0 mkdir /tmp/etc mount_unionfs /tmp/etc /etc sed -i '' 's/#PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config service sshd onestart passwd ###### GPT INSTALL ###### gpart create -s gpt ada0 gpart add -s 122 -t freebsd-boot -l boot0 ada0 gpart add -s 2048M -t freebsd-swap -l swap0 ada0 gpart add -t freebsd-zfs -l disk0 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot /dev/gpt/disk0 ###### MBR INSTALL ###### gpart create -s mbr ada0 gpart add -t freebsd ada0 gpart create -s bsd ada0s1 gpart add -s 51G -t freebsd-zfs ada0s1 gpart add -s 8G -t freebsd-swap ada0s1 glabel label disk0 /dev/ada0s1a glabel label swap /dev/ada0s1b gpart set -a active -i 1 ada0 gpart bootcode -b /boot/mbr ada0 dd if=/boot/zfsboot of=/dev/ada0s1 count=1 dd if=/boot/zfsboot of=/dev/ada0s1a skip=1 seek=1024 zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot /dev/label/disk0 ###### ZRAID/MIRROR GPT INSTALL ###### gpart create -s gpt ada0 gpart add -s 122 -t freebsd-boot -l boot0 ada0 gpart add -s 2048M -t freebsd-swap -l swap0 ada0 gpart add -t freebsd-zfs -l disk0 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 gpart create -s gpt ada1 gpart add -s 122 -t freebsd-boot -l boot1 ada1 gpart add -s 2048M -t freebsd-swap -l swap1 ada1 gpart add -t freebsd-zfs -l disk1 ada1 ## -- repeat for adaN gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 gmirror label -b load -F swap /dev/gpt/swap0 /dev/gpt/swap1 zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot mirror /dev/gpt/disk0 /dev/gpt/disk1 ## -- zpool create zroot [mirror|raidz1|raidz2|raidz3] /dev/gpt/disk0 ... /dev/gpt/diskN ###### COMMON INSTALL COMMANDS ######### zfs set mountpoint=/zroot zroot zfs set checksum=fletcher4 zroot zfs create -o mountpoint=legacy zroot/ROOT mount -t zfs zroot/ROOT /mnt tar -C /mnt -xppf /usr/freebsd-dist/base.txz tar -C /mnt -xppf /usr/freebsd-dist/kernel.txz zpool set bootfs=zroot/ROOT zroot zfs set freebsd:boot-environment=1 zroot/ROOT cp /tmp/zpool.cache /mnt/boot/zfs/ echo 'hostname="host"' >> /mnt/etc/rc.conf echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf echo 'zfs_load="YES"' >> /mnt/boot/loader.conf echo 'vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf ###### GPT INSTALL ###### cat << EOF > /mnt/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/gpt/swap0 none swap sw 0 0 EOF ###### MBR INSTALL ###### cat << EOF > /mnt/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/label/swap none swap sw 0 0 EOF ###### ZRAID/MIRROR GPT INSTALL ###### echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf cat << EOF > /mnt/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 EOF ###### CHROOT INSTALL COMMANDS ###### chroot /mnt passwd root tzsetup cd /etc/mail make aliases exit reboot ###### DISCUSSION ###### ## -- notes on swap -- swap can go on ZFS filesystem but can no longer be used as a dump device. ## -- mountpoint=legacy for zroot/ROOT release/9.1.0/tools/tools/zfsboottest/zfsboottest.sh "Dataset's mountpoint property should be set to 'legacy'" I would imagine this protects you against importing the pool on another system, and forgetting the altroot command. When you are set legacy, you MUST either have: echo vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf OR an entry in fstab (I have not tested this) ## -- File system layout -- There are a lot of really cool things you can do with zfs layout Please see [link to zfs Layout Wiki page] The only advanced thing we did here is moved the root file system so that /tmp /usr and /var do not inherit zfs commands from / ## -- notes on zpool.cache -- in 9.1-RELEASE -- without zpool.cache in this configuration, the the system will NOT boot. ## -- Exporting zroot -- in 9.1-RELEASE -- If you export zroot, your system will not boot. You MUST boot from liveCD, and re-import it to get it to boot.