From owner-freebsd-cloud@freebsd.org Sat Jan 5 12:40:02 2019 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62898149A143; Sat, 5 Jan 2019 12:40:02 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: from mail-vs1-xe2a.google.com (mail-vs1-xe2a.google.com [IPv6:2607:f8b0:4864:20::e2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 757EB80D55; Sat, 5 Jan 2019 12:40:01 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: by mail-vs1-xe2a.google.com with SMTP id n13so24233477vsk.4; Sat, 05 Jan 2019 04:40:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=AQ1EbwmDWQV0b+XFN98hU/MksRovIw46xxVuqASzUZY=; b=rMMmbpjGytK0MXcj+ZrGf6W8XGLZB2gj7v/fYENnoJ1H3pqHLdf0xAmpfSzes69kS1 ZRmyTywEm0I2JlBd4r3U724F7i9usCCEku+p6EBAbhP1+cIxUnDmBNDfSA6CNQMSr8Cr 80JmBvNcv+LJrAxJibATk+m35fyuJe5M+lduA0xrcq723Nzs1LO4J/SDWmGGTQ6CqzpA Zeebcvy+gEq5FoBytvkR/8MA6Nz6nSe1KmhM7eNBORBW4vOgLY4v9T5g9PLqPJ5+V9EG o7o5zg18KY9GM1UhKDvCmPFjngihNTXViQ/TCQHMlKuV6B7OCFMz2YtPNhyDopoMJdbV Vs+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=AQ1EbwmDWQV0b+XFN98hU/MksRovIw46xxVuqASzUZY=; b=gO2RkeHXByJ729NH0uItYUNck93JjwcSQanHN5OGyTjOQawy0Hefgpzk8jZBvWNU4r I/Fe9vq/Gk1ygQpqZRnEFaVuucll/O+ECItg40mngu+Ay16nUSOhunua4oO19dV5VO4g kIciBxAqnM6+cj3nLD8/mCp/2/R5olsOGIBGooj0ib05kaS/Y7/kQ4ZZ2sN87NUL94tt c91osT9PfWiD1O3cfbAyrfu/V4fjwhj0vbPWWRMtujRJzyo+QWs+vMbLlmsGdQ+1l/6J UviDF5umHhn+jVtwh/CPhzzG8+bCUehb/6s2Msbl1LO1C3/NyiDtzLfEXSm+labAbTvT aXBA== X-Gm-Message-State: AA+aEWbcOIPN0Ys43uUe2J+LTMaR6C9NfUIiXOmDJkSKu+EN03jqxKP6 u6vHGIQ4c+uBZXaFv2jy1zdgy4871x1w1EAmrio= X-Google-Smtp-Source: AFSGD/U0XWrsWxiFsQVEHUvaS/NjXqPns5rTcbjQFEROy+HqxkVNfZOIJoBwuOVyh7FWu/VtFe3arLUB3aaNsHwfpNY= X-Received: by 2002:a67:1505:: with SMTP id 5mr19585653vsv.20.1546692000537; Sat, 05 Jan 2019 04:40:00 -0800 (PST) MIME-Version: 1.0 References: <01000167dd7b16cd-58ee733f-c662-42d5-9c75-2301fbc8377a-000000@email.amazonses.com> <01000167f27b4c13-41c15731-ae4c-4658-820d-80c4d4fbb01c-000000@email.amazonses.com> In-Reply-To: <01000167f27b4c13-41c15731-ae4c-4658-820d-80c4d4fbb01c-000000@email.amazonses.com> From: Ben Woods Date: Sat, 5 Jan 2019 23:39:57 +1100 Message-ID: Subject: Re: Any plans for EC2 AMIs with RootOnZFS? To: Colin Percival Cc: freebsd-cloud@freebsd.org, "freebsd-fs@freebsd.org" X-Rspamd-Queue-Id: 757EB80D55 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=rMMmbpjG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of woodsb02@gmail.com designates 2607:f8b0:4864:20::e2a as permitted sender) smtp.mailfrom=woodsb02@gmail.com X-Spamd-Result: default: False [-4.17 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[5]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.43)[-0.431,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[a.2.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.73)[ip: (-9.71), ipnet: 2607:f8b0::/32(-2.19), asn: 15169(-1.67), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jan 2019 12:40:02 -0000 On Fri, 28 Dec 2018 at 12:43, Colin Percival wrote: > I've created an experimental FreeBSD 12.0 ZFS AMI in the us-east-1 region: > ami-0786f5b55d5aa573f > > Since I'm basically a ZFS newbie (I run it on my laptop, but only with the > default setup from the installer, and I've never used any of the fancy ZFS > features) I don't know if I've set everything up properly. So far I have > one piece of feedback, which is that I should have marked canmount=off for > zroot. > Hi Colin, Thanks a lot for doing this - much appreciated. I have just given it a try, and have the following feedback for you: 1. ZFS DATASETS AND PROPERTIES SHOULD MATCH INSTALLER DEFAULT To avoid any confusion or unexpected behaviour, I believe the list of ZFS datasets and their properties should exactly match the bsdinstall defaults (especially for releases). zpool create: https://svnweb.freebsd.org/base/head/usr.sbin/bsdinstall/scripts/zfsboot?revision=342283&view=markup#l1308 zfs datasets and properties: https://svnweb.freebsd.org/base/head/usr.sbin/bsdinstall/scripts/zfsboot?revision=342283&view=markup#l141 Looking at how you created the zpool and zfs datasets (using the "zpool history zroot" command), the following changes would achieve this outcome: --- zfs-commands.old +++ zfs-commands.new @@ -1,19 +1,15 @@ -zpool create -o altroot=/mnt zroot ada0p2 -zfs set compress=on zroot +zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot ada0p2 zfs create -o mountpoint=none zroot/ROOT zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp -zfs create -o canmount=off -o mountpoint=/usr zroot/usr +zfs create -o mountpoint=/usr -o canmount=off zroot/usr zfs create zroot/usr/home -zfs create -o exec=off -o setuid=off zroot/usr/src -zfs create zroot/usr/obj -zfs create -o mountpoint=/usr/ports -o setuid=off zroot/usr/ports -zfs create -o exec=off -o setuid=off zroot/usr/ports/distfiles -zfs create -o exec=off -o setuid=off zroot/usr/ports/packages -zfs create -o canmount=off -o mountpoint=/var zroot/var +zfs create -o setuid=off zroot/usr/ports +zfs create zroot/usr/src +zfs create -o mountpoint=/var -o canmount=off zroot/var zfs create -o exec=off -o setuid=off zroot/var/audit zfs create -o exec=off -o setuid=off zroot/var/crash zfs create -o exec=off -o setuid=off zroot/var/log -zfs create -o atime=on -o exec=off -o setuid=off zroot/var/mail -zfs create -o exec=on -o setuid=off zroot/var/tmp +zfs create -o atime=on zroot/var/mail +zfs create -o setuid=off zroot/var/tmp zpool set bootfs=zroot/ROOT/default zroot 2. LOADER.CONF SETTINGS The following settings should be included in /boot/loader.conf: kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" 3. SYSCTL.CONF SETTINGS The following settings should be included in /etc/sysctl.conf: vfs.zfs.min_auto_ashift=12 4. THE GPT TABLE IS INITIALLY CORRUPT I suspect this is because I used a 50Gb disk, and it hasn't been expanded correctly. The following error was in DMESG and I suspect is related: GEOM: ada0: the secondary GPT header is not in the last LBA. I was able to fix this with the following commands on firstboot: # gpart recover ada0 # gpart resize -i 2 ada0 # zpool online -e zroot ada0p2 The full background of my investigation and resolution is shown below: # gpart show => 40 20971440 ada0 GPT (50G) [CORRUPT] 40 1024 1 freebsd-boot (512K) 1064 984 - free - (492K) 2048 20967424 2 freebsd-zfs (10G) 20969472 2008 - free - (1.0M) # gpart recover ada0 ada0 recovered # gpart show => 40 104857520 ada0 GPT (50G) 40 1024 1 freebsd-boot (512K) 1064 984 - free - (492K) 2048 20967424 2 freebsd-zfs (10G) 20969472 83888088 - free - (40G) # gpart resize -i 2 ada0 ada0p2 resized # gpart show => 40 104857520 ada0 GPT (50G) 40 1024 1 freebsd-boot (512K) 1064 984 - free - (492K) 2048 104855512 2 freebsd-zfs (50G) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 9.50G 1.25G 8.25G - - 0% 13% 1.00x ONLINE - # zpool online -e zroot ada0p2 # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 49.5G 1.25G 48.3G - - 0% 2% 1.00x ONLINE - Thanks again for your efforts. Regards, Ben -- From: Benjamin Woods woodsb02@gmail.com