From owner-freebsd-cloud@freebsd.org Tue Jan 8 07:46:41 2019 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FB701492E44; Tue, 8 Jan 2019 07:46:41 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1664873592; Tue, 8 Jan 2019 07:46:40 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: by mail-wr1-x443.google.com with SMTP id v13so3000703wrw.5; Mon, 07 Jan 2019 23:46:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=DRhuQGO45qsh81ol50JqZokobpbrXGIt57d5gOKcm+U=; b=BVrGINXZeZrPQAr8DrnzhYke62qS8cQg3R72c48TfTepucwlpVoatYNLFNGd0dCWXa Xwfra4u2B9OIGJx38djD2YaqFQ4ZupggcEwj9aDhy/rYj/QF6F0qwlYpIuS/aw7aBRUc xL0E9e/r8NruNNv04LT4ABUrn4ImeQ5EGjTq4cvV30EvOrc1q88OTSwwGvmnP3YSNjOI 4sjQcFrbNAhMsoCrym86DrO3ouIonbrp5yG/CZmHyeiDkIgF473RnPm6RJ3RTH3GaBS1 CZOi5nYsUNeiPoqUK/nN6vDAXVnCWV7THZ/JtkOTKhhCeOnG1vuaFChPoXICAeLGhnSj qVLg== X-Gm-Message-State: AJcUukecZwnXCdbfAR4wMNsFbMzGgo3vhMKk2dQH66veSLKzc1dcX7a6 29CFa+nWu/3ckcC2fl2gLldjQfr6IxI= X-Google-Smtp-Source: ALg8bN7VUvla2P8+pKn7BN7dM4kuEjWvo7CgL+4lpRTuRUWOG9y0mSnwYcX7ock7AbD4UMJETR5t3w== X-Received: by 2002:adf:f984:: with SMTP id f4mr577682wrr.234.1546933598698; Mon, 07 Jan 2019 23:46:38 -0800 (PST) Received: from asommer-mac.local ([2001:4c50:28e:400:f405:a433:ddbb:8f7d]) by smtp.googlemail.com with ESMTPSA id o82sm8401478wmo.29.2019.01.07.23.46.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jan 2019 23:46:37 -0800 (PST) Subject: Re: Any plans for EC2 AMIs with RootOnZFS? To: freebsd-fs@freebsd.org, Colin Percival , Ben Woods , freebsd-cloud@freebsd.org References: From: Andreas Sommer Message-ID: <94db0514-c19c-0472-dc61-7a217fe9cb50@googlemail.com> Date: Tue, 8 Jan 2019 08:46:37 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 1664873592 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.43 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[googlemail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.88)[-0.876,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.55)[ip: (0.93), ipnet: 2a00:1450::/32(-1.91), asn: 15169(-1.68), country: US(-0.08)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; SUBJECT_ENDS_QUESTION(1.00)[] X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2019 07:46:41 -0000 Hi Colin, could you share your config to automatically generate the ZFS-based AMI? I have an existing setup using Packer and bsdinstall, used on several staging systems. For an official AMI, I wouldn't expect anything but the default bsdinstall ZFS-on-root setup. Here's an example (does some more stuff, see repo's purpose): https://github.com/AndiDog/zfs-lockup/blob/master/http/installerconfig_zfs Cheers, Andreas On 2018-12-28 13:00, freebsd-fs-request@freebsd.org wrote: > Message: 4 > Date: Fri, 28 Dec 2018 01:43:45 +0000 > From: Colin Percival > To: Ben Woods , freebsd-cloud@freebsd.org, > "freebsd-fs@freebsd.org" > Subject: Re: Any plans for EC2 AMIs with RootOnZFS? > Message-ID: > <01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@email.amazonses.com> > > Content-Type: text/plain; charset=utf-8 > > [Adding freebsd-fs in the hope of finding more ZFS people.] > > On 12/23/18 3:51 PM, Colin Percival wrote: >> On 12/23/18 7:12 AM, Ben Woods wrote: >>> Is there any plans to provide official FreeBSD AMIs in the AWS Marketplace >>> which use RootOnZFS out of the box? >> >> Nothing concrete yet, but it's on my radar. Now that FreeBSD AMIs are being >> published by a separate (release engineering) account, I'm more comfortable >> with working on more "experimental" AMIs. (And if this turns out to be widely >> useful I'll talk to the release engineering team about adding it to the set >> they publish.) > > I've created an experimental FreeBSD 12.0 ZFS AMI in the us-east-1 region: > ami-0786f5b55d5aa573f > > Since I'm basically a ZFS newbie (I run it on my laptop, but only with the > default setup from the installer, and I've never used any of the fancy ZFS > features) I don't know if I've set everything up properly. So far I have > one piece of feedback, which is that I should have marked canmount=off for > zroot. > > Can people please try out this AMI and let me know if there's anything else > (ZFS-related) which I should fix? Some time next week I'll fix whatever > people notice and build AMIs for all the regions. > From owner-freebsd-cloud@freebsd.org Wed Jan 9 09:31:03 2019 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57E9F149CB6B for ; Wed, 9 Jan 2019 09:31:03 +0000 (UTC) (envelope-from 0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com) Received: from a8-237.smtp-out.amazonses.com (a8-237.smtp-out.amazonses.com [54.240.8.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5E187752E7 for ; Wed, 9 Jan 2019 09:31:02 +0000 (UTC) (envelope-from 0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=dqtolf56kk3wpt62c3jnwboqvr7iedax; d=tarsnap.com; t=1547026027; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=cORvlez6PwnYDnQ+lyJVp20/IK4HZibdIldXXP9pMtU=; b=Y3YxESELAL+ZCNdhQipp67B0KSTUSwWzcPwgS4JFQvbN6H7DaSPddEckey1GHTDw eEPkY9AqXO2lb34q/Wy2jaqBGaHOvMFGyQTZlhgXnsvnpaTejVxxjVFRr3cyuQnlMFU niGca732M2aDJCwbtKFOiQdptQL3vUYuXNDly91I= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1547026027; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=cORvlez6PwnYDnQ+lyJVp20/IK4HZibdIldXXP9pMtU=; b=F7eHjfULDxxzdY12mO5YD85UZA0nKVFIwnNtOXdrJK8g4qhAIMxO834tPp2qQf/t NxMl5z1JC6XoOOjAPnmpSq8+cTrM1u4/9iip41XZ3Kk50HgX6WtqjG3qPpSZ+6cvWCR cvSrW+P7SOtQWLGH1KGT7+T7bfb2irVKVQDZmro4= Subject: Re: Any plans for EC2 AMIs with RootOnZFS? To: Ben Woods Cc: "freebsd-fs@freebsd.org" , freebsd-cloud@freebsd.org References: <01000167dd7b16cd-58ee733f-c662-42d5-9c75-2301fbc8377a-000000@email.amazonses.com> <01000167f27b4c13-41c15731-ae4c-4658-820d-80c4d4fbb01c-000000@email.amazonses.com> From: Colin Percival Openpgp: preference=signencrypt Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= mQGhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgStCVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+iGEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEuQINBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFiEkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@email.amazonses.com> Date: Wed, 9 Jan 2019 09:27:07 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2019.01.09-54.240.8.237 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Rspamd-Queue-Id: 5E187752E7 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tarsnap.com header.s=dqtolf56kk3wpt62c3jnwboqvr7iedax header.b=Y3YxESEL; dkim=pass header.d=amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=F7eHjfUL; spf=pass (mx1.freebsd.org: domain of 0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com designates 54.240.8.237 as permitted sender) smtp.mailfrom=0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com X-Spamd-Result: default: False [-1.73 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:54.240.0.0/18]; DKIM_TRACE(0.00)[tarsnap.com:+,amazonses.com:+]; MX_GOOD(-0.01)[feedback-smtp.us-east-1.amazonses.com,feedback-smtp.us-east-1.amazonses.com,feedback-smtp.us-east-1.amazonses.com]; NEURAL_HAM_SHORT(-0.92)[-0.924,0]; FORGED_SENDER(0.30)[cperciva@tarsnap.com,0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; IP_SCORE(-2.10)[ip: (-2.50), ipnet: 54.240.8.0/21(-4.59), asn: 14618(-3.34), country: US(-0.08)]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14618, ipnet:54.240.8.0/21, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; FROM_NEQ_ENVFROM(0.00)[cperciva@tarsnap.com,0100016831efd1f5-6da3fa16-a86b-46ea-a1f4-8d9bb073153c-000000@amazonses.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[tarsnap.com:s=dqtolf56kk3wpt62c3jnwboqvr7iedax,amazonses.com:s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tarsnap.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[237.8.240.54.list.dnswl.org : 127.0.15.0]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2019 09:31:03 -0000 Hi Ben, On 1/5/19 4:39 AM, Ben Woods wrote: > On Fri, 28 Dec 2018 at 12:43, Colin Percival wrote: >> I've created an experimental FreeBSD 12.0 ZFS AMI in the us-east-1 region: >> ami-0786f5b55d5aa573f > > Thanks a lot for doing this - much appreciated. > > I have just given it a try, and have the following feedback for you: > > 1. ZFS DATASETS AND PROPERTIES SHOULD MATCH INSTALLER DEFAULT Oops. I thought I was doing this, but I guess the wiki page I was following (https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot) didn't quite match... > Looking at how you created the zpool and zfs datasets (using the "zpool > history zroot" command), the following changes would achieve this outcome: > [snip] Thanks! > 2. LOADER.CONF SETTINGS > The following settings should be included in /boot/loader.conf: > kern.geom.label.disk_ident.enable="0" > kern.geom.label.gptid.enable="0" > > 3. SYSCTL.CONF SETTINGS > The following settings should be included in /etc/sysctl.conf: > vfs.zfs.min_auto_ashift=12 Got it. > 4. THE GPT TABLE IS INITIALLY CORRUPT > I suspect this is because I used a 50Gb disk, and it hasn't been expanded > correctly. > The following error was in DMESG and I suspect is related: > GEOM: ada0: the secondary GPT header is not in the last LBA. Yes. This message appears whenever you launch an instance with a larger-than-default root disk; with UFS, the AMI promptly fixes the GPT and resizes the UFS filesystem. We can't avoid having the message (well, not without fiddling with the kernel bits to silence the warning) but we should fix the autoresizing to work with ZFS as well as UFS. > I was able to fix this with the following commands on firstboot: > # gpart recover ada0 > # gpart resize -i 2 ada0 > # zpool online -e zroot ada0p2 Can you look at /etc/rc.d/growfs (which currently only works for UFS, using the 'growfs' utility) and see if you can teach it to handle ZFS filesystems? Alternatively, maybe someone can teach growfs(8) to say "hey, this looks like a disk with ZFS... let's run 'zpool online -e' instead"? Either way, I don't think this is something I can fix for a FreeBSD 12.0 ZFS AMI, but if the growfs rc.d script can be made to do the right thing with ZFS then at least future releases will work. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid