From owner-freebsd-arm@freebsd.org Thu May 6 19:12:52 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DDE9F62B6A0 for ; Thu, 6 May 2021 19:12:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fbjrz55Dhz501T for ; Thu, 6 May 2021 19:12:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1620328370; bh=o5xnJQxC7HiKLYDJSgdWvoACrIp0MJuWaHPvU2YK9oG=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=GA3StKvVbslKx/hW3sSPlVhoW1fp9kDRN3fmUBnN0c60ooSmn2vVNKhy6ToGbv5yZyDTr6OL+0xjixJmjtlpPI2EJrzh4T2wV+65PNeUSNuIP7G2OaQlTQU0kN1TU1uFqnhMC+aP4b98fJFRfBaP2/dJFvv1kO/YG7FHInuaXxVG8yZpndE8K22rasXZfjpsWUy3N+8UoJopeh9qg/Ze0iwFeJpCU8XxYU/xU7F6P9ngzqFMKA/o7Ll1cZqNnZ8M8GfZXIl4iUNmjyXg/5z1gfgbdoH35FSfjdPMciX8+YZCahX8+ixNmFkCHqjH6TVdoJErZC52vKeGPWJpErJ9FQ== X-YMail-OSG: LkWBElcVM1mMV72uG8ia5juaaG2sQGiZSy8WKFTAmH.NNSzuIx_unX5DbpwQhU3 bgC_Rl5wkNCEK2wK1qrkBug6x61D9bcrIzNutBPi8K2LcE7yBoXF7Wm99TquC8ai9hi.xN27UKD0 gULA8YKXtgx_aQLSG4PoHISNGg_yvMLPTJJ7kvM.WhCGzhfDEAXpKSXwdcDDcxWOKuXNRwHali4R cpaPThBrWnRKRbiQT4Rdfom0YZpAHz9BgQWXoqoQhE02x3RzTiONeSiDRZMcvB9xKw38nbb.amGD zb5M7eKKv99__Mpx4R.KfCn4GMrnJ7syiFY.zjlVWXpj6s5._iizWTOSxLYXzHuL6U4Yhm_CpQ8i 4ZJr1VW4jZpL4dib_iNzLfDmU1UzDaR_Fkq902ZU5VImhybYB995S0IsB3.6Lpx6kx4jClESb1G6 UuYg0_KVrNfi357c7w39CYNoX2_z.gXt3OZjUrw8Mh6K_m.qW.7UDLv4a.OX3JZpCCIdHX32uI56 sNEcF.VAeK54owiw6R5r2h_r1GPu0XdGdlYmYTGJNfyPIRDE_HvAo8.eE3cG7.Yecvu.1HWM3ISO UhH.2nfqkDXz7IQ1wRDgVaB1Ig_xhvNdd0ZoSLSTU.6kIHFEOrBjKXqE5kVDzi9Cw9osaEdPWpdd ELb1WAy7ykubs1bM4nVAS_fOzFuJX9lh0TU8Qr4LJ27WHXnKAv.ucRbb3tQMs.S0eg2lY0XFxjtk IYgrdqrYLQfK_SJdYl5F4E8VQGUcVjMMZ2tUNERBpKpIVb2hlM..75qveEiLVyIiurXmPx88mxH7 o4S6EC.w05jYKrEifUGWemJ.MRFPov402DV400jo3wg4nLnPEYgzjrq.bufK8vjGiTPW3taloBrV r4oBTMofzF2eTYmpd2gjiFcoGhzR8gF2cCwJc9hHTHtlR0BCMEJzISOg_kzPT6CAGrOKYHH8kmDb GETp6qyITuQG_g4lg6G6n41hg28Wnzlzi3Tf7Uz3k4IT1A8CTJdxDLoFTK5DlcVbIKg68I.X2j95 pu53nBCT.ZXX0cmxs0_19qhhMfuHdY2lLRKaAVNVKmgMTZ6BOA44qRMtwQ4Kwy18VyaJUVPT1abY AlFgS2F7IhgB5BeSFWJy3xURuLoKRTGOj1eZYsEyNU2CkMfHJ5k0KkI7WSoawHKvfDUMx89z8.8F 9tZJem_gGC0MuCBMu21mH13m4vltt8ZrNyp0Y1QTNgTtrlU3MbpvpN258I4susc5Vfe6Qb1KJb1Q GH93HkxcXAHP40w4CZv.M4KO3fGtOm1I5qS3NMKE6FZCyeHeZj7L0Kyu4mlno0OfvGw7Mnco1i68 VcxRTcXlPTolOaR90UdcFMfZAuQ0Cm9bRSDJbIj1bR7RKB1N6La9VmFEhoOohQQP_45RyASIZgA0 uWzp03gBxUYfZYJ88qCEmtM9ucBrMOHpbHkNsVKfX7C5CPKIg8.PCZDBNnmTNm9HU7m7sEOvqoAz NVEsIOKYZLL_oz2UqEZEcAZbBlmK21w52vP6TCZlRHP4z7wgljHUCO3t21vj30ECtoX0TxTmgQgI ZX8Co3bS1FAtlkBUye3Yd1qgV7sfjTneXeTPTPrv4.dOEmVoBJJGumClInyRFhgxvy9.xxLpwtk4 SXAsfxBCvQJkDDl0JWLB6VLOHaXTgXgF07VliJhA6qroxQh83DsmEmcLuzbYCHEJXl1SiQ8J2mrk l5BYT7AtoIxZgH7nLJHWAUFFJhwaZZ_Q4lv3ADT7J0j8szZI4JARVV3oFgpJySZc1zL9.hqQimC3 Fi7x27UVj7chkUeRit6y1EOhVwFpmw3HMQzMGwIncNuWB.4XTm36Ky88ov7kfnULs7dZk7JXp2qt Ir1oaa2R_Ao.o5Js0pQv2eGFRE9kfzuW6KI113qWVlAsrc333WhMTTfd3ZCxJOv2_XpIywqcrMFV iOKuKtWFjhsBNMjKr.bCW7BNMb95RfzVlO8j3HoM7Ymu9FnTV79lIbgFHifzKbUMJ_SW1XShlaGy XFLIA3yxMqwHaNJb_h9yVQdfgKpLDvtTvsQtQwZCo6dhxmfRnFQuUmobTtystNVZgwe.FBVetmFy h2Z8oRnOeMq1T5xN_TQqK5Psd34PhfL6wELGKtK_QSsp3039qhHaOTUZnuwBPBQTyITMRFvNY41t _1iOst4pFWmCQ19hud8mAwWQwX__EaCbWH9aNVrftxFiEMQg.ulzPD4O2b.uA8bzst18BHwhG4kV itWIWfhGRvxfaBfoneEv3oWD_ZeP2abX9Xn1rS0Six8QXRgfiHQYQNfnlSQGICPl2Mt5HDUfjtz3 i59__6Bqk4GcEr39je6r726OZpy3xo3Q1VQ8MkDAVG7tEE.E3x_4z9cRmq5WseSZEFb2_RfoIZxK 9wNdH5fJAmZd3uKI5QRBYfRKzGNwWFw2MIwhMOaQ87yxCh.av0g8aAM5cKfPmaz96n1yvI1F9mXq SZUQyOYQW1Sz.8MRWBq5W69l5qGKU1m2y1LawnmedgisNPj1ATX_rwo3nFJkxI8C1coDwSgI7vXy Q2wLSh9DIp56lxAERhI14RWyY5NRZQoMA3oKVcd4qAwkp98AIdIswyvP28cfnaP8r76cN3Eg7TQh 8gJAgCJjfu73YBA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 May 2021 19:12:50 +0000 Received: by kubenode548.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 17f3247fef2a0c0963accff32ec777d3; Thu, 06 May 2021 19:12:45 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: rpi4 zfs-on-root boot-to-usb3 [Example sequence that lead to booting zfs-on-root under GPT partitioning] From: Mark Millard In-Reply-To: Date: Thu, 6 May 2021 12:12:43 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: tech-lists X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Rspamd-Queue-Id: 4Fbjrz55Dhz501T X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.80)[-0.801]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.206:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.206:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.206:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting FreeBSD to ARM processors List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 May 2021 19:12:52 -0000 On 2021-May-6, at 05:55, Mark Millard wrote: > On 2021-May-6, at 04:12, tech-lists wrote: >=20 >> How can zfs-on-root boot-to-usb3 on rpi4 be accomplished? >>=20 >> I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to be >> problems with it that I can't work around. What's really needed is an >> installer, but these aren't made for arm64.aarch64 rpi4 from what I = can >> see (I'm no expert though, it's entirely feasible i've missed >> something). >>=20 >> Maybe one way of doing it would be to have a usb key (as ufs2) for = the >> system to boot on, then have /home /usr/obj and other larger dirs on = the >> usb3-zfs disk. >=20 > I used bsdinstall from booting a releng/13.0's release/13.0.0.0 > microsd card in a 8 GiBYte RPi4B to produce the: > . . . Various details shown will just be my specific choices. (The RPi4B's that I have access to have the 2021-Apr-29 default(/critical) EEPROM image.) Taking notes as I go (and readjusting as I progress and figure things out, eliminating failing attempts as well) . . . Booting based on a microsd card with releng/13.0 's release/13.0.0 as its basis. The context has a working network with internet access. # uname -apKU FreeBSD generic 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 06:06:55 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 Plug in USB3 SSD. Ends up as da0. # /bin/sh # Just for my familiarity # set -o vi # mkdir -p /usr/freebsd-dist # cd /usr/freebsd-dist # fetch = http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST MANIFEST 782 B 6147 kBps = 00s # cd ~ # bsdinstall Continue with default keymap : Select Enter hostname as ZFStest : OK [*] base-dbg [*] kernel-dbg [ ] ports [ ] src [*] tests then: OK (Note: I use git for src and ports.) Main Site : OK Auto (ZFS) : OK Pool Name : Select Enter name for zpool ztstp : OK Swap Size : Select Enter swap size 24g : OK Proceed with Installation : Select Stripe - No Redundancy : OK [*] da0 : OK Last Chance for da0 : YES Downloads . . . Extracts . . . New Password: . . . Retype New Password: . . . genet0 : OK configure IPv4 : YES configure DHCP : YES configure IPv6 : YES try SLAAC : YES Resolver Configuration : OK time is UTC? : YES America : OK United States of America : OK Pacific : OK Does PDT look reasonable? : Yes May 2021 6 : Set Date 11 07 00 : Set Time [ ] local_unbound [*] sshd [ ] moused [ ] ntpdate [*] ntpd [*] powerd [*] dumpdev Then : OK No hardening options enabled : OK Add uses? : Yes . . . details omitted . . . OK ? yes Add another user? no Handbook : OK [*] en : OK Apply configuration and exit installer : OK open a shell : No # shutdown -p now At this point it still can not boot an RPi4B for lack of rpi firmware and U-Boot. I have such available on other machine based on the latest ports instead of quarterly. There are RPi4B's in the world that need the more modern U-Boot compared to the quarterly that releng/13.0 is tied to by default. But you likely could install rpi-firmware and u-boot-rpi-arm64 on the microsd card and then copy over materials from there. In my context . . . # gpart show -p da1 =3D> 40 468862048 da1 GPT (224G) 40 532480 da1p1 efi (260M) 532520 2008 - free - (1.0M) 534528 50331648 da1p2 freebsd-swap (24G) 50866176 417994752 da1p3 freebsd-zfs (199G) 468860928 1160 - free - (580K) # mount -onoatime -tmsdosfs /dev/da1p1 /mnt # cp -aRx /usr/local/share/rpi-firmware/* /mnt/ # cp -aRx /mnt/config_arm64.txt /mnt/config.txt=20 # cp -aRx /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin /mnt/ # umount /mnt Back to the RPi4B, no microsd card but plugging in the USB3 SSD and booting and logging in: Dec 31 16:00:48 ZFStest login[1351]: ROOT LOGIN (root) ON ttyu0 FreeBSD 13.0-RELEASE (GENERIC) #0 releng/13.0-n244733-ea31abc261f: Fri = Apr 9 03:54:53 UTC 2021 # gpart show -p =3D> 40 468862048 da0 GPT (224G) 40 532480 da0p1 efi (260M) 532520 2008 - free - (1.0M) 534528 50331648 da0p2 freebsd-swap (24G) 50866176 417994752 da0p3 freebsd-zfs (199G) 468860928 1160 - free - (580K) # uname -apKU FreeBSD ZFStest 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 I end up adding to /etc/rc.conf: # ntpd_sync_on_start=3D"YES" ntpd_user=3D"root" The first boot's time will be messed up for lack of the ntpd_sync_on_start=3D"YES" . # shutdown -r now After login: # ls -Tld /etc/rc.conf -rw-r--r-- 1 root wheel 279 Dec 31 16:12:37 1969 /etc/rc.conf # touch /etc/rc.conf There are other files around with such an odd timestamp. # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT ztstp 199G 1.09G 198G - - 0% 0% 1.00x = ONLINE - # zfs list NAME USED AVAIL REFER MOUNTPOINT ztstp 1.09G 192G 96K /ztstp ztstp/ROOT 1.08G 192G 96K none ztstp/ROOT/default 1.08G 192G 1.08G / ztstp/tmp 96K 192G 96K /tmp ztstp/usr 416K 192G 96K /usr ztstp/usr/home 128K 192G 128K /usr/home ztstp/usr/ports 96K 192G 96K /usr/ports ztstp/usr/src 96K 192G 96K /usr/src ztstp/var 680K 192G 96K /var ztstp/var/audit 96K 192G 96K /var/audit ztstp/var/crash 96K 192G 96K /var/crash ztstp/var/log 200K 192G 200K /var/log ztstp/var/mail 96K 192G 96K /var/mail ztstp/var/tmp 96K 192G 96K /var/tmp # more /etc/sysctl.conf=20 # $FreeBSD$ # # This file is read when going to multi-user and its contents piped = thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for = details. # # Uncomment this to prevent users from seeing information about = processes that # are being run under another UID. #security.bsd.see_other_uids=3D0 vfs.zfs.min_auto_ashift=3D12 I'll note that in: # more /boot/efi/config.txt=20 [all] arm_64bit=3D1 dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don dtoverlay=3Dmmc dtoverlay=3Ddisable-bt device_tree_address=3D0x4000 kernel=3Du-boot.bin [pi4] hdmi_safe=3D1 armstub=3Darmstub8-gic.bin The hdmi)safe=3D1 line restricts the HDMI display resolution/scaling. Any of the following replacements for that line will avoid that but in some contexts one could end up in a "blind display" context instead, which is why hdmi_safe is enabled by default. hdmi_safe=3D0 or: #hdmi_safe=3D1 or just delete the line. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)