From owner-freebsd-arm@freebsd.org Thu May 6 20:09:36 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 393A862C9F5 for ; Thu, 6 May 2021 20:09:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fbl6R20ZTz51rX for ; Thu, 6 May 2021 20:09:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1620331773; bh=iIFheja4XncH7d3hP5PZc2BbkLdgAhIrszs+3bBXf8f=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=n5idhYTbQ40j8vVIQLyA8GbFt0LPhbBRyrahzas30JslTlkASkd+LXtmBeG6MKYclsf2Oh3z7Oj84BQmZwaKWEJdfuHSt0XNbhJMk2/2DGL0WUqSTTKm8bUmjX7qkHWE6gznslj9i1s0N+D2gqR/DaLrrlJ9ndC0Dvo2A5eyhFdlpSwymG1WKZgVi7MK3jXlWI6KOm+larRMhHDmXVSWIGH4sCQEEhwg5x23BK1wprIxlcOcRTDyEHmjGpgzBGbQD3OCzi2IM5Nq0zXWImFhZrCOopA3ZDrAsMLpznD+8UbRhC2hD2W95ET1xcFL/IJnYnrPiBEIwvB3apDgmPDjvg== X-YMail-OSG: 43ZAnOwVM1kyNKNTVrJs.IlhERUvwIjQU.9mbTDWW7Yb5VKrljZMTiirbFZM0HX UVWKIj1nHCrAe9qoy.WMCBkc75MVOzn.Nzt1gkDQBTKovpyOnFMY5VIvE0JrwC0J_8hSDpYZh5FG qrK8LsK3HuyLoiScd9dDQWL9HFXJnWQGB.k_qaw1xgbFvkFVXMUgIZqk0pKBesfItNJQ1hbwDN_I qBXAscYY9c1D1Yy0NFI6PHp0OjuB3vZ2_X24b_9myih2kAQnXZ6IrwnVPSQZuEt4d_oGNzcN4RO2 Wa9enRLtaCMVftU5k2U6Aufil7Cz1mJDQ9uJ_C0EudZYBQVMSzZbapNiQzuylyAaQo1YB8XzpY8B ei0KlhiMU6SNtM68y3VRoA8dtE0aAQS.sbouOuPHV0frPDgsUYgImkkDh6XYylC0_8JWnZJAkLpp Kctqd2LmPcCv2oZEIws0mtpJ0wBH4O.G4kCRirSjoGv6Bd8K.dIZY7yUvXml6Zl4Uf_KGvSuxZzZ cXnsJuWdTGrBvFf2wrbH2wkwMW_AvP.mQrPbypVItJDCk1xR_TXu6eP4p4f4CgKdz8rfHaigKCHm KliLsR88TZdTJS6G8As8bjzo1tbCrnXn4gteRdy4zRvK60jUmqQQYeMqPgX_yfTqYKHIhHVKl1IO WZzLWqYBi7S5xKnkAHfQkD3KBopBRMuQZ9tbG.IdGwziSiEYWJ1lHKvsLag.M5mC2PIRQgSa2BMX Cty5iOgNQhqS__GE_wrmN_bGS9z6_BI4j0HAmsgrM7mAdYIMTQzNJRiWegwz27Yve3zNOPdy6gBl T.uo.eq7Y.Pj2XHEH.tThbKcY6YQavcL0a.0woayGEQRvPMGIqav0mV9D.xk0GbE5xObyGmBnz2S chd1PZcHwY5hcISe6u62cYmxJZLjTseFoXtwmlXQgNQ7tZJooC5g6AuvOoSi8NYjWRXEyG9F.DBG gpr_Rnxm0eMw_j5u1EbMdosZXgkDjNZKWH1d.HE8TlZOsxMbaErMSKmSwHjyH2Za0gVxrds19vit Rnr1mimkuIJRYDeHGJttOYK06lOiq2XfA_vjK.2ePZQpwKlG0wpu_nU2q8WG8aVU862e31Uopyd3 pUWVUC1eCpHFHlslHLnM.xYwvQrA25dicfPq3gAMexZcMTEs6CNyJj7PA5lxqd9In9A62ooRXtWl I2307AXcpvS0ajJstU.ANSA3NQIQqadpbB8VDjTFcbKECRDiypyRk8XL8E3zToBfyB83Fk60FnSa A0nUsSMCoNAi6sjO4xV_baEjTsYOHzxjehjt4Qw.9HuWFiOu2bfcWIVu_TRJo4IRpIVbfZHWT3iD MZZO.287BfPoGkXjc2ezpJmjRkhG6ylwL9TWdPiqbBv2cjfs8_q_WHtJoaNlNOsdVIvsY7cfnig2 hwRxZb4UPQGA4iUcFVav7JLFNz5Q217.y5C.cetYeboH_m5PujS5YA.clbEMdoUS55FHt58eUCnu 497_C9qdK_xOAOKnnvie624XVF6mSkZqci9ORXJ16vR30ENxDkldB84p4LBAxisYIosWLZkDw.8a XNII.Hlp92AJLCqD9CpzGcnTngKanyldBBNat5qBksUoloym_lh_vcFBErfvycDJCeVLPMJ9Zacl .7jf.Ef523OLh2gFHfDrGtzRk1DcWLTgXZMxEexR8oigKhKC66krlJAQ7W5cq7V9J43hL.mZNgYq 8QBx9GdawNNwAoOE8G2jD.SudTJOrf6LDc4NDwCnKidPjGAPjHLwP2s_EFJXRNP.pVpOOmmgg0eO NTapOtGsMAONkPXNmMc2WaGovxOD61ZXxiY.d.LztH.DDTo5.ovzmFy5b_xK_iFj1NeWTVH7NRxn 8I4aE5lepSFgiVmqP0xgIjvTKcPxt.JuZvttaye.1a7SIMrog.QDqbE6AnsJBSD952VAaJVpuRfX rtSL9XZLVWzbnu1NxoK1TZr4XJZKEfLrSuWb0PGdqavHDhgOlJHoeXoJ9jhQp.eSvc3OOuo0Z0Jp ODBwam7v5aiZln3Ov_AZdNcriVEM4_ilQX0_ONKeLQI4yJDOn89u6f1TILDhoVfKfKHmYnj2fYDd x9kITMq1S9_66AEQJGKU_5vDPBPN1DKYvjW3t6l64yOXNDggqH2047Bb_3dKhU.JX1woobWo.0e1 CnHsK070TyxQUuCengLdKOBrr.jRqs2YpKcLNc8RPZLZlytTRQ2_hlOe8TVGJRdboFk2GcvL6ZJg Zgqy1K7FJugqmm_fA0U6K4PtgllpxZtRZeul8N.iREJWqpHcIawLAc4DDidk8husDg7ke8lOMT4B MjrRx9FjNOqUh9GciXfn.XCHY917BXxpYOJB.Q2jGIn_R71fgY9tUV.tJM04q0oimac2WzNwSFOj _HKsW5t_w9v8e8yhIaC1HsC84jvi2H5EYCYJxMvzPRhnldQySgBjLsoW77SdQSMNKf7qbLRRGbi_ YWAtBA3SEvsGjzW.KIXr49sBiyvaiOT3__iteViL5QdcZXmshoIiQXcyCgjOofbOax1P5qE8IoXl 5vSbAv9m2NrcNPaQpVmjficaWm.6C0jkW8gXqFb8gnkoLkDaAfncK7sn5V9O1WmZ7z9U.b0tiX5M .jgrHIvgteHL4PVgoJ4oOAJJwPJArGAoq1TURdNJBrxqnhFosm6I.t_HvDBTYkE.cJTg- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 May 2021 20:09:33 +0000 Received: by kubenode581.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID fd89a4a2e220b4ef441a0646408e5320; Thu, 06 May 2021 20:09:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: rpi4 zfs-on-root boot-to-usb3 [Example sequence that lead to booting zfs-on-root under GPT partitioning] From: Mark Millard In-Reply-To: Date: Thu, 6 May 2021 13:09:31 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <96F0BF83-D0FB-442F-B8A1-54194A724BD1@yahoo.com> References: To: tech-lists X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Rspamd-Queue-Id: 4Fbl6R20ZTz51rX X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.206:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.206:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.206:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting FreeBSD to ARM processors List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 May 2021 20:09:36 -0000 On 2021-May-6, at 12:12, Mark Millard wrote: > On 2021-May-6, at 05:55, Mark Millard wrote: >=20 >> On 2021-May-6, at 04:12, tech-lists wrote: >>=20 >>> How can zfs-on-root boot-to-usb3 on rpi4 be accomplished? >>>=20 >>> I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to be >>> problems with it that I can't work around. What's really needed is = an >>> installer, but these aren't made for arm64.aarch64 rpi4 from what I = can >>> see (I'm no expert though, it's entirely feasible i've missed >>> something). >>>=20 >>> Maybe one way of doing it would be to have a usb key (as ufs2) for = the >>> system to boot on, then have /home /usr/obj and other larger dirs on = the >>> usb3-zfs disk. >>=20 >> I used bsdinstall from booting a releng/13.0's release/13.0.0.0 >> microsd card in a 8 GiBYte RPi4B to produce the: >> . . . >=20 > Various details shown will just be my specific > choices. (The RPi4B's that I have access to have > the 2021-Apr-29 default(/critical) EEPROM image.) >=20 > Taking notes as I go (and readjusting as I > progress and figure things out, eliminating > failing attempts as well) . . . >=20 > Booting based on a microsd card with releng/13.0 's > release/13.0.0 as its basis. The context has a > working network with internet access. >=20 > # uname -apKU > FreeBSD generic 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 06:06:55 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 >=20 > Plug in USB3 SSD. Ends up as da0. >=20 > # /bin/sh # Just for my familiarity > # set -o vi >=20 > # mkdir -p /usr/freebsd-dist > # cd /usr/freebsd-dist > # fetch = http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST > MANIFEST 782 B 6147 = kBps 00s > # cd ~ >=20 > # bsdinstall >=20 > Continue with default keymap : Select >=20 > Enter hostname as ZFStest : OK >=20 > [*] base-dbg > [*] kernel-dbg > [ ] ports > [ ] src > [*] tests > then: OK >=20 > (Note: I use git for src and ports.) >=20 > Main Site : OK >=20 > Auto (ZFS) : OK >=20 > Pool Name : Select >=20 > Enter name for zpool ztstp : OK >=20 > Swap Size : Select >=20 > Enter swap size 24g : OK >=20 > Proceed with Installation : Select >=20 > Stripe - No Redundancy : OK >=20 > [*] da0 : OK >=20 > Last Chance for da0 : YES >=20 > Downloads . . . > Extracts . . . >=20 > New Password: . . . > Retype New Password: . . . >=20 > genet0 : OK >=20 > configure IPv4 : YES > configure DHCP : YES > configure IPv6 : YES > try SLAAC : YES > Resolver Configuration : OK >=20 > time is UTC? : YES >=20 > America : OK > United States of America : OK > Pacific : OK > Does PDT look reasonable? : Yes > May 2021 6 : Set Date > 11 07 00 : Set Time >=20 > [ ] local_unbound > [*] sshd > [ ] moused > [ ] ntpdate > [*] ntpd > [*] powerd > [*] dumpdev > Then : OK >=20 > No hardening options enabled : OK >=20 > Add uses? : Yes > . . . details omitted . . . > OK ? yes > Add another user? no >=20 > Handbook : OK > [*] en : OK >=20 > Apply configuration and exit installer : OK > open a shell : No I suggested an inappropriate later stage if one wants to try the same vintage of rpi-firmware and u-boot as releng/13.0 itself uses. One can copy over materials before the shutdown by getting them from /boot/msdos/ . Note that you might not want to copy over /boot/msdos/EFI/... as the install will already have such. So, something like: # mount -onoatime -tmsdosfs /dev/da0p1 /mnt # cp -aRx /boot/msdos/[LRa-z]* /mnt/ # umount /mnt then do the shutdown and remove the microsd card. > # shutdown -p now The following is only if one did not copy from /boot/msdosfs/ or one needs more recent materials, such as U-Boot for *C0T revision processors. > At this point it still can not boot an RPi4B > for lack of rpi firmware and U-Boot. >=20 > I have such available on other machine based > on the latest ports instead of quarterly. There > are RPi4B's in the world that need the more > modern U-Boot compared to the quarterly that > releng/13.0 is tied to by default. But you > likely could install rpi-firmware and > u-boot-rpi-arm64 on the microsd card and > then copy over materials from there. The above is dumb suggestion unless newer material is needed. See before the shutdown above. The below is me getting more recent materials (well, U-Boot) from another context. You might not do similarly. > In my context . . . >=20 > # gpart show -p da1 > =3D> 40 468862048 da1 GPT (224G) > 40 532480 da1p1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 50331648 da1p2 freebsd-swap (24G) > 50866176 417994752 da1p3 freebsd-zfs (199G) > 468860928 1160 - free - (580K) >=20 > # mount -onoatime -tmsdosfs /dev/da1p1 /mnt > # cp -aRx /usr/local/share/rpi-firmware/* /mnt/ > # cp -aRx /mnt/config_arm64.txt /mnt/config.txt=20 > # cp -aRx /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin /mnt/ > # umount /mnt (Note: It is possible to be more selective in what to copy over. I did not complicate the sequence with such handling.) Back to the normal flow on the RPi4B given appropriate RPi* materials copied over to the msdos file system . . . > Back to the RPi4B, no microsd card but plugging in the > USB3 SSD and booting and logging in: >=20 > Dec 31 16:00:48 ZFStest login[1351]: ROOT LOGIN (root) ON ttyu0 > FreeBSD 13.0-RELEASE (GENERIC) #0 releng/13.0-n244733-ea31abc261f: Fri = Apr 9 03:54:53 UTC 2021 >=20 > # gpart show -p > =3D> 40 468862048 da0 GPT (224G) > 40 532480 da0p1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 50331648 da0p2 freebsd-swap (24G) > 50866176 417994752 da0p3 freebsd-zfs (199G) > 468860928 1160 - free - (580K) >=20 > # uname -apKU > FreeBSD ZFStest 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 >=20 > I end up adding to /etc/rc.conf: >=20 > # > ntpd_sync_on_start=3D"YES" > ntpd_user=3D"root" >=20 > The first boot's time will be messed up for > lack of the ntpd_sync_on_start=3D"YES" . >=20 > # shutdown -r now >=20 > After login: >=20 > # ls -Tld /etc/rc.conf > -rw-r--r-- 1 root wheel 279 Dec 31 16:12:37 1969 /etc/rc.conf > # touch /etc/rc.conf >=20 > There are other files around with such an odd timestamp. >=20 > # zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT > ztstp 199G 1.09G 198G - - 0% 0% 1.00x = ONLINE - >=20 > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > ztstp 1.09G 192G 96K /ztstp > ztstp/ROOT 1.08G 192G 96K none > ztstp/ROOT/default 1.08G 192G 1.08G / > ztstp/tmp 96K 192G 96K /tmp > ztstp/usr 416K 192G 96K /usr > ztstp/usr/home 128K 192G 128K /usr/home > ztstp/usr/ports 96K 192G 96K /usr/ports > ztstp/usr/src 96K 192G 96K /usr/src > ztstp/var 680K 192G 96K /var > ztstp/var/audit 96K 192G 96K /var/audit > ztstp/var/crash 96K 192G 96K /var/crash > ztstp/var/log 200K 192G 200K /var/log > ztstp/var/mail 96K 192G 96K /var/mail > ztstp/var/tmp 96K 192G 96K /var/tmp >=20 > # more /etc/sysctl.conf=20 > # $FreeBSD$ > # > # This file is read when going to multi-user and its contents piped = thru > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for = details. > # >=20 > # Uncomment this to prevent users from seeing information about = processes that > # are being run under another UID. > #security.bsd.see_other_uids=3D0 > vfs.zfs.min_auto_ashift=3D12 >=20 > I'll note that in: >=20 > # more /boot/efi/config.txt=20 > [all] > arm_64bit=3D1 > dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don > dtoverlay=3Dmmc > dtoverlay=3Ddisable-bt > device_tree_address=3D0x4000 > kernel=3Du-boot.bin >=20 > [pi4] > hdmi_safe=3D1 > armstub=3Darmstub8-gic.bin >=20 > The hdmi)safe=3D1 line restricts the HDMI display > resolution/scaling. Any of the following > replacements for that line will avoid that but > in some contexts one could end up in a "blind > display" context instead, which is why hdmi_safe > is enabled by default. >=20 > hdmi_safe=3D0 > or: > #hdmi_safe=3D1 > or just delete the line. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)