Date: Wed, 28 Aug 2019 05:55:41 -0400 From: Ricky Zhang <zhangchaowang@gmail.com> To: Ronald Klop <ronald-lists@klop.ws> Cc: freebsd-arm@freebsd.org Subject: Re: How to change rootfs from official RPI3 image Message-ID: <A8C596BB-AC79-4CD2-B70B-E1B3CD36743D@gmail.com> In-Reply-To: <op.z662cwh9kndu52@sjakie> References: <0FC8815E-58D7-4196-BF7E-0D6B127B314D@gmail.com> <1668312327.7.1566381369480@localhost> <C5D66D8B-05F2-445A-92A5-CD463B393EEA@gmail.com> <61580006.14.1566389702495@localhost> <D815D0C1-E17A-412F-AEF0-772E7B1DE857@gmail.com> <op.z662cwh9kndu52@sjakie>
next in thread | previous in thread | raw e-mail | index | archive | help
I don=E2=80=99t get any errors. It just waited indefinitely.=20 I already tried to set ZFS mount point with legacy option or real mount = point path. But both tries failed to boot. The module opensolaris_load=3D=E2=80=9CYES=E2=80=9D is required in = /boot/loader.conf. If I don=E2=80=99t add that, ZFS module didn=E2=80=99t = load properly during the boot. I saw an unknown file system error. In Linux, there is a initramfs where the kernel loads it as initial = rootfs during the boot process. After the boot setup is done in Linux, = the kernel switch from initramfs to real rootfs. Since I don=E2=80=99t = see similar things in FreeBSD, I assume there might be a bug in kernel = moudles to load ZFS from external USB SSD.=20 I can=E2=80=99t figure out a good solution to boot from rootfs in ZFS on = USB external SSD. For now, I split the SSD into two slices. The first = slice is UFS which contains roofs and home, while the second slice is = ZFS pool. With this workaround, I can start to learn to use Jail with = the help from iocage. PS: TBH, there are so many hardware limitations on RPI 3B+.=20 - The 1GB memory is too small for ZFS or running jail. I can=E2=80=99t = even clone FreeBSD port form Git repo due to OOM.=20 - The Ethernet is built on USB 2.0. The network speed is slow.=20 - USB 2.0 is slow when do disk I/O.=20 In addition, ARM64 is still 2nd tier support from FreeBSD. The port = security update may be out of date. Based on my observations, RPI 3B+ is not suitable for my experiment = purpose where I want to deploy a word press jail. I may keep it to do = some security experiments, instead. =20 thanks, Ricky > On Aug 27, 2019, at 2:44 PM, Ronald Klop <ronald-lists@klop.ws> wrote: >=20 > On Sat, 24 Aug 2019 12:15:35 +0200, Ricky Zhang = <zhangchaowang@gmail.com> wrote: >=20 >> Hi Ronald, >>=20 >> After migrating to UFS, I read from BSD Journal May/2019 = (https://www.freebsdfoundation.org/past-issues/freebsd-for-makers/). = RPI3 can boot from ZFS rootfs. >>=20 >> So I tried it on my own (see the step details below in method 2): >>=20 >> = https://github.com/rickyzhang82/FreeBSDWiki/wiki/1.-Migrate-SD-card-rootfs= -to-SSD-for-RPI3#method-2-sd-card-kernel-boot--ssd-zfs-rootfs >>=20 >> Unfortunately, it hangs when try to mount root from dataset = tank/rootfs. >>=20 >> I read between lines that the the author in the BSD Journal only = mount /var and /usr from ZFS dataset. The rest of rootfs still comes = from SD card ufs partition. >>=20 >> My way moved the whole rootfs and home to tank/rootfs and tank/home = dataset. When it boots, I saw my USB SSD LED is flashing periodically. = The screen prints =E2=80=9CTry to mount root from tank/roofs[]...". It = seems to me that the kernel struggled to find the zpool from SSD. I saw = that zfs module gets loaded properly from the screen. I have added two = module in /boot/loader.conf as Internet suggest: >>=20 >> ... >> opensolaris_load=3D"YES" >> zfs_load=3D"YES=E2=80=9D >> ... >>=20 >> If I unplug USB SSD without any settings change, hit Ctrl-C. I can = override mount root from command line at run time with = "ufs:ufs/rootfs=E2=80=9D rathe than =E2=80=9Czfs:tank/rootfs". Then I = can boot from SD card as before. Once I login, I can run zfs list and = show both tank/rootfs and tank/home data set. So that confirmed my ZFS = pool can be found. But somehow the boot process messed up somewhere. >>=20 >> Do you have any suggestion? >>=20 >> thanks >> Ricky >=20 > Mmm, I'm not sure. It is a pretty trivial setup. I think the error is = in the details. Trying this question on the freebsd-fs@ mailinglist = might help. > Do you get an error? Or is it waiting indefinitely? >=20 > What is the output of 'zfs get mountpoint'? >=20 > If you set vfs.root.mountfrom=3D"zfs:tank/rootfs" than you don't need = to specify / in /etc/fstab. > And ZFS will automatically mount the subdirs of rootfs depending on = the mountpoint setting of the ZFS volumes. >=20 > You might need to set: > zfs set mountpoint=3Dlegacy tank/rootfs > zfs set mountpoint=3D/home tank/home >=20 > If you want to test this you can use 'zpool import -R /mnt/tank tank' = to temporarily mount everything under the /mnt/tank subdirs. >=20 > BTW opensolaris_load is not necessary. It is a dependency of zfs_load. >=20 > Ronald. >=20 >=20 >>=20 >>=20 >>> On Aug 21, 2019, at 8:15 AM, Ronald Klop <ronald-lists@klop.ws> = wrote: >>>=20 >>> Thanks. I will not take the credits for the idea though. :-) >>> The FreeBSD wiki has some more examples of similar setups: >>> https://wiki.freebsd.org/ZFSOnRoot describes how to do /boot on UFS = and the rest on ZFS, which is similar to /boot on SD-card and the rest = on SSD. >>> https://wiki.freebsd.org/RootOnZFS/UFSBoot >>>=20 >>> Nice that you made a write up also. More exposure for FreeBSD is = always a good thing. :-) >>>=20 >>> Good luck and keep hacking. >>>=20 >>> Ronald. >>> Van: Ricky Zhang <zhangchaowang@gmail.com> >>> Datum: woensdag, 21 augustus 2019 13:16 >>> Aan: Ronald Klop <ronald-lists@klop.ws> >>> CC: freebsd-arm@freebsd.org >>> Onderwerp: Re: How to change rootfs from official RPI3 image >>>=20 >>> Your idea is brilliant. It solved the dilemma. >>>=20 >>> Ricky@router ~ $ sudo cat /etc/fstab >>> # Custom /etc/fstab for FreeBSD embedded images >>> /dev/ufs/rootfs /bootdir ufs rw 1 1 >>> /dev/label/gpt/ssdrootfs / ufs rw 1 1 >>> /dev/msdosfs/MSDOSBOOT /boot/msdos msdosfs rw,noatime 0 0 >>> tmpfs /tmp tmpfs rw,mode=3D1777,size=3D50m 0 0 >>>=20 >>> Ricky@router ~ $ mount >>> /dev/label/gpt/ssdrootfs on / (ufs, local, soft-updates) >>> devfs on /dev (devfs, local, multilabel) >>> /dev/ufs/rootfs on /bootdir (ufs, local, soft-updates) >>> /dev/msdosfs/MSDOSBOOT on /bootdir/boot/msdos (msdosfs, local, = noatime) >>> tmpfs on /tmp (tmpfs, local) >>>=20 >>> I wrote the whole thing down in my wiki: = https://github.com/rickyzhang82/FreeBSDWiki/wiki/1.-Migrate-SD-card-rootfs= -to-SSD-for-RPI3 >>>=20 >>> Thanks >>>=20 >>> Ricky >>>=20 >>> On Aug 21, 2019, at 5:56 AM, Ronald Klop <ronald-lists@klop.ws> = wrote: >>>=20 >>> Sorry, I did only reply to the mailinglist. I will use reply-all = now. >>>=20 >>> You are right about uboot finding /boot/kernel and = /boot/loader.conf. >>>=20 >>> What you need to do is mount the SD-card on /bootdir; in fstab: >>> /dev/yoursdcard /bootdir ufs rw,noatime 1 2 >>>=20 >>> And a symlink from /boot -> /bootdir/boot on your SSD. >>>=20 >>> Then installworld/installkernel will do the proper thing. >>> So the complete /boot stays on the SD-card. >>>=20 >>> Regards, >>>=20 >>> Ronald. >>> Van: Ricky Zhang <zhangchaowang@gmail.com> >>> Datum: woensdag, 21 augustus 2019 04:58 >>> Aan: freebsd-arm@freebsd.org >>> Onderwerp: Re: How to change rootfs from official RPI3 image >>>=20 >>> > BTW: Yes, u-boot is opensource: /usr/ports/sysutils/u-boot-rpi3 >>> > There are sysutils/u-boot-* ports for different system. With = sysutils/u-boot-master as the main part of it. >>> > >>> > https://www.freshports.org/sysutils/u-boot-rpi3 = <https://www.freshports.org/sysutils/u-boot-rpi3>; >>> > http://www.denx.de/wiki/U-Boot <http://www.denx.de/wiki/U-Boot>; >>> > >>> > Regards, >>> > Ronald. >>>=20 >>>=20 >>> Hi Ronald, >>>=20 >>> Sorry, if I messed up the mailing list thread. I didn=E2=80=99t = receive your email directly. Instead, I got your reply from daily = digest. I have to copy subject and quote your reply manually in my email = client. I have no idea how to fix it after reading all FAQ = (https://www.freebsd.org/doc/en_US.ISO8859-1/articles/mailing-list-faq/art= icle.html#etiquette = <https://www.freebsd.org/doc/en_US.ISO8859-1/articles/mailing-list-faq/art= icle.html#etiquette>). Any other mailing list I subscribed didn=E2=80=99t = work this way... >>>=20 >>> In any case, the magic works. I did rsync: >>>=20 >>> rsync -aAXvr --progress --delete /* /mnt/USB \ >>> --exclude=3D'/boot/msdos/*' \ >>> --exclude=3D'/dev/*' \ >>> --exclude=3D'/proc/*' \ >>> --exclude=3D'/net/*' \ >>> --exclude=3D'/tmp/*' \ >>> --exclude=3D'/mnt/*' \ >>> --exclude=3D'/media/*' >>>=20 >>> I confirmed that it mount ssd as rootfs: >>>=20 >>> Ricky@router ~ $ df -h >>> Filesystem Size Used Avail Capacity Mounted = on >>> /dev/label/gpt/ssdrootfs 407G 5.5G 369G 1% / >>> devfs 1.0K 1.0K 0B 100% /dev >>> /dev/msdosfs/MSDOSBOOT 50M 13M 37M 26% = /boot/msdos >>> tmpfs 50M 4.0K 50M 0% /tmp >>>=20 >>> As you said, kernel still comes from SD card. I don=E2=80=99t fully = understand the FreeBSD boot process. Neither am I familiar with UEFI. >>>=20 >>> - I guess /boot/msdos/uboot.bin finds the SD card roofs system. Load = the kernel from /boot/kernel in SD card and scan /boot/loader.conf to = find the rootfs. Please correct me if I=E2=80=99m wrong. >>> - Should I remove /boot folder from SSD to avoid confusion? >>> - Are there any guide how to compile and deploy kernel? >>>=20 >>>=20 >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-arm@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-arm >>> To unsubscribe, send any mail to = "freebsd-arm-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A8C596BB-AC79-4CD2-B70B-E1B3CD36743D>