From nobody Fri Apr 1 22:39:29 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8A21F1A4F86E for ; Fri, 1 Apr 2022 22:39:35 +0000 (UTC) (envelope-from lm@lukaszmoskala.pl) Received: from lukaszmoskala.pl (mail2.lukaszmoskala.pl [IPv6:2001:470:71:5c5::129:2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KVZqB3fpMz4fHc for ; Fri, 1 Apr 2022 22:39:34 +0000 (UTC) (envelope-from lm@lukaszmoskala.pl) Received: by lukaszmoskala.pl (Postfix, from userid 5555) id E50D460628; Sat, 2 Apr 2022 00:39:31 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 lukaszmoskala.pl E50D460628 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lukaszmoskala.pl; s=mail; t=1648852771; bh=5Xlq3mVPqQi0CpuA9G/EawRYBbiC8k2H4T4LcP45bU4=; h=Date:To:Cc:References:From:Subject:In-Reply-To:From; b=Hs+oe+RnQqlzIpH/2301t177oOLwIS0ErVX71lgb6KnQ+Tr0YyWhWH4FUBAWObWZx aNnNl7IHonU6Yve9v6ZEKM71wLYlTWUGxLv93xCxpSxNAsc1u3dS9vTunsozZ7Z1z1 P3jpX30AKoNGnQfsubxvExAzjtbedDVltkhu10eo= X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail2.lukaszmoskala.pl X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,NICE_REPLY_A, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.4 Received: from [IPV6:2001:470:612d:1337:f171:8c08:6007:5a65] (unknown [IPv6:2001:470:612d:1337:f171:8c08:6007:5a65]) by lukaszmoskala.pl (Postfix) with ESMTPSA id F2C10604C8; Sat, 2 Apr 2022 00:39:29 +0200 (CEST) Message-ID: <767ee719-287f-323d-ef18-4da3ef464d6c@lukaszmoskala.pl> Date: Sat, 2 Apr 2022 00:39:29 +0200 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Content-Language: en-US To: Mark Millard Cc: freebsd-arm@freebsd.org References: <18284b32-6d1d-864a-63a9-21b5fe72deb1@lukaszmoskala.pl> <83A1232D-38B7-46E1-8991-043AAC36CE42@yahoo.com> From: =?UTF-8?Q?=c5=81ukasz_Moska=c5=82a?= Subject: Re: Booting rock64 from USB SSD In-Reply-To: <83A1232D-38B7-46E1-8991-043AAC36CE42@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4KVZqB3fpMz4fHc X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=lukaszmoskala.pl header.s=mail header.b=Hs+oe+Rn; dmarc=pass (policy=none) header.from=lukaszmoskala.pl; spf=pass (mx1.freebsd.org: domain of lm@lukaszmoskala.pl designates 2001:470:71:5c5::129:2 as permitted sender) smtp.mailfrom=lm@lukaszmoskala.pl X-Spamd-Result: default: False [-2.75 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[lukaszmoskala.pl:s=mail]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:71:5c5::129:2]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[lukaszmoskala.pl:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[lukaszmoskala.pl,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-arm]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(1.25)[subject]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 9555 Lines: 221 W dniu 1.04.2022 o 22:52, Mark Millard pisze: > On 2022-Apr-1, at 06:51, Łukasz Moskała wrote: > >> Hi everyone, >> >> I want to boot my rock64 from SSD (because all sd cards I could find are crap). >> >> I followed those instructions to flash u-boot to SPI: https://github.com/ayufan-rock64/linux-build/blob/master/recipes/flash-spi.md >> >> I was then able to boot FreeBSD from SSD (dd FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SSD, just like it would be to SD card), but ethernet wasn't working. Actually, it could receive packets but no outgoing packets were sent. Not even ARP packets - switch didn't detect any device on that port, but link was UP >> >> I assumed it was something to do with u-boot, so I erased u-boot from flash, then DD FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SD card, and it worked without problems. >> >> Do I understand correctly that this means that FreeBSD on rock64 uses custom u-boot? Can I somehow flash it to SPI? >> >> Possible workaround that I can think of would be to put /boot on sd card, install u-boot to sd card, then put / on SSD. > > My memory is that USB2 booting could be more normal, in that the > U-Boot from ports could access USB2 storage but not USB3 storage, > at least when I set this up long ago. I wanted USB3 use, thus the > below details. > > I have U-Boot on the /media/ device below. As I remember, it could not > deal with the USB3 port, which is the one I wanted to use. The > FreeBSD kernel was the first stage that could deal with the USB3 port > back when I set up the Rock64 that I have access to. > > I've not had to change the basic structure since I set it up. There > could be aspects that could be different these days and I'd not know. > Think of any comments as potentially being old information. > > Note: mmcsd0 here is actually an eMMC device instead of the microsd > card slot contents, leaving the microsd card slot free for other uses. > But I'd used a microsd card this way before using the eMMC device > (as I remember anyway). > > # gpart show -p > => 63 244277185 mmcsd0 MBR (116G) > 63 32705 - free - (16M) > 32768 102312 mmcsd0s1 fat32lba [active] (50M) > 135080 28760 - free - (14M) > 163840 241172480 mmcsd0s2 freebsd (115G) > 241336320 2940928 - free - (1.4G) > > => 0 241172480 mmcsd0s2 BSD (115G) > 0 230686720 mmcsd0s2a freebsd-ufs (110G) > 230686720 10485760 - free - (5.0G) > > => 40 1953525088 da0 GPT (932G) > 40 532480 da0p1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 7340032 da0p2 freebsd-swap (3.5G) > 7874560 1048576 - free - (512M) > 8923136 23068672 da0p3 freebsd-swap (11G) > 31991808 2097152 - free - (1.0G) > 34088960 33554432 da0p4 freebsd-swap (16G) > 67643392 1740636160 da0p5 freebsd-ufs (830G) > 1808279552 4194304 da0p6 freebsd-swap (2.0G) > 1812473856 141051272 - free - (67G) > > For reference, from "gpart show -pl" : > > 67643392 1740636160 da0p5 Rock64root (830G) > > (The USB drive can boot other systems as well, with > widely varying amounts of RAM. Thus the efi partition > and the odd set of freebsd-swap partitions.) > > So, /media/ below is mmcsd0s1's fat32lba and /mnt/ is > mmcsd0s2a's freebsd-ufs. da0 is the USB3 SSD media, > which I do not give other details of here. (I manually > mounted these for this note.) > > # ls -Tld /media/*/*/* /mnt/* /mnt/etc/* > -r-xr-xr-x 1 root wheel 1243772 Jan 28 12:33:00 2022 /media/EFI/BOOT/bootaa64.efi > -r-xr-xr-x 1 root wheel 50618 Jan 28 12:32:28 2022 /media/dtb/rockchip/rk3328-rock64.dtb > -r--r--r-- 1 root wheel 6170 Feb 1 04:48:34 2020 /mnt/COPYRIGHT > drwxr-xr-x 23 root wheel 1536 Jan 28 15:26:41 2022 /mnt/boot > drwxr-xr-x 2 root wheel 512 Apr 26 14:39:22 2020 /mnt/etc > -rw-r--r-- 1 root wheel 37 Dec 31 16:00:18 2009 /mnt/etc/hostid > drwx------ 2 root wheel 33280 Nov 27 09:46:08 2019 /mnt/lost+found > > # ls -Tld /mnt/boot/dtb/overlays/rk3328-* > -r--r--r-- 1 root wheel 238 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-analog-sound.dtbo > -r--r--r-- 1 root wheel 1281 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-dwc3.dtbo > > (I make no use of rk3328-analog-sound.dtbo .) > > /mnt/boot/loader.conf has, in part, > > # A msdosfs /MNTPNT/dtb/rockchip/rk3328-rock64.dtb > # copy of the ufs /boot/dtb/rockchip/rk3328-rock64.dtb > # uses the intended DTB in u-boot and the kernel --and > # avoids needing to tell the kernel where to find a > # copy . . . > #rk3328_rock64_load="YES" > #rk3328_rock64_type="dtb" > #rk3328_rock64_name="/boot/dtb/rockchip/rk3328-rock64.dtb" > # > # rk3328 USB3-related: > fdt_overlays="rk3328-dwc3.dtbo" > # ucom is not automatically being loaded when umodem is loaded at boot. > ucom_load="YES" > umodem_load="YES" > # > vfs.root.mountfrom="ufs:/dev/gpt/Rock64root" > kern.cam.boot_delay=10000 > vfs.mountroot.timeout=10 > vfs.root_mount_always_wait=1 > > === > Mark Millard > marklmi at yahoo.com > Hi Mark, With some idea from what you did, I managed to do this: =================== BEGIN COMMAND LIST ============================ sudo pkg install u-boot-rock64 truncate -s 512M rock64-sd.img truncate -s 5G rock64-disk.img sudo mdconfig -a -t vnode -f FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img #returned md0 sudo mdconfig -a -t vnode -f rock64-sd.img #returned md1 sudo mdconfig -a -t vnode -f rock64-disk.img #returned md2 sudo gpart create -s gpt md1 sudo gpart add -t efi -b 32768 -s 50M -i 1 md1 sudo gpart add -t freebsd-ufs -i 2 md1 sudo newfs -O 2 -L bootfs /dev/md1p2 sudo dd if=/dev/md0p1 of=/dev/md1p1 bs=1M mkdir mp_old mp_new sudo mount -o ro /dev/md0p2 mp_old sudo mount -o rw /dev/md1p2 mp_new sudo rsync -rxav mp_old/boot/ mp_new/boot/ echo 'fdt_overlays="rk3328-dwc3.dtbo"' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.root.mountfrom="ufs:/dev/ufs/rootfs"' | sudo tee -a mp_new/boot/loader.conf echo 'kern.cam.boot_delay=10000' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.mountroot.timeout=10' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.root_mount_always_wait=1' | sudo tee -a mp_new/boot/loader.conf sudo gpart create -s gpt md2 sudo gpart add -t freebsd-ufs -a 4096 -i 1 md2 sudo newfs -O 2 -t -L rootfs /dev/md2p1 mkdir mp_newroot sudo mount -o rw /dev/md2p1 mp_newroot sudo rsync -rxav --exclude=boot mp_old/ mp_newroot/ sudo rm mp_newroot/etc/fstab sudo mkdir mp_newroot/media/sd echo '/dev/ufs/rootfs / ufs rw 1 1' | sudo tee -a mp_newroot/etc/fstab echo '/dev/ufs/bootfs /media/sd ufs rw,noatime 1 2' | sudo tee -a mp_newroot/etc/fstab echo '/dev/msdosfs/EFI /media/sd/boot/efi msdosfs rw,noatime 0 0' | sudo tee -a mp_newroot/etc/fstab echo 'tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0' | sudo tee -a mp_newroot/etc/fstab cd mp_newroot sudo ln -sf media/sd/boot boot cd .. sudo umount mp_newroot sudo umount mp_new sudo umount mp_old sudo dd if=/usr/local/share/u-boot/u-boot-rock64/idbloader.img of=/dev/md1 seek=64 bs=512 conv=sync sudo dd if=/usr/local/share/u-boot/u-boot-rock64/u-boot.itb of=/dev/md1 seek=16384 bs=512 conv=sync sudo mdconfig -d -u 2 sudo mdconfig -d -u 1 sudo mdconfig -d -u 0 =================== END COMMAND LIST ============================ (my apologies if some lines wrap improperly) then, DD rock64-sd.img to sdcard and rock64-disk.img to disk. It would be possible to do that directly on disk and sdcard itself, but in my case, machine on which I ran these commands doesn't have sd card reader. Anyway, it successfully booted with SSD plugged into USB3 port, despite those messages: Root mount waiting for: usbus0 usbus2 usbus3 usbus4 CAM usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR after which da0 was detected and boot continued. Network is working, and everything seems okay (uptime is 6 minutes as of moment I'm writing this). I'm using official pine64 usb-sata adapter, but I suppose it doesn't really matter. I don't have any other adapters to check. I copied 2.5GiB file to it and it seems stable. I got a lot of "devmatch: Can't read linker hints file." That was fixed by kldxref -R /media/sd/boot root@generic:~ # df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/rootfs 108G 4.2G 95G 4% / devfs 1.0K 1.0K 0B 100% /dev /dev/ufs/bootfs 432M 82M 316M 21% /media/sd /dev/msdosfs/EFI 50M 2.7M 47M 5% /media/sd/boot/efi tmpfs 50M 4.0K 50M 0% /tmp root@generic:~ # mount /dev/ufs/rootfs on / (ufs, local) devfs on /dev (devfs) /dev/ufs/bootfs on /media/sd (ufs, local, noatime) /dev/msdosfs/EFI on /media/sd/boot/efi (msdosfs, local, noatime) tmpfs on /tmp (tmpfs, local) root@generic:~ # gpart show -p => 40 1048496 mmcsd0 GPT (7.4G) [CORRUPT] 40 32728 - free - (16M) 32768 102400 mmcsd0p1 efi (50M) 135168 913368 mmcsd0p2 freebsd-ufs (446M) => 40 234441568 da0 GPT (112G) 40 4056 - free - (2.0M) 4096 234437512 da0p1 freebsd-ufs (112G) Thanks for your help, and I hope that whatever I made here will be usefull to someone. -- Łukasz Moskała