Date: Sun, 19 Jul 2020 16:39:19 -0700 From: Mark Millard <marklmi@yahoo.com> To: Robert Crowston <crowston@protonmail.com> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: FYI: RPi4 (8 GiByte) USB3 vs. head -r363123: still a no-go for booting a USB3 / in my experiments Message-ID: <37CA77CB-8B78-4219-B6F8-64158826BC26@yahoo.com> In-Reply-To: <wV8i_sck3uM7ZuLiK-OrDeHjPMuvCee2dHGczONShToQ3GCWe_5Bz-mwTXw-YSKQPi2Gxq4Z7UxckXjU6RnK4DYTpvQwyJAPu3oS1lSQYNo=@protonmail.com> References: <BF48971B-7D34-415B-BDA2-00CF82ED1342.ref@yahoo.com> <BF48971B-7D34-415B-BDA2-00CF82ED1342@yahoo.com> <dku0j_wjTui_MeLV0obVoPHTA6AJ0JfAamjVb503rQbeTbdd2xUoG8JrZQB5ENSp4PeSj6estzC0pBYBw7expbvw_v4gd6pDIVo7aOBZApc=@protonmail.com> <64A12BF8-9E2D-441D-8765-52B6D3907A58@yahoo.com> <7J4Opq5TWFEhuN7xv3tq206mMjbDK67p35GadBDuaJRKtDuObVfk7M8ItEEUgumC_jKkoMqC2ZJVUCpNifY579oBq1UDthM1wXjG3tBBTj0=@protonmail.com> <5C8DD645-5C24-4155-AA4C-819AF3AF3B3D@yahoo.com> <wV8i_sck3uM7ZuLiK-OrDeHjPMuvCee2dHGczONShToQ3GCWe_5Bz-mwTXw-YSKQPi2Gxq4Z7UxckXjU6RnK4DYTpvQwyJAPu3oS1lSQYNo=@protonmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-Jul-19, at 15:08, Robert Crowston <crowston at protonmail.com> =
wrote:
>> Certainly makes it interesting to figure out what
>> combination of things should be used overall.
>=20
> Sorry, yes, the dtb I mostly use is ahead of FreeBSD's pkg content. =
The 8 GB rpi4 model I obtained refuses to boot with the start4.elf and =
other boot firwmare from November, so I just obtained the newest version =
for the whole boot partition. i.e., the newest versions of those files =
from the Raspberry Pi Foundation. That broke ethernet (by moving to =
rgmii-rxid) which I then fixed in D25251. I appreciate this complicates =
the situation.
The uefi context requires more recent than even June,
no official tab yet. I'd started with what I use there
but may need to explore alternatives for u-boot-rpi4
based contexts for u-boot-rpi4 based operation.
>> Until such a time, is there a good way to limit a RPi4 to
>> the lower <=3D 3 GiBytes overall for the u-boot case?
>=20
> You could set hw.physmem=3D"3G" in /boot/loader.conf. I don't know if =
that guarantees FreeBSD will specifically see the lowest 3 GB. The bus =
tag should not be too hard to fix though.
Hmm. Seemed to do nothing:
# vmstat
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us =
sy id
0 0 0 301M 7.5G 996 0 3 0 1.5K 5 0 0 10236 1.8K 18K 2 =
4 94
So, using set hw.physmem=3D"3G" at the loader prompt and then
booting got me to what I report below. But, for all I know
it could be random variations not related to the set that
I typed. What I report below seems to have done fairly
extensive file system damage as seen in attempting to
boot afterwards. So far this way of operating does not
seem viable, at least given the firmware vintage I'm using.
login: root
ld-elf.so.1: /usr/bin/login: Shared object has no run-time symbol table
FreeBSD/arm64 (Rock64orRPi4) (ttyu0)
login: root
ld-elf.so.1: /usr/bin/login: Shared object has no run-time symbol table
FreeBSD/arm64 (Rock64orRPi4) (ttyu0)
login: panic: ufs_dirbad: /: bad dir ino 12760706 at offset 0: mangled =
entry
cpuid =3D 2
time =3D 1595201136
KDB: stack backtrace:
db_trace_self() at db_trace_self_wrapper+0x28
pc =3D 0xffff00000082eb1c lr =3D 0xffff00000010e688
sp =3D 0xffff00006cb7c010 fp =3D 0xffff00006cb7c210
db_trace_self_wrapper() at vpanic+0x194
pc =3D 0xffff00000010e688 lr =3D 0xffff00000047b428
sp =3D 0xffff00006cb7c220 fp =3D 0xffff00006cb7c270
vpanic() at panic+0x44
pc =3D 0xffff00000047b428 lr =3D 0xffff00000047b290
sp =3D 0xffff00006cb7c280 fp =3D 0xffff00006cb7c320
panic() at ufs_lookup_ino+0xbb4
pc =3D 0xffff00000047b290 lr =3D 0xffff00000079943c
sp =3D 0xffff00006cb7c330 fp =3D 0xffff00006cb7c410
ufs_lookup_ino() at vfs_cache_lookup+0xc4
pc =3D 0xffff00000079943c lr =3D 0xffff000000552ae0
sp =3D 0xffff00006cb7c420 fp =3D 0xffff00006cb7c490
vfs_cache_lookup() at lookup+0x5f4
pc =3D 0xffff000000552ae0 lr =3D 0xffff00000055d7e8
sp =3D 0xffff00006cb7c4a0 fp =3D 0xffff00006cb7c510
lookup() at namei+0x464
pc =3D 0xffff00000055d7e8 lr =3D 0xffff00000055ccd4
sp =3D 0xffff00006cb7c520 fp =3D 0xffff00006cb7c610
namei() at kern_chdir+0x4c
pc =3D 0xffff00000055ccd4 lr =3D 0xffff0000005794a0
sp =3D 0xffff00006cb7c620 fp =3D 0xffff00006cb7c790
kern_chdir() at do_el0_sync+0x3bc
pc =3D 0xffff0000005794a0 lr =3D 0xffff00000084e0cc
sp =3D 0xffff00006cb7c7a0 fp =3D 0xffff00006cb7c830
do_el0_sync() at handle_el0_sync+0x90
pc =3D 0xffff00000084e0cc lr =3D 0xffff000000831224
sp =3D 0xffff00006cb7c840 fp =3D 0xffff00006cb7c980
handle_el0_sync() at 0x212168
pc =3D 0xffff000000831224 lr =3D 0x0000000000212168
sp =3D 0xffff00006cb7c990 fp =3D 0x0000ffffffffebf0
KDB: enter: panic
[ thread pid 1299 tid 100145 ]
Stopped at 0x403b2910
db>=20
> -- RHC.
>=20
> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 =
Original Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=
=80=90
> On Sunday, 19 July 2020 22:51, Mark Millard <marklmi@yahoo.com> wrote:
>=20
>>=20
>>=20
>> On 2020-Jul-19, at 04:30, Robert Crowston <crowston atprotonmail.com> =
wrote:
>>=20
>>> The ethernet problem could be caused by the recent change in =
upstream dtb, to change the ethernet driver to operate in rgmii-rxid =
mode instead of rgmii mode. See
>>> https://reviews.freebsd.org/D25251. I have not noticed this =
performance problem you report myself, when using upstream dtbs from =
June 2020.
>>=20
>> https://reviews.freebsd.org/D25251 was added to head as
>> -r362353 and my testing was based on head -r3630123 .
>> So my report is based on D25251.
>>=20
>> I do see that sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts
>> is base on Linux 5.5 (checked-in to FreeBSD 4 months ago).
>>=20
>> But the port Makefile sysutils/rpi-firmware/Makefile indicates:
>> FW_TAG=3D 2042453 for PORTVERSION=3D 1.20190925.g20200109 .
>> Looking on github 2042453 is the 2019-Nov-22 commit for
>> "kernel: Bump to 4.19.85". It seems to be from between official
>> Tags 1.0290925 and 1.20200114 .
>>=20
>> The most recent official Tag is 1.20200601+arm64 and has
>> materials in part based on "kernel: Bump to 5.4.42". (That
>> is a raspberrypi/linux number.)
>>=20
>> The most recent "kernel: Bump to" for master is for 5.4.51
>> (committed on 2020-Jul-13). One of the items in that is:
>>=20
>> kernel: ARM: dts: Make bcm2711 dts more like 5.7
>>=20
>> (I assume 5.7 is an upstream linux version indication.)
>> So newer than where
>> sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts is based.
>>=20
>> Certainly makes it interesting to figure out what
>> combination of things should be used overall. Your wording
>> suggests to me 1.20200601+arm64 (i.e., f382cc1), if your
>> wording is for an officially tagged version.
>>=20
>>> You are right that we are not handling the 3 GB DMA limit in the =
pcie driver. Unfortunately, it did not seem easy to thread the =
appropriate bus tag through without rewriting half the inherited driver =
stack, and in my testing the USB driver always allocated its DMA buffers =
in the lower 3 GB without being told. But obviously it is the wrong to =
rely on luck, so I=E2=80=99ll have a think about it.
>>=20
>> Until such a time, is there a good way to limit a RPi4 to
>> the lower <=3D 3 GiBytes overall for the u-boot case? (There
>> is an explicit selection for the uefi case.) Having a mode
>> of operation that avoids the unreliable status could prove
>> important for the duration.
>>=20
>>> =E2=80=94 RHC.
>>>=20
>>>=20
>>> On Sun, Jul 19, 2020 at 03:18, Mark Millard marklmi@yahoo.com wrote:
>>>=20
>>>>> On 2020-Jul-18, at 14:37, Robert Crowston <crowston at =
protonmail.com> wrote:
>>>>> I believe this differential (https://reviews.freebsd.org/D25261) =
would resolve it, but I haven't got around to addressing the comments =
there yet.
>>>>> -- RHC.
>>>>=20
>>>> Yep, a -r363123 kernel with that also it booted / from
>>>> the USB3 SSD. (The kernel came from the mmcsd0.)
>>>> Notes for this u-boot-rpi4 based context:
>>>> Unlike for uefi v1.17, genet0 and mmcsd0 show up.
>>>> There is a big asymmetry for sending vs. receiving
>>>> over genet0 (192.168.1.126 here):
>>>>=20
>>>> iperf3 -c 192.168.1.120 --get-server-output -B 192.168.1.126
>>>>=20
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>=20
>>>> Connecting to host 192.168.1.120, port 5201
>>>> [ 5] local 192.168.1.126 port 56649 connected to 192.168.1.120 port =
5201
>>>> [ ID] Interval Transfer Bitrate Retr Cwnd
>>>> [ 5] 0.00-1.00 sec 22.7 MBytes 191 Mbits/sec 85 38.4 KBytes
>>>> [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec 146 25.7 KBytes
>>>> [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec 226 27.1 KBytes
>>>> [ 5] 3.00-4.00 sec 23.3 MBytes 195 Mbits/sec 229 95.0 KBytes
>>>> [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec 228 83.7 KBytes
>>>> [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec 209 7.13 KBytes
>>>> [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec 207 49.8 KBytes
>>>> [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec 217 1.43 KBytes
>>>> [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec 224 71.3 KBytes
>>>> [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 242 8.55 KBytes
>>>>=20
>>>> [ ID] Interval Transfer Bitrate Retr
>>>> [ 5] 0.00-10.00 sec 234 MBytes 197 Mbits/sec 2013 sender
>>>> [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
>>>>=20
>>>> Server output:
>>>>=20
>>>> ---------------
>>>>=20
>>>> Server listening on 5201
>>>>=20
>>>> -------------------------
>>>>=20
>>>> Accepted connection from 192.168.1.126, port 57882
>>>> [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port =
56649
>>>> [ ID] Interval Transfer Bitrate
>>>> [ 5] 0.00-1.00 sec 16.4 MBytes 138 Mbits/sec
>>>> [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec
>>>> [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec
>>>> [ 5] 3.00-4.00 sec 23.1 MBytes 194 Mbits/sec
>>>> [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec
>>>> [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec
>>>> [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec
>>>> [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec
>>>> [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec
>>>> [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec
>>>> [ 5] 10.00-10.27 sec 6.30 MBytes 197 Mbits/sec
>>>>=20
>>>> [ ID] Interval Transfer Bitrate
>>>> [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
>>>> iperf Done.
>>>> Rock64orRPi4# iperf3 -R -c 192.168.1.120 --get-server-output -B =
192.168.1.126
>>>> Connecting to host 192.168.1.120, port 5201
>>>> Reverse mode, remote host 192.168.1.120 is sending
>>>> [ 5] local 192.168.1.126 port 25404 connected to 192.168.1.120 port =
5201
>>>> [ ID] Interval Transfer Bitrate
>>>> [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec
>>>> [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec
>>>>=20
>>>> [ ID] Interval Transfer Bitrate Retr
>>>> [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
>>>> [ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
>>>>=20
>>>> Server output:
>>>>=20
>>>> ---------------
>>>>=20
>>>> Server listening on 5201
>>>>=20
>>>> -------------------------
>>>>=20
>>>> Accepted connection from 192.168.1.126, port 41483
>>>> [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port =
25404
>>>> [ ID] Interval Transfer Bitrate Retr Cwnd
>>>> [ 5] 0.00-1.00 sec 83.7 MBytes 702 Mbits/sec 62 1.61 MBytes
>>>> [ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 92 1.61 MBytes
>>>> [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 93 175 KBytes
>>>> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 95 88.4 KBytes
>>>> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 94 268 KBytes
>>>> [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 89 291 KBytes
>>>> [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 92 4.28 KBytes
>>>> [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 92 1.27 MBytes
>>>> [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 94 1.61 MBytes
>>>> [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 96 1.42 MBytes
>>>> [ 5] 10.00-10.26 sec 29.3 MBytes 931 Mbits/sec 24 1.61 MBytes
>>>>=20
>>>> [ ID] Interval Transfer Bitrate Retr
>>>> [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
>>>> iperf Done.
>>>> My attempt to duplicate an about 11 GiByte .tar file failed:
>>>>=20
>>>> cp -aRx /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/mmjnk.tar
>>>>=20
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>=20
>>>> g_vfs_done():gpt/Rock64root[READ(offset=3D274911461376, =
length=3D32768)]error =3D 5
>>>> c/usr/obj/clang-armv7-on-UFS /dev/gpt/Rock64root (/) cylinder =
checksum failed: cg 270, cgp: 0x0 !=3D bp: 0xa023cccf
>>>> aarch64.tar: Input/output error
>>>> UFS /dev/gpt/Rock64root (/) cylinder checksum failed: cg 270, cgp: =
0x0 !=3D bp: 0xa023cccf
>>>> pid 1123 (ntpd), jid 0, uid 0: exited on signal 6 (core dumped)
>>>> pid 1240 (login), jid 0, uid 0: exited on signal 11 (core dumped)
>>>> Same media as used for Rock64 experiments. I've had no
>>>> evidence of problems there.
>>>> However uefi based booting the RPi4 also has problems
>>>> unless uefi is set to force 3 GiBytes of RAM (at most).
>>>> The problem has usually been visible as the diff
>>>> after the copy showing 4 KiByte or slightly less
>>>> having differences according to diff or cmp or the
>>>> like. There is evidence of the content and where
>>>> changing without updates to the files, suggesting
>>>> garbage read data.
>>>> Again, same media for 3GiByte RAM or on the Rock64:
>>>> no evidence of problems for such operations.
>>>> My initial guess is that the handling of the RPi4
>>>> DMA limitations is not yet correct overall,
>>>> apparently for both uefi and u-boot style booting.
>>>>=20
>>>>> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 =
Original Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=
=80=90
>>>>> On Wednesday, 15 July 2020 10:09, Mark Millard via freebsd-arm =
freebsd-arm@freebsd.org wrote:
>>>>> . . .
>=20
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?37CA77CB-8B78-4219-B6F8-64158826BC26>