Date: Sun, 19 Jul 2020 22:08:11 +0000 From: Robert Crowston <crowston@protonmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: FYI: RPi4 (8 GiByte) USB3 vs. head -r363123: still a no-go for booting a USB3 / in my experiments Message-ID: <wV8i_sck3uM7ZuLiK-OrDeHjPMuvCee2dHGczONShToQ3GCWe_5Bz-mwTXw-YSKQPi2Gxq4Z7UxckXjU6RnK4DYTpvQwyJAPu3oS1lSQYNo=@protonmail.com> In-Reply-To: <5C8DD645-5C24-4155-AA4C-819AF3AF3B3D@yahoo.com> References: <BF48971B-7D34-415B-BDA2-00CF82ED1342.ref@yahoo.com> <BF48971B-7D34-415B-BDA2-00CF82ED1342@yahoo.com> <dku0j_wjTui_MeLV0obVoPHTA6AJ0JfAamjVb503rQbeTbdd2xUoG8JrZQB5ENSp4PeSj6estzC0pBYBw7expbvw_v4gd6pDIVo7aOBZApc=@protonmail.com> <64A12BF8-9E2D-441D-8765-52B6D3907A58@yahoo.com> <7J4Opq5TWFEhuN7xv3tq206mMjbDK67p35GadBDuaJRKtDuObVfk7M8ItEEUgumC_jKkoMqC2ZJVUCpNifY579oBq1UDthM1wXjG3tBBTj0=@protonmail.com> <5C8DD645-5C24-4155-AA4C-819AF3AF3B3D@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> Certainly makes it interesting to figure out what
> combination of things should be used overall.
Sorry, yes, the dtb I mostly use is ahead of FreeBSD's pkg content. The 8 G=
B rpi4 model I obtained refuses to boot with the start4.elf and other boot =
firwmare from November, so I just obtained the newest version for the whole=
boot partition. i.e., the newest versions of those files from the Raspberr=
y Pi Foundation. That broke ethernet (by moving to rgmii-rxid) which I then=
fixed in D25251. I appreciate this complicates the situation.
> Until such a time, is there a good way to limit a RPi4 to
> the lower <=3D 3 GiBytes overall for the u-boot case?
You could set hw.physmem=3D"3G" in /boot/loader.conf. I don't know if that =
guarantees FreeBSD will specifically see the lowest 3 GB. The bus tag shoul=
d not be too hard to fix though.
-- RHC.
=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Sunday, 19 July 2020 22:51, Mark Millard <marklmi@yahoo.com> wrote:
>
>
> On 2020-Jul-19, at 04:30, Robert Crowston <crowston atprotonmail.com> wro=
te:
>
> > The ethernet problem could be caused by the recent change in upstream d=
tb, to change the ethernet driver to operate in rgmii-rxid mode instead of =
rgmii mode. See
> > https://reviews.freebsd.org/D25251. I have not noticed this performance=
problem you report myself, when using upstream dtbs from June 2020.
>
> https://reviews.freebsd.org/D25251 was added to head as
> -r362353 and my testing was based on head -r3630123 .
> So my report is based on D25251.
>
> I do see that sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts
> is base on Linux 5.5 (checked-in to FreeBSD 4 months ago).
>
> But the port Makefile sysutils/rpi-firmware/Makefile indicates:
> FW_TAG=3D 2042453 for PORTVERSION=3D 1.20190925.g20200109 .
> Looking on github 2042453 is the 2019-Nov-22 commit for
> "kernel: Bump to 4.19.85". It seems to be from between official
> Tags 1.0290925 and 1.20200114 .
>
> The most recent official Tag is 1.20200601+arm64 and has
> materials in part based on "kernel: Bump to 5.4.42". (That
> is a raspberrypi/linux number.)
>
> The most recent "kernel: Bump to" for master is for 5.4.51
> (committed on 2020-Jul-13). One of the items in that is:
>
> kernel: ARM: dts: Make bcm2711 dts more like 5.7
>
> (I assume 5.7 is an upstream linux version indication.)
> So newer than where
> sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts is based.
>
> Certainly makes it interesting to figure out what
> combination of things should be used overall. Your wording
> suggests to me 1.20200601+arm64 (i.e., f382cc1), if your
> wording is for an officially tagged version.
>
> > You are right that we are not handling the 3 GB DMA limit in the pcie d=
river. Unfortunately, it did not seem easy to thread the appropriate bus ta=
g through without rewriting half the inherited driver stack, and in my test=
ing the USB driver always allocated its DMA buffers in the lower 3 GB witho=
ut being told. But obviously it is the wrong to rely on luck, so I=E2=80=
=99ll have a think about it.
>
> Until such a time, is there a good way to limit a RPi4 to
> the lower <=3D 3 GiBytes overall for the u-boot case? (There
> is an explicit selection for the uefi case.) Having a mode
> of operation that avoids the unreliable status could prove
> important for the duration.
>
> > =E2=80=94 RHC.
> >
> >
> > On Sun, Jul 19, 2020 at 03:18, Mark Millard marklmi@yahoo.com wrote:
> >
> > > > On 2020-Jul-18, at 14:37, Robert Crowston <crowston at protonmail.c=
om> wrote:
> > > > I believe this differential (https://reviews.freebsd.org/D25261) wo=
uld resolve it, but I haven't got around to addressing the comments there y=
et.
> > > > -- RHC.
> > >
> > > Yep, a -r363123 kernel with that also it booted / from
> > > the USB3 SSD. (The kernel came from the mmcsd0.)
> > > Notes for this u-boot-rpi4 based context:
> > > Unlike for uefi v1.17, genet0 and mmcsd0 show up.
> > > There is a big asymmetry for sending vs. receiving
> > > over genet0 (192.168.1.126 here):
> > >
> > > iperf3 -c 192.168.1.120 --get-server-output -B 192.168.1.126
> > >
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > > Connecting to host 192.168.1.120, port 5201
> > > [ 5] local 192.168.1.126 port 56649 connected to 192.168.1.120 port 5=
201
> > > [ ID] Interval Transfer Bitrate Retr Cwnd
> > > [ 5] 0.00-1.00 sec 22.7 MBytes 191 Mbits/sec 85 38.4 KBytes
> > > [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec 146 25.7 KBytes
> > > [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec 226 27.1 KBytes
> > > [ 5] 3.00-4.00 sec 23.3 MBytes 195 Mbits/sec 229 95.0 KBytes
> > > [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec 228 83.7 KBytes
> > > [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec 209 7.13 KBytes
> > > [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec 207 49.8 KBytes
> > > [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec 217 1.43 KBytes
> > > [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec 224 71.3 KBytes
> > > [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 242 8.55 KBytes
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.00 sec 234 MBytes 197 Mbits/sec 2013 sender
> > > [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
> > >
> > > Server output:
> > >
> > > ---------------
> > >
> > > Server listening on 5201
> > >
> > > -------------------------
> > >
> > > Accepted connection from 192.168.1.126, port 57882
> > > [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port 56=
649
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-1.00 sec 16.4 MBytes 138 Mbits/sec
> > > [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec
> > > [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec
> > > [ 5] 3.00-4.00 sec 23.1 MBytes 194 Mbits/sec
> > > [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec
> > > [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec
> > > [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec
> > > [ 5] 10.00-10.27 sec 6.30 MBytes 197 Mbits/sec
> > >
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
> > > iperf Done.
> > > Rock64orRPi4# iperf3 -R -c 192.168.1.120 --get-server-output -B 192.1=
68.1.126
> > > Connecting to host 192.168.1.120, port 5201
> > > Reverse mode, remote host 192.168.1.120 is sending
> > > [ 5] local 192.168.1.126 port 25404 connected to 192.168.1.120 port 5=
201
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
> > > [ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
> > >
> > > Server output:
> > >
> > > ---------------
> > >
> > > Server listening on 5201
> > >
> > > -------------------------
> > >
> > > Accepted connection from 192.168.1.126, port 41483
> > > [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port 25=
404
> > > [ ID] Interval Transfer Bitrate Retr Cwnd
> > > [ 5] 0.00-1.00 sec 83.7 MBytes 702 Mbits/sec 62 1.61 MBytes
> > > [ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 92 1.61 MBytes
> > > [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 93 175 KBytes
> > > [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 95 88.4 KBytes
> > > [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 94 268 KBytes
> > > [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 89 291 KBytes
> > > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 92 4.28 KBytes
> > > [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 92 1.27 MBytes
> > > [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 94 1.61 MBytes
> > > [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 96 1.42 MBytes
> > > [ 5] 10.00-10.26 sec 29.3 MBytes 931 Mbits/sec 24 1.61 MBytes
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
> > > iperf Done.
> > > My attempt to duplicate an about 11 GiByte .tar file failed:
> > >
> > > cp -aRx /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/mmjnk.tar
> > >
> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > >
> > > g_vfs_done():gpt/Rock64root[READ(offset=3D274911461376, length=3D3276=
8)]error =3D 5
> > > c/usr/obj/clang-armv7-on-UFS /dev/gpt/Rock64root (/) cylinder checksu=
m failed: cg 270, cgp: 0x0 !=3D bp: 0xa023cccf
> > > aarch64.tar: Input/output error
> > > UFS /dev/gpt/Rock64root (/) cylinder checksum failed: cg 270, cgp: 0x=
0 !=3D bp: 0xa023cccf
> > > pid 1123 (ntpd), jid 0, uid 0: exited on signal 6 (core dumped)
> > > pid 1240 (login), jid 0, uid 0: exited on signal 11 (core dumped)
> > > Same media as used for Rock64 experiments. I've had no
> > > evidence of problems there.
> > > However uefi based booting the RPi4 also has problems
> > > unless uefi is set to force 3 GiBytes of RAM (at most).
> > > The problem has usually been visible as the diff
> > > after the copy showing 4 KiByte or slightly less
> > > having differences according to diff or cmp or the
> > > like. There is evidence of the content and where
> > > changing without updates to the files, suggesting
> > > garbage read data.
> > > Again, same media for 3GiByte RAM or on the Rock64:
> > > no evidence of problems for such operations.
> > > My initial guess is that the handling of the RPi4
> > > DMA limitations is not yet correct overall,
> > > apparently for both uefi and u-boot style booting.
> > >
> > > > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Ori=
ginal Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=
=E2=80=90
> > > > On Wednesday, 15 July 2020 10:09, Mark Millard via freebsd-arm free=
bsd-arm@freebsd.org wrote:
> > > > . . .
>
> =3D=3D
>
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?wV8i_sck3uM7ZuLiK-OrDeHjPMuvCee2dHGczONShToQ3GCWe_5Bz-mwTXw-YSKQPi2Gxq4Z7UxckXjU6RnK4DYTpvQwyJAPu3oS1lSQYNo=>