Date: Sun, 19 Jul 2020 14:51:51 -0700 From: Mark Millard <marklmi@yahoo.com> To: Robert Crowston <crowston@protonmail.com> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: FYI: RPi4 (8 GiByte) USB3 vs. head -r363123: still a no-go for booting a USB3 / in my experiments Message-ID: <5C8DD645-5C24-4155-AA4C-819AF3AF3B3D@yahoo.com> In-Reply-To: <7J4Opq5TWFEhuN7xv3tq206mMjbDK67p35GadBDuaJRKtDuObVfk7M8ItEEUgumC_jKkoMqC2ZJVUCpNifY579oBq1UDthM1wXjG3tBBTj0=@protonmail.com> References: <BF48971B-7D34-415B-BDA2-00CF82ED1342.ref@yahoo.com> <BF48971B-7D34-415B-BDA2-00CF82ED1342@yahoo.com> <dku0j_wjTui_MeLV0obVoPHTA6AJ0JfAamjVb503rQbeTbdd2xUoG8JrZQB5ENSp4PeSj6estzC0pBYBw7expbvw_v4gd6pDIVo7aOBZApc=@protonmail.com> <64A12BF8-9E2D-441D-8765-52B6D3907A58@yahoo.com> <7J4Opq5TWFEhuN7xv3tq206mMjbDK67p35GadBDuaJRKtDuObVfk7M8ItEEUgumC_jKkoMqC2ZJVUCpNifY579oBq1UDthM1wXjG3tBBTj0=@protonmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-Jul-19, at 04:30, Robert Crowston <crowston at protonmail.com> = wrote: > The ethernet problem could be caused by the recent change in upstream = dtb, to change the ethernet driver to operate in rgmii-rxid mode instead = of rgmii mode. See=20 > https://reviews.freebsd.org/D25251. I have not noticed this = performance problem you report myself, when using upstream dtbs from = June 2020. https://reviews.freebsd.org/D25251 was added to head as -r362353 and my testing was based on head -r3630123 . So my report is based on D25251. I do see that sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts is base on Linux 5.5 (checked-in to FreeBSD 4 months ago). But the port Makefile sysutils/rpi-firmware/Makefile indicates: FW_TAG=3D 2042453 for PORTVERSION=3D 1.20190925.g20200109 . Looking on github 2042453 is the 2019-Nov-22 commit for "kernel: Bump to 4.19.85". It seems to be from between official Tags 1.0290925 and 1.20200114 . The most recent official Tag is 1.20200601+arm64 and has materials in part based on "kernel: Bump to 5.4.42". (That is a raspberrypi/linux number.) The most recent "kernel: Bump to" for master is for 5.4.51 (committed on 2020-Jul-13). One of the items in that is: kernel: ARM: dts: Make bcm2711 dts more like 5.7 (I assume 5.7 is an upstream linux version indication.) So newer than where sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts is based. Certainly makes it interesting to figure out what combination of things should be used overall. Your wording suggests to me 1.20200601+arm64 (i.e., f382cc1), if your wording is for an officially tagged version. > You are right that we are not handling the 3 GB DMA limit in the pcie = driver. Unfortunately, it did not seem easy to thread the appropriate = bus tag through without rewriting half the inherited driver stack, and = in my testing the USB driver always allocated its DMA buffers in the = lower 3 GB without being told. But obviously it is the wrong to rely on = luck, so I=E2=80=99ll have a think about it. Until such a time, is there a good way to limit a RPi4 to the lower <=3D 3 GiBytes overall for the u-boot case? (There is an explicit selection for the uefi case.) Having a mode of operation that avoids the unreliable status could prove important for the duration. > =E2=80=94 RHC. >=20 > On Sun, Jul 19, 2020 at 03:18, Mark Millard <marklmi@yahoo.com> wrote: >>=20 >>=20 >> > On 2020-Jul-18, at 14:37, Robert Crowston <crowston at = protonmail.com> wrote: >> > >> > I believe this differential (https://reviews.freebsd.org/D25261) = would resolve it, but I haven't got around to addressing the comments = there yet. >> > >> > -- RHC. >>=20 >> Yep, a -r363123 kernel with that also it booted / from >> the USB3 SSD. (The kernel came from the mmcsd0.) >>=20 >>=20 >> Notes for this u-boot-rpi4 based context: >>=20 >>=20 >> Unlike for uefi v1.17, genet0 and mmcsd0 show up. >>=20 >>=20 >> There is a big asymmetry for sending vs. receiving >> over genet0 (192.168.1.126 here): >>=20 >> # iperf3 -c 192.168.1.120 --get-server-output -B 192.168.1.126 >> Connecting to host 192.168.1.120, port 5201 >> [ 5] local 192.168.1.126 port 56649 connected to 192.168.1.120 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 22.7 MBytes 191 Mbits/sec 85 38.4 KBytes >> [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec 146 25.7 KBytes >> [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec 226 27.1 KBytes >> [ 5] 3.00-4.00 sec 23.3 MBytes 195 Mbits/sec 229 95.0 KBytes >> [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec 228 83.7 KBytes >> [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec 209 7.13 KBytes >> [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec 207 49.8 KBytes >> [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec 217 1.43 KBytes >> [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec 224 71.3 KBytes >> [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 242 8.55 KBytes >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 234 MBytes 197 Mbits/sec 2013 sender >> [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver >>=20 >> Server output: >> ----------------------------------------------------------- >> Server listening on 5201 >> ----------------------------------------------------------- >> Accepted connection from 192.168.1.126, port 57882 >> [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port = 56649 >> [ ID] Interval Transfer Bitrate >> [ 5] 0.00-1.00 sec 16.4 MBytes 138 Mbits/sec >> [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec >> [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec >> [ 5] 3.00-4.00 sec 23.1 MBytes 194 Mbits/sec >> [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec >> [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec >> [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec >> [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec >> [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec >> [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec >> [ 5] 10.00-10.27 sec 6.30 MBytes 197 Mbits/sec >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate >> [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver >>=20 >>=20 >> iperf Done. >> Rock64orRPi4# iperf3 -R -c 192.168.1.120 --get-server-output -B = 192.168.1.126 >> Connecting to host 192.168.1.120, port 5201 >> Reverse mode, remote host 192.168.1.120 is sending >> [ 5] local 192.168.1.126 port 25404 connected to 192.168.1.120 port = 5201 >> [ ID] Interval Transfer Bitrate >> [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec >> [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender >> [ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver >>=20 >> Server output: >> ----------------------------------------------------------- >> Server listening on 5201 >> ----------------------------------------------------------- >> Accepted connection from 192.168.1.126, port 41483 >> [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port = 25404 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 83.7 MBytes 702 Mbits/sec 62 1.61 MBytes >> [ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 92 1.61 MBytes >> [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 93 175 KBytes >> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 95 88.4 KBytes >> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 94 268 KBytes >> [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 89 291 KBytes >> [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 92 4.28 KBytes >> [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 92 1.27 MBytes >> [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 94 1.61 MBytes >> [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 96 1.42 MBytes >> [ 5] 10.00-10.26 sec 29.3 MBytes 931 Mbits/sec 24 1.61 MBytes >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender >>=20 >>=20 >> iperf Done. >>=20 >>=20 >> My attempt to duplicate an about 11 GiByte .tar file failed: >>=20 >> # cp -aRx /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/mmjnk.tar >> g_vfs_done():gpt/Rock64root[READ(offset=3D274911461376, = length=3D32768)]error =3D 5 >> c/usr/obj/clang-armv7-on-UFS /dev/gpt/Rock64root (/) cylinder = checksum failed: cg 270, cgp: 0x0 !=3D bp: 0xa023cccf >> aarch64.tar: Input/output error >> UFS /dev/gpt/Rock64root (/) cylinder checksum failed: cg 270, cgp: = 0x0 !=3D bp: 0xa023cccf >> pid 1123 (ntpd), jid 0, uid 0: exited on signal 6 (core dumped) >> pid 1240 (login), jid 0, uid 0: exited on signal 11 (core dumped) >>=20 >> Same media as used for Rock64 experiments. I've had no >> evidence of problems there. >>=20 >> However uefi based booting the RPi4 also has problems >> unless uefi is set to force 3 GiBytes of RAM (at most). >> The problem has usually been visible as the diff >> after the copy showing 4 KiByte or slightly less >> having differences according to diff or cmp or the >> like. There is evidence of the content and where >> changing without updates to the files, suggesting >> garbage read data. >>=20 >> Again, same media for 3GiByte RAM or on the Rock64: >> no evidence of problems for such operations. >>=20 >> My initial guess is that the handling of the RPi4 >> DMA limitations is not yet correct overall, >> apparently for both uefi and u-boot style booting. >>=20 >> > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 = Original Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2= =80=90 >> > On Wednesday, 15 July 2020 10:09, Mark Millard via freebsd-arm = <freebsd-arm@freebsd.org> wrote: >> > >> >. . . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5C8DD645-5C24-4155-AA4C-819AF3AF3B3D>