Date: Sat, 23 Dec 2023 15:32:01 -0800 From: Mark Millard <marklmi@yahoo.com> To: Mike Karels <mike@karels.net> Cc: Steve Bernacki <steve@copacetic.net>, "freebsd-arm@freebsd.org" <freebsd-arm@FreeBSD.org> Subject: Re: FreeBSD 14.0-RELEASE and Raspberry Pi CM4 4GB Message-ID: <C0796CB1-07F3-4B53-96D1-58ADEFF03254@yahoo.com> In-Reply-To: <8CBB5E92-942E-43FA-B188-8288480B8D95@yahoo.com> References: <445940f7-e8f1-4dbc-87be-99bfd705141d@copacetic.net> <F037F5CF-07D7-499B-AA36-274EEADC34D0@karels.net> <fa4facd0-1223-42f6-85fc-cec4b6bf2874@copacetic.net> <2E1B887B-EB3C-4F47-A6EE-8256149F7C84@karels.net> <D2A938B6-2E24-4956-8966-17391A153342@yahoo.com> <8CBB5E92-942E-43FA-B188-8288480B8D95@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 23, 2023, at 15:27, Mark Millard <marklmi@yahoo.com> wrote: > On Dec 22, 2023, at 17:28, Mark Millard <marklmi@yahoo.com> wrote: >=20 >> On Dec 22, 2023, at 14:48, Mike Karels <mike@karels.net> wrote: >>=20 >> On 22 Dec 2023, at 16:14, Steve Bernacki wrote: >>>=20 >>>> Hi Mike, >>>>=20 >>>> Indeed, I'm getting a lot of retransmits: >>>>=20 >>>> [ 5] local 172.16.200.2 port 55551 connected to 172.16.200.182 = port 5201 >>>> [ ID] Interval Transfer Bitrate Retr Cwnd >>>> [ 5] 0.00-1.00 sec 36.2 MBytes 304 Mbits/sec 60 9.98 = KBytes >>>> [ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec 143 111 = KBytes >>>> [ 5] 2.00-3.00 sec 34.9 MBytes 293 Mbits/sec 141 7.13 = KBytes >>>> [ 5] 3.00-4.00 sec 33.9 MBytes 284 Mbits/sec 198 99.5 = KBytes >>>> [ 5] 4.00-5.00 sec 34.9 MBytes 292 Mbits/sec 167 1.43 = KBytes >>>> [ 5] 5.00-6.00 sec 34.2 MBytes 287 Mbits/sec 221 2.85 = KBytes >>>> [ 5] 6.00-7.00 sec 34.1 MBytes 286 Mbits/sec 169 100 = KBytes >>>> [ 5] 7.00-8.00 sec 35.2 MBytes 295 Mbits/sec 159 7.13 = KBytes >>>> [ 5] 8.00-9.00 sec 34.3 MBytes 287 Mbits/sec 138 4.28 = KBytes >>>> [ 5] 9.00-10.00 sec 33.3 MBytes 279 Mbits/sec 182 2.85 = KBytes >>>> - - - - - - - - - - - - - - - - - - - - - - - - - >>>> [ ID] Interval Transfer Bitrate Retr >>>> [ 5] 0.00-10.00 sec 347 MBytes 291 Mbits/sec 1578 = sender >>>> [ 5] 0.00-10.00 sec 346 MBytes 291 Mbits/sec = receiver >>>>=20 >>>> Thanks, >>>> Steve >>>=20 >>> One other question: are you running powerd? I booted without it, = and my >>> throughput dropped to 600-640 Mb/s. Repeating the test, = retransmissions >>> went down but throughput was about the same. Note, the RPi 4, and = probably >>> the CM 4, boots at a lower clock frequency by default, and powerd = raises it >>> under load. I'm running powerd with -M 1800, overclocking a little. >>=20 >> I explore here fixed frequencies: 2000 MHz, 600 MHz, 1500 MHz, 1800 = MHz >> (no powerd use) >=20 > Well, my later assumption about the likes of the = hw.cpufreq.sdram_freq_min Sorry: (RPi* config.txt notation) sdram_freq_min (since 400 is observed = to occur) (FreeBSD does not expose a hw.cpufreq.sdram_freq_min .) > being due to RPi* firmware looks to be wrong. The RPi* documentation > changed from 400 MHz to 3200 MHz for RPi4B sdram_freq_min at: >=20 > Before (400), Jun 8, 2021: > = https://github.com/raspberrypi/documentation/blob/974995fabb184a2435a98e68= c1e728b346112f89/configuration/config-txt/overclocking.md >=20 > After (3200), Jun 9, 2021: > = https://github.com/raspberrypi/documentation/blob/920ff905995541f7ef1c6048= 2924a392143e9192/configuration/config-txt/overclocking.md >=20 > The RPi* firmware should be setting things up to have 3200 MHz. Since > that is not what FreeBSD ends up with in modern snapshots with the > FreeBSD supplied config.txt , Likely FreeBSD has taken control of > such. >=20 > This might just be one example parameter that is overridden. >=20 > FYI: the 2023-Dec-16 stable/14 snapshot that I'm using > has: >=20 > # strings /boot/efi/start4.elf | grep VC_BUILD_ID_ > VC_BUILD_ID_USER: dom > VC_BUILD_ID_TIME: 10:50:39 > VC_BUILD_ID_VARIANT: start > VC_BUILD_ID_TIME: Mar 17 2023 > VC_BUILD_ID_BRANCH: bcm2711_2 > VC_BUILD_ID_HOSTNAME: buildbot > VC_BUILD_ID_PLATFORM: raspberrypi_linux > VC_BUILD_ID_VERSION: 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) >=20 > Far more recent than 2021. >=20 >> Based on: >>=20 >> # uname -apKU >> FreeBSD generic 14.0-STABLE FreeBSD 14.0-STABLE #0 = stable/14-n266002-2ef9079ece5a: Sat Dec 16 08:49:23 UTC 2023 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch6 >>=20 >> # more /boot/efi/config.txt=20 >> [all] >> arm_64bit=3D1 >> dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don >> dtoverlay=3Dmmc >> dtoverlay=3Ddisable-bt >> device_tree_address=3D0x4000 >> kernel=3Du-boot.bin >>=20 >> [pi4] >> hdmi_safe=3D1 >> armstub=3Darmstub8-gic.bin >> # >> over_voltage=3D6 >> sdram_freq_min=3D3200 >> arm_freq_min=3D2000 >> force_turbo=3D1 >>=20 >> # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq >> dev.bcm2835_cpufreq.0.freq_settings: 2000/-1 >> dev.cpu.0.freq_levels: 2000/-1 >> dev.cpu.0.freq: 2000 >>=20 >>=20 >> # iperf3 -c 192.168.1.157 >> Connecting to host 192.168.1.157, port 5201 >> [ 5] local 192.168.1.159 port 52424 connected to 192.168.1.157 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 243 328 = KBytes >> [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 18.5 = KBytes >> [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 149 173 = KBytes >> [ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 150 456 = KBytes >> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 159 456 = KBytes >> [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 160 538 = KBytes >> [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 143 1.43 = KBytes >> [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 215 167 = KBytes >> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 194 580 = KBytes >> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 157 552 = KBytes >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 1720 = sender >> [ 5] 0.00-10.01 sec 1.10 GBytes 941 Mbits/sec = receiver >>=20 >> iperf Done. >>=20 >>=20 >> Note: The amd64 system running main [so: 15] and the RPi4B are >> on the same ethernet switch. >>=20 >>=20 >> With the 4 overclocking lines in config.txt commented out : >>=20 >> # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq >> dev.bcm2835_cpufreq.0.freq_settings: 1500/-1 600/-1 >> dev.cpu.0.freq_levels: 1500/-1 600/-1 >> dev.cpu.0.freq: 600 >>=20 >> Note: the default context lacks 1800 (based on the RPi* firmware = vintage >> in the snapshot). Later I show having 1800 instead of 1500. >>=20 >> # iperf3 -c 192.168.1.157 >> Connecting to host 192.168.1.157, port 5201 >> [ 5] local 192.168.1.159 port 42060 connected to 192.168.1.157 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 70.8 MBytes 594 Mbits/sec 18 195 = KBytes =20 >> [ 5] 1.00-2.00 sec 73.8 MBytes 619 Mbits/sec 8 293 = KBytes =20 >> [ 5] 2.00-3.00 sec 73.6 MBytes 618 Mbits/sec 19 250 = KBytes =20 >> [ 5] 3.00-4.00 sec 73.6 MBytes 618 Mbits/sec 9 366 = KBytes =20 >> [ 5] 4.00-5.00 sec 73.3 MBytes 615 Mbits/sec 9 447 = KBytes =20 >> [ 5] 5.00-6.00 sec 73.3 MBytes 615 Mbits/sec 16 303 = KBytes =20 >> [ 5] 6.00-7.00 sec 73.2 MBytes 614 Mbits/sec 0 455 = KBytes =20 >> [ 5] 7.00-8.00 sec 73.6 MBytes 618 Mbits/sec 1 328 = KBytes =20 >> [ 5] 8.00-9.00 sec 73.5 MBytes 616 Mbits/sec 16 246 = KBytes =20 >> [ 5] 9.00-10.00 sec 73.3 MBytes 615 Mbits/sec 0 435 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 732 MBytes 614 Mbits/sec 96 = sender >> [ 5] 0.00-10.01 sec 732 MBytes 613 Mbits/sec = receiver >>=20 >> iperf Done. >>=20 >> Assigning 1500: >>=20 >> # sysctl dev.cpu.0.freq=3D1500 >> dev.cpu.0.freq: 600 -> 1500 >>=20 >> # sysctl dev.cpu.0.freq=3D1500 >> dev.cpu.0.freq: 600 -> 1500 >> root@generic:~ # iperf3 -c 192.168.1.157 >> Connecting to host 192.168.1.157, port 5201 >> [ 5] local 192.168.1.159 port 28904 connected to 192.168.1.157 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 113 MBytes 949 Mbits/sec 4 472 = KBytes =20 >> [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 6 464 = KBytes =20 >> [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 5 452 = KBytes =20 >> [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 3 443 = KBytes =20 >> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 4 421 = KBytes =20 >> [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 4 397 = KBytes =20 >> [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 378 = KBytes =20 >> [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 5 355 = KBytes =20 >> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 2 476 = KBytes =20 >> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 5 446 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 41 = sender >> [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec = receiver >>=20 >>=20 >>=20 >> Adding arm_boost=3D1 to config.txt in order to have 1800 instead of = 1500 >> (needed due to the RPi* firmware vintage in FreeBSD snapshots): >>=20 >> # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq >> dev.bcm2835_cpufreq.0.freq_settings: 1800/-1 600/-1 >> dev.cpu.0.freq_levels: 1800/-1 600/-1 >> dev.cpu.0.freq: 600 >>=20 >> # sysctl dev.cpu.0.freq=3D1800 >> dev.cpu.0.freq: 600 -> 1800 >>=20 >> # iperf3 -c 192.168.1.157 >> Connecting to host 192.168.1.157, port 5201 >> [ 5] local 192.168.1.159 port 27499 connected to 192.168.1.157 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 169 104 = KBytes =20 >> [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 320 = KBytes =20 >> [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 157 52.8 = KBytes =20 >> [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 143 87.0 = KBytes =20 >> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 143 121 = KBytes =20 >> [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 159 104 = KBytes =20 >> [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 138 238 = KBytes =20 >> [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 152 276 = KBytes =20 >> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 145 115 = KBytes =20 >> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 162 283 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 1518 = sender >> [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec = receiver >>=20 >> iperf Done. >>=20 >>=20 >>=20 >> =46rom this it appears that the Retr counts do not seem to make >> much of a difference to the Bitrate's achieved. But the arm >> frequency does if 600 is involved. >>=20 >>=20 >> My understanding is that arm_boost=3D1 was later made the default >> in later vintages of the rpi* firmware. arm_boots only causes >> 1800 for Rev 1.4+ . Pi 400's have 1800 available by default, at >> least for modern enough RPi* firmware. >>=20 >> https://www.raspberrypi.com/documentation/computers/config_txt.html >> is not necessarily accurate for the older RPi* firmware that FreeBSD >> uses in its snapshots/releases. >>=20 >>> IIRC >>> the standard clock is 1500 for the RPi 4. But the throughput is = about the >>> same using the standard clock with powerd. >>>=20 >>> Mike >>>=20 >>>> On 12/22/2023 9:23 AM, Mike Karels wrote: >>>>> On 22 Dec 2023, at 6:20, Steve Bernacki wrote: >>>>>=20 >>>>>> I recently purchased a RPI CM4 with 4GB and 32GB eMMC to replace = my aging FreeBSD firewall. I managed to install FreeBSD 14.0-RELEASE-p3 = on it, and both Ethernet devices (genet0 and ue0) were properly = identified. However, network throughput on my gigabit network is pretty = bad; iperf3 reports a maximum transfer speed of 291 Mbits/sec. Flashing = OpenWRT on the same hardware using the same ethernet port, I'm able to = achieve 923 Mbits/sec. >>>>>>=20 >>>>>> Does anyone have any suggestions on how to improve throughput = under FreeBSD? >>>>>>=20 >>>>>> Thank you >>>>>> Steve >>>>> I just tested with an RPi4 (4 GB) and 14.0 using iperf3. It looks = like I'm getting >>>>> a rather variable number of retransmissions. On my first run = (client on RPi 4), >>>>> I got 460 Mb/s with a lot of retransmissions, but the next couple = of runs, including >>>>> one receiving, I got about 940 Mb even with some retransmissions. = The peers were >>>>> fairly fast FreeBSD 13.2 and 15-current systems. Are you seeing = retransmissions? >>>>>=20 >>>>> I'll try to look into this, but I'm not sure when I'll get to it. >>>>>=20 >>>>> Mike >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C0796CB1-07F3-4B53-96D1-58ADEFF03254>