Date: Fri, 22 Dec 2023 17:28:04 -0800 From: Mark Millard <marklmi@yahoo.com> To: Mike Karels <mike@karels.net> Cc: Steve Bernacki <steve@copacetic.net>, "freebsd-arm@freebsd.org" <freebsd-arm@FreeBSD.org> Subject: Re: FreeBSD 14.0-RELEASE and Raspberry Pi CM4 4GB Message-ID: <D2A938B6-2E24-4956-8966-17391A153342@yahoo.com> In-Reply-To: <2E1B887B-EB3C-4F47-A6EE-8256149F7C84@karels.net> References: <445940f7-e8f1-4dbc-87be-99bfd705141d@copacetic.net> <F037F5CF-07D7-499B-AA36-274EEADC34D0@karels.net> <fa4facd0-1223-42f6-85fc-cec4b6bf2874@copacetic.net> <2E1B887B-EB3C-4F47-A6EE-8256149F7C84@karels.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 22, 2023, at 14:48, Mike Karels <mike@karels.net> wrote:
On 22 Dec 2023, at 16:14, Steve Bernacki wrote:
>=20
>> Hi Mike,
>>=20
>> Indeed, I'm getting a lot of retransmits:
>>=20
>> [ 5] local 172.16.200.2 port 55551 connected to 172.16.200.182 port =
5201
>> [ ID] Interval Transfer Bitrate Retr Cwnd
>> [ 5] 0.00-1.00 sec 36.2 MBytes 304 Mbits/sec 60 9.98 =
KBytes
>> [ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec 143 111 =
KBytes
>> [ 5] 2.00-3.00 sec 34.9 MBytes 293 Mbits/sec 141 7.13 =
KBytes
>> [ 5] 3.00-4.00 sec 33.9 MBytes 284 Mbits/sec 198 99.5 =
KBytes
>> [ 5] 4.00-5.00 sec 34.9 MBytes 292 Mbits/sec 167 1.43 =
KBytes
>> [ 5] 5.00-6.00 sec 34.2 MBytes 287 Mbits/sec 221 2.85 =
KBytes
>> [ 5] 6.00-7.00 sec 34.1 MBytes 286 Mbits/sec 169 100 =
KBytes
>> [ 5] 7.00-8.00 sec 35.2 MBytes 295 Mbits/sec 159 7.13 =
KBytes
>> [ 5] 8.00-9.00 sec 34.3 MBytes 287 Mbits/sec 138 4.28 =
KBytes
>> [ 5] 9.00-10.00 sec 33.3 MBytes 279 Mbits/sec 182 2.85 =
KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate Retr
>> [ 5] 0.00-10.00 sec 347 MBytes 291 Mbits/sec 1578 =
sender
>> [ 5] 0.00-10.00 sec 346 MBytes 291 Mbits/sec =
receiver
>>=20
>> Thanks,
>> Steve
>=20
> One other question: are you running powerd? I booted without it, and =
my
> throughput dropped to 600-640 Mb/s. Repeating the test, =
retransmissions
> went down but throughput was about the same. Note, the RPi 4, and =
probably
> the CM 4, boots at a lower clock frequency by default, and powerd =
raises it
> under load. I'm running powerd with -M 1800, overclocking a little.
I explore here fixed frequencies: 2000 MHz, 600 MHz, 1500 MHz, 1800 MHz
(no powerd use)
Based on:
# uname -apKU
FreeBSD generic 14.0-STABLE FreeBSD 14.0-STABLE #0 =
stable/14-n266002-2ef9079ece5a: Sat Dec 16 08:49:23 UTC 2023 =
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC =
arm64 aarch6
# more /boot/efi/config.txt=20
[all]
arm_64bit=3D1
dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don
dtoverlay=3Dmmc
dtoverlay=3Ddisable-bt
device_tree_address=3D0x4000
kernel=3Du-boot.bin
[pi4]
hdmi_safe=3D1
armstub=3Darmstub8-gic.bin
#
over_voltage=3D6
sdram_freq_min=3D3200
arm_freq_min=3D2000
force_turbo=3D1
# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels =
dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 2000/-1
dev.cpu.0.freq_levels: 2000/-1
dev.cpu.0.freq: 2000
# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[ 5] local 192.168.1.159 port 52424 connected to 192.168.1.157 port =
5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 243 328 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 18.5 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 149 173 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 150 456 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 159 456 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 160 538 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 143 1.43 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 215 167 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 194 580 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 157 552 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 1720 =
sender
[ 5] 0.00-10.01 sec 1.10 GBytes 941 Mbits/sec =
receiver
iperf Done.
Note: The amd64 system running main [so: 15] and the RPi4B are
on the same ethernet switch.
With the 4 overclocking lines in config.txt commented out :
# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels =
dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 1500/-1 600/-1
dev.cpu.0.freq_levels: 1500/-1 600/-1
dev.cpu.0.freq: 600
Note: the default context lacks 1800 (based on the RPi* firmware vintage
in the snapshot). Later I show having 1800 instead of 1500.
# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[ 5] local 192.168.1.159 port 42060 connected to 192.168.1.157 port =
5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 70.8 MBytes 594 Mbits/sec 18 195 KBytes =
=20
[ 5] 1.00-2.00 sec 73.8 MBytes 619 Mbits/sec 8 293 KBytes =
=20
[ 5] 2.00-3.00 sec 73.6 MBytes 618 Mbits/sec 19 250 KBytes =
=20
[ 5] 3.00-4.00 sec 73.6 MBytes 618 Mbits/sec 9 366 KBytes =
=20
[ 5] 4.00-5.00 sec 73.3 MBytes 615 Mbits/sec 9 447 KBytes =
=20
[ 5] 5.00-6.00 sec 73.3 MBytes 615 Mbits/sec 16 303 KBytes =
=20
[ 5] 6.00-7.00 sec 73.2 MBytes 614 Mbits/sec 0 455 KBytes =
=20
[ 5] 7.00-8.00 sec 73.6 MBytes 618 Mbits/sec 1 328 KBytes =
=20
[ 5] 8.00-9.00 sec 73.5 MBytes 616 Mbits/sec 16 246 KBytes =
=20
[ 5] 9.00-10.00 sec 73.3 MBytes 615 Mbits/sec 0 435 KBytes =
=20
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 732 MBytes 614 Mbits/sec 96 =
sender
[ 5] 0.00-10.01 sec 732 MBytes 613 Mbits/sec =
receiver
iperf Done.
Assigning 1500:
# sysctl dev.cpu.0.freq=3D1500
dev.cpu.0.freq: 600 -> 1500
# sysctl dev.cpu.0.freq=3D1500
dev.cpu.0.freq: 600 -> 1500
root@generic:~ # iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[ 5] local 192.168.1.159 port 28904 connected to 192.168.1.157 port =
5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 949 Mbits/sec 4 472 KBytes =
=20
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 6 464 KBytes =
=20
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 5 452 KBytes =
=20
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 3 443 KBytes =
=20
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 4 421 KBytes =
=20
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 4 397 KBytes =
=20
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 378 KBytes =
=20
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 5 355 KBytes =
=20
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 2 476 KBytes =
=20
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 5 446 KBytes =
=20
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 41 =
sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec =
receiver
Adding arm_boost=3D1 to config.txt in order to have 1800 instead of 1500
(needed due to the RPi* firmware vintage in FreeBSD snapshots):
# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels =
dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 1800/-1 600/-1
dev.cpu.0.freq_levels: 1800/-1 600/-1
dev.cpu.0.freq: 600
# sysctl dev.cpu.0.freq=3D1800
dev.cpu.0.freq: 600 -> 1800
# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[ 5] local 192.168.1.159 port 27499 connected to 192.168.1.157 port =
5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 169 104 KBytes =
=20
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 320 KBytes =
=20
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 157 52.8 KBytes =
=20
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 143 87.0 KBytes =
=20
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 143 121 KBytes =
=20
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 159 104 KBytes =
=20
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 138 238 KBytes =
=20
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 152 276 KBytes =
=20
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 145 115 KBytes =
=20
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 162 283 KBytes =
=20
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 1518 =
sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec =
receiver
iperf Done.
=46rom this it appears that the Retr counts do not seem to make
much of a difference to the Bitrate's achieved. But the arm
frequency does if 600 is involved.
My understanding is that arm_boost=3D1 was later made the default
in later vintages of the rpi* firmware. arm_boots only causes
1800 for Rev 1.4+ . Pi 400's have 1800 available by default, at
least for modern enough RPi* firmware.
https://www.raspberrypi.com/documentation/computers/config_txt.html
is not necessarily accurate for the older RPi* firmware that FreeBSD
uses in its snapshots/releases.
> IIRC
> the standard clock is 1500 for the RPi 4. But the throughput is about =
the
> same using the standard clock with powerd.
>=20
> Mike
>=20
>> On 12/22/2023 9:23 AM, Mike Karels wrote:
>>> On 22 Dec 2023, at 6:20, Steve Bernacki wrote:
>>>=20
>>>> I recently purchased a RPI CM4 with 4GB and 32GB eMMC to replace my =
aging FreeBSD firewall. I managed to install FreeBSD 14.0-RELEASE-p3 on =
it, and both Ethernet devices (genet0 and ue0) were properly identified. =
However, network throughput on my gigabit network is pretty bad; iperf3 =
reports a maximum transfer speed of 291 Mbits/sec. Flashing OpenWRT on =
the same hardware using the same ethernet port, I'm able to achieve 923 =
Mbits/sec.
>>>>=20
>>>> Does anyone have any suggestions on how to improve throughput under =
FreeBSD?
>>>>=20
>>>> Thank you
>>>> Steve
>>> I just tested with an RPi4 (4 GB) and 14.0 using iperf3. It looks =
like I'm getting
>>> a rather variable number of retransmissions. On my first run =
(client on RPi 4),
>>> I got 460 Mb/s with a lot of retransmissions, but the next couple of =
runs, including
>>> one receiving, I got about 940 Mb even with some retransmissions. =
The peers were
>>> fairly fast FreeBSD 13.2 and 15-current systems. Are you seeing =
retransmissions?
>>>=20
>>> I'll try to look into this, but I'm not sure when I'll get to it.
>>>=20
>>> Mike
>=20
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D2A938B6-2E24-4956-8966-17391A153342>