Date: Wed, 30 Aug 2023 17:09:34 -0700 From: Mark Millard <marklmi@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Message-ID: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> In-Reply-To: <FF64898C-2968-4362-81FB-BDA5FE7C4A6B@yahoo.com> References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> <DF47C0CB-7456-4677-9F41-3FB57D655288@yahoo.com> <D57A4777-DC48-4374-9FB0-43E468DF7D4C@yahoo.com> <FF64898C-2968-4362-81FB-BDA5FE7C4A6B@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 30, 2023, at 01:49, Mark Millard <marklmi@yahoo.com> wrote:
> On Aug 30, 2023, at 01:22, Mark Millard <marklmi@yahoo.com> wrote:
>=20
>> On Aug 30, 2023, at 01:17, Mark Millard <marklmi@yahoo.com> wrote:
>>=20
>>> On Aug 29, 2023, at 12:52, Mark Millard <marklmi@yahoo.com> wrote:
>>>=20
>>>> Wei Hu <weh_at_microsoft.com> wrote on
>>>> Date: Tue, 29 Aug 2023 12:55:35 UTC :
>>>>=20
>>>>> Thanks for the update. Seems the numbers are the same on zfs and =
ufs. That's=20
>>>>> good to know.=20
>>>>>=20
>>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, =
my original
>>>>> intention was to find out why scp on Linux is performing much =
better than FreeBSD
>>>>> under the same hardware env.=20
>>>>>=20
>>>>> Is it possible to try Linux in your ARM64 setting? I am using =
Ubuntu 22.04 on ext4=20
>>>>> file system.
>>>>=20
>>>>=20
>>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
>>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
>>>> Hyper-V UEFI reports for the disk created:
>>>>=20
>>>> 1. SCSI Disk 0,0
>>>> The boot loader did not load an operating system.
>>>>=20
>>>> (It then reports the network adapter attempt found no
>>>> boot image, but that is expected.)
>>>>=20
>>>> That leaves me wondering if Hyper-V Quick Create
>>>> established a VM file holding Intel/AMD material
>>>> despite the aarch64 context.
>>>>=20
>>>> Establishing a Ubuntu more directly is not familiar and
>>>> will have to be a background activity and, so, likely
>>>> will not be timely. If I did any experiments outside
>>>> Hyper-V (native booting), they would be with slower
>>>> USB3 SSD media than I use for FreeBSD.
>>>>=20
>>>> I did notice that Hyper-V Quick Create did not create
>>>> a fixed sized disk but a dynamic sized one. That is
>>>> different than what I did for FreeBSD.
>>>>=20
>>>> Also, it was not obvious if you were after aarch64
>>>> Hyper-V testing vs. native-boot testing vs. both. So
>>>> I may have gone the wrong direction from the start.
>>>> It is possible that I'd find establishing a native-boot
>>>> easier and then be able to have a VM file created from
>>>> the media, more like what I did with FreeBSD.
>>>>=20
>>>> The Ubuntu activity likely would not be analogous to
>>>> the FreeBSD builds having -mcpu=3D optimization used.
>>>>=20
>>>> Back to $work.
>>>>=20
>>>=20
>>> I found a sequence of UI operations that worked for
>>> installing Ubuntu server 22.04.3 into Hyper-V in
>>> Windows 11 Pro on the Windows Dev Kit 2023 via
>>> use of a downloaded *.iso .
>>>=20
>>> The kernel that results predates 6.0:
>>>=20
>>> $ uname -ap
>>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 =
UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
>>>=20
>>> Using my usual rule of rebooting before the first scp:
>>>=20
>>> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:FreeBSD-14-TEST.img
>>> . . .
>>> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 431.3MB/s 00:11=20
>>>=20
>>> $ rm FreeBSD-14-TEST.img
>>> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:FreeBSD-14-TEST.img
>>> . . .
>>> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 482.2MB/s 00:10
>>>=20
>>> Definitely faster than the FreeBSD results that I reported
>>> earlier, including faster than the ThreadRipper 1950X with
>>> Optane in a PCIe slot (more like 300 MiBytes/sec).
>>>=20
>>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual =
hard
>>> disk under Hyper-V.
>>>=20
>>> For reference:
>>>=20
>>> $ lsblk -f
>>> NAME FSTYPE FSVER LABEL UUID =
FSAVAIL FSUSE% MOUNTPOINTS
>>> loop0 squashfs 4.0 =
0 100% /snap/core20/1977
>>> loop1 squashfs 4.0 =
0 100% /snap/lxd/24326
>>> loop2 squashfs 4.0 =
0 100% /snap/snapd/19459
>>> sda =
=E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 =
1G 1% /boot/efi
>>> =E2=94=94=E2=94=80sda2 ext4 1.0 =
48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
>>>=20
>>>=20
>>>=20
>>> An experiment would be to have a small amount if RAM relative
>>> the file size. That would force it to actually write to media
>>> for some part of the file copy.
>>=20
>> The wording was poor: "force it" here is just from the
>> Ubuntu viewpoint. I make no claim to know if Hyper-V
>> is actually writing the material out to media at the
>> time vs. later.
>>=20
>>> So using 1024 MiByte of RAM assigned in Hyper-V:
>>>=20
>>> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:FreeBSD-14-TEST.img
>>> . . .
>>> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 407.5MB/s 00:12
>>>=20
>>> $ rm FreeBSD-14-TEST.img
>>> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:FreeBSD-14-TEST.img
>>> . . .
>>> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 404.7MB/s 00:12
>>>=20
>>> Still definitely faster than the FreeBSD results that I
>>> reported earlier, including faster than the ThreadRipper
>>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).
>=20
> One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
> of assigned RAM: use of localhost:/dev/null
>=20
> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:/dev/null
> . . .
> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
=20
>=20
> $ scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
markmi@localhost:/dev/null
> . . .
> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 492.9MB/s 00:10
>=20
>=20
> The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS =
context):
>=20
> # scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
root@localhost:/dev/null
> . . .
> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
=20
>=20
> # scp =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
root@localhost:/dev/null
> . . .
> =
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img =
=
100% 5120MB 198.7MB/s 00:25
>=20
>=20
> Note: At most one VM running at a time, never both in overlapping =
times.
Avoiding having a cipher involved and even localhost
involved: use dd . . .
FreeBSD examples for Windows Dev Kit 2023 Hyper-V context,
24576 MiByts of RAM assigned):
# dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D1m status=3Dprogress
2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D1k status=3Dprogress
5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D512 status=3Dprogress
5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec)
Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context,
only 1024 MiBytes of RAM assigned:
$ dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D1M status=3Dprogress
4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s
$ dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D1K status=3Dprogress
4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s
markmi@ubwdk23s:~$ dd =
if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.=
img of=3D/dev/null bs=3D512 status=3Dprogress
4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F>