From nobody Thu Aug 31 01:45:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbkWt4nmjz4rWJM for ; Thu, 31 Aug 2023 01:45:46 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbkWs6qDmz4vp8 for ; Thu, 31 Aug 2023 01:45:45 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x82f.google.com with SMTP id d75a77b69052e-4121788397fso6959271cf.0 for ; Wed, 30 Aug 2023 18:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1693446345; x=1694051145; darn=freebsd.org; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:from:to:cc:subject:date:message-id :reply-to; bh=IvgBO4RWaXoZXAYKoqKC8OdNgp3E6yA1iJ+jV21d4Io=; b=nDfdGPojdHLKN174AcK0uu+/VRZfLj44Q7lJqH20OhTg5+iqdfjEoPL2skWWovT/QD DlH+9DoVJhPuwkwiF48P38v5FZOFuQECI2f4i8Mu2J1Z2tYEy1lb3vIMWUExEYXk9ykX aiImIwa/Bm20hFlkRIY6oHKbU3NZjPmr2KkiOL1Hv89kSnKmdOSTYdtCRxeMKlMk3uls 1cPRv0YBRdqZvC3el15X4mwEfbf+z60dtkdKhGM37T4hZ35qaRIs+BrfYA4eQe93zSso +bszGGIumWJKsL8FBBmC/HBJuwS16D15XVw73HO5XQ7boHc+BHsrPKTWrC5hcIg4mX3b A90A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693446345; x=1694051145; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IvgBO4RWaXoZXAYKoqKC8OdNgp3E6yA1iJ+jV21d4Io=; b=XzcvhPAfOIZSj5EsKKZMHQFIXgRv7+OYOtOvpoAj0wXQ+E9aGuPLntmn4MNn5/rrun xCLniEptU7ts9t5L5RJcbMdSN8hZsTP4nx2UwCG9upII99eU16R0/WRv6cJS/ioIBIh2 E7Pv6e9t5YH27cGzBA8yj36ZnvJjqr5V6BMCwrW5BePZg1Gf0kfq7jAI62qiCa15V2Vg kKQLXH7W6K6S2ahy6vAzBPgEXz+HS5xGoTW81QiNjJjfavmZmqVe1TCnkcJwBPqUI/Oo gFzIVUPeLLe3Vbl4KywcPsGusNeTR/FqZU2DMZAef9Trx6LaPzcPXWhcfvc1AHCQ4dO7 eBBA== X-Gm-Message-State: AOJu0YylzmrEk8NZTV6O5PPU0oG2b5tk8dIvOngGl+gaez+LYDv2RJeL xh3epuSz/W/TDsUKJ1FrcHy4Zg== X-Google-Smtp-Source: AGHT+IGtnSGGptsWOa3yiwcFysli+IM5ThAZpbGV9gSezDGReldX+6jfG53kphMUXluDfh8h0NffEQ== X-Received: by 2002:ac8:4d5e:0:b0:410:8ac5:3084 with SMTP id x30-20020ac84d5e000000b004108ac53084mr1284173qtv.7.1693446344722; Wed, 30 Aug 2023 18:45:44 -0700 (PDT) Received: from smtpclient.apple ([72.89.196.186]) by smtp.gmail.com with ESMTPSA id fu9-20020a05622a5d8900b00410910c6164sm149205qtb.50.2023.08.30.18.45.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 30 Aug 2023 18:45:44 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Mark Saad List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Date: Wed, 30 Aug 2023 21:45:33 -0400 Message-Id: <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> Cc: weh@microsoft.com, FreeBSD Hackers In-Reply-To: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> To: Mark Millard X-Mailer: iPhone Mail (20G75) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RbkWs6qDmz4vp8 All Why not take scp out of the picture and try iperf? Why , we could be looki= ng at rss by default in Linux . --- Mark Saad | nonesuch@longcount.org > On Aug 30, 2023, at 8:10 PM, Mark Millard wrote: >=20 > =EF=BB=BFOn Aug 30, 2023, at 01:49, Mark Millard wrote= : >=20 >>> On Aug 30, 2023, at 01:22, Mark Millard wrote: >>>=20 >>>> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>>=20 >>>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>>=20 >>>>> Wei Hu wrote on >>>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>>=20 >>>>>> Thanks for the update. Seems the numbers are the same on zfs and ufs.= That's=20 >>>>>> good to know.=20 >>>>>>=20 >>>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, my= original >>>>>> intention was to find out why scp on Linux is performing much better t= han FreeBSD >>>>>> under the same hardware env.=20 >>>>>>=20 >>>>>> Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 2= 2.04 on ext4=20 >>>>>> file system. >>>>>=20 >>>>>=20 >>>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >>>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >>>>> Hyper-V UEFI reports for the disk created: >>>>>=20 >>>>> 1. SCSI Disk 0,0 >>>>> The boot loader did not load an operating system. >>>>>=20 >>>>> (It then reports the network adapter attempt found no >>>>> boot image, but that is expected.) >>>>>=20 >>>>> That leaves me wondering if Hyper-V Quick Create >>>>> established a VM file holding Intel/AMD material >>>>> despite the aarch64 context. >>>>>=20 >>>>> Establishing a Ubuntu more directly is not familiar and >>>>> will have to be a background activity and, so, likely >>>>> will not be timely. If I did any experiments outside >>>>> Hyper-V (native booting), they would be with slower >>>>> USB3 SSD media than I use for FreeBSD. >>>>>=20 >>>>> I did notice that Hyper-V Quick Create did not create >>>>> a fixed sized disk but a dynamic sized one. That is >>>>> different than what I did for FreeBSD. >>>>>=20 >>>>> Also, it was not obvious if you were after aarch64 >>>>> Hyper-V testing vs. native-boot testing vs. both. So >>>>> I may have gone the wrong direction from the start. >>>>> It is possible that I'd find establishing a native-boot >>>>> easier and then be able to have a VM file created from >>>>> the media, more like what I did with FreeBSD. >>>>>=20 >>>>> The Ubuntu activity likely would not be analogous to >>>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>>=20 >>>>> Back to $work. >>>>>=20 >>>>=20 >>>> I found a sequence of UI operations that worked for >>>> installing Ubuntu server 22.04.3 into Hyper-V in >>>> Windows 11 Pro on the Windows Dev Kit 2023 via >>>> use of a downloaded *.iso . >>>>=20 >>>> The kernel that results predates 6.0: >>>>=20 >>>> $ uname -ap >>>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC= 2023 aarch64 aarch64 aarch64 GNU/Linux >>>>=20 >>>> Using my usual rule of rebooting before the first scp: >>>>=20 >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 431.3MB/s 00:11=20 >>>>=20 >>>> $ rm FreeBSD-14-TEST.img >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 482.2MB/s 00:10 >>>>=20 >>>> Definitely faster than the FreeBSD results that I reported >>>> earlier, including faster than the ThreadRipper 1950X with >>>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>>=20 >>>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard >>>> disk under Hyper-V. >>>>=20 >>>> For reference: >>>>=20 >>>> $ lsblk -f >>>> NAME FSTYPE FSVER LABEL UUID FSAVAI= L FSUSE% MOUNTPOINTS >>>> loop0 squashfs 4.0 0= 100% /snap/core20/1977 >>>> loop1 squashfs 4.0 0= 100% /snap/lxd/24326 >>>> loop2 squashfs 4.0 0= 100% /snap/snapd/19459 >>>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>>> =E2=94=94=E2=94=80sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6= d8efc6ffe 99.3G 14% / >>>>=20 >>>>=20 >>>>=20 >>>> An experiment would be to have a small amount if RAM relative >>>> the file size. That would force it to actually write to media >>>> for some part of the file copy. >>>=20 >>> The wording was poor: "force it" here is just from the >>> Ubuntu viewpoint. I make no claim to know if Hyper-V >>> is actually writing the material out to media at the >>> time vs. later. >>>=20 >>>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>>=20 >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 407.5MB/s 00:12 >>>>=20 >>>> $ rm FreeBSD-14-TEST.img >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 404.7MB/s 00:12 >>>>=20 >>>> Still definitely faster than the FreeBSD results that I >>>> reported earlier, including faster than the ThreadRipper >>>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >>=20 >> One more variation in ubuntu under Hyper-V, still with 1024 MiBytes >> of assigned RAM: use of localhost:/dev/null >>=20 >> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img markmi@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>=20 >> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img markmi@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >>=20 >>=20 >> The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS con= text): >>=20 >> # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img root@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>=20 >> # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img root@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >>=20 >>=20 >> Note: At most one VM running at a time, never both in overlapping times. >=20 > Avoiding having a cipher involved and even localhost > involved: use dd . . . >=20 >=20 > FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, > 24576 MiByts of RAM assigned): >=20 > # dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1m status=3Dprogress > 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s > 5120+0 records in > 5120+0 records out > 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) > CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=3DFreeBSD-14.0-ALPHA2-ar= m-armv7-GENERICSD-20230818-77013f29d048-264841.img of=3D/dev/null bs=3D1k st= atus=3Dprogress > 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s > 5242880+0 records in > 5242880+0 records out > 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) > CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=3DFreeBSD-14.0-ALPHA2-ar= m-armv7-GENERICSD-20230818-77013f29d048-264841.img of=3D/dev/null bs=3D512 s= tatus=3Dprogress > 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s > 10485760+0 records in > 10485760+0 records out > 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) >=20 >=20 > Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, > only 1024 MiBytes of RAM assigned: >=20 > $ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1M status=3Dprogress > 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s > 5120+0 records in > 5120+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s > $ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1K status=3Dprogress > 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s > 5242880+0 records in > 5242880+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s > markmi@ubwdk23s:~$ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-2023081= 8-77013f29d048-264841.img of=3D/dev/null bs=3D512 status=3Dprogress > 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s > 10485760+0 records in > 10485760+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s >=20 >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20 >=20