From nobody Thu Aug 31 00:09:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbhPJ4hSQz4rQjC for ; Thu, 31 Aug 2023 00:09:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-25.consmr.mail.gq1.yahoo.com (sonic312-25.consmr.mail.gq1.yahoo.com [98.137.69.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbhPG6RB9z4kNv for ; Thu, 31 Aug 2023 00:09:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="q/ZuzZ87"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693440592; bh=MJllwp50E9EKQgorX6ya2RHTvsQroKdlZcbpayQtdcI=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=q/ZuzZ87z2Q5fhuiLzQ+ksYsVu4ydGosDvfTuIYKUKN58yzEsPdrAA28Izo+NHHJtnMdsmFBTHUzgtV5fSnpLmXJiB/Bwax+xgbgRXPeW1T54opabODIvkYGYthAK2RlGjYIZJOTJPq8inbZQWf6Ke4xt18essmXJZfESqRMb5RPfTz9purX/EVHzbfJKaD9wzNlfl43sdYtVGlPaEqQMSrWYXiwHZoZ7EeQIbaoFxOMWLQLOONbQJbEislwxkt+mdsTVYLNk0Y0Isf5Zw7YPpluqwUfNkmPYbxqN9n7MUWCYlpz4VjVYp+QNrisdo9LeK6JFFPPZ9OYFcS+5mRXWw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693440592; bh=cpTRQJDTnF4yE7F29Jmq9hq3eUkjgIX+kL8Xa9JwJG2=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=CParrwLiJvBkoMTutBSVljrfhH5vFVCCI1YnbF5lqTo4HWKkzRukMpbqGIQJvP9WxPM3LVnKgrJkUpFnzh1Ky7mZaJMZRQXB2+0lk2X25n8+cEF1UXtW/cu+oukRXRRjTKWyS7AFj1tF/DY4UWLIxHaL2p02pj03kNS85jWJG/Zu8C5LmmrhOEtrQvtBm8N+KssqBkQLMVlLljpITv6bwtGxZ++EzaMZ3W5QgCHUnQImVGG+Jn0t2LyBa0evp7LGrmo3YrdprQMDkcsDITWZPT9rjV8HvI0VR+xo3/RbbKwIymYkNmWN09ID/2ybGP3luI9gbFBx7sQ3V6EXdR8TCw== X-YMail-OSG: RBCr75cVM1kVCL_sinPIe87PexJauemzUCDTxaXq46uxMAkgzYh11GLJ.DXMDN8 Pp5wsJ_m6puHHOS8d5We3AOKDMTbBkAZAFGSYld4GTVGNgGGffJ5PHv5yugW.lIwbzDe7K6mNKyV BqVjSOOORm3xp_iB.VgNthhIrn1kTx1085bu4DikOTiUsC5niennEH2y9.7LE2rDF8k6ouE0ON4g UNMJs38H9KVQDz7ET.dANzxsWWgT2ae0FFtvI2uzzn8k90_qfyCUMRW8.pmHm2cOgnnpDk_K_cmo qlt.3xxEO1lm2oltr9iQvlDDEutA9wSrF45P2GhiU_nO3U0_mzC0caZxYx2eWsqX6PFJG7V0gK3q 2FdhjABL2TJ47I9XHc4qkwPRm8i3OZNij22MHzIJtp2MWlEl_VjCrNP5Q_hPvlu4XFxuBtRjbkDP tF6O9b.x_nyX4ungTFFvuLwFMCyEIipqQyMhPlgW1A_BwGym6RpciCCxgJTA4bCBhwxzZotGL0Zm 8VKBkGwuxa4OcT6ufFYY3LbLVdWCPZBm0Qnnm..5TljJ3HluhbqMgxIdKpiBhg5sl3WCPqQMLM7x pyxX.Qdn.m7hlxRcNFQH5ZL9pyc.yIysxBeDHSxDVFfQDpvWnU5z.uQixj_hlrpHmquKODzEqqVG 9GEpcvWvGmZtA5kmRxDy52PvmOIjAjOQy2O.pd4qbohXebSV.VXmXvY0DoOtG3AP.xU3Ilix4ET6 17FbLfAUHYL_fURcp1JjrjQbJUn39ijFbb2iSMpNQFeZxULmfSUEhOR8JvwhZihPNghUQHQxB.WW G1N6p71uuoPL5sgzCtAe3rw0L6dQ0dCsmDA2USiHRQLo4mePwfLcscsqEfDefGNRYCHMY3ss7Ww. .gxoP73jgz2e4LO_g4VxcvgKM1l4s8WRP0ZGNBsirIZJkFRDj5I03KWjz.3ejvap4CD28ZDJ7PpE oXNqXDqZFGHKsdp1Y5YEpZ2I19BQyxicGkxrB5AjZ.mlPo_m3qksACzj3hLcv5zeSFLV9fB2SLWE biPv3JvOKGzpZRdNQPJbeeitMj6YczEfxqwAKi_5EDlJjLoTtajmAsKjVGBHVweC.FliEkXVY9j2 p_Eh3IJyUt9diJxbxchSC9Qv20kuNhOt3ATLHPzfhg22_lE7pnjCWL_VvdcIu5UGWftRrck3Kkgm .budIrHOyCCHiI1f968ZrWeWWrZH7iQZYoewMVgsFsHtr.USu2PBKe8sDMxVNmXUdfonj1a35Rfg 08AyQMUJ.HbaYC0kUCgEuuwQg9J.J3nNYC9jZBCpR.5vhpbdMSTrCuPGJCj6PWt0BPe6NaQgZjWn UvPGL52HFhOOT4TwQ3H5jvrxYdZ2nA42fIRjDsl_wdJGq3k2tOmsAcdSZrrTmklGWJB1ztMeTHao 3no8cveV4FVibTdUiKMOEr3lk0sUW_pN_9c_JiXzdyedirMis2608odB04Lml_LuNbGjFOVXBftF f7PgOgnAK0rJPQv5d39D7LE6N0Uhgr1GINW8gr6Cl0JZrrF3RsiUONpv9rCZBcOBH593XgSdUFKJ 5B5CirNMJlaVKVllZNqTU6oBbiTz8ESRuj1NI5bZvxzz0gyQtsEXqpm2kHV9P6tRUcxqan7q7C.s C3P1Hg5t79IuIZHb.kFyab_6ztGTdu1aVocb0ypPHdw2jhEekK2NU6Y_Bumev8hufj_ip33zF6eb f3Td9hzPB9qyhuBtYEiV61EnbziLfoKqKgclzH_EjEc1abw62cVUYR_8ee0XAdQOdrAdbEdb58uN Nf0tfyiSik_aj32QQ6VJ4QsbQoT4YLTEy_QOdiA.ETE7KAude80IKl0pbCf0yODwnOJQj3stZc1w neQjYBBmkSHbsxXrZb7a7p5qAzrmm986MHvoenul.u6LqsxW0KsFhhk1z5lV1YoaL.eswDarNVh1 QMHahJ10undOnc3jdXKfrU8S7TIu5vSgA_RAYxKICwjuAqt.3pwlC02vVO_hAa4LK0cCXIHKcwd1 QxS3q3BpcWRJNb5Ufb0cXAJ6H6SmAjms8naX48CqkFIOwRHjnfSP8uDyefal9E0WXcizrXiHX7nr VXmzt2JZp2j0HVX8s_jamVMoWkootyMMc.upisnSbZkrE2cLON.VAWhY1x3ZaCnhsXVlbH2Hm9BF PNT4Vo08uHImGkRcYkyy7k0EAiUOq8eW9kGX5GRWqApRPu6wprcQ5u_s05jRYOPOEYioe9Y7p.du 1YvzfjBqhCxLbbvRWAmDS0ROqwMx_GXBZCS3N.FUYGjdFnKxOedDeJkaivZ.fZb72nJqYrLXoipo - X-Sonic-MF: X-Sonic-ID: 9ccea03b-b7c5-4f51-8721-18eab296f352 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 00:09:52 +0000 Received: by hermes--production-bf1-865889d799-5m62n (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e706621ac190da6f12d857cb6413dd52; Thu, 31 Aug 2023 00:09:47 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Date: Wed, 30 Aug 2023 17:09:34 -0700 References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: Message-Id: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.206:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RbhPG6RB9z4kNv On Aug 30, 2023, at 01:49, Mark Millard wrote: > On Aug 30, 2023, at 01:22, Mark Millard wrote: >=20 >> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>=20 >>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>=20 >>>> Wei Hu wrote on >>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>=20 >>>>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>>>> good to know.=20 >>>>>=20 >>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, = my original >>>>> intention was to find out why scp on Linux is performing much = better than FreeBSD >>>>> under the same hardware env.=20 >>>>>=20 >>>>> Is it possible to try Linux in your ARM64 setting? I am using = Ubuntu 22.04 on ext4=20 >>>>> file system. >>>>=20 >>>>=20 >>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >>>> Hyper-V UEFI reports for the disk created: >>>>=20 >>>> 1. SCSI Disk 0,0 >>>> The boot loader did not load an operating system. >>>>=20 >>>> (It then reports the network adapter attempt found no >>>> boot image, but that is expected.) >>>>=20 >>>> That leaves me wondering if Hyper-V Quick Create >>>> established a VM file holding Intel/AMD material >>>> despite the aarch64 context. >>>>=20 >>>> Establishing a Ubuntu more directly is not familiar and >>>> will have to be a background activity and, so, likely >>>> will not be timely. If I did any experiments outside >>>> Hyper-V (native booting), they would be with slower >>>> USB3 SSD media than I use for FreeBSD. >>>>=20 >>>> I did notice that Hyper-V Quick Create did not create >>>> a fixed sized disk but a dynamic sized one. That is >>>> different than what I did for FreeBSD. >>>>=20 >>>> Also, it was not obvious if you were after aarch64 >>>> Hyper-V testing vs. native-boot testing vs. both. So >>>> I may have gone the wrong direction from the start. >>>> It is possible that I'd find establishing a native-boot >>>> easier and then be able to have a VM file created from >>>> the media, more like what I did with FreeBSD. >>>>=20 >>>> The Ubuntu activity likely would not be analogous to >>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>=20 >>>> Back to $work. >>>>=20 >>>=20 >>> I found a sequence of UI operations that worked for >>> installing Ubuntu server 22.04.3 into Hyper-V in >>> Windows 11 Pro on the Windows Dev Kit 2023 via >>> use of a downloaded *.iso . >>>=20 >>> The kernel that results predates 6.0: >>>=20 >>> $ uname -ap >>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 = UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >>>=20 >>> Using my usual rule of rebooting before the first scp: >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >>>=20 >>> $ rm FreeBSD-14-TEST.img >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >>>=20 >>> Definitely faster than the FreeBSD results that I reported >>> earlier, including faster than the ThreadRipper 1950X with >>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>=20 >>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual = hard >>> disk under Hyper-V. >>>=20 >>> For reference: >>>=20 >>> $ lsblk -f >>> NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS >>> loop0 squashfs 4.0 = 0 100% /snap/core20/1977 >>> loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 >>> loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 >>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>> =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >>>=20 >>>=20 >>>=20 >>> An experiment would be to have a small amount if RAM relative >>> the file size. That would force it to actually write to media >>> for some part of the file copy. >>=20 >> The wording was poor: "force it" here is just from the >> Ubuntu viewpoint. I make no claim to know if Hyper-V >> is actually writing the material out to media at the >> time vs. later. >>=20 >>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >>>=20 >>> $ rm FreeBSD-14-TEST.img >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >>>=20 >>> Still definitely faster than the FreeBSD results that I >>> reported earlier, including faster than the ThreadRipper >>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >=20 > One more variation in ubuntu under Hyper-V, still with 1024 MiBytes > of assigned RAM: use of localhost:/dev/null >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >=20 >=20 > The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS = context): >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >=20 >=20 > Note: At most one VM running at a time, never both in overlapping = times. Avoiding having a cipher involved and even localhost involved: use dd . . . FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, 24576 MiByts of RAM assigned): # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1m status=3Dprogress 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s 5120+0 records in 5120+0 records out 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1k status=3Dprogress 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s 5242880+0 records in 5242880+0 records out 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s 10485760+0 records in 10485760+0 records out 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, only 1024 MiBytes of RAM assigned: $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1M status=3Dprogress 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s 5120+0 records in 5120+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1K status=3Dprogress 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s 5242880+0 records in 5242880+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s markmi@ubwdk23s:~$ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s 10485760+0 records in 10485760+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s =3D=3D=3D Mark Millard marklmi at yahoo.com