From nobody Tue Aug 29 10:27:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZkBf2jhvz4rdCj for ; Tue, 29 Aug 2023 10:27:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-21.consmr.mail.gq1.yahoo.com (sonic301-21.consmr.mail.gq1.yahoo.com [98.137.64.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZkBb0LkHz4d8j for ; Tue, 29 Aug 2023 10:27:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ak5BAemP; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693304837; bh=u8WeiJ96HfIVHyVGHSkXkMvM0LWn18B9TY0Ku5O2lNI=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=ak5BAemPPfkmEYcY7sLh8NbqKMx9mMbYq4kVxVgrFzv7RTyt+1+zQ+uap9a1T+Q8k42no0oj+TIcH/3P/HKv7HGb6RSQfPJ5fz/q2lYFVIIftUiGsESfDY7/1/o40ZfANOhNBlFRxjpogjHN+VRuFQ8hwtrFbw+qolWdv7EU2hMAjLL7PB40GVE97vx7e/+Ty3O7SXJm5CMAvc7IvIkEfrjRaUHF/1F9RvU905DOF7N5Me+cJhgCJRWFfYwqnLRv7gfVZP/zqEYyZdbEcuGqpHP26ok1oSJz/UOvisCAPGXjYlwJmXDmTTWNiw6/ZoQKf0rBVahuQTxKBWXtW7krlw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693304837; bh=U8rJLQk6dTToYGeVbTSuDSbanCbh8M4pIpeTvwerElI=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=JsL4amujgrlqPXfBL8gCGD/A+eJ3Haa4Q53LyBoz61s0LY4n7GARdhqv7U5ED1OZwLI/f/+qpG2fu2t7LKBxSbuIl9jf92hnHZae0eR+n5JFAISqL4zeu6+QnWfczToiM3DHcWi4jJgGr6OY4bhXAEBVwXJQrIf6fKO8y61DGBe9JyLFu1YzhKQkXeWbYVS5jvXZa5B5TOJeYGEAwRSr5BO+xQ6yBIUAMYrAc/0PMwOtL9O2X/3/5BAP2pfow98GiH8zOPsuK5uJB7r+lB8TH/IOPilGoH4dvdEU1lhszrKNAxc9ywM0eTkCOgeqMfeyTNYB1aUGydeNjaUxtaZGZg== X-YMail-OSG: QkgXs.0VM1mtyGJjo2dZr5qzjMv0PXv2GspoUMji8bEXJljb22uA.g6PofTMo_8 PXH9otrYtbHTjMIRRHbP8iXLv9HafpT0UPvxWclR3e.ZsBqRlh.qrE8PC_T0Hf8Vifh_1s13uyK9 I5w0UdRZtBOoqG5XnkXNvhjYNpfEzo2T8ARsQrqD4Nvc1y4PPWrzUabkJNFPY0bjc_JzmHRRNVEv GgHLlt60KETGNvzGr2GxewIaLbBYnK0u8H.d2AKXxrzbaitzS3LHsizTdyhjzYiuyNZ9LX9lB.pu ZHj3KhRWFuZf1uiwX_BNWqFg5cEGEUNJhs__D5AJSQhPGGJ83lgIC3CkmSfmFfBGeE2.Ec._z62f ZrK6DDlrEs0Na8azAMg_ZQYTjW_cd7dlmfj4myhWn2.R5BukGhdFqmkl4LLczxfFnaAZMYYgD5Df m8F4hA8y6H5T_zrALvGC8AH0dOpjjA2z1WBpO_K.2Jyg5_EHl9nebgTwdRQtldnlKdWRitT32emC VtavYumInA91UaLpkhMESx3Eu.DTprsvjd8s.xlOUvyVrSNKyN9I50rSDJXfcSvcQYa4J.AjmcQn lw3bhM1FpQGxtJpfPPV5vKToFDaj4WVI.pH9ReHQtLLJ8Vad5vJQl543awyGKehENmRAlepoLw88 XnxJHfwFM7HVutHM5uAMn1k6WPqTREjy80iUUMoOOkz_XYj4Uk19O8qQMT9RuzevZCbaCudqi8dz KVkft1c_0e4S.JfkBQoscLdVbIl18LClfHlFrX.m_6nUZCY4cS7xnXy6Uw6F3DAtGjBvTr_RNxPF FoMV8BBIvQDBg_.jaZlwPONfIPj23dzFL6.a7_eNUHAauryIcJagpApZBbcsLq.CCNK6qqNLgxft NrNrvRmlDjs6tj5WCmZPMqVKTgfSrBh7lfwqSWXdyTCwesfargZPUp3mpeCn4hrIZPEtiDDQkCtG ebM.Bx.5S1k4eMmYK_5qsBO1BBgl5Nd_5anJV8wDGt5XQO2aiQoMiJt4i5mlpBM.4ZWC1CWtA0Ie DHer3T1kYlo2h0pIZuXEzep3f67PA04CBGGDBcL3qi5gUf8.4gtxbISqtbWHzrUb78AZWf0NdPO9 E3SRNIsP1aEmrYk8dI1m3UGj2Ap1gibTODn1TvTKo8Rr3UdzzT1dMnna6HGqzYx4NOretASKnlUN oLhLPc4h4R61EfjetaTEq8mteEvrVXOKLGWA3yVaG3CvnAq5PWNRMwedckWx46UdEzHVTLJtrWmE AEW1F.Ea_KrKX9u2dVGUJtMP4PNJ0OCGhVwo4DzFRd.xEjf3lg0CHTEYqZ3SexiiE4mwOxxZwIdM dr6H9UP2gqPBVB9wKrjU0EGb5FP.CLqOtFTby066FHluh20d40ayXomj82d2lDgo979hRNrOGk4h Yl8bcvD1UG_Db_q0.ZP0CaMyiJv_kY8NT4AUfhmph7XPncSiL0deqzDW1biVo_x_HXA59hyBYwVQ xVaKmqD_B6d_De2uyh2OoqyrO9mXLCjyDmzP6d2pc6QDcj4ANBF9O.Jw1HWLb0iuhgbv.uo.UEmF kvfYNQywi1lA.mRhQROPP7hKJf3dU7SBtMDEPshyNCpBdhihTOvZe_iu4e.sYrveiOkXQO1Imtm5 DgOZGrQjFPB4M218Tl8QYakGfPeWPkOGmBHNibX4vxfXA6J75s7efWRal89PS44YjbNvCZSjc8WO 01Y2Y3OaYXDgSiVzUFFdGicaHrZ9VoWOosGglAuNVO0kiheoBiu1LKFpvO0WgjuOjQGHKTmIzxJY la.UEYBNJslAYDB80kD8AtHPkiijButFbwToDHnUhwQobsPTVsmoDl_bi5E4NXDzv6gNr9VHeu2t hCa1N.0hdorkRrQXfJbIgL_33wPmqOVjgPfo6J5SXFVZ04Dyd_bk8DESWMxejHtvLeLY1jtLfDRm .HWzlwrMTIvugEWM.3BTPw_JYNjTGhouogvwrPfKW_TDQ_77KIdV3Whi0p6.gVZSI.hvpLQddK6t vFqPjm27lsXCvpRa5q2HzV39hdQprirO.aslUBLb7UjAuu30.DKT3DA.g3pgs1XlPQESckQ56nyq NSx_jnU.P6HXHCHAFyaigFuDDllAThTggw0ELiLxCgMOlX9_DdBRDgHgUwmBXTzTRcaDTziYtiNq TY7USjXV66oU1p4Hp9.xQTO_qucE3geIrNE77uMg4OqdIEpQmjV7UtH1Y5zl.mBvYIhbKo2vEWnM VbIrr_mE4cpPDNvTFjuCv4lwuEeg2zh93RY4_hlcRhC0HhwJ8WGtmSlviD3rY841PDuuS_lB47a9 RhUMb X-Sonic-MF: X-Sonic-ID: 7c059123-b4c3-41c5-83fe-157d27477191 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Tue, 29 Aug 2023 10:27:17 +0000 Received: by hermes--production-ne1-7b767b77cc-ht9fv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 529693c29ed31783430621bcd3b8e4af; Tue, 29 Aug 2023 10:27:11 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: RE: Very slow scp performance comparing to Linux Message-Id: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> Date: Tue, 29 Aug 2023 03:27:00 -0700 Cc: FreeBSD Hackers To: weh@microsoft.com X-Mailer: Apple Mail (2.3731.700.6) References: <98C8E07C-2247-4439-8836-ED350CC83F16.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.147:from]; BLOCKLISTDE_FAIL(0.00)[98.137.64.147:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZkBb0LkHz4d8j Wei Hu wrote on Date: Tue, 29 Aug 2023 07:07:39 UTC : > Sorry for the top posting. But I don't want to make it look too messy. = Here is the > Information that I have missed in my original email. >=20 > All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz = K8-class CPU). >=20 > FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: > 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 = nodbg-n264692-59e706ffee52-dirty... = /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 >=20 > Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: > 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 = x86_64 x86_64 x86_64 GNU/Linux >=20 > I did a couple more tests as suggested by others in this thread. In = recap: >=20 > Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s > Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 = MB/s >=20 > Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs = 48.8 Gb/s >=20 > Would these numbers suggest that > 1. ext4 caches a lot more than ufs? > 2. there is a tcp performance gap in the network stack between FreeBSD = and Ubuntu? >=20 > Would you also try run scp on ufs on your bare metal arm host? I am = curious to now how different between ufs and zfs. For this round I'm rebooting between the unxz and the 1st scp. So I'll also have zfs results again. I'll also do a 2nd scp (no reboot) to see if it gets notably different results. . . . Well, I just got FreeBSD main [so: 15] running under HyperV on the Windows Dev Kit 2023. So reporting for there first. This was via an ssh session. The context is ZFS. The VM file size is fixed, as is the RAM size. 6 cores (of 8) and 24576 MiBytes assigned (of 32 GiBytes) to the one FreeBSD instance. The VM file is on the internal NVMe drive in the Windows 11 Pro file system in the default place. (I was having it copy the hardrive media to the VM file when I started this process. Modern HyperV no longer seems to support direct use of USB3 physical media. I first had to produce a copy of the material on smaller media so that a fixed VM file size from a copy to create the VM file would fit in the NVMe's free space.) # uname -apKU FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 (The ZFS content is a copy of the USB3 interfaced ZFS Optane media's content previously reported on. So the installed system was built with -mcpu=3D based optimization, as noted before.) # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 193.6MB/s 00:26 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.0MB/s 00:25 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. For reference: # gpart show -pl =3D> 40 468862055 da0 GPT (224G) 40 32728 - free - (16M) 32768 102400 da0p1 wdk23sCA78Cefi (50M) 135168 421703680 da0p2 wdk23sCA78Czfs (201G) 421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G) 468860928 1167 - free - (584K) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x = ONLINE - (UFS would have notably more allocated and less free for the same size partition.) The below is be based on the HoneyComb (16 cortex-a72's) since I've got the HyperV context going on the Windows Dev Kit 2023 at the moment. UFS first: # uname -apKU FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 129.7MB/s 00:39 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 130.9MB/s 00:39 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. Note: This is via a U.2 Optane 960 GB media and an M.2 adapter instead of being via a PCIe Optane 960 GB media in the PCIe slot. ZFS second: # uname -apKU FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 121.1MB/s 00:42 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img (root@localhost) Password for root@CA72-16Gp-ZFS: FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 124.6MB/s 00:41 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. Note: This is via a PCIe Optane 960 GB media in the PCIe slot. UFS was slightly faster then ZFS for the HoneyComb context but there is the M.2 vs. PCIe difference as well. =3D=3D=3D Mark Millard marklmi at yahoo.com