From nobody Mon Aug 28 16:15:59 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZFzl43C6z4rj6w for ; Mon, 28 Aug 2023 16:16:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZFzk08Ysz3Fl6 for ; Mon, 28 Aug 2023 16:16:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="FbdJUr/9"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693239376; bh=FoGrBo8oyvguUafMwQkKKBOmSfxpL5XpstnN0H5sYeo=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=FbdJUr/9K1vFP+hwVbQnohKY2/LTtSbZqtzXxiijXAQekEPlRDk/ajW9Fu3SBBitSJ9w3PCsuCiKD9OMhQoC098jVRIR0cgJrd6BJD7ER14PnDndZQf/wP5XHHwhdczWccSTQZ8BBAYKJdklYSD70+sYWYm2dY8LMZlPjzcJ+ck/uhd+Tj3vjmu25P1s6RaW0N0IluZXUIMQ4TXflJKZaFWxa4WzfrFPNOHMShTzZivwjAAEJBA4e/9/xKlDYskbSM42BKrtHRDMZCbQiVLidc7yJpIvxwVSVHpsHXAxSPkOS37j4oMoLEEut1QodzST86HwLrKhEQtzqlsS1LQG5Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693239376; bh=P+csdyG70ToJfSH1CdCz3GCRgcXotIEU2nY0BgD9R6M=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=nWICjwev8rbKRaO0Ac+6V1jpgx2NYY9YEGWfTwaFmuFRNEsJesSzb0AcPUl9FGYJPig0f2lbjvQHUNGLv9UAfJKmJhmpggMmeYrug38b7CyhmOu0LkQlbnRXY6XURB/A4BcDb+bjNdQs+IuUomsAFjjzlNeiKbKKbG+vMgbuUOT1TXVyQgOSEa9wgg5ZOvqkBiYRXWUbkX5EYB32RscLZHkvwZ2TxWfMHiGasKVXZNI8wPhdzNgDnbVggcMD8z/xWhpFnirsm6oaw/QoDJykTyc5jfC/3jabCkL54UbH7YrCncxXDi2WLM1diY6e9kwwKbEN7rXKuVsdpYw9HRIHRA== X-YMail-OSG: _NowpoMVM1lPxWDPvZaEky_VGkwK3M3eSUJzEiV3mtnj7BrM.a4tpqip2rBjuOU 1JxbEFc3ztb6eTdFiUrvkkc2._k8CmkGiaKgQs4NNAwd4pmZ1Wt7.GKRyLvJKqRZJmmHxkkaG.2A n53fVctIXyLNLjk_NVMKlna.zMKT6XoDYtT6yUNl6AvawjCQpjgq4AEVC09U03Uu2GY06h6W6Fub 7Jz9y2FMV63gjPiyfwzHuaobMx80QCsTFB.RkPbSrcZ64bxARiR9SiX0.ddZv9Q.o6ohnOxQo8H. INv7FtVwhX9DArxyYgr0ilT43ISv8Zfrl_tQemyKHTeOm9wh2NQ1LK7z4iQ9B7nCyFzH9fAcjSrY 22I0BxJXsyt3R_XuTkK6Y5fjD5_Vgas3.UREqboTVCB_.HpQkgl2XdXZBjrFw6hFhxVlLWnjAyYw gbHYHl3CMR.dM9DxYvK278QFuY5dSzrll8p9pUSjSJ0uHLEq.yz9qwHG3N_KllxxAxUk5vEweKl4 IKFioWS2tTqjrwsF0MAIhY5nMtZvgD49CvcEXDtkriKR.UE5KSK3URt8ia1luuD0D.hWHjPhHxVJ S3NOMqW9zaFeER14xIqs64Km5uDHb5qpc490T6Nm3BFEVe_nbKe2b6.M7YFFPtpUeXnSjWte5DhM ojOTZduPzY1KtFnliCZsBgN8mfUn.x7EHCvNposgeydo.xn_loYVmjX0XZXdNe1ijJJBakvSEG7w lljk3P4T58h.jwAbGIPwi9T246UXF9nQzEByu.gxHzk6mrA9jQ77eJWeATtRpTs3W2l12e15Rt4p 42aA9Bb51e84E2zObjbWMvb7NSNfFzg.fWTsXsNo1_vj18O3WoiG78XPdc2dKLwB.UyCq0rd0Ps5 HEFZiZSfCLUuxUaIi9Q0iBz0Ma3rlrWjs3rfe4lmFPjvO.w43cd6lr.pyYPuNzdObIfywXQlG8fq XOiEbaJVETNyG10BfTpcEGVcf3.7udWzbAV6uLOHfD2Qj4.krJ9.7uOk7EEqSM9enY6.xroBGJbO c5_NnJCYjpucZu9bKrRDC8GrivttLJRE0EfNbeYX1WTWJnkI201sSiHa4qSFAVh_WYRp.VdEbGNr FXspYB1bTBX3OORqLSxlRLin_WpiA8YYTESzBP99aD5JdnIhzGW1azhpkFXJcH9biml_7Z1TxIce PKmF302Vh2hIyuEg0uLmj1GV3R56SKw6W56FG7ayoj6WBQtTlJOa5f8LjDoQ2DXYwo58KCnduR90 rAYinThmp8wbqGSAkIsi2Z.eueT8LTYQbte0BpvO9XJXP.mAR2aBpAsVrrzvCOyjFWJWv.5Hoagx QVS.LGXfM8qWqI.fcMhGqxgl2OHo63dT5xQoCuOLZO0S4ZvYdtwsrN7U5M8e2ZfOZ4pO93gYDSaS MSQ66ynDrnNLav7jFrzVRIFD8XJyuW_IpWh_DyFwjP19zJrfFUxIeX3oRDiYtNTaQQ.vklxj7vJr 10_xajJN_A5uiZ15xEn5nt30H4RfJTJRpsMBhPdb4rc0PYkGYqElyFoj98UnSwOOvK1mw4ED67Vi 0Hg6P1tDEq69DDa7uhqfsqoqZpJugMX4AEt0HQ1Nnhp6c6tU7VIoiFkogZMTHEk4_SyHx_QmIyzc 1Fukdfy3asLtu2njiHH7_s0zcX5WwRCMJmedwuWw7MrtbEoeCynwh1U3Vk7zaT.itHjJyc3z4DUP ZQ_rjAsLRlYqrJDiDttP1CYsQDziJy_NGmq3c3atU5E4oQENSYIp4wdt7XmY1CuR72yYN6b4.kSh fZs59ujRh6qml.0sMwMRVj2XLJ.vJbS.XnPvv0OueO.TVUCXg526pkxx3WTQ56e9a8bW40sLxF4R DAkVMcP.FJ_vvkDdpRqED9E7jwLgcYtIvS.VlpI6Naap_dxuGqpGwdzyEbD4pM7uo2Pf0XRrBlQQ 26uwHxYid78sj8JCE.8DlI40BKXEwmQ.auUQJZdZcP6xtFeHofwyA8SGkyJzEUo2FQOfeu8ZABg_ Xfas_tBc4G_vwLsdleoKLJmoVidlwaRq68yWTIcEZdF5caX8TKPO_YvCKdjqssUMZAqTJF0KDvTj Yg1tRJTehnDF7huGuLKAJdH7iDcxEdDJ3lKRIokrC4yDHeHfM.WmP9zGw.c1yC8lqWXgogikQFIz IoJ1QoysaAptUEJQg2_7tsiQ6lGJd8JhJk6hsS7Z11_WKyyttArR_mrpKgFqkM2JB6Pm_PCJqwJm 5NYNlbh9G.Yh1BnvpdEIsRHnC9xouGERQOQk2jmAfRal2hV3OJWK1E6Qjdv4GE6pUmXnz819HvlH OsQ-- X-Sonic-MF: X-Sonic-ID: ab00da86-4758-4ee9-a994-18a53a7cb8ff Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 16:16:16 +0000 Received: by hermes--production-ne1-7b767b77cc-27nt8 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1338cfc330c34f563f69ce6a8193088b; Mon, 28 Aug 2023 16:16:11 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Date: Mon, 28 Aug 2023 09:15:59 -0700 References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> Message-Id: <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZFzk08Ysz3Fl6 On Aug 28, 2023, at 08:43, Mark Millard wrote: > Wei Hu wrote on > Date: Mon, 28 Aug 2023 07:32:35 UTC : >=20 >> When I was testing a new NIC, I found the single stream scp = performance was almost 8 time slower than Linux on the RX side. = Initially I thought it might be something with the NIC. But when I = switched to sending the file on localhost, the numbers stay the same.=20 >>=20 >> Here I was sending a 2GB file from sender to receiver using scp. = FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux = kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. = The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. >>=20 >> Sender Receiver throughput >> Linux FreeBSD 70 MB/s >> Linux Linux 550 MB/s >> FreeBSD FreeBSD 70 MB/s >> FreeBSD Linux 350 MB/s >> FreeBSD localhost 70 MB/s >> Linux localhost 550 MB/s >>=20 >> =46rom theses test, it seems I can rule out the issue on NIC and its = driver. Looks the FreeBSD kernel network stack is much slower than Linux = on single stream TCP, or there are some problem with scp? >>=20 >> I also tried turning on following kernel parameters on FreeBSD = kernel. But it makes no difference, neither do the other tcp cc = algorithms such as htcp and newreno. >>=20 >> net.inet.tcp.soreceive_stream=3D"1" >> net.isr.maxthreads=3D"-1" >> net.isr.bindthreads=3D"1" >>=20 >> net.inet.ip.intr_queue_maxlen=3D2048 >> net.inet.tcp.recvbuf_max=3D16777216 >> net.inet.tcp.recvspace=3D419430 >> net.inet.tcp.sendbuf_max=3D16777216 >> net.inet.tcp.sendspace=3D209715 >> kern.ipc.maxsockbuf=3D16777216 >>=20 >> Any ideas? >=20 >=20 > You do not give explicit commands to try. Nor do you specify your > hardware context that is involved, just that HyperV is involved. >=20 > So, on a HoneyComb (16 cortex-A72's) with Optane boot media in > its PCIe slot I, no HyperV or VM involved, tried: I should have listed the non-debug build in use: # uname -apKU FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 120.2MB/s 00:42 >=20 > It is not a high performance system. 64 GiBytes of RAM. >=20 > So instead trying a ThreadRipper 1950X that also has Optane in a > CPIe slot for its boot media, no HyperV or VM involved, I should have listed the non-debug build in use: # uname -apKU FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #116 = main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:20 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000 (Same source tree content.) > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 299.7MB/s 00:17 >=20 > (These systems do not run with any tmpfs areas, not even /tmp . So > I'm not providing that kind of example, at least for now.) >=20 > 128 GiBytes of RAM. >=20 > Both systems are ZFS based but with a simple single partition. > (Used for bectl BE not for other types of reasons to use ZFS. > I could boot UFS variants of the boot media and test that > kind of context.) >=20 > So both show between your FreeBSD figure and the Linux figure. > I've no means of checking how reasonable the figures are relative > to your test context. I just know the results are better than > you report for localhost use. Adding a Windows Dev Kit 2023 booting via USB3 (but via a U.2 adapter to Optane media), again ZFS, again no VM involved: # uname -apKU FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 168.7MB/s 00:30 Note: the cortex-a72 and cortex-a78c/x1c builds were optimized via -mcpu=3D use. The ThreadRipper build was not. Note: I've not controlled for if the reads of the input *.img data were gotten from memory caching of prior activity or not. I could do so if you want: reboot before scp command. =3D=3D=3D Mark Millard marklmi at yahoo.com