From nobody Tue Aug 29 07:07:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZdmN4CQ8z4rS2S for ; Tue, 29 Aug 2023 07:07:48 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on20712.outbound.protection.outlook.com [IPv6:2a01:111:f403:704b::712]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZdmN1QnTz4GXN for ; Tue, 29 Aug 2023 07:07:48 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IJaSkIEE/hh2Ad/kv638kUpKam2D1wwKl6GX66T+dhMg+jOtpNNlfh0dUsAGP81UTRBfThHAtvL9PWAaUmfD1Sz6Sl2ulEgvoygQRm5Nu/d74IAdxLyw9NxFduBL83Vgy9kgAZk93mF1Wo4u7m5fX+pU6PtFb+J9P34m6tToTxwhlPlADeEEd+MWyzx8ND1Yy1thgT2bA5vcrS7y56EQx8zeCxOdvZUpDWu0FuUmtqbmqxs6Z/vpGew6OQXJP7fTlgeufrPa7wp7d3EGl0rbCKiiDRXisScUoeeAr1kdZ25moA+AluMGjr0qn0nc0NbE22R0VlPXwf0+PYFwoPJq4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=H4dFVorlrFwJVqn1AfVgWtwlXrgjA+ezcz+OASsuVis=; b=HkEtsgKNDxGpxl4itJWqIOud0HWXjThba+DFWzqmOd7bPRnkBJkkgeY1Bornc5cjP5l3y0ejw80Zl35Z/0MkgaiPeuv/22OtKYhj3knW6CdwHjqt7PI+ba1a9kx0qrAGZMCBcru3P1fGfVuWURp9mUjnPzUFN8lbiJxbkFHbEkTzVZeNtYWnZwMpGfJjOAs945BNSxaYKcN+/qyaqMRI0hWjKHH5r/p0XgiHv5bFF+dGiZb2s6YkbCvfJ9bXDwXCi4FRuCLe1xvc9pIk3dpy7Ke7Z66N9UfnGS8ZGbrgSetqgbT0CZJvUXTQJrP/bi1Z47BWIvfhW+RiTtXglYFX0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H4dFVorlrFwJVqn1AfVgWtwlXrgjA+ezcz+OASsuVis=; b=Kn4+sCziyZWWP9k87F0va7HiBWpnDTMhLCgS41fFke5ji79N0lrtY2uGal4cw/QsjWpB3NfWfiKZfVfMgSXoPyJ6FIw9egp43kfa2QRjZNhzHAHqKXiXapiZI+b9u3zueql03+zj/fE0c2Ix/OoKSz6YqV+tG+jzgvr9yK7wghA= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by TYZP153MB0416.APCP153.PROD.OUTLOOK.COM (2603:1096:400:25::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.8; Tue, 29 Aug 2023 07:07:40 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.007; Tue, 29 Aug 2023 07:07:40 +0000 From: Wei Hu To: Mark Millard , FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AQHZ2cZijEFfDgyXnUKWUbsPmO6Saq//4k+AgADxB4A= Date: Tue, 29 Aug 2023 07:07:39 +0000 Message-ID: References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> In-Reply-To: <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=4386a806-7a1f-497d-b24c-85ca4c88d627;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-29T06:38:38Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|TYZP153MB0416:EE_ x-ms-office365-filtering-correlation-id: ae46fe46-d4e6-42ec-448b-08dba85ea1e4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1z2EXRkbxCjzJafiFDclWLLaTG+Q8strjcLkseJi0FxUyRdg1nBkdMfgvNfWtFczFxSXJsdDeu+V9Cqvg+3n9X+drAHSe8/1jEzMTvhFClD/7rv8INvOEojM2crg6PJW8u0eHUdWTp6+kTp4hN/Gfw53QWvW1KA0lxgxbH6imj0BVy/kS8npZe6TVR1XUd1mm71LmqEVANIhdUsfFWPHYt9dm9CJp9RgNau1JTvQOvDIGh9k2Gbxi/VsYN01eKcatvk8lVQteUNTUNcjsN1fIyiiiqy9Dk1xk7gC3Me9YNAwWSOgnT1sTuIm9y5XmW/XZPoLibbTWQ/i0r4MD3Yt8Qoyg9KvcVt8oUTFpvod14qGabYV8GCeiJ6x1sxVWdp2kL9DtZn2sH00EpD5iOogmY3A9a2bmO+ccJ+ruLmC6Mhly7nzcEuC66Tdql/oMw6GtRzfJ38L4hgztcIbjX4fbmLDXU/fqLS4kEnBTtNwOpb4ocMx6CTFFi+TRGWGiJjWB/Zaq2j/HI//t0I1kPeXUb8JDXyI0ddTDfy5wGs+RfhEaUjXF1q32gX684yw8I2WaCaDtgV8QcSmFE7IhB9PSr4WhbVdafg9blbRwYDg+miGaXtwyCDKgnuC9OnrYKpykqiY+FirO6pworylBgebAmuFn+TfiB/OEV+XDs8U9v7WPik20AfZUdPpiM1sUxk85ULjQm7k0vjyH1t3M0LujQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(376002)(396003)(136003)(366004)(346002)(451199024)(186009)(1800799009)(9686003)(7696005)(6506007)(71200400001)(53546011)(19627235002)(10290500003)(83380400001)(76116006)(26005)(2906002)(478600001)(64756008)(316002)(8990500004)(66446008)(66476007)(8936002)(66946007)(41300700001)(66556008)(110136005)(52536014)(5660300002)(8676002)(33656002)(82950400001)(122000001)(55016003)(82960400001)(38100700002)(38070700005)(86362001)(12101799020);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RyW2qB9giIXRPEi1UDzm9H/h10jdbYKKTa0ns8eUzF2Ni812yKjT5gGa8Kj5?= =?us-ascii?Q?3T4hVgRj67xZXvGCCQ5YlbEFaho5Kq1XWpsSKk+hCgIcNV+aKIG1gY+UgrZK?= =?us-ascii?Q?CAkCZjYl82GlvnvVCnM+1mDUy7jrMh4X1a/qxta/q+eCPT/ZEIDqhjf7zC3m?= =?us-ascii?Q?VdnzizI/EW0wLakINfELM5d6dTF7NoxVJ9NqHg2i3YmMqqYGR+eymhijptOa?= =?us-ascii?Q?vx8HATUFXj4vlfeHoUJ9QK+IDg4TpOQXbiIn8BNoQys+Fl1lPrMVKuKDNhg1?= =?us-ascii?Q?VNZBQxyDvia2EtAnVVnBBtl8N2F45YhG2jZUYupOSsafmfpO71JPaXHfW7hn?= =?us-ascii?Q?2/mOe9BI7fSrgdOq0//FumadcqzivZRs3jcY0+0cOZvpf0XFq+QgGu6Uqvem?= =?us-ascii?Q?8bbnJ4ZL35SFfZMcI4G1kJRb6LKir/BWf/NSkCxla+FyAOibLPy3oN4rQpKm?= =?us-ascii?Q?MwQxhMKhIVMgRWUtYCx1G6EmvrSVNtKaP2NWtC+IhPpc3e1Lhe7zMl3jFjpw?= =?us-ascii?Q?PIrX1jK9Veekek6PMHyD/9LCjS5bsAqOZVbNo97eDznOwVRD9vmsAweT8N6U?= =?us-ascii?Q?72ZHLSmtz6VfLz+nrSauxIQ7G9XCb5p5xJmEp9+eQW91CsGcMaoKXDo1udPJ?= =?us-ascii?Q?4wdjcp1jkUoR/QpkjYtZw4mJbKW3tQDa/s3h85n30PWifFnFWi1wWS+TCO7N?= =?us-ascii?Q?2dn+pB54acdxXuRppTT2E8d2Vhj8jqwf9kT6sijEVfhMiyxvYtt4igSN7Yx7?= =?us-ascii?Q?UU9SG9PTiOCULr3fJ1K3Kn4dNd7y0y9pIf2+TpwborBP3v4tTE/YCOsxj9Nu?= =?us-ascii?Q?/nNrsZ4WRL3FhTNBFsk9gtRbRZxsO43+HtWSljG4RAJjH+1ab60gNbM+t9bd?= =?us-ascii?Q?omZYFQ430DFFmfVff63oxQQBHji0/u97VR57jhppfEuvtrQ+Q65U1dwg5R8V?= =?us-ascii?Q?VRytf0yQy/QRO5vwq/UtqkprISyiPZhr1dnWd2tPhaWjT5HKqpK4sPwzV/TI?= =?us-ascii?Q?Uncl/H6xvGdGUvjHfusXWfTyPAe8++jKRsOjZEQN3nI4cpxL8Gs8ByBSeQoG?= =?us-ascii?Q?oeOAryQzlMU7JIPqrqKJruitTrCccHHC36pBDES7340mO8FpH+jHaxbF8be8?= =?us-ascii?Q?y7SQksExN/XULypqxT/Hd//DVotIkbHxoehIKL6OcwndCdWu5MrRi3EbQ89x?= =?us-ascii?Q?6+0PQakgH7sbWr4oMESgC101rAcltXwavw2ELhH4Ra41N5oLa7f7E/63jGuJ?= =?us-ascii?Q?78APtsM9yOghwllvmembrnq7RCcV3jyUNUH47UpIZfpRlg8FycfGaRQKmdGJ?= =?us-ascii?Q?z49USHgfEG5fDC7NNBjeKARsH2DA1YEzAUmcKbhMkZh6NyF5NXK666RXK6Og?= =?us-ascii?Q?Z+tqDLilD2XI9IYqfcarU2KtOJ6/7ECMU4EN3YNv02xVdFLo2jgKLQmZ+zJM?= =?us-ascii?Q?Q2NrOzCz4f2w2OCh8rwXPeo2Sh8A47m9CPbnacVvqK3S7Cg7NJGGacZWkq8N?= =?us-ascii?Q?Xncaq4iG9EOXUnuPeITsqh4yWE6WE0qtO+PzyPKLyJ+4se5tuxjrLuU4IzHR?= =?us-ascii?Q?iL4M6EGDSDz7cwalceY=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: ae46fe46-d4e6-42ec-448b-08dba85ea1e4 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2023 07:07:39.9099 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mOkx/uHPIwdjFOP0gBUgAlKtPqNgnloxwJzuBiWRx01OhMIIBQ+f4leSaLBs1UMLSSr2NDEzLD/q0MgkMbSXsQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZP153MB0416 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Queue-Id: 4RZdmN1QnTz4GXN Hi Mark, Sorry for the top posting. But I don't want to make it look too messy. Here= is the Information that I have missed in my original email. All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-clas= s CPU). FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 nodbg-n264692-59e706ffee52= -dirty... /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 x86_64 = x86_64 x86_64 GNU/Linux I did a couple more tests as suggested by others in this thread. In recap: Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 MB/s Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs 48.8 Gb/= s Would these numbers suggest that 1. ext4 caches a lot more than ufs? 2. there is a tcp performance gap in the network stack between FreeBSD and = Ubuntu? Would you also try run scp on ufs on your bare metal arm host? I am curious= to now how different between ufs and zfs. Thanks, Wei > -----Original Message----- > From: Mark Millard > Sent: Tuesday, August 29, 2023 12:16 AM > To: Wei Hu ; FreeBSD Hackers hackers@freebsd.org> > Subject: Re: Very slow scp performance comparing to Linux >=20 > On Aug 28, 2023, at 08:43, Mark Millard wrote: >=20 > > Wei Hu wrote on > > Date: Mon, 28 Aug 2023 07:32:35 UTC : > > > >> When I was testing a new NIC, I found the single stream scp performanc= e > was almost 8 time slower than Linux on the RX side. Initially I thought i= t might > be something with the NIC. But when I switched to sending the file on > localhost, the numbers stay the same. > >> > >> Here I was sending a 2GB file from sender to receiver using scp. FreeB= SD is a > recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. > Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has > 16 vcpus, while Ubuntu VM has 4 vcpu. > >> > >> Sender Receiver throughput > >> Linux FreeBSD 70 MB/s > >> Linux Linux 550 MB/s > >> FreeBSD FreeBSD 70 MB/s > >> FreeBSD Linux 350 MB/s > >> FreeBSD localhost 70 MB/s > >> Linux localhost 550 MB/s > >> > >> From theses test, it seems I can rule out the issue on NIC and its dri= ver. > Looks the FreeBSD kernel network stack is much slower than Linux on singl= e > stream TCP, or there are some problem with scp? > >> > >> I also tried turning on following kernel parameters on FreeBSD kernel.= But it > makes no difference, neither do the other tcp cc algorithms such as htcp = and > newreno. > >> > >> net.inet.tcp.soreceive_stream=3D"1" > >> net.isr.maxthreads=3D"-1" > >> net.isr.bindthreads=3D"1" > >> > >> net.inet.ip.intr_queue_maxlen=3D2048 > >> net.inet.tcp.recvbuf_max=3D16777216 > >> net.inet.tcp.recvspace=3D419430 > >> net.inet.tcp.sendbuf_max=3D16777216 > >> net.inet.tcp.sendspace=3D209715 > >> kern.ipc.maxsockbuf=3D16777216 > >> > >> Any ideas? > > > > > > You do not give explicit commands to try. Nor do you specify your > > hardware context that is involved, just that HyperV is involved. > > > > So, on a HoneyComb (16 cortex-A72's) with Optane boot media in its > > PCIe slot I, no HyperV or VM involved, tried: >=20 > I should have listed the non-debug build in use: >=20 > # uname -apKU > FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT > 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 > aarch64 1500000 1500000 >=20 > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 120.2MB/s 00:42 > > > > It is not a high performance system. 64 GiBytes of RAM. > > > > So instead trying a ThreadRipper 1950X that also has Optane in a CPIe > > slot for its boot media, no HyperV or VM involved, >=20 > I should have listed the non-debug build in use: >=20 > # uname -apKU > FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 > #116 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:20 PDT 2023 > root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main- > src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000 >=20 > (Same source tree content.) >=20 > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 299.7MB/s 00:17 > > > > (These systems do not run with any tmpfs areas, not even /tmp . So I'm > > not providing that kind of example, at least for now.) > > > > 128 GiBytes of RAM. > > > > Both systems are ZFS based but with a simple single partition. > > (Used for bectl BE not for other types of reasons to use ZFS. > > I could boot UFS variants of the boot media and test that kind of > > context.) > > > > So both show between your FreeBSD figure and the Linux figure. > > I've no means of checking how reasonable the figures are relative to > > your test context. I just know the results are better than you report > > for localhost use. >=20 > Adding a Windows Dev Kit 2023 booting via USB3 (but via a > U.2 adapter to Optane media), again ZFS, again no VM involved: >=20 > # uname -apKU > FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT > 2023 root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 > aarch64 1500000 1500000 >=20 > # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818- > 77013f29d048-264841.img root@localhost:FreeBSD-14-TEST.img > . . . > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 168.7MB/s 00:30 >=20 >=20 > Note: the cortex-a72 and cortex-a78c/x1c builds were optimized via -mcpu= =3D > use. The ThreadRipper build was not. >=20 >=20 > Note: I've not controlled for if the reads of the input *.img data were g= otten > from memory caching of prior activity or not. I could do so if you want: = reboot > before scp command. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com